Web Scraping & Data Extraction
The data you need is on a website and not in a file. I build extractors that collect it reliably, keep collecting it on a schedule, and hand it over in a shape you can actually use.
Writing a scraper that works once is an afternoon. Writing one that still works in six months — through layout changes, pagination quirks, rate limits and half-loaded pages — is the actual job, and it is where almost every DIY attempt quietly fails.
What you get
- Sites that resist the simple approachJavaScript-rendered pages, infinite scroll, paginated tables, multi-step forms, and content behind a login you legitimately hold credentials for.
- Delivered where you will use itCSV, Excel, JSON, straight into Postgres or MySQL, appended to a Google Sheet, or exposed as a small API. The extraction is only useful at the point it reaches your workflow.
- Built to survive a redesignResilient selectors, response validation against an expected shape, and an alert when a page stops looking like itself — so a broken scraper announces itself instead of silently writing empty rows for a month.
- Scheduled and incrementalRuns hourly, nightly or weekly, fetching only what has changed. Cheaper to run, faster to finish, and far lighter on the site being read.
- Clean before it landsDeduplicated, typed and normalised on the way in, so what arrives is analysis-ready rather than a second cleaning project.
My Web Scraping & Data Extraction process
Check it is permissible and possible
Terms of service, robots.txt, whether the data is personal, and whether an official API or export already exists. This happens before any code, because it sometimes ends the project — and that is a cheaper outcome than finding out later.
Map the target and the fields
Which pages hold what, how they paginate, what a complete record looks like, and how to tell a genuinely missing value from a failed fetch.
Build the extractor
Plain HTTP where the site allows it, a headless browser only where it is genuinely required — it is far slower and more fragile, so it is a fallback rather than a default.
Validate against reality
Sample records checked by hand against the live pages. Counts reconciled. A scraper that returns plausible wrong data is worse than one that crashes.
Schedule, monitor and hand over
Cron or a queue, failure alerts, the code in your repository, and documentation of every field and where it came from.
Tools I use for this
Chosen per project against your constraints and budget — never one stack forced onto every problem.
Questions I get asked about this
Is web scraping legal?
It depends on three things, and I check all of them before writing any code. What the data is — publicly posted factual information such as product listings and prices sits on much firmer ground than personal data, which brings GDPR and similar regimes into play regardless of how public it looks. How you get it — reading pages a site serves to anyone is different from circumventing a login, a paywall or a technical access control, and the second is where genuine legal risk lives. What the terms say — a site's terms of service may prohibit automated access, which is a contractual matter and can carry real consequences even where no law is broken. I will tell you plainly which category your project is in. If it needs the target's permission, the honest move is usually to ask them, and it succeeds more often than people expect. I do not build scrapers that break access controls or harvest personal data without a lawful basis.
The site loads everything with JavaScript — can you still extract it?
Yes, and usually not the way people expect. The obvious route is a headless browser that renders the page like Chrome would, and that always works. But a page rendering data with JavaScript is fetching that data from somewhere, and that somewhere is very often a JSON endpoint the page calls in the background. Reading that endpoint directly is typically ten to fifty times faster, uses a fraction of the resources, and is far more stable — a JSON shape changes much less often than a page layout. So I look for the underlying request first and fall back to a browser only when there genuinely is not one, or when the content needs real interaction to appear.
What happens when they redesign the site and it breaks?
It will break eventually — that is a property of scraping, not a defect in the build, and anyone promising otherwise is not being straight with you. What is controllable is how you find out. Extractors validate each record against an expected shape, so a page returning something unrecognisable fails loudly instead of writing blank rows. Row counts are compared against the previous run, because a scraper returning 12 records where it returned 4,000 yesterday is broken even when nothing errored. You get an alert naming the field. Within the support window I fix breakages as part of the engagement; after it, a small maintenance arrangement covers it.
How do you deal with rate limits and blocking?
By being a well-behaved client first. Requests are spaced deliberately, run outside the target's peak hours where timing does not matter, cache aggressively so the same page is never fetched twice, and identify honestly rather than impersonating a browser to disguise what they are. Most blocking that people encounter is a direct response to hammering a site far harder than any human would. Where a site actively defends against automated access — serious CAPTCHA challenges, sophisticated fingerprinting — I read that as the operator declining, and I will tell you rather than escalating an arms race. At that point the productive options are an official API, a data licence, or asking. I do not build CAPTCHA-solving or detection-evasion systems.
How do I actually receive the data, and how often?
Whatever fits where the data is going. A one-off extraction is usually a CSV or Excel file, or a Google Sheet if several people need to look at it. An ongoing feed is better written straight into a database table or appended to a sheet on a schedule, so nothing depends on anyone remembering to run something. Where it feeds another system, I expose it as a small JSON API or push it directly through that system's own API. Frequency follows how fast the source actually changes — hourly for prices and stock, daily for listings, weekly for reference data. Scraping more often than the data changes just costs money and irritates the source.
Need data that only exists on a website?
Send me the URL and the fields you want. You will get a straight answer on whether it is feasible, permissible, and worth doing.
Start with a free discovery call