How to scrape hundreds of websites every day without breaking your pipeline

Why pipelines break at hundreds of sites

A pipeline that works for ten sites usually fails at three hundred for the same few reasons.

• Every site fails differently. Some return a 403, some a captcha page with a 200 status, some an empty shell that only fills in after JavaScript runs.
• Failures are silent. A page that loads but contains no products looks like a success unless you check.
• One slow site holds up the run. Without per-site timeouts, a single hanging target delays everything behind it.
• Costs are hard to predict. If rendering, proxies and retries are billed differently, your monthly bill depends on which sites happened to be difficult that month.
• Infrastructure becomes a job. Browsers, proxy pools and their updates need someone to look after them.

None of these are solved by writing a better parser. They are solved by structuring the pipeline so that failure is expected, visible and cheap.

1. Separate fetching from parsing

The most useful design decision is to keep two stages apart:

1. Fetch: get the page and store the raw HTML with the URL, a timestamp and the status.
2. Parse: turn stored HTML into rows.

When a site changes its markup, you fix the parser and re-run it against the HTML you already have. You do not need to fetch again, and you do not lose the day's data. When a site blocks you, the problem is isolated to the fetch stage.

Store the raw response. Storage is cheap compared with a lost day.

2. Treat the fetch layer as a service you call, not a stack you run

For a handful of targets, running your own headless browsers and proxies is reasonable. At hundreds of targets you spend more time on browser versions, proxy rotation and block handling than on the data.

A web unblocking API moves that layer out of your code. With UnblockingAPI, a single /unblock endpoint fetches a public URL and returns rendered HTML, status code and metadata. JavaScript rendering, rotating residential proxies and country-level routing are part of the request. It tries a plain fetch first and uses a real browser when the site needs one, and it learns which approach a site needs. You do not configure that per site. See how it works and the docs.

The practical effect for a pipeline is that every target looks the same to your code: a URL in, a result out.

UnblockingAPI is built for public web data only. It does not support login flows or private content, so plan around that if any of your targets sit behind an account.

3. Make the cost per request predictable

At volume, pricing structure matters more than the headline rate. Some scraping APIs charge between 1 and 75 credits per request depending on rendering and proxy type, so a single hard site can consume what fifty easy ones would. That makes budgeting a guess.

UnblockingAPI charges one credit per successful request. Rendering, premium proxies, country routing and retries are included. Blocks, failed requests and timeouts are not billed.

Here is a worked example. Say you track 300 sites and fetch 10 pages from each, once a day.

• 300 × 10 = 3,000 requests a day
• 3,000 × 30 days = 90,000 credits a month

What that costs depends on the plan:

• Pay as you Go: €1.00 per 1,000 credits, so about €90 for the month. It starts with 500 free credits and needs no card.
• Scout: €49 a month for 55,000 credits. Too small for this workload.
• Builder: €199 a month for 250,000 credits. This fits with room to grow, and is the better choice once volume is steady and above what you want to buy by the thousand.

Notice that Pay as you Go is cheaper than Builder at this exact volume. Choose the plan from your real monthly count, and revisit it as your site list grows. Annual plans are ten months for twelve. Prices exclude VAT. Current prices are on the pricing page, and the credit estimate guide covers larger volumes.

The point is that because every successful request is one credit, you can do this arithmetic before you start, and it stays true when the hard sites turn up.

4. Schedule jobs so one failure stays small

Run the daily job as many small independent tasks, not one long loop.

• One task per URL, or per small batch. Put them on a queue. A stuck site then blocks one task, not the run.
• Set a timeout per task. Decide how long a page is worth waiting for, and give up after that.
• Retry with limits. Retry a failed task a small, fixed number of times with a delay. Since failed requests are not billed, retries do not add cost on the fetch side, but they still add time, so cap them.
• Spread the load. If many targets are on the same domain, space the requests out instead of sending them all at once. Be a considerate visitor.
• Make tasks idempotent. Running the same task twice should overwrite, not duplicate.

Any scheduler works here: cron to enqueue, a queue and a few workers to fetch. The structure matters more than the tools.

5. Validate every response, not just the status code

A 200 does not mean you got the data. Add a cheap check after each fetch and before parsing:

• Does the page contain an element you always expect, such as a product grid or a price?
• Is the HTML size within a normal range for this site?
• Did you get roughly the same number of items as yesterday?

When a check fails, mark that site as degraded and keep going. Do not let a bad page overwrite good data from the day before.

6. Get structured data without writing selectors for every site

Parsers are the other half of the maintenance burden. For sites you scrape often, a template can replace hand-written selectors. UnblockingAPI's Visual Editor lets you load a page, tap the fields you want and save a template that becomes an API endpoint on the same account and key. It returns JSON through the same endpoint, and you can pin a version so a change does not alter your output unexpectedly. A public template library covers common sites.

For long-tail sites with unusual layouts, you may still write your own parsing. The fetch/parse split from step 1 keeps that contained.

7. Watch the pipeline, not just the sites

Track a few numbers per run and per site:

• Success rate by site
• Time to complete the run
• Items extracted per site compared with the previous run
• Credits used

A sudden drop in items from one site is the earliest sign it changed. A drop across many sites at once usually points at your own pipeline. Alert on those two patterns and you will catch most problems before your users do.

A short checklist

1. Store raw HTML separately from parsed output.
2. Call a fetch service instead of running browsers and proxies.
3. Work out monthly credits from sites × pages × days before you commit to a plan.
4. Queue small tasks with timeouts and capped retries.
5. Validate content, not just status codes.
6. Use templates where a site is stable and important.
7. Alert on per-site item counts.

Try it on your own list

Pay as you Go gives you 500 free credits with no card. Take ten of your real targets, fetch each once and look at what comes back. If you would rather have help with the architecture or migrating an existing scraper, the team offers developer setup services, and support comes from the engineers who build the product.