How to scrape JavaScript-rendered pages without managing Playwright

What you'll build

By the end of this guide you will have a workflow that sends a public URL to UnblockingAPI, receives fully rendered HTML, parses the content with a standard HTML parser, validates the extracted fields and stores or processes the results. The architecture looks like this:

UnblockingAPI handles browser rendering, JavaScript execution and page delivery. Your application handles parsing, validation and everything downstream. Common use cases include:

• — Single-page applications. React, Vue and Angular apps that render all content client-side.
• — Dynamic product catalogs. Prices, availability and variants loaded via JavaScript after page load.
• — Search result pages. Results populated by client-side API calls rather than server-rendered HTML.
• — Public dashboards. Data rendered via JavaScript charting and visualization libraries.
• — Tabbed and accordion content. Page sections where the content is populated by JavaScript and present in the rendered DOM.

Why JavaScript-rendered pages are different

Traditional server-rendered pages deliver all content in the initial HTML response. A tool like curl, fetch or axios can retrieve the page and you will find the data you need inside the HTML. JavaScript-rendered pages work differently.

Server-rendered HTML

The server generates complete HTML and sends it in the response. All visible content is in the initial document. A plain HTTP request returns the same HTML a browser would display. This is the simplest case — no rendering needed.

Client-rendered HTML

The server sends a minimal HTML shell — often just a <div id="root"></div> — along with JavaScript bundles. The browser downloads and executes the JavaScript, which makes API calls, processes the responses and renders the actual content into the DOM. A plain HTTP request returns only the empty shell.

API-loaded content

Some pages start with a server-rendered structure but load key data sections asynchronously. Prices, reviews, availability or personalized recommendations appear only after JavaScript fetches data from internal APIs and inserts it into the page. The initial HTML contains the layout but not the data.

Hydrated applications

Frameworks like Next.js and Nuxt can server-render the page and then "hydrate" it — the client-side JavaScript reattaches to the existing HTML to make it interactive. In these cases, the initial HTML may contain the data you need, but some fields might still be populated or updated after hydration completes. Check the raw HTML before enabling rendering — you may not need it.

Data in the initial HTML vs the rendered DOM

Before enabling rendering, check whether the data you need is already available in the raw HTML response. Many sites embed structured data that a plain HTTP request can access — you only need a browser when the content genuinely requires JavaScript execution.

The following script fetches a page with a plain HTTP request and checks whether expected content markers exist in the raw HTML:

const res = await fetch("https://example.com/products/widget");
const html = await res.text();

const markers = {
  title: html.includes('class="product-title"'),
  price: html.includes('class="product-price"'),
  jsonLd: html.includes('application/ld+json'),
  nextData: html.includes('__NEXT_DATA__'),
  initialState: html.includes('__INITIAL_STATE__'),
};

console.log(markers);
// { title: false, price: false, jsonLd: true, nextData: false, initialState: false }
// JSON-LD is present — check if it contains the fields you need

If the target data is present in JSON-LD, __NEXT_DATA__ or __INITIAL_STATE__, you can extract it from the raw HTML without rendering. These embedded data blocks often contain the same product, pricing or listing information that the page would display visually.

If the content markers return false and no embedded data blocks contain what you need, the page likely requires JavaScript rendering.

How to detect whether a page needs JavaScript rendering

Use this checklist to determine if a target page requires browser rendering:

1. Fetch the page with curl or a plain HTTP client and save the response.
2. Open the same URL in a browser. Right-click and "View Page Source" — this shows the initial HTML the server sent.
3. Right-click and "Inspect" — this shows the live DOM after JavaScript execution.
4. Compare the two. If your target data is in View Source, you can use a plain HTTP request. If it only appears in the Inspector, you need rendering.

Common signals that a page needs rendering:

• • The HTML body contains only a root element like <div id="root"> or <div id="__next">
• • Large JavaScript bundles are loaded in the <head> or before the closing </body>
• • The raw HTML contains no visible text content that matches what the browser displays
• • Content loads progressively — spinners, skeleton screens or placeholder elements appear first

The basic Playwright approach

The standard approach for scraping JavaScript-rendered pages is to launch a headless browser, navigate to the page, wait for the content to load, and extract the data. Here is a minimal Node.js example using Playwright:

import { chromium } from "playwright";

const browser = await chromium.launch();
const page = await browser.newPage();

await page.goto("https://example.com/products/widget", {
  waitUntil: "networkidle",
});

const title = await page.$eval(
  ".product-title",
  (el) => el.textContent?.trim()
);

const price = await page.$eval(
  ".product-price",
  (el) => el.textContent?.trim()
);

console.log({ title, price });
await browser.close();

This works well for local development and small-scale tasks. But running Playwright in production requires managing several things:

• • Downloading and maintaining compatible browser binaries
• • Memory for concurrent browser workloads — actual usage depends on page complexity, browser configuration and how browsers, contexts and pages are reused
• • Browser lifecycle management — launching, closing, crash recovery and resource cleanup
• • Anti-detection measures — fingerprinting, proxy rotation and TLS configuration
• • Infrastructure scaling — more pages means more browser instances and more server resources

At production scale, browser infrastructure becomes the bottleneck — not the data extraction itself.

First request with UnblockingAPI

Instead of managing your own browser, send the URL to UnblockingAPI with browser rendering enabled. The API executes JavaScript and returns the fully rendered HTML. The examples below use the parameter conventions from the existing guides — see the API documentation for the full specification.

curl \
  -H "X-Api-Key: YOUR_API_KEY" \
  "https://api.unblockingapi.com/unblock?url=https://example.com/products/widget&render=true"

Key parameters:

The response includes the rendered HTML and metadata:

{
  "job_id": "abc-123",
  "url": "https://example.com/products/widget",
  "status": "succeeded",
  "http_response_code": 200,
  "response_time_ms": 2845,
  "render": true,
  "location": null,
  "response_format": "html",
  "response": "<!doctype html><html>...rendered content...</html>"
}

The response field contains the fully rendered HTML — equivalent to what you would see in a browser's DevTools Elements panel after JavaScript has executed. See the API documentation for the full response specification.

Waiting for dynamic content

Some pages load content in stages. The initial page may render quickly, but key data — prices, availability, reviews — arrives later through asynchronous API calls or deferred JavaScript execution. A common mistake is to read the HTML too early, before the data has been inserted into the DOM.

Common wait strategies in Playwright

When running Playwright locally, you might use waitForSelector, waitForLoadState("networkidle") or waitForFunction to delay extraction until the content appears. These work but require you to know the exact selector or condition in advance, and they run inside your own browser instance.

Managed rendering

When you enable browser rendering, UnblockingAPI loads the page in a real browser and returns the rendered HTML. Check the API documentation for available parameters that control wait behavior and rendering options.

Extracting data from rendered HTML

Once you have the rendered HTML, parse it with a standard HTML parser. UnblockingAPI returns raw HTML — your application is responsible for extracting the fields you need. Here is an example using Cheerio:

import * as cheerio from "cheerio";

function extractProduct(html) {
  const $ = cheerio.load(html);

  // Try JSON-LD first — structured data is more reliable than DOM selectors
  const jsonLd = $('script[type="application/ld+json"]')
    .toArray()
    .map((el) => {
      try { return JSON.parse($(el).html()); } catch { return null; }
    })
    .filter(Boolean)
    .find((obj) => obj["@type"] === "Product" || obj["@graph"]?.find((n) => n["@type"] === "Product"));

  if (jsonLd) {
    const product = jsonLd["@type"] === "Product"
      ? jsonLd
      : jsonLd["@graph"].find((n) => n["@type"] === "Product");

    return {
      source: "json-ld",
      title: product.name,
      price: product.offers?.price,
      currency: product.offers?.priceCurrency,
      availability: product.offers?.availability,
      sku: product.sku,
    };
  }

  // Fall back to DOM selectors
  return {
    source: "dom",
    title: $(".product-title").text().trim() || null,
    price: $(".product-price").text().trim() || null,
    currency: null,
    availability: $(".stock-status").text().trim() || null,
    sku: $('[data-sku]').attr("data-sku") || null,
  };
}

The extraction order matters. JSON-LD and embedded structured data are more stable than DOM selectors — they are part of the page's data contract, not its visual presentation. Use this priority:

1. script[type="application/ld+json"] — JSON-LD structured data
2. Embedded application state — __NEXT_DATA__, __INITIAL_STATE__, inline JSON in script tags
3. Semantic HTML attributes — data-* attributes, itemprop, aria-*
4. CSS selectors — class names and element structure
5. Text content — last resort, most fragile

Lazy loading and infinite scroll

Many modern pages defer content loading to improve initial page speed. Images load as you scroll into view, product listings appear in batches, and additional content is fetched when you reach the bottom of the page. This creates challenges for automated data collection.

Lazy-loaded content

Images and content blocks that use loading="lazy", Intersection Observer or scroll-triggered loading may not appear in the rendered HTML unless the browser scrolls to their position. For images, look for data-src or data-lazy attributes — these contain the real image URL that gets swapped in during scroll.

Infinite scroll

Pages that load more items as you scroll typically fetch data from an internal API endpoint. Rather than simulating scroll events, find the underlying API and request it directly. Look in your browser's Network tab for XHR or Fetch requests that fire when you scroll — they often accept pagination parameters like ?page=2 or ?offset=20.

If the underlying API is accessible, fetching it directly through UnblockingAPI (without render=true) is typically faster and more reliable than rendering the full page.

Sessions, cookies and state

Some JavaScript-rendered pages rely on cookies, session state or local storage to display content. A first-time visitor may see a cookie consent banner, a location selector or region-specific pricing that depends on a previously set cookie.

Some pages display different content depending on cookies or stored state — a consent banner on first visit, a region prompt, or default pricing until a location is chosen. How cookies and browser state are handled during a rendered request affects what the returned HTML contains. Check the API documentation for details on session and cookie behavior.

Why Playwright scrapers get blocked

Running Playwright locally gives you a real browser, but websites can still detect automated browser sessions. This is not a Playwright-specific problem — it affects any automated browser tool.

Common detection signals include:

• — WebDriver flag. Automated browsers set navigator.webdriver = true by default. Scripts can detect this property.
• — Browser fingerprint. Headless browsers have distinct patterns — missing plugins, unusual screen dimensions, specific rendering quirks — that fingerprinting scripts can identify.
• — CDP detection. The Chrome DevTools Protocol used by Playwright can be detected by JavaScript running on the page.
• — Network-level fingerprints. Characteristics of the TLS and HTTP/2 handshake can be analyzed and correlated with known automation setups, depending on how the browser and its network stack are configured.
• — IP reputation. Datacenter IP addresses are commonly associated with automated traffic and may receive different treatment than residential IPs.
• — Behavioral patterns. Automated browsers navigate, scroll and interact differently from human users — consistent timing, no mouse movement, instant navigation.

Addressing each of these individually is possible, but the maintenance burden adds up. Anti-detection methods evolve, fingerprinting libraries update, and the techniques that worked last month may not work today. This is the infrastructure layer that managed rendering services handle for you.

Playwright vs managed browser infrastructure

Both approaches have legitimate use cases. The right choice depends on your scale, infrastructure and what you are building.

When self-managed Playwright makes sense

• • Local development and prototyping — quick iteration without API keys
• • Small-scale projects where blocking is not a problem
• • When you need precise browser control — screenshots, PDF generation, multi-step interactions
• • When you already have working browser infrastructure

When managed infrastructure makes sense

• • Production workloads where browser maintenance outweighs the API cost
• • When anti-detection is a recurring engineering problem
• • When you want to focus engineering time on data extraction, not browser operations
• • When scaling up means increasing API concurrency, not provisioning more servers

For a deeper look at how UnblockingAPI's rendering pipeline works, see How it works.

Data quality, retries and scaling

Browser rendering alone does not guarantee correct data. The page may return a challenge page, an error message with a 200 status, content that varies by session, or HTML where your selectors no longer match. Separate fetch failures from parse failures and handle each independently.

Validating extracted data

After extracting fields, verify that the data is usable before storing or processing it:

function validateProduct(product) {
  const required = ["title", "price"];
  const missing = required.filter((field) => !product[field]);

  if (missing.length > 0) {
    console.warn("Missing fields:", missing.join(", "), product.url);
    return false;
  }

  // Check for challenge pages or error content
  if (product.title.toLowerCase().includes("access denied")) {
    console.warn("Possible challenge page:", product.url);
    return false;
  }

  return true;
}

Retrying failed requests

Not every request will succeed on the first attempt. Network issues, temporary rate limits and intermittent rendering failures are normal. Use exponential backoff with a maximum retry count:

async function fetchRendered(url, apiKey, maxRetries = 3) {
  for (let attempt = 1; attempt <= maxRetries; attempt++) {
    const res = await fetch(
      `https://api.unblockingapi.com/unblock?${new URLSearchParams({
        url,
        render: "true",
      })}`,
      { headers: { "X-Api-Key": apiKey } }
    );

    if (res.ok) {
      const data = await res.json();
      if (data.status === "succeeded") return data;
    }

    if (attempt < maxRetries) {
      const delay = Math.min(1000 * Math.pow(2, attempt), 10000);
      await new Promise((r) => setTimeout(r, delay));
    }
  }

  return null; // All retries exhausted
}

Common failure modes

• — Empty HTML. The page returned a shell without content — rendering may not have completed, or the page requires specific conditions to display data.
• — Selectors missing. The site has redesigned and your CSS selectors no longer match. Track parse success rates to catch this early.
• — 403 or challenge page. The site returned an access-denied response or a verification challenge. Site-specific results may vary.
• — Timeout. The page took too long to render. Some pages are slow by design — heavy JavaScript, slow API calls or large asset downloads.
• — Locale-dependent content. The page returned different content based on the detected location. Use the location parameter to control this.
• — Inconsistent DOM. The page structure varies between loads — A/B tests, personalization or randomized layouts.

Scaling considerations

At scale, use a queue-based architecture: your scheduler adds URLs to a queue, worker processes fetch and parse them, and results flow into your data store. This separates concerns and lets you control concurrency, retry independently, and monitor each stage. Respect your plan's concurrency and rate limits — see Pricing for details.

What UnblockingAPI handles

UnblockingAPI handles the rendering and delivery layer. Your application handles the data extraction layer.

Common workflows

JavaScript rendering applies to many types of pages. Here are common workflows developers build with rendered HTML.

SPA content extraction

Single-page applications built with React, Vue or Angular render all content client-side. The initial HTML contains only a root element and JavaScript bundles. With render=true, you receive the same fully rendered DOM the browser would display.

Dynamic pricing pages

Product pages that load prices via JavaScript after the initial page render. The price may depend on the user's location, selected variant or promotional state. See the e-commerce product pages guide for detailed pricing extraction patterns.

Search result pages

Search engines and directory sites may populate results through client-side API calls. The rendered HTML includes the results present after the initial page load — including links, snippets and structured data that a plain HTTP request would miss. Note that some result pages load additional items only on scroll or interaction, so the first render may not contain everything.

Public dashboards and reports

Data visualizations, charts and public analytics dashboards often render content through JavaScript charting libraries. The underlying data may be extractable from embedded JSON or data attributes in the rendered DOM.

Content behind tabs or accordions

Pages that organize content into tabs, accordions or collapsible sections may defer loading that content until the user interacts with the UI element. Whether rendering helps depends on how the page is built: if the content is present in the DOM but visually hidden, it will be in the rendered HTML. If the content is only fetched when the user clicks the tab or expands the accordion, rendering the page alone will not trigger that interaction — inspect the DOM to see which case applies. For real estate and property data workflows, see the property listing pages guide.

Responsible use

Frequently asked questions