How to scrape e-commerce product pages and pricing data with UnblockingAPI

What you'll build

By the end of this guide you will have a workflow that fetches public product pages, extracts pricing and product details, and stores timestamped snapshots for monitoring and analysis. The architecture looks like this:

UnblockingAPI handles the fetching layer — your application decides what to do with the data. Common workflows include:

• — Price monitoring. Track product prices over time and alert on changes.
• — Competitor pricing. Compare the same product across retailers.
• — Stock and availability alerts. Detect out-of-stock, restock and preorder changes.
• — Catalog intelligence. Monitor product assortments, new arrivals and delisted items.
• — Historical pricing datasets. Build time-series data for analytics and research.
• — Product data feeds. Deliver normalized product records as JSON, CSV or webhooks.

What product data can public e-commerce pages contain?

Product pages can expose a wide range of fields. Not every page contains all of them, and availability varies by retailer, region and page type. Understanding what is typically present helps you design a parser that extracts the fields you need and gracefully handles the ones that are missing.

Core product information

Most product pages include a title, description and at least one image. Beyond that, many expose structured identifiers — SKU, GTIN (EAN, UPC), manufacturer part number and brand — that are essential for matching products across sources. The canonical URL and category breadcrumb help deduplicate and classify records.

Pricing data

Pricing is rarely a single number. A page may show a current price, an original or list price, a sale price, a discount amount or percentage, and a currency. Some retailers show unit pricing, price ranges (for variant sets), member pricing, subscription pricing or quantity-based tiered pricing. The displayed price can vary by country, currency, tax presentation, selected variant and seller — so your parser should capture what is visible rather than assuming a universal format.

Availability and inventory signals

Availability text such as "In stock", "Only 3 left", "Pre-order" or "Backordered" is usually visible on the page, but it represents a signal rather than a precise warehouse count. Other availability data includes estimated delivery dates, pickup availability and variant-level stock status. Products may also be available in one region but not another.

Product variants and options

One product page can represent dozens of sellable variants — differing by size, colour, material, pack size or configuration. Each variant may have its own SKU, price, availability and image. Some pages embed all variant data in the initial HTML or JavaScript state; others load variant details only after user interaction.

Seller and marketplace data

On marketplace pages, the visible product listing may include a seller name, seller price, shipping cost, seller rating, fulfilment provider and number of competing offers. These fields are important for marketplace seller monitoring and competitive analysis.

Ratings and review signals

Average rating, review count and sometimes a rating distribution are visible on many product pages. Changes in review volume over time can signal product momentum. Note that review content itself may require pagination or separate endpoints and should be handled responsibly.

Why e-commerce product pages are difficult to fetch

A plain HTTP request to a product page often returns an incomplete response. The reasons are structural, not accidental:

• • Client-side rendering. Many e-commerce sites are single-page applications that render product details through JavaScript after the initial HTML loads.
• • Dynamically loaded prices. Prices and availability are often fetched by client-side API calls and injected into the DOM after page load.
• • Geo-specific content. Country, currency, tax display, shipping eligibility and product assortment can all vary based on the visitor's location.
• • Lazy-loaded availability. Stock status, delivery estimates and variant-level inventory may load asynchronously or only after interaction.
• • Embedded application data. Product details may live in hydration payloads, data attributes or inline script blocks rather than visible DOM elements.
• • Changing DOM structures. Selectors break when retailers redesign pages, run A/B tests or ship new frontend frameworks.
• • Challenge pages. Some responses return a 200 status but contain a bot-detection challenge instead of product data. Your pipeline needs to validate content, not just status codes.

It helps to distinguish three levels of success: fetch success (the request returned a response), usable page success (the response contains actual product content) and parsing success (your selectors extracted the expected fields). Treating these as separate states makes debugging far easier.

Send your first product-page request

Pass a public product URL with rendering and location controls:

curl \
  -H "X-Api-Key: YOUR_API_KEY" \
  "https://api.unblockingapi.com/unblock?url=https://example.com/product/widget-pro&render=true&location=de"

This guide uses example.com throughout. Real-world success depends on the target page, region and rendering mode. See the full response format reference for all available parameters.

Understand the response

A successful response contains the rendered HTML, timing data and request metadata:

{
  "job_id": "a1b2c3",
  "url": "https://example.com/product/widget-pro",
  "status": "succeeded",
  "http_response_code": 200,
  "response_time_ms": 912,
  "render": true,
  "location": "de",
  "response_format": "html",
  "response": "<!doctype html><html>..."
}

The response field contains the full HTML string. Your application loads and parses it to extract product details.

Extracting product data

Before reaching for CSS selectors, check whether the page already exposes structured data that is easier and more stable to parse.

Look for structured data first

Many e-commerce pages embed structured data for search engines. The most common formats are JSON-LD with schema.org/Product and schema.org/Offer types, Open Graph meta tags, and embedded application state in inline script blocks. When available, JSON-LD is often the most reliable source because it is machine-readable and less likely to break when the visual layout changes.

A practical extraction order:

1. JSON-LD script[type="application/ld+json"]
2. Embedded application state (inline JSON, hydration payloads)
3. Stable semantic attributes (data- attributes, itemprop)
4. CSS selectors on DOM elements
5. Rendered visible text as a last resort

import * as cheerio from "cheerio";

// After fetching via UnblockingAPI:
const $ = cheerio.load(data.response);

function isProduct(item) {
  const types = [].concat(item["@type"] || []);
  return types.includes("Product");
}

// Extract all JSON-LD blocks
const products = [];
$('script[type="application/ld+json"]').each((_, el) => {
  try {
    const json = JSON.parse($(el).html());
    // Handle top-level arrays, @graph wrappers, or single objects
    const items = Array.isArray(json)
      ? json
      : json["@graph"] || [json];
    for (const item of items) {
      if (isProduct(item)) products.push(item);
    }
  } catch {
    // Malformed JSON — skip this block
  }
});

if (products.length > 0) {
  const p = products[0];
  console.log({
    name: p.name,
    sku: p.sku,
    brand: p.brand?.name,
    price: p.offers?.price,
    currency: p.offers?.priceCurrency,
    availability: p.offers?.availability,
  });
}

Parse product data from HTML

When structured data is not available or is incomplete, parse the rendered HTML directly. Use defensive patterns — trim whitespace, handle missing elements, and preserve raw text alongside cleaned values:

import * as cheerio from "cheerio";

const res = await fetch(
  "https://api.unblockingapi.com/unblock?" +
    new URLSearchParams({
      url: "https://example.com/product/widget-pro",
      render: "true",
      location: "de",
    }),
  { headers: { "X-Api-Key": "YOUR_API_KEY" } }
);

const data = await res.json();

if (data.status !== "succeeded") {
  console.error("Fetch failed:", data.error);
  process.exit(1);
}

const $ = cheerio.load(data.response);

const title = $("h1").first().text().trim() || null;
const rawPrice = $('[data-testid="price"], .price').first().text().trim() || null;
const sku = $('[data-sku], [itemprop="sku"]').first().attr("content")
  || $('[data-sku]').first().attr("data-sku")
  || null;
const availability = $('[itemprop="availability"]').attr("content")
  || $(".stock-status").first().text().trim()
  || null;
const canonical = $('link[rel="canonical"]').attr("href") || data.url;

console.log({ title, rawPrice, sku, availability, canonical });

Selectors like [data-testid="price"] are examples only. Inspect the target page to find the correct selectors for your source.

Normalize prices correctly

Raw price text from a product page is messy. Across different markets you will encounter decimal commas (49,99), decimal points (49.99), thousands separators (1.299,00 or 1,299.00), non-breaking spaces, currency symbols and codes in varying positions, and promotional text mixed in with the number.

Other complications include distinguishing a sale price from a list price, VAT-inclusive versus VAT-exclusive display, unit pricing ("per kg"), "from" pricing for configurable products, and subscription or member-only pricing that is publicly visible but conditional.

function normalizePrice(raw) {
  if (!raw) return null;
  // Strip currency symbols, letters and whitespace
  let cleaned = raw.replace(/[^0-9.,\-]/g, "").trim();
  if (!cleaned) return null;

  // European thousands: 1.299,99 or 1.299
  if (/\d\.\d{3}(,|$)/.test(cleaned)) {
    cleaned = cleaned.replace(/\./g, "").replace(",", ".");
  }
  // US thousands: 1,299.99 or 1,299
  else if (/\d,\d{3}(\.|$)/.test(cleaned)) {
    cleaned = cleaned.replace(/,/g, "");
  }
  // Simple comma decimal: 49,99 → 49.99
  else if (/,\d{1,2}$/.test(cleaned)) {
    cleaned = cleaned.replace(",", ".");
  }

  const value = parseFloat(cleaned);
  return Number.isFinite(value) ? value : null;
}

Store both the raw displayed text and the normalized numeric value. When building historical datasets, also record the currency, price type (current, original, sale), tax-display context where known, source URL, selected variant and seller. Never overwrite the source currency with a converted value — store conversions as separate fields.

Handle product variants

A product page may represent many sellable variants — different sizes, colours, materials or configurations — each with its own price and availability. The default visible variant is not necessarily the complete product record.

Variant data already in the page

Many pages embed all variant data in the initial response — inside JSON-LD offers, inline JavaScript objects, hidden DOM elements or data attributes. When this data is available, you can extract every variant in a single request:

// Extract variants from embedded JSON state
const scriptContent = $("script:contains('productVariants')")
  .first().html();

if (scriptContent) {
  const match = scriptContent.match(
    /productVariants\s*[:=]\s*(\[.*?\])/s
  );
  if (match) {
    try {
      const variants = JSON.parse(match[1]);
      for (const v of variants) {
        console.log({
          variantId: v.id,
          size: v.size,
          color: v.color,
          price: v.price,
          available: v.available,
        });
      }
    } catch { /* malformed — fall back to DOM parsing */ }
  }
}

Variant data requires interaction

Some pages load variant details only after the user selects an option — choosing a size, changing a colour, or updating a query parameter. Strategies for these cases include fetching variant-specific URLs separately (when variant selection changes the URL), using the wait parameter to wait for variant data to load after rendering, or accepting that the default variant is the only reliably extractable data point.

Store a stable parent-product identifier alongside each variant identifier. Avoid treating every variant as an unrelated product — matching should prioritise identifiers such as SKU, GTIN or manufacturer part number where available.

Historical pricing and monitoring

Build a price-history pipeline

The workflow is straightforward: fetch the product page on a schedule, extract and normalize the current price, identify the product and variant, and save a timestamped snapshot. Over time, these snapshots become a historical pricing dataset you can query for trends, averages and anomalies.

A practical data model for each snapshot:

{
  "product_id":      "SKU-12345",
  "variant_id":      "SKU-12345-BLK-M",
  "source_url":      "https://example.com/product/widget-pro",
  "seller":          "Example Store",
  "captured_at":     "2026-07-22T14:30:00Z",
  "price":           29.99,
  "original_price":  39.99,
  "currency":        "EUR",
  "availability":    "InStock",
  "shipping_price":  4.95,
  "raw_price_text":  "29,99 €",
  "response_status": "succeeded",
  "parser_version":  1
}

Tracking parser_version helps you identify when a price shift is real versus caused by a change in your extraction logic. When you deploy updated selectors or normalization rules, increment the version so you can filter or flag snapshots that straddle the change.

Detect price changes without false alerts

The simplest comparison is straightforward:

const priceChanged =
  previous.currency === current.currency &&
  previous.price !== current.price;

Production logic usually needs more context. False alerts can be triggered by formatting-only changes, currency switches when a location parameter changes, comparing different variants of the same product, seller changes on marketplace pages, tax-display toggles, and shipping-cost changes that affect the total. Distinguish list price from sale price, detect promotion start and end dates, and record anomalies separately from confirmed price changes.

Monitor product availability

Availability monitoring follows the same snapshot-and-compare pattern. Parse stock status from schema.org values, visible status text or CSS classes, then compare each snapshot with the previous one:

function parseAvailability($) {
  // Check schema.org value first
  const schema = $('[itemprop="availability"]').attr("content") || "";
  if (schema.includes("InStock")) return "in_stock";
  if (schema.includes("OutOfStock")) return "out_of_stock";
  if (schema.includes("PreOrder")) return "preorder";

  // Fall back to visible text
  const text = $(".stock-status, .availability")
    .first().text().trim().toLowerCase();
  if (text.includes("in stock")) return "in_stock";
  if (text.includes("out of stock")) return "out_of_stock";
  if (text.includes("pre-order")) return "preorder";
  if (text.includes("back")) return "backorder";

  return "unknown";
}

Useful signals include stock status changes, restock events, variant-level availability, delivery estimate changes and regional availability differences. Remember that availability text is a displayed signal, not an exact inventory count.

Competitor price monitoring

Comparing prices across retailers requires both fetching and product matching. Fetch the same product from multiple public storefronts, then match records using stable identifiers — GTIN, EAN, UPC, manufacturer part number or SKU. Title-based matching works as a fallback but produces more false matches, especially for generic product names.

Beyond price, align on variant (same size, same colour), seller (marketplace pages may show different sellers), shipping cost, tax treatment, and regional pricing. Account for promotion windows and bundle pricing. Flag products where no confident match exists rather than silently comparing unrelated items.

If you need hands-on help tuning requests for specific targets, our developers can help.

Country-level product pricing

Use the location parameter to route requests through a specific country when you need region-specific product data. Different locations can produce legitimately different responses because of:

• • Currency and price formatting
• • Local promotions and sale events
• • Regional inventory and product assortment
• • Tax display (VAT-inclusive vs exclusive)
• • Shipping eligibility and delivery estimates
• • Language and localised content
• • Local seller availability on marketplaces
• • Market-specific URLs and redirects

Regional pricing research — comparing how the same product is priced across markets — is one of the most common reasons to use location routing for e-commerce data. See how it works for a deeper look at the routing pipeline.

Product pages, catalogs and feeds

Product-detail pages vs category pages

Product-detail pages are best for exact prices, full identifiers, variants, availability, descriptions, images, seller details and structured data. Use them when you need complete, granular product records.

Category and search-result pages are best for discovering products, monitoring assortments, broad price snapshots, new-product detection and URL discovery. They typically show less detail per product but cover many products in a single request.

Most catalog-monitoring pipelines use a two-stage architecture:

Category-page prices may differ from product-detail-page prices and should be validated against the full product page when accuracy matters. For a worked example of the same two-stage pattern applied to property listings, see How to scrape public property listing pages.

Crawl catalogs and pagination

Category pages use page-number pagination, cursor-based pagination, load-more buttons or infinite scroll. When crawling, strip tracking parameters and use canonical URLs to avoid storing duplicates. Handle changed sort orders and product disappearance gracefully — a product missing from page 3 today may have moved to page 2 rather than been delisted.

Keep product discovery and product-detail extraction as separate pipeline stages. Discovery collects URLs; extraction fetches full details. This separation makes it easy to resume, deduplicate and schedule each stage independently.

Build product data feeds

The output of your pipeline is a normalized product data feed — records that can be delivered as JSON files, CSV exports, database rows, webhook payloads or an internal API that downstream systems consume. A feed is typically built from fetched pages, parser output, normalized records, timestamps, validation checks and deduplication.

UnblockingAPI provides the fetched page content, not a ready-made structured product dataset. Your application is responsible for parsing, normalizing, validating and distributing the data in whatever format your downstream consumers need — whether that is a Google Sheet, a BI tool, an analytics pipeline or a pricing engine.

Data quality and scaling

Validate everything

Product data pipelines are fragile at the parsing layer. Common failure modes include:

• • Missing identifiers or prices after a page redesign
• • Blocked or challenge content returned with a 200 status code
• • Wrong region or currency because of stale location settings
• • Selector drift after A/B tests or frontend updates
• • Changed JSON-LD structures or missing schema types
• • Duplicate products from URL variations
• • Parser regressions after extraction-logic updates

Practical safeguards: validate that required fields are present and within expected ranges, track your parser version alongside each snapshot, retain raw responses when storage allows, log anomalies separately from normal operations, and run periodic sample reviews. Most importantly, distinguish fetch failures from parsing failures — a successful fetch with unexpected HTML is a parser issue, not an API issue.

Scale the workflow

As your product list grows, structure the pipeline around queues, scheduled jobs and per-target configuration. Use deduplication to avoid fetching the same URL twice in one cycle. Cache responses where appropriate. Implement retries with backoff for both fetch and parse failures, and track them as separate states — a product that fetches successfully but fails to parse should not be retried at the API level.

Index your database on product identifier, source URL and timestamp for efficient history queries. Use change detection to skip re-processing unchanged pages, and build idempotent operations so interrupted runs can resume cleanly. Site-specific success depends on the target page, region, rendering mode and anti-bot behavior.

What UnblockingAPI handles

UnblockingAPI handles the fetching layer. Your application handles the product-data layer.

Common e-commerce data workflows

Competitor price monitoring

Track how competing retailers price the same product by fetching their public product pages on a schedule. Match products using stable identifiers, normalize prices, and flag meaningful changes. Account for variant alignment, seller differences and regional pricing to avoid false comparisons.

Historical pricing datasets

Build time-series pricing data by collecting and storing repeated price snapshots. These datasets power trend analysis, seasonal-pricing research, dynamic-pricing models and reporting. The data accumulates over time — each fetch cycle adds a new data point.

Stock and restock monitoring

Detect when products go out of stock, come back in stock or move to preorder. This is useful for supply-chain intelligence, demand signals and restocking alerts on high-demand products.

Product catalog intelligence

Monitor product assortments to detect new arrivals, delisted products, category changes and assortment shifts. Category-page crawling combined with product-detail extraction gives a comprehensive view of a retailer's public catalogue.

Marketplace seller monitoring

On marketplace pages, track seller-level pricing, seller ratings, fulfilment methods and offer counts. Changes in seller presence can signal competitive shifts or supply-chain changes.

Regional pricing research

Compare how the same product is priced across countries using the location parameter. Regional pricing data is valuable for market-entry analysis, transfer-pricing research and understanding local competitive dynamics.

Promotion and MAP monitoring

Track promotional pricing events, sale start and end dates, and coupon-based discounts. For brands, monitoring publicly displayed retail prices against internal pricing policies can help identify policy compliance patterns. Note that pricing-policy compliance is a business decision, not a legal claim — use this data to inform internal processes.

Product matching and comparison

Build cross-retailer product-comparison tools by matching products on stable identifiers (GTIN, SKU, MPN), normalizing prices, and presenting side-by-side views. Product matching is your application's responsibility — UnblockingAPI provides the raw page data, and your logic determines which products are equivalent.

Responsible use

Frequently asked questions