How to Scrape JavaScript-Rendered Pages – Three Approaches
Learn how to extract data from client-rendered pages: find the underlying API, read the embedded state blob, or drive a real browser.
A client-rendered page returns almost nothing useful to a plain HTTP request. The content appears only after scripts run and fetch data. There are three ways to reach it, and they differ by an order of magnitude in cost.
1. Confirm the Diagnosis First
Before changing tools, check whether the content is genuinely absent from the response:
curl -s "https://example.com/product/123" | grep -c "the-price-text"
Zero matches with content visible in a browser confirms client rendering. A non-zero count means the data is present and the parser, not the fetch, is the problem.
2. Approach One: Call the Underlying API
Scripts that render content fetch it from somewhere. Open the network panel, filter to XHR or fetch, reload, and look for the request returning the data.
| Advantage | Detail |
|---|---|
| Speed | One request, no rendering |
| Stability | Data contracts change less often than layouts |
| Cleanliness | JSON needs no parsing heuristics |
| Cost | Comparable to any plain HTTP call |
Reproduce the request with its required headers. Note that internal endpoints carry no stability guarantee and their terms of use still apply.
3. Approach Two: Read the Embedded State
Many frameworks ship the initial data inside the HTML, in a script tag, so the client can render without a second round trip. Common shapes include a JSON-LD block, a state assignment such as window.__DATA__ = {...}, or a framework-specific payload element.
Extracting that blob and parsing it as JSON gives structured data from a single plain fetch — the same benefits as the API approach, without needing to discover an endpoint.
4. Approach Three: Drive a Browser
When neither an endpoint nor an embedded blob exists, render the page.
page.goto(url, wait_until="domcontentloaded")
page.wait_for_selector("article.product")
html = page.content()
Wait on a condition that proves the content arrived — a selector, a network response, or a count of elements — rather than a fixed sleep, which is either too short or wasteful.
Block images, fonts, and analytics when only text is needed; on heavy pages that alone can halve load time.
5. Choosing Between Them
- Look for the API. Best cost and stability.
- Look for the embedded state. Nearly as good, no endpoint discovery needed.
- Render in a browser. Always works, costs the most per page.
Most teams discover the first two only after building the third, and then delete a lot of code.
6. Pitfalls of the Browser Path
- Fixed sleeps produce flaky runs; wait on conditions.
- Infinite scroll requires a scroll-and-wait loop with a termination test.
- Lazy loading means content only exists once an element is in view.
- Per-page cost — memory and time per session are much higher, so concurrency needs planning.
7. Verifying What You Got
After extraction, assert that the values look right: expected record count, non-empty required fields, plausible types. A page that renders a challenge or an empty state can parse without error and yield a clean, empty result — a failure that silently looks like success.
8. Summary
Client-rendered pages have three access paths: the API the page calls, the state embedded in the HTML, and a real browser. Cost rises sharply along that order, so check the cheaper options before reaching for the browser.
Whichever path is used, wait on conditions rather than time, and validate the extracted records — the most expensive failure is the one that returns zero rows without raising anything.