Scrapeless Wiki

What Is JavaScript Rendering? – Definition and How It Works

Informational P1 what is javascript rendering
Overlaps a live page. A published page already targets this keyword: https://www.scrapeless.com/en/blog/javascript-rendering. Publishing both risks cannibalisation — decide before this goes live.

Learn what JavaScript rendering is, how CSR, SSR and hydration differ, why it breaks naive scrapers, and how to get the data without a browser.

JavaScript rendering means the page's content is assembled in the browser by code, not delivered as finished HTML. It is the single most common reason a scraper returns an empty result from a page that clearly has content.

1. What Is JavaScript Rendering?

The server sends a minimal shell; JavaScript fetches data and builds the DOM.

  • Key idea: the HTML you download is not the HTML you see.
  • Mechanism: the shell loads a bundle, the bundle calls an API, the response is turned into DOM nodes.
  • Consequence: an HTTP client sees the shell, a browser sees the page.

What a fetch returns

<div id="root"></div>
<script src="/static/js/main.8f2a.js"></script>

Everything a user reads is created after that script runs.

2. The Rendering Strategies

Strategy HTML from server Data in initial HTML Scraper sees content
Static (SSG) Complete Yes Yes
Server-side (SSR) Complete Yes Yes
Client-side (CSR) Empty shell No No
Hydration Complete, then re-bound Usually yes Usually yes
Streaming SSR Progressive Partly Partly

Hydration is the case worth knowing: frameworks like Next.js and Nuxt render real HTML on the server and then "hydrate" it in the browser to attach interactivity. The content is in the initial response — and very often the data is too, as JSON in a __NEXT_DATA__ or similar script tag.

3. Why Scrapers Break

The failure is rarely an error. requests.get() succeeds, returns 200, and the parser finds nothing:

soup.select(".product")   # []

No exception, no signal. Unless something asserts that a product listing should contain products, the pipeline stores empty rows and looks healthy.

The second failure mode is timing. Even in a browser, reading the DOM before the data request completes captures an empty container. This is why waiting on a condition — the element existing, the response arriving — matters more than waiting on a fixed number of seconds.

4. Getting the Data Without a Browser

Before launching Chromium, check three things. In most cases one of them works.

1. The internal API. Open DevTools → Network → Fetch/XHR and reload. The page fetches its data from somewhere. That endpoint returns clean JSON, is smaller than the page, and is usually more stable than the markup around it.

GET /api/v2/products?category=shoes&page=1

2. Embedded state. Search the HTML for __NEXT_DATA__, __NUXT__, window.__INITIAL_STATE__ or a <script type="application/json">. Frameworks routinely ship the full dataset inside the page.

import json, re
m = re.search(r'__NEXT_DATA__" type="application/json">(.*?)</script>', html, re.S)
data = json.loads(m.group(1))

3. Structured data. application/ld+json blocks carry product, article and review data in a documented schema, placed there deliberately for machines.

Finding any of these turns a browser job into an HTTP job — typically an order of magnitude faster and far cheaper.

5. When You Do Need a Browser

  • Data arrives only after interaction — clicking, scrolling, submitting.
  • The API requires signed parameters generated by the page's own JavaScript.
  • Rendering depends on client state you cannot reconstruct.
  • The rendered appearance is itself what you are measuring.
  • The page is protected in ways that require a genuine browser environment.

Then wait on conditions, not on time:

page.wait_for_selector(".product", timeout=15000)

networkidle is convenient and unreliable on pages with polling or long-lived connections, which never go idle.

6. Telling Which You Are Facing

  • View source (not Inspect). Source is what the server sent; Inspect shows the live DOM. If the content is in Inspect but not in source, it is client-rendered.
  • Disable JavaScript and reload. What remains is what a plain HTTP client gets.
  • curl the URL and search for a known string from the page.

That last check takes ten seconds and settles the question definitively.

7. Real-World Examples

  • A 200 response with zero parsed rows, because the shell was all there was.
  • A React app whose entire catalogue sat in __NEXT_DATA__, making the browser unnecessary.
  • A browser scraper capturing empty containers because it read the DOM at load rather than after the data request.
  • A six-hour Selenium crawl reduced to twenty minutes by calling the API the page called.
  • A networkidle wait timing out forever on a page with an open analytics WebSocket.

8. Summary

JavaScript rendering means content is constructed in the browser rather than delivered as HTML. Client-side rendering leaves an HTTP client with an empty shell; server rendering and hydration usually do not, and hydrated pages often embed their data as JSON in the response.

The failure it causes is silent — a successful request that parses to nothing — so scrapers need assertions that catch empty results rather than storing them.

Before reaching for a browser, look for the internal API in the Network tab, embedded state blobs, and JSON-LD. One of them usually holds the data in cleaner form. When a browser really is required, wait on specific conditions rather than fixed delays or network idle.