How to Handle Infinite Scroll When Scraping – A Working Method

Informational P2 how to handle infinite scroll when scraping

A practical walkthrough: find the batch request, replay it, and — when you cannot — write a scroll loop that terminates correctly.

There are two methods, and they differ by two orders of magnitude in cost. Try the first properly before accepting the second.

1. Find the Batch Request

Open developer tools, filter the network panel to fetch and XHR, scroll once, and watch what fires.

Note the URL, the method, the pagination parameter — cursor, offset, or page — and any headers the request carries beyond the ordinary. That request is the entire feature, and it can usually be replayed directly.

2. Replay It

import httpx

def batches(url, params):
    with httpx.Client(timeout=30.0, headers={"accept": "application/json"}) as client:
        while True:
            response = client.get(url, params=params)
            response.raise_for_status()
            payload = response.json()
            items = payload.get("items") or []
            if not items:
                return
            yield from items
            cursor = payload.get("next_cursor")
            if not cursor:
                return
            params = {**params, "cursor": cursor}

One request per batch, no browser, and termination driven by the server's own signal.

3. Check Whether the List Is Virtualised

Before considering the browser route, determine whether scrolling could ever work:

Scroll a long way down the page.
Count the item elements present in the DOM.

If the count stops growing while the scrollbar continues, off-screen items are being removed. The full set never exists in the DOM, so scrolling cannot produce a complete collection and the endpoint is the only option.

4. If You Must Scroll, Terminate on Progress

seen = 0
for _ in range(MAX_ROUNDS):
    page.mouse.wheel(0, 20000)
    page.wait_for_timeout(400)
    count = page.locator("article.item").count()
    if count == seen:
        break            # no new items: the list is exhausted or stalled
    seen = count

The condition that matters is "did the count grow", not "have I scrolled enough times". Fixed iteration counts either stop early or waste time.

5. Wait for the Right Thing

Better than a fixed pause: wait for the count to exceed its previous value, or for the batch response to arrive. Both prove progress rather than assuming it.

Where the page shows a loading indicator, waiting for it to disappear is also reliable and easy to express.

6. Deduplicate and Cap

Batches overlap when the underlying order shifts between requests, so deduplicate on a stable identifier as records are written.

Always cap total rounds and total records. A page that keeps returning content — or a sentinel that keeps triggering — otherwise loops indefinitely.

7. Validate the Collection

Assert on the total collected against expectation, on the overlap rate, and on whether the cap was reached. A traversal that stopped early looks exactly like a short day unless it is checked.

Log when a cap is hit; a truncated collection reported as complete is the worst outcome.

8. Summary

Handle infinite scroll by finding the batch request in the network panel and replaying it — one request per batch, terminated by the server's own cursor.

Where the list is virtualised or no endpoint is reachable, scroll in a loop that terminates when the item count stops growing, deduplicate as you write, cap the rounds, and validate the total before trusting it.