How to Handle Pagination When Scraping – Patterns and Pitfalls
Learn how to page through listings reliably: finding the mechanism, terminating correctly, deduplicating, and detecting silent truncation.
Pagination is where collection quietly goes wrong. A loop that terminates early looks like a small dataset; one that never terminates looks like a hang; one that repeats pages looks like a busy day.
1. Identify the Mechanism First
| Signal on the page | Mechanism |
|---|---|
?page=2 or ?offset=20 in links |
Offset or page number |
| An opaque token in the next link | Cursor |
| A "load more" button | A background request, usually JSON |
| Content appearing on scroll | Infinite scroll, also a background request |
Link headers on the response |
Server-supplied navigation |
For the last three, open the network panel: the underlying request is almost always a clean JSON call that can be made directly, which is faster and far more stable than driving the interface.
2. Terminate on Evidence
Reliable stopping conditions, in order of preference:
- The server says there is no next page — absent cursor or missing next link.
- A page returns zero new records after deduplication.
- A hard cap on pages is reached.
Do not stop on a short page, since some sources return uneven page sizes legitimately. Do not rely on a stated total, which can change mid-traversal.
3. Always Cap the Loop
A cap is not optional. A next link that always resolves, a cursor the server keeps returning, or an offset the site ignores will otherwise loop forever.
Log when a cap is hit rather than exiting silently — a truncated collection reported as complete is worse than a failure.
4. Deduplicate Across Pages
Consecutive pages frequently overlap, particularly on sites where results are re-ranked between requests. Deduplicate on a stable identifier as records are written.
Measure the overlap rate too. A sudden rise usually means the sort order is unstable and pages are being reshuffled between requests, which changes what "complete" means.
5. Watch for Depth Limits
Many sites cap how deep pagination goes, returning empty pages or repeating the last one beyond a threshold.
Where a limit is hit, the answer is usually to narrow the query — by category, price band, or date range — so each narrower traversal stays within the limit. That collects the full set through several bounded traversals rather than one blocked one.
6. Preserve Session and Pace
Pagination is the point where clients most often exceed a site's tolerance, because it is a burst of similar requests to one host.
Keep the session and its cookies across pages, keep the same egress address for the traversal, and space requests rather than firing pages in parallel.
7. Validate the Result
Assert on the collection as a whole: an expected minimum record count, an overlap rate within normal bounds, and a page count consistent with previous runs.
A traversal that collected three pages where it usually collects sixty should fail rather than write.
8. Summary
Handle pagination by identifying the underlying mechanism — usually a JSON request behind the interface — terminating on server evidence rather than heuristics, capping the loop, and deduplicating as you write.
Depth limits are common and are solved by narrowing the query rather than paging harder, and every traversal deserves a completeness check before its output is trusted.