Scrapeless Wiki

How Does Pagination Work in APIs? – Patterns and Trade-Offs

Informational P2 how does pagination work in apis

Learn the four pagination patterns — offset, cursor, page number, and link headers — their trade-offs, and how to consume each correctly.

Pagination splits a large result set into pages. Which pattern an API uses determines how a client iterates, and whether the iteration is correct when data changes mid-traversal.

1. Offset and Limit

GET /items?limit=100&offset=200

Simple and familiar. Its weakness is drift: if records are inserted or deleted while paging, offsets shift and a client can skip or repeat rows.

Performance also degrades on large offsets, because the database must count past the skipped rows.

2. Cursor-Based

GET /items?limit=100&cursor=eyJpZCI6IDQxOTJ9

The server returns an opaque cursor marking where the page ended. The next request resumes from there.

Stable under insertion and deletion, and efficient at any depth. The trade is that random access is impossible — you cannot jump to page fifty without walking there.

3. Page Number

GET /items?page=3&per_page=100

Offset paging with friendlier arithmetic, and the same drift problem. Common on interfaces built for humans.

Some APIs return navigation in headers rather than the body:

Link: <https://api.example.com/items?page=4>; rel="next",
      <https://api.example.com/items?page=99>; rel="last"

The client follows rel="next" until it is absent. This is the most robust pattern to consume, because the server owns the URL construction entirely.

5. Comparison

Pattern Stable under change Deep pages Random access
Offset No Slow Yes
Cursor Yes Fast No
Page number No Slow Yes
Link header Depends on underlying scheme Depends Depends

6. Consuming Pagination Correctly

  1. Follow the server's own next link where provided.
  2. Stop on an explicit signal — an absent cursor or empty page — not on a computed total, which can change.
  3. Deduplicate on write, since overlap happens with offset schemes.
  4. Cap the loop, so a broken next-link never becomes infinite.
  5. Respect rate limits between pages; pagination is where clients most often exceed them.

7. Common Mistakes

  • Assuming a fixed page size. Servers may return fewer, and some ignore the requested size entirely.
  • Trusting a total count that shifts mid-traversal.
  • Stopping at the first short page, which some APIs return legitimately mid-set.
  • Parallelising offsets, which multiplies drift and load.
  • Reusing a cursor across sessions, when cursors can expire.

8. Summary

APIs paginate by offset, cursor, page number, or server-supplied links. Cursors are stable and efficient but disallow random access; offsets allow it and drift when data changes.

Follow the server's next link where one exists, stop on an explicit end signal, deduplicate, and cap the loop — that consumption pattern works correctly regardless of which scheme is underneath.