What Is Crawlee? – Definition and How It Works

Learn what Crawlee provides — request queues, autoscaling, session pools, and a unified API across HTTP and browser crawlers — and where it fits.

Crawlee is an open-source crawling library for Node.js and Python. Its distinguishing idea is one programming model spanning plain HTTP crawling and browser-based crawling, so switching between them is a configuration change rather than a rewrite.

1. The Core Pieces

Shape

const crawler = new CheerioCrawler({
  async requestHandler({ $, request, enqueueLinks, pushData }) {
    await pushData({ url: request.url, title: $('h1').text().trim() });
    await enqueueLinks({ selector: 'a.next' });
  },
});
await crawler.run(['https://example.com/catalogue']);

Swapping CheerioCrawler for a browser crawler keeps the handler's structure intact.

2. Persistent, Resumable Queues

The request queue is stored rather than held in memory, so a crawl interrupted halfway resumes where it stopped instead of restarting.

At any meaningful scale this is the feature that saves the most time, because long crawls are interrupted routinely.

3. Autoscaling

Rather than fixing concurrency, the pool measures system load and adjusts. On a browser crawl, memory is the binding constraint and the pool backs off before exhausting it.

Per-domain limits still need setting explicitly — autoscaling protects your machine, not the destination.

4. Session Pools

Sessions bundle cookies and related state, with automatic retirement after repeated failures. This matters because a session carrying a poor history performs worse than a fresh one, and rotating them systematically is otherwise fiddly to implement.

5. Where It Fits

• Crawls of hundreds to millions of pages that need queue durability.
• Mixed workloads where most pages are static and some require rendering.
• Projects wanting storage, deduplication, and scaling without assembling them.
• Node.js projects specifically, where framework options are thinner than in Python.

6. Where Something Simpler Fits

For a handful of URLs, a fetch loop with a parser is shorter and clearer. Frameworks pay off when the queue, retries, concurrency control, and storage would otherwise be written by hand.

7. Practical Notes

• Set per-domain concurrency explicitly; autoscaling is about your resources.
• Use the HTTP crawler by default, escalating to a browser only for pages that need it.
• Store the raw response alongside extracted data so extraction can be improved later.
• Watch the queue's storage growth on large crawls.
• Honour robots.txt and keep request rates proportionate.

8. Summary

Crawlee supplies persistent request queues, autoscaling, session management, and storage behind one API that spans HTTP and browser crawling.

It is worth adopting when crawl scale makes queue durability and resumability matter, and unnecessary when a short script would do — with the usual caveat that per-destination politeness remains your responsibility.

Potential Related Glossary

• Web Crawling
• Parsing
• Headless Browser
• IP Rotation