What Is a Service Worker? – Definition and How It Works
Learn what service workers do, how they intercept requests, and why they interfere with scraping and caching.
A service worker is a script the browser runs in the background, separate from any page, able to intercept and respond to network requests from the pages it controls.
1. What It Can Do
| Capability | Effect |
|---|---|
Intercept fetch |
Serve from cache, rewrite, or synthesise responses |
| Cache Storage | Programmatic, persistent asset cache |
| Offline operation | Serve content with no network at all |
| Background sync | Defer work until connectivity returns |
| Push notifications | Receive server-initiated messages |
The interception capability is the consequential one: a service worker sits between the page and the network and can answer requests itself.
2. The Lifecycle
navigator.serviceWorker.register('/sw.js');
Then: install → activate → control. A newly registered worker does not control existing pages until they reload, and an updated worker waits for the old one to release control unless it claims it immediately.
That staging is why service worker behaviour differs between a first visit and a subsequent one — and why automation that always starts fresh sees only the first-visit path.
3. Why It Matters for Scraping
Responses may not come from the network. A service worker can serve a cached response, so what the page displays need not correspond to any request you observed. A HAR capture can look inconsistent with the rendered content for exactly this reason.
Content can appear with no request. Offline-capable pages render from Cache Storage, so waiting for network activity to settle is not a reliable signal that content has arrived.
First visit differs from later visits. With a persistent profile the worker is installed and serving; with a fresh profile it is not. That produces genuinely different behaviour between runs, which reads as flakiness.
It survives navigation. A registered worker persists in the profile, so state carries between runs in a way that is easy to overlook when debugging.
4. Detecting and Inspecting
const regs = await navigator.serviceWorker.getRegistrations();
regs.map(r => ({scope: r.scope, active: !!r.active}));
const caches_ = await caches.keys(); // named caches the worker created
A page with registrations and populated caches is one where content may bypass the network entirely.
5. Disabling It for Deterministic Collection
Where you want every response to come from the network — which is usually the case for collection — turning service workers off removes a large source of nondeterminism:
context = browser.new_context(service_workers="block")
That forces requests to the network, makes HAR captures complete, and eliminates first-visit-versus-later differences. For scraping, determinism is worth more than the caching benefit.
The trade: a site that genuinely requires its service worker to render will break, so verify content still appears after blocking.
6. When to Leave It Enabled
- The site is a progressive web app whose content genuinely depends on the worker.
- You are reproducing real user behaviour, including cache effects.
- You are testing your own application's offline path.
In those cases, use a persistent profile so the worker's installed state is stable rather than varying run to run.
7. Fingerprinting and Security Notes
Service worker availability, and the state of the persistent-storage and notification permissions, form part of a browser's API and permission signature — one more reason for a profile to look coherent.
Security-wise, a service worker has broad power over its scope, which is why registration requires a secure context and is limited to the script's own path scope. For anyone running automated browsers, that persistence is another reason to treat a profile directory as sensitive state rather than a disposable temp folder.
8. Summary
A service worker is a background script that intercepts network requests for the pages it controls, enabling caching, offline operation, and push messaging.
For collection it is mainly a source of nondeterminism — content can be served from cache with no observable request, and first visits behave differently from later ones — so blocking service workers makes captures complete and runs repeatable, provided you verify the content still renders without them.