What Is networkidle? – Definition and Why to Avoid It
Learn what the networkidle wait state means, why it is unreliable on modern pages, and which conditions to wait on instead.
networkidle is a navigation wait state that resolves when there has been no network activity for a short period. It sounds like exactly what a scraper wants, and it is the wait state most likely to cause trouble.
1. What It Waits For
A quiet network — no in-flight requests for a defined interval, conventionally around half a second.
The assumption behind it is that a page which has stopped fetching has finished loading. On a modern page that assumption frequently does not hold.
2. Why It Fails
| Page behaviour | Effect on networkidle |
|---|---|
| Analytics beacons on a timer | Never quiet |
| Polling for updates | Never quiet |
| Server-sent events or websockets | Never quiet |
| Lazy-loaded media as you scroll | Quiet, then not |
| Advertisement refresh cycles | Never quiet |
Any of these means the condition either never resolves or resolves long after the content was ready. Both outcomes are bad: a timeout, or a slow run.
3. It Also Does Not Mean Your Data Arrived
Even where the network does go quiet, that says nothing about whether the specific element you need exists. A page can be entirely idle with an empty results list.
Waiting on network state is therefore an indirect proxy for a question that can be asked directly.
4. What to Use Instead
page.goto(url, wait_until="domcontentloaded")
page.wait_for_selector("article.product")
Navigate with the cheap condition, then wait for evidence that the content is present. That evidence can be:
- A selector matching.
- An element count exceeding a threshold.
- A specific network response arriving.
- Text appearing on the page.
Each of those proves the thing you actually care about.
5. Waiting on a Response
Where content arrives from a known endpoint, waiting for that response is the most precise condition available:
with page.expect_response(lambda r: "/api/products" in r.url) as info:
page.goto(url, wait_until="domcontentloaded")
response = info.value
That also hands you the JSON directly, frequently making DOM extraction unnecessary.
6. When networkidle Is Acceptable
On pages known to go quiet — static documentation, simple server-rendered pages — it works and is harmless.
It is also occasionally useful during exploration, when you do not yet know what to wait for. It should not survive into production code.
7. Fixed Waits Are Worse
A fixed pause is the alternative people reach for, and it is worse: too short on a slow run, wasteful on a fast one, and it makes total runtime a function of the pause rather than of the work.
Condition-based waiting is faster and more reliable simultaneously.
8. Summary
networkidle resolves when the network goes quiet, which many modern pages never do because of analytics, polling, and persistent connections.
Navigate with domcontentloaded and then wait for a condition that proves your content exists — a selector, a count, or the response itself. That is both more reliable and faster than waiting on network state or on the clock.