Scrapeless Wiki

What Causes a Request Timeout? – Diagnosis and Fixes

Troubleshooting P2 what causes a request timeout

Learn the difference between connect, read, and total timeouts, which layer owns each one, how to diagnose a hang, and how to set sensible values.

"Timeout" describes several different failures that happen at different stages. Identifying which stage expired turns an opaque hang into a specific, fixable problem.

1. The Three Timeouts

Timeout Covers A failure means
Connect TCP handshake, plus TLS The host is unreachable, filtered, or not listening
Read Waiting for response bytes Connected, but the server is slow or stalled
Total The whole operation Everything combined took too long

Most client libraries let these be set separately, and doing so is what makes the failure diagnostic rather than generic.

2. Causes by Stage

Connect timeouts come from DNS resolving to an unreachable address, a firewall dropping packets silently, the service not listening, or a saturated accept queue.

Read timeouts come from slow server-side work, an upstream dependency stalling, an enormous response body, or a connection that stayed open while nothing was sent.

Total timeouts usually reflect a chain β€” several redirects, a slow handshake, then slow generation β€” where no single stage was unreasonable.

3. Diagnosing a Hang

Time each phase separately rather than guessing:

curl -s -o /dev/null -w "dns %{time_namelookup}  connect %{time_connect}  tls %{time_appconnect}  ttfb %{time_starttransfer}  total %{time_total}\n" https://example.com/

A large gap between connect and ttfb points at server-side slowness. A hang before connect points at network or firewall. A slow dns value points at resolution.

4. Setting Sensible Values

  • Always set one. A missing timeout means a hang can last indefinitely and freeze a worker.
  • Connect should be short β€” a few seconds. Reachability is fast or it is broken.
  • Read should reflect real work. A report endpoint legitimately takes longer than a lookup.
  • Keep client timeouts longer than server ones, so the server produces a clear error instead of the client abandoning a request that would have succeeded.
  • Match ordering across hops. A gateway with a shorter limit than its origin produces gateway timeouts instead of application errors.

5. Timeouts in Collection Pipelines

Long timeouts hide problems and waste capacity: a worker blocked for two minutes on one page is not collecting anything else. Short timeouts turn slow-but-working pages into failures.

The workable compromise is a modest read timeout with a small number of spaced re-attempts, plus a per-host concurrency cap so one slow destination cannot consume the whole pool.

  • 408 β€” the server gave up waiting for the client's request.
  • 504 β€” a gateway gave up waiting for an upstream.
  • 524 β€” an edge proxy gave up waiting for the origin.
  • No status at all β€” the client's own timeout fired, which is a local exception rather than an HTTP response.

That last distinction matters when reading logs: a client-side timeout never had a status code to record.

7. Real-World Examples

  • A firewall change causing connect timeouts from one network only.
  • A read timeout on an export endpoint that generates a large file synchronously.
  • A pipeline stalling because no timeout was set and one host stopped responding.
  • Intermittent timeouts under load, where queueing rather than processing consumes the budget.

8. Summary

Request timeouts are three distinct failures: connect, read, and total. Separating them identifies whether the problem is reachability, server speed, or accumulation across a chain.

Always set explicit values, keep connect short and read realistic, order timeouts consistently across hops, and cap per-host concurrency so a single slow destination cannot stall everything else.