What Is a Thread Pool? – Definition and How It Works
Learn what a thread pool is, how to size one for I/O versus CPU work, and the queue behaviour that causes stalls.
A thread pool is a fixed set of reusable worker threads fed by a task queue. It exists so that creating a thread per task — expensive, and unbounded — is avoided.
1. Why Not a Thread Per Task
Each thread costs memory for its stack and scheduler overhead. A thousand concurrent tasks means a thousand threads, and the machine spends its time context-switching rather than working.
A pool bounds that: a fixed number of workers, a queue for the rest. Concurrency is capped by design, which is the property that keeps a system predictable under load.
2. Basic Use
from concurrent.futures import ThreadPoolExecutor, as_completed
with ThreadPoolExecutor(max_workers=16) as pool:
futures = {pool.submit(fetch, url): url for url in urls}
for fut in as_completed(futures):
url = futures[fut]
try:
handle(url, fut.result())
except Exception as exc:
log_failure(url, exc)
Two details matter more than they look. as_completed yields results as they finish rather than in submission order, so a slow task does not hold up fast ones. And calling .result() inside a try is mandatory — an exception in a worker is stored on the future and vanishes silently if the result is never retrieved.
3. Sizing: I/O Versus CPU
The right size depends entirely on what the tasks do.
| Workload | Sizing guide |
|---|---|
| CPU-bound | Around the core count; more only adds switching |
| I/O-bound | Far more than cores — threads are mostly waiting |
| Mixed | Separate pools, sized separately |
For network work, threads spend nearly all their time blocked, so dozens or hundreds can be useful. The binding constraint is usually not local CPU at all — it is the target's tolerance, the proxy pool, or connection limits.
Which means: size an I/O pool by what the remote side accepts, not by your core count.
4. In Python, Mind the GIL
Python's global interpreter lock means threads do not execute Python bytecode in parallel. A thread pool therefore gives no speedup for CPU-bound Python work — use a process pool for that.
For I/O-bound work threads are fine, because the lock is released while waiting on a socket. Fetching pages in threads works; parsing them in threads mostly does not, since parsing is CPU work.
The practical consequence for a collection pipeline: threads for fetching, processes for heavy parsing.
5. Queue Behaviour Causes the Surprises
The queue in front of a pool has properties that determine failure mode:
Unbounded queue — submissions never block, and memory grows without limit when producers outpace workers. The failure is an out-of-memory kill far from the cause.
Bounded queue — submission blocks or is rejected when full, applying backpressure. Almost always the right choice.
No queue — every submission needs a free worker or is rejected.
Backpressure is what stops a fast producer from converting a slow consumer into a crash.
6. Blocking Inside a Worker
Two failure modes worth knowing because they present as an inexplicable hang:
No timeout on I/O. One request without a timeout occupies a worker forever, and enough of them exhaust the pool. Every network call in a pooled task needs a timeout.
Pool exhaustion by nesting. A task that submits to the same pool and waits for the result can deadlock — all workers waiting on tasks that need a worker. Use a separate pool for nested work, or restructure so tasks do not wait on each other.
7. Thread Pools Versus Async
For I/O concurrency, async offers a lighter alternative: thousands of concurrent operations on one thread, without per-thread stacks.
Thread pools remain preferable when the libraries you need are synchronous, when the work mixes I/O with CPU, or when the code is easier to reason about sequentially. Async is preferable at very high concurrency, and requires the whole path to be non-blocking — one synchronous call blocks the entire event loop.
8. Summary
A thread pool caps concurrency with a fixed set of reusable workers fed by a queue, avoiding the cost of a thread per task.
Size it by core count for CPU work and by what the remote side tolerates for I/O work, use a bounded queue so backpressure replaces unbounded memory growth, set a timeout on every network call so one hang cannot exhaust the pool, and retrieve every future's result so worker exceptions are not swallowed.