A thread pool exists so you do not create a thread per task. It has a size, and choosing that size is one of the more consistently mishandled configuration decisions in server software.
The common approach is to size it to expected concurrency: a thousand concurrent requests, so a large pool. That is the wrong basis, and the right one depends on what the threads actually do.
What changed in 2026
- Lightweight runtime threads changed the calculus. Where available, they removed much of the pooling motivation for I/O-bound work.
- Separate pools per workload type became standard advice. Mixing CPU and I/O work in one pool got recognised as a common cause of unexplained latency.
- Bounded queues with explicit rejection became the default. Unbounded queuing was widely recognised as converting overload into memory exhaustion.
- Observability improved. Queue depth and rejection metrics became standard rather than something you added.
Two opposite answers
CPU-bound work — computation with no waiting. The threads compete for cores. Beyond roughly the core count, additional threads add context switching without adding throughput; each thread gets a smaller slice and the total work done stays the same or falls.
A pool near the number of available cores is correct here. More is actively worse.
I/O-bound work — threads spend most of their time waiting on network or disk. A waiting thread uses no CPU, so a pool can be much larger than the core count and still be useful. The constraint becomes memory per thread and whatever the downstream resource can handle.
| Workload |
Pool size basis |
| Pure CPU |
Around the core count |
| Mostly waiting |
Much larger; bounded by the downstream limit |
| Mixed |
Separate pools |
| Database calls |
Bounded by the connection pool, not the core count |
That last row is important. A thread pool larger than your database connection pool means threads queue for connections rather than doing work — you have moved the bottleneck without relieving it, per connection limits.
Separate the workloads
Mixing CPU-heavy and I/O-heavy tasks in one pool produces the worst of both.
Size it for CPU work and I/O tasks queue behind computation, wasting time that could have been spent waiting concurrently. Size it for I/O work and CPU tasks oversubscribe the cores, making everything slower.
Separate pools let each be sized correctly, and they provide isolation: a burst of expensive computation cannot starve the pool serving requests.
The same reasoning applies to isolating by importance — a pool for user-facing requests and another for background work means background load cannot consume the capacity users need.
Bound the queue
A thread pool has a queue for tasks waiting for a thread. Whether that queue is bounded is a more consequential decision than the pool size.
An unbounded queue never rejects. Under sustained overload it grows until memory is exhausted, and long before that, tasks sit in the queue so long that their results are useless — the client timed out ten seconds ago.
A bounded queue rejects when full. That is unpleasant and it is honest: the system is telling you it cannot accept more work, promptly, rather than accepting it and failing later.
Rejection needs a policy. Fail the request with a clear error is usually right, since the caller can retry or degrade. Executing the task on the calling thread is another option that applies natural backpressure to whoever is submitting.
The metrics to watch are queue depth and rejection rate. Consistently deep queues mean the pool is undersized or the downstream is the constraint; rising rejections mean genuine overload — see backpressure explained.
Common mistakes
- Sizing to expected concurrency. Wrong basis entirely.
- Large pools for CPU-bound work. Context switching without throughput.
- One pool for everything. Wrong size for both workloads.
- Unbounded queues. Overload becomes memory exhaustion.
- A thread pool larger than the connection pool. Moves the bottleneck.
- No queue depth or rejection metrics. Invisible saturation.
- Ignoring per-thread memory. Large pools consume real memory.
FAQ
How do I size an I/O-bound pool?
Start from what the downstream resource supports — connection pool size, external API rate limit — and size to that rather than to a formula. The thread pool should not be able to overwhelm what it calls.
Should I use one global pool?
Rarely. Separate pools by workload type and by importance give better sizing and isolation, and the overhead of having several is negligible.
What about lightweight threads?
Where a runtime provides them, pooling for I/O-bound work becomes largely unnecessary — you spawn per task. CPU-bound work still benefits from bounding concurrency to core count.
How do I choose a queue size?
Small enough that a queued task still has a chance of being useful when it runs. Longer than a client's timeout is pointless — the result will be discarded.
Where to go next
For the concurrency model choice, read async vs threads. For shedding load rather than queueing it, backpressure explained, and for the downstream constraint that usually caps pool size, connection limits.