Why vLLM?
Serving a language model well is mostly a memory management problem. vLLM's contribution is treating the KV cache like virtual memory — paging it — which raises how many requests you can hold concurrently far more than a faster kernel would.
- 2026-03-15
- Programmatic DIB
- LLM inference · Serving · vLLM
The bottleneck is memory, not compute
Every in-flight request holds a key-value cache proportional to its context length. Naive serving allocates that cache contiguously for the maximum possible length, so most of the reserved memory is never used — and the wasted space, not the GPU's arithmetic throughput, is what caps concurrency.
vLLM's PagedAttention allocates the cache in fixed-size blocks that need not be contiguous, the same trick operating systems use for virtual memory. Fragmentation collapses, more requests fit at once, and throughput rises without touching the model.
Why this matters beyond throughput
Blocks that are not tied to a single sequence can be shared. Requests with a common prefix — a system prompt, a shared document — can reference the same cached blocks instead of each holding a copy, which is a large saving in exactly the workloads production systems actually run.
- LLM serving is capped by KV cache memory before it is capped by compute.
- PagedAttention pages the cache in blocks, removing fragmentation.
- Shared prefixes can share blocks rather than duplicating them.