What is vLLM, and why does throughput jump when you switch to it?

If you have moved a model onto vLLM and watched throughput climb several times over without touching the model itself, this post explains where that came from.

Two ideas do most of the work.

1. The KV cache is the bottleneck, not the maths

When a model generates text, it keeps a KV cache — the attention keys and values for every token so far — so it does not recompute the whole prefix on each new token. That cache is large, and it grows with every token generated.

The naive way to hold it is one contiguous block of GPU memory per request, sized for the longest output the request might produce. That is where the waste lives:

• A request allowed 2,048 tokens that stops at 100 has reserved 20× what it used.
• The reserved-but-unused memory cannot be lent to another request.
• You size the whole server for the worst case, so concurrency stays low.

Teams usually meet this as “the GPU has plenty of free memory but I still get out-of-memory errors when I raise concurrency.”

2. PagedAttention: treat GPU memory like virtual memory

vLLM’s central idea borrows from operating systems. Instead of one contiguous reservation per request, the KV cache is split into fixed-size blocks, and a block table maps each request’s logical token positions to physical blocks that can live anywhere in memory.

The consequences are immediate:

• Almost no internal fragmentation. A request consumes blocks as it generates, not up front. Waste drops to at most one partly-filled block per sequence.
• Sharing becomes free. Two requests with the same prompt prefix — a shared system prompt, a few-shot preamble, a RAG document — can point at the same physical blocks instead of holding two copies. For workloads with a long fixed prefix this alone is large.
• Concurrency rises. More requests fit in the same card, which is the actual source of the throughput number.

The name comes from exactly this: paging, applied to attention.

3. Continuous batching: stop waiting for the slowest request

The second idea is about scheduling. Classic static batching collects N requests, runs them together, and returns when all N finish. Since generation lengths vary wildly, the batch runs at the speed of its slowest member and the GPU idles on the finished slots.

Continuous batching works at the iteration level instead. After every single decoding step the scheduler can evict a finished sequence and admit a waiting one. The batch is refilled continuously rather than drained and refilled.

For mixed traffic — some requests wanting 20 tokens, some wanting 2,000 — this is the difference between a GPU that is busy and a GPU that is waiting.

What this means in practice

A few consequences worth internalising before you tune anything:

Throughput and latency are a dial, not a fixed pair. More concurrent sequences means higher total tokens/second and higher per-request latency. Decide which one your product is actually paying for.

Long shared prefixes are worth engineering for. If every request carries the same 800-token system prompt, prefix sharing is doing real work for you. Keep the shared part byte-identical and at the front — a per-request timestamp at position zero silently destroys the sharing.

--max-model-len is a memory decision. It bounds the KV cache per sequence and therefore how many sequences fit at once. Setting it to the model maximum “just in case” quietly cuts your concurrency.

Watch preemption. When memory runs out, vLLM preempts and later recomputes or swaps sequences. A rising preemption count in the metrics means you are over-subscribed — that is your signal to lower concurrency or raise memory, before latency spikes show up in user complaints.

For Thai-language workloads specifically

One detail that catches teams here: Thai tokenizes poorly in most multilingual models. Thai has no spaces between words, and tokenizers trained mostly on English frequently emit one token per character or per short cluster. A Thai prompt can consume noticeably more tokens than its English translation of the same meaning.

That lands directly on everything above — more tokens per request means more KV cache per request, which means fewer concurrent sequences on the same card. If you are budgeting GPU memory from English benchmarks and serving Thai traffic, measure your real token counts before sizing the deployment.

Where to go next

• The vLLM repository and its docs
• The PagedAttention paper, Efficient Memory Management for Large Language Model Serving with PagedAttention
• And on 6 September, ask the maintainers directly at vLLM Bangkok Day — they wrote this.