The Serving Stack: Throughput, Memory, and Hardware Sizing

Loading a model and calling generate in a loop works, and it will serve perhaps one user acceptably. The gap between that and a production server is larger than it appears, and it is mostly about how the accelerator is kept busy.

The underlying constraint is that generation is memory-bandwidth-bound, not compute-bound. Producing each token requires reading the entire model's weights from memory, and the arithmetic performed on them is trivial by comparison. So the accelerator spends most of its time waiting on memory rather than computing.

That has a consequence that drives everything else: processing one request uses a fraction of the available compute. If you read all the weights anyway, you may as well do the arithmetic for many requests at once, because the expensive part, the memory read, is shared.

So batching is not an optimisation, it is the whole game. A server that batches well can serve many times the throughput of one that does not, on identical hardware.

What a production inference server provides beyond a loop: batching, memory management for the key-value cache, request scheduling, streaming, and continuity when requests arrive and complete at different times. vLLM and SGLang are the common open implementations, and the inference internals themselves are covered in more depth in the existing LLM inference lesson.

The single most important thing a modern inference server does, and the reason throughput improved so dramatically when it arrived.

Static batching waits to collect a batch, runs it to completion, then starts the next. The problem is that requests in a batch finish at different times, because generation lengths differ. A batch containing one request generating 800 tokens and seven generating 50 leaves seven slots idle for most of the batch, and no new work can start until the longest finishes.

Continuous batching, sometimes called in-flight batching, operates per token rather than per batch. After every generation step, completed requests leave the batch and waiting requests join it immediately. The accelerator stays saturated because a finished slot is refilled at the next step rather than at the end of the batch.

The practical effect is large: throughput improves by a substantial multiple on realistic traffic with variable generation lengths, and the benefit grows as length variance grows.

Two consequences worth internalising.

Throughput and latency stop being the same question. A larger running batch raises total tokens per second and slows each individual request, so the server exposes a max batch size that trades one against the other.

And traffic shape matters. Continuous batching helps most with many concurrent requests of varying length, which is what real traffic looks like, and helps least with a single request at a time, which is what a benchmark often measures.

Accelerator memory divides into three parts, and knowing the split is what makes sizing possible.

Model weights are fixed and computable in advance: parameters times bytes per parameter. A 7-billion parameter model in 16-bit precision is roughly 14 gigabytes; the same model at 4-bit is roughly 3.5.

The key-value cache holds the attention state for every token in every active sequence, and it grows with batch size and sequence length. This is the variable term, and on a busy server it frequently exceeds the weights.

Activations and overhead take the remainder, including the framework's own working memory.

The operational consequence: the KV cache is what limits concurrency. Once it fills, no further requests can be admitted regardless of available compute, so the server queues. Which means the practical question is not can this model fit but how many concurrent sequences fit alongside it.

That is why quantizing the weights raises concurrency: memory freed from weights becomes cache, admitting more simultaneous requests.

The KV cache has a memory management problem that a specific technique solved, and it is worth understanding because it explains a large throughput difference between servers.

The naive approach allocates a contiguous block per sequence, sized for the maximum possible length. Since most sequences are far shorter than the maximum, most of that reservation is never used. The waste is severe, frequently a majority of allocated cache memory, and it directly limits how many sequences can run at once.

Paged attention, introduced with vLLM, applies the idea operating systems use for virtual memory. Divide the cache into fixed-size blocks, allocate blocks to a sequence as it grows, and keep a table mapping logical positions to physical blocks. Blocks need not be contiguous.

What this buys. Near-elimination of the reservation waste, so far more sequences fit in the same memory. And block sharing: sequences with a common prefix, several requests using the same long system prompt, or several samples from one prompt, can point at the same physical blocks rather than duplicating them.

That second property connects directly to the prompt caching idea from the fine-tuning cursus, and it is why a shared system prompt across many requests costs far less memory than it appears to.

The practical takeaway for someone choosing a stack: this is not a marginal optimisation, and a server without equivalent memory management will serve materially fewer concurrent users on identical hardware.

Quantization in serving is a different decision from quantization in a vector index, though the principle is shared: fewer bits per value, less memory, some quality cost.

The formats in common use. 16-bit is the usual baseline for serving. 8-bit halves memory with quality loss small enough that it is frequently the default. 4-bit quarters it against the 16-bit baseline, with a quality cost that is real but often acceptable, particularly for larger models where there is more redundancy to lose.

What quantization buys beyond memory. Because generation is memory-bandwidth-bound, reading fewer bytes per weight directly increases token throughput. So it is a speed improvement as well as a capacity one, which is the opposite of the usual intuition that compression costs performance.

And the freed memory becomes KV cache, raising concurrency, which compounds the throughput gain.

The judgement to make. Quantization is not uniformly safe: quality loss is task-dependent, and it tends to show up first on the hardest inputs rather than uniformly. So a model that scores similarly on a benchmark after quantization may be noticeably worse on your specific edge cases.

The practical instruction is the same as everywhere else in this catalogue: measure on your own evaluation set, at several quantization levels, before choosing. A benchmark number from elsewhere is not evidence about your task.

Working from a traffic estimate to a hardware decision, which is the calculation people most want and most often skip.

Start with the weights. Parameters times bytes per parameter at your chosen precision. A 7-billion model at 8-bit is roughly 7 gigabytes; a 70-billion model at 4-bit is roughly 35.

Then decide how much memory remains for cache, which is total accelerator memory minus weights minus overhead. On an 80-gigabyte accelerator running a 7-billion model at 8-bit, that leaves roughly 65 gigabytes for cache and working memory.

Estimate cache per sequence. It scales with context length, model architecture and precision, and the server will report it. Divide available cache by per-sequence cost to get maximum concurrency.

Then check throughput against demand. Measure tokens per second at your target batch size, and compare against required tokens per second, which is requests per second times average output length.

And size for peak, not average, then add redundancy, since a single accelerator is a single point of failure.

The two mistakes this prevents. Sizing on weights alone, which produces a deployment that fits the model and serves three users. And sizing on average traffic, which produces a deployment that queues at every peak.

Measure rather than trusting the arithmetic: run your own load test at realistic concurrency and length distribution, because the estimate above is a starting point and not an answer.

Inference serving has its own vocabulary, and using the right metric prevents optimising the wrong thing.

Time to first token is how long before the user sees anything. This is what perceived responsiveness actually depends on in a streaming interface, and it is dominated by prompt processing, so it scales with input length.

Inter-token latency is the gap between subsequent tokens, which determines how fast the response appears to stream. Beyond a certain speed it stops mattering, since it already exceeds reading pace.

Total generation time matters for non-streaming consumers such as another program.

Throughput in tokens per second across all requests is the capacity metric, and it is what determines cost per token.

And queue time is the one teams forget: how long a request waits before the server admits it at all. Under load this dominates everything else, and it is invisible in a single-request benchmark.

The trade to be aware of: raising batch size raises throughput and raises inter-token latency, because each step now does more work. So the same configuration change improves your cost per token and makes each user's experience slower.

Which means you cannot tune without deciding which you are optimising, and that decision belongs to the product rather than to infrastructure. Report both, always, since a throughput improvement with no latency number is half of the result.

The hardware decision follows from the memory arithmetic more than from raw compute figures.

Memory capacity is the first filter. The model plus a useful amount of KV cache must fit, and if it does not, you are into multi-accelerator serving with the added complexity of tensor parallelism, which splits a model across devices and adds interconnect as a new bottleneck.

Memory bandwidth is the second, and it matters more than compute for generation. Two accelerators with similar compute and different bandwidth will differ substantially in tokens per second, which is not obvious from a specification sheet that leads with compute.

Interconnect matters only if you are splitting a model across devices, and then it matters a great deal.

Compute capability matters for prompt processing, which is compute-bound, and therefore for time to first token on long inputs.

The practical shape of the decision. Prefer one accelerator with enough memory over two smaller ones, because single-device serving avoids an entire class of complexity. Quantize to fit on a smaller device rather than adding a second, where quality permits. And rent before buying, since utilisation is uncertain until you have real traffic and the cost case depends entirely on it.

The common error is choosing on compute figures, which are prominent in marketing and are the wrong metric for the memory-bandwidth-bound workload that generation actually is.