Why the Same AI Request Has Two Different Speeds

What is the difference between prefill and decode?

Prefill processes the entire prompt in one parallel computation step and sets how long the first word takes. Decode then generates each further word one at a time and sets how fast the answer keeps running. The two phases run into different bottlenecks.

Two requests to the same model. Same hardware, same day. One answer arrives instantly, then drips out word by word, painfully slowly. The other takes a while before the first word shows up, then runs through fast.

Same model, same system, two completely different waiting experiences. That is not chance. It comes down to how long the prompt is. And how long the answer turns out to be. The model does two different things for the same request, at two different speeds.

Everything at once, then word by word

The model processes the prompt in a single pass: all tokens at the same time, in one parallel computation step. The longer the prompt, the more work sits in that one pass. That is the wait before the first word.

The answer comes about differently. The model generates exactly one token, looks at everything that is on the page by now, and only then generates the next one. Every further word costs its own step. That is the dripping afterwards.

These two phases have names: prefill and decode.

Here is where this term sits in the series:

Under the Hood

During prefill, the model processes all prompt tokens at once, in a single computation pass. That fully saturates the GPU’s matrix-multiply units: the phase is compute-bound and sets the time to first token (TTFT).

During decode, exactly one new token is produced per step. Each step re-reads the entire model weights and the KV cache from memory, doing comparatively little arithmetic per byte read. The phase is memory-bound and sets the gap between tokens (inter-token latency, ITL, also called time per output token, TPOT).

How strongly a phase saturates which GPU resource can be expressed as arithmetic intensity: compute operations per byte moved. The roofline model plots this intensity against achievable performance. High intensity (prefill) hits the compute ceiling, low intensity (decode) hits the memory ceiling. This single distinction explains why an optimization like quantization or continuous batching helps one phase and barely touches the other.

This term builds on the KV cache: Why a Model Gets More Expensive the Longer It Talks

Where It Breaks

Anyone who plans capacity around a single number called “response time” regularly optimizes the wrong lever. More concurrent requests in a batch raise total throughput, which helps most when many requests with short prompts are queued up, i.e. decode-heavy requests. But when a user is waiting on the summary of a long contract, what counts is how fast the first word arrives, i.e. the prefill-heavy side. More batching can make exactly that worse, because the request sits longer in the queue.

For German-language operations there is a second effect: German text with compound nouns and umlauts breaks into noticeably more tokens in common tokenizers than the same content in English. A single word shows it: Auftragsverarbeitungsvertrag breaks into eight pieces in cl100k, the tokenizer of the 2023 GPT-4 generation, namely Auf, tr, ags, ver, arbeit, ungs, ver, trag. The English data processing agreement gets tokenized just the same, but each of its three words exists as its own token in the vocabulary: three tokens total. Newer tokenizers narrow the gap, to six against three in the same example; they do not close it. That primarily lengthens the prompt, i.e. the compute-bound prefill phase. Anyone sizing an AI project’s cost from English-language benchmarks regularly underestimates how much extra prefill work a German contract or transcript generates.

The fix is to measure TTFT and ITL separately instead of one blurred “response time,” and to plan capacity around goodput: the share of throughput that actually meets the latency targets set for the system. The concrete levers for that, continuous batching and paged attention, are their own chapters in this series.

Prefill processes the prompt all at once, decode produces the answer word by word. Measuring both with the same number optimizes the wrong lever.

Next term: sampling, temperature, and top-p. Why the same model never gives exactly the same answer to the same question twice.

Frequently Asked Questions

What is the difference between prefill and decode? Prefill processes the entire prompt in one parallel computation pass and determines how long it takes for the first word of the answer to appear. Decode then generates each further token one at a time and determines how fast the answer keeps running after that.

What is TTFT (time to first token)? TTFT is the wait from sending a request to the first visible word of the answer. It is determined by the prefill phase.

Why does decode take longer per token than prefill? Prefill processes all prompt tokens at once and uses the GPU’s full compute throughput. Decode generates one token at a time and, at every single step, has to re-read the entire model weights and the KV cache from memory.

What do “compute-bound” and “memory-bound” mean? Compute-bound means the GPU’s raw arithmetic throughput is the bottleneck. Memory-bound means the speed of reading data from memory is the bottleneck. Prefill is typically compute-bound, decode is memory-bound.

Part of the series AI Engineering Explained. Related: Why a Model Gets More Expensive the Longer It Talks (KV cache) and What Does an AI Project Cost?