A server can reuse the opening of a request instead of recomputing it. That only works if the opening is identical, token for token. From the first differing token on, the saving is gone.
For every single token, the GPU re-reads the entire set of model weights just to compute that one token. Speculative decoding lets a small model guess ahead and saves exactly that wait.
Fewer bits save space for model weights. Comparing the intended tasks shows whether answers stay good enough and whether the model actually runs faster.
A short answer can tie up a lot of GPU memory. Paged attention splits the KV cache into blocks that a request receives when it needs them.
Eight requests start at the same time on the same server. The shortest one is done after three tokens. Its answer still does not arrive until the longest of the eight has written its last word.
An extraction reliably pulls values out of contracts, until one field is missing. The same call still returns a number, cleanly formatted, just invented. Why a language model is not a database, and what that means for checking its output.
