Why GPU Memory Is Already Full Before It Has To Be
A short answer can tie up a lot of GPU memory. Paged attention splits the KV cache into blocks that a request receives when it needs them.
Tag: inference
Why a Finished Answer Still Has to Wait
Eight requests start at the same time on the same server. The shortest one is done after three tokens. Its answer still does not arrive until the longest of the eight has written its last word.
Why the Same AI Request Has Two Different Speeds
One request answers instantly, then drips out word by word. Another takes a while before the first word appears, then runs through fast. The reason: two phases with different bottlenecks.
Why a Model Gets More Expensive the Longer It Talks
Same model, same question. Every answer costs more the longer the conversation runs. The reason has a name: KV cache.
