--- title: "Why a Model Gets More Expensive the Longer It Talks" source: https://bitvaria.com/en/kv-cache --- 20\. Juni 2026 · Daniel Schleipfer · [AI](https://bitvaria.com/category/ai/)  · 5 min read # Why a Model Gets More Expensive the Longer It Talks Same model, same question. Every answer costs more the longer the conversation runs. The reason has a name: KV cache. ![Same model, same question. Every answer costs more the longer the conversation runs. The reason has a name: KV cache.](https://bitvaria.com/_astro/kv-cache.DoXy8WTH.jpg) > **What is the KV cache?** > > The KV cache is a buffer in GPU memory. The model stores the computed results of all previous tokens there so it can read them on the next step instead of recomputing everything. It grows with every token and is usually the first thing to run out. Same model, same question. Yet every answer gets slower and more expensive the longer the conversation runs. Ten messages exchanged, and the difference is measurable. The model is not getting worse. The reason lies in what it holds onto: for every token it has processed once, it keeps intermediate results around. And that stockpile grows. ## Compute once, read back forever For every new token, the model needs the computation results of all tokens before it. Without an intermediate store, it would have to recompute the entire conversation history at every step. Instead, it computes each token exactly once and places the result in GPU memory. On the next step it reads those stored results directly; only the new token gets computed. The price: that store grows with every token, and the memory it lives in is finite. In long conversations, the limit is not set by computation but by space. This intermediate store has a name: **KV cache**. Here is where this term sits in the series: The series, and where this term sits This term sits at "Inference" in the group "The Machine". ## Under the Hood Without the cache, the model would recompute attention over the entire sequence so far for every new token. Instead, it stores the key and value tensors of all processed tokens in VRAM and reads them on the next step. The cache size works out roughly as: ``` 2 × layers × kv_heads × head_dim × seq_len × batch × bytes_per_value ``` For a modern 8B model with grouped query attention (8 KV heads, 32 layers), a single conversation at 100,000 tokens reaches around 13 GB of cache while the model weights in FP16 occupy about 16 GB. Sixteen parallel conversations at 4,000 tokens each come to around 8 GB. Older architectures without GQA run up to four times higher. That is why VRAM fills up at the cache first, not at compute. A larger context window moves this limit up, but does not make it disappear. *This term assumes familiarity with tokens: [What Is a Token?](https://bitvaria.com/en/was-ist-ein-token)* ## Where It Breaks When VRAM fills up, the system has to evict cache entries. If the wrong entries get thrown out, the model recomputes those tokens on the next call. Time and compute cost doubled. When too many long conversations hit a system at once, throughput drops. Not because the model computes slower, but because memory cannot fit more requests. For organizations working with long contexts, this lands directly. Contract analyses, multi-hour customer conversations, large knowledge documents: the longer the document, the larger the cache, the sooner the limit. Estimating AI costs from short test documents underestimates real production costs considerably. The defenses are fixed-size cache pages that reduce fragmentation (paged attention), and shared starting blocks for requests with the same opening prefix (prefix reuse). Both come in the next part. * * * **The KV cache keeps every token that was computed once, so the past never has to be computed twice. And it is the first thing to run out.** Next term: **prefill and decode**. Why the same request runs at two different speeds, and which one the user actually feels. ## Frequently Asked Questions **What is the KV cache?** The KV cache (key-value cache) is a buffer in GPU memory. The model stores the computed results of all processed tokens there and reads them on the next step instead of recomputing everything. It grows with every token and for longer conversations is often larger than the model weights themselves. **Why does an AI conversation get more expensive over time?** With every new message the KV cache grows. The model needs more GPU memory, and on busy systems throughput drops because fewer parallel requests fit in memory. **What is paged attention?** Paged attention manages the KV cache in fixed-size pages, similar to how an operating system divides virtual memory. This reduces memory fragmentation and lets requests that share the same conversation start reuse the same cache block. **Why does VRAM usage spike for long texts?** The KV cache grows linearly with sequence length and the number of parallel requests. For long documents or conversations it can exceed the size of the model weights themselves. That is usually the bottleneck that determines how many requests can run concurrently. * * * *Part of the series [AI Engineering Explained](https://bitvaria.com/en/ai-engineering-begriffe). Related: [What Is a Token?](https://bitvaria.com/en/was-ist-ein-token) and [What Does an AI Project Cost?](https://bitvaria.com/en/was-kostet-ki-projekt-mittelstand)* - [ki](https://bitvaria.com/tag/ki/) - [ai-engineering](https://bitvaria.com/tag/ai-engineering/) - [kv-cache](https://bitvaria.com/tag/kv-cache/) - [inference](https://bitvaria.com/tag/inference/) - [llm](https://bitvaria.com/tag/llm/) Share: [Back to Blog](https://bitvaria.com/blog/)