Why the Same Long Instruction Sometimes Costs Almost Nothing and Sometimes Full Price

What is prompt caching?

A server keeps the intermediate computation state for the opening of a request and reuses it for the next request with the same opening, instead of computing it again. The opening must be identical, token for token.

A company puts the same long instruction in front of every request: a role, rules, and a contract set as reference text. Some requests then bill cheaply. Others cost the full price even though the instruction is the same. The difference is a single character.

What Stays the Same Does Not Need Recomputing

Before a model writes the first token of an answer, it reads the whole request in one pass. That is the first of the two phases. It produces the intermediate state the model needs for every further token: the KV cache.

If two requests begin with the same text, that intermediate state is the same for the shared opening. A server can therefore keep it and take it over the second time. It only computes the rest anew.

This holds only for a shared opening, though. The intermediate state of each token depends on all tokens before it. If one token differs, everything after it is invalid.

This approach has a name: prompt caching.

Under the Hood

A prompt cache stores the KV intermediate state of a prefix, meaning the first n tokens. For a new request, the server looks for the longest stored prefix that matches it and computes only from the first difference on. The comparison is exact. Similar is not enough: a different token at position 3 makes the cache useless for the whole request.

At Anthropic a fixed order underlies this: tool definitions first, then the system prompt, then the messages. A change at one level invalidates it and all levels after it. The benefit shows in the price list. A cache read costs one tenth of the normal input price there for most models, even less for some newer ones. The first write of the cache costs 1.25 times as much. The default lifetime is five minutes, measured from the last use. Very short prompts cannot be cached at all; the lower limit is between 512 and 4,096 tokens depending on the model. Each provider sets and changes its own prices and limits. They appear here only as an order of magnitude.

For self-hosted models, paged attention provides the basis. Requests with the same opening share the same memory pages of the KV cache instead of creating them several times. If several tenants share one server, this cache needs separating too. Otherwise response times reveal whether someone else already sent the same opening. vLLM offers a per-request addition for this, the cache_salt.

Where the Cache Misses, and Where It Hits Wrong

Prompt caching keeps intermediate computation states. The model still writes the answer anew afterwards, and it stays unchanged. That is why it never returns a wrong answer. It does have one hard condition: the opening must match.

A typical mistake is a variable building block in the wrong place. A timestamp, a user name or a request ID near the front makes every request differ right there. The cache then never hits. Nobody notices, because the answers are correct. Only the bill is higher. The remedy is an ordering. Fixed parts such as role, rules and long documents go first. Variable ones such as date and question go last. At Anthropic, the cache marker also belongs at the end of the fixed part. If it sits after the variable content, the fixed part is never stored on its own. Whether a hit happened is shown by the usage figures in the response. Whoever does not check them may be saving nothing.

Then there is the lifetime. If the same instruction arrives less often than the cache lives, the cache is written anew every time. Writing costs more than reading. With one request per hour, the default lifetime does not pay off.

Semantic caching goes one step further and keeps finished answers. A new question is converted into a vector of numbers that describes its meaning. If it lies close enough to that of an earlier question, the earlier answer comes back. The model does not run at all. A hit therefore saves more than with prompt caching, but it is not exact. A threshold set too generously returns the answer to a different question. “Notice period for customer A” and “notice period for customer B” can lie close together in that number space. The answers differ, though.

In the Mittelstand this matters twice over. Answers that relate to a tenant, a contract or a person must never come from another tenant’s cache. A shared answer cache that does not enforce separation by tenant is a data leak. Answers with a time reference also go stale without anyone noticing.

Prompt caching keeps what does not change: only an identical opening is not recomputed. Whoever does not know the order of their request pays the full price.

Next term: structured output. How a model delivers answers in a fixed format, and what happens when the format is wrong.

Frequently Asked Questions

What is prompt caching? A server keeps the intermediate computation state for the opening of a request and reuses it for the next request with the same opening, instead of computing it again. The opening must be identical, token for token.

What is the difference between prompt caching and semantic caching? Prompt caching keeps the intermediate computation state of an identical opening; the model still writes a fresh answer afterwards. Semantic caching keeps finished answers and returns them for questions similar in meaning. The model then does not run at all, but the answer can be wrong.

Why does the cache miss even though the prompt is almost the same? The comparison runs token by token from the start. If something variable such as a timestamp sits near the front, the request already differs there. Everything after that is recomputed. Fixed parts belong at the front, variable ones at the end.

How long does a prompt cache last? The provider decides. At Anthropic the default lifetime is five minutes, measured from the last use, optionally one hour. After that the opening is computed in full again.

Sources

• Anthropic, Prompt caching, documentation, retrieved 25 September 2026, rechecked 2 October 2026: hits only for fully identical segments up to the marked block; order tools, system, messages, changes invalidate that level and all following ones; default lifetime five minutes, measured from the start of the last use; writes 1.25x, reads 0.1x of the base input price (standard models; Claude Opus 5.5 0.05x, Claude Fable 5.1 0.025x); lower limit 512 to 4,096 tokens depending on the model, shorter prompts are not cached and return no error; writes happen only at the cache marker, and a marker placed after changing content never hits (section “Common mistake: Breakpoint on content that changes every request”).
• Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention, SOSP 2023: shared memory pages for requests with the same opening.
• vLLM, Automatic Prefix Caching, design documentation, retrieved 2 October 2026: cache_salt separates the cache per trust group and prevents timing attacks via latency differences.
• Redis, Semantic cache with redis-py, retrieved 2 October 2026: question as a vector, a hit only below a distance threshold, separation by tenant via a filter inside the query.

Part of the series AI Engineering Explained. Related: Why the Same AI Request Has Two Different Speeds (Prefill/Decode), Why a Model Gets More Expensive the Longer It Talks (KV Cache) and Why GPU Memory Is Already Full Before It Has To Be (Paged Attention).