---
title: "Why an Answer Arrives Slower Than the GPU Allows"
source: https://bitvaria.com/en/speculative-decoding
---

28\. Sept. 2026 · Daniel Schleipfer · [AI](https://bitvaria.com/en/category/ai/)  · 8 min read

# Why an Answer Arrives Slower Than the GPU Allows

For every single token, the GPU re-reads the entire set of model weights just to compute that one token. Speculative decoding lets a small model guess ahead and saves exactly that wait.

![For every single token, the GPU re-reads the entire set of model weights just to compute that one token. Speculative decoding lets a small model guess ahead and saves exactly that wait.](https://bitvaria.com/_astro/speculative-decoding.9Qt7bc5r.jpg)

On this page

1.  [A Small Model Guesses Ahead, the Large One Checks](#a-small-model-guesses-ahead-the-large-one-checks)
2.  [Under the Hood](#under-the-hood)
3.  [When the Draft Model Guesses Wrong Too Often](#when-the-draft-model-guesses-wrong-too-often)
4.  [Frequently Asked Questions](#frequently-asked-questions)
5.  [Sources](#sources)

> **What is speculative decoding?**
> 
> A technique where a small draft model proposes several tokens and the large target model checks them in one pass, instead of writing each token itself one at a time. Correct proposals are kept, and the first wrong one is corrected.

A model writes its answer token by token. For each of them, the GPU re-reads the entire set of model weights, just for that one token. This read costs more time than the computation itself. It is what makes [the slower of the two phases in a request](https://bitvaria.com/en/prefill-decode) slow. Much of the card’s compute capacity sits unused at every step.

## A Small Model Guesses Ahead, the Large One Checks

The series, and where this term sits

This term sits at "Efficiency" in the group "The Machine".

Checking several tokens at once costs the GPU barely more than checking one. The read operation for the weights stays the same. That holds for one token just as much as for several.

This observation flips the order. A small, fast model proposes several tokens at once first. The large model then checks all the proposals together, in a single pass. Every correct token it accepts without writing it itself. On the first wrong proposal it corrects. It discards the rest of the guess.

This approach has a name: **speculative decoding**.

## Under the Hood

The proposing model is called the draft model, the checking one the target model. The draft model still generates its tokens one at a time, as usual, but it is small and cheap. The target model verifies the proposed sequence in a parallel pass, the same computation pattern as [prefill](https://bitvaria.com/en/prefill-decode): several positions at once. With a handful of proposed tokens, that pass remains bound by reading the weights, not by compute. It therefore takes barely longer than the pass for a single token.

The check is not a simple word-by-word comparison. For each proposed token, the target model compares its own probability with the draft model’s and accepts the token with a probability derived from that ratio. This sampling rule (rejection sampling) mathematically guarantees that the accepted token sequence follows the same probability distribution as an answer the target model would have generated entirely on its own. Individual answers can still differ in wording, as with any sampling. Chen et al. point out explicitly that different handling of random numbers and different numerical precision mean identical outputs are not to be expected. Unlike [quantization](https://bitvaria.com/en/quantization), the speedup is therefore not a trade-off against answer quality. It is purely a time gain.

Two independent papers establish the order of magnitude. Leviathan et al. (Google, ICML 2023) measured a 2x to 3x speedup over standard decoding on a T5-XXL model, with the same output distribution. Chen et al. (DeepMind, 2023) reached a 2x to 2.5x speedup on the 70-billion-parameter Chinchilla model, using a purpose-trained 4-billion-parameter draft model.

## When the Draft Model Guesses Wrong Too Often

The speedup depends mainly on the draft model’s acceptance rate. That is the share of its proposals the target model accepts. It is high when both models judge similarly which token fits next. When the small model usually guesses well, the large one rarely steps in, and the savings are large. When it often guesses wrong, the large one steps in often, and the advantage shrinks. Two more factors play in: how fast the small model runs compared with the large one, and how many tokens it proposes per round. The same acceptance rate therefore does not mean the same gain on every piece of hardware. In the worst case, an operator pays the small model’s compute time without getting much back.

The acceptance rate depends on the text. A small model trained on web text does well on everyday sentences. On specialized language, such as contracts or a company’s internal documents, it can drop sharply. The two models then judge differently what comes next. Hong et al. measure such a drop for domain questions put to a domain-trained target model. A figure from a paper therefore says nothing about the actual task. It has to be measured on the actual text.

Speculative decoding also does not lower costs by itself. It buys time with more compute for the small model. A deployment [aimed mainly at throughput](https://bitvaria.com/en/continuous-batching), many requests at once rather than one fast answer, does the math differently. The GPU time the small model needs is then missing for further requests.

* * *

**Speculative decoding lets a small model guess ahead. The large model only checks, instead of writing every token itself. When the guess lands, the answer arrives faster. Its quality does not change.**

Next term: **prompt caching**. How a server reuses computation for a conversation’s already-processed opening, instead of recomputing it.

## Frequently Asked Questions

**What is speculative decoding?** A technique where a small draft model proposes several tokens and the large target model checks them in one pass, instead of writing each token itself one at a time. Correct proposals are kept, and the first wrong one is corrected.

**What is the difference between the draft model and the target model?** The draft model is small and cheap and proposes tokens one at a time. The target model is the larger model whose answer counts. It checks several proposed tokens together in one pass.

**Does speculative decoding change the quality of the answer?** No. The check follows a sampling rule that guarantees the accepted token sequence follows the same probability distribution as an answer the target model would have generated on its own.

**What does the speedup from speculative decoding depend on?** Mainly on the draft model’s acceptance rate, the share of its proposals the target model accepts. Beyond that, on its run time relative to the target model and on the number of tokens per round. The acceptance rate depends on the text and must be measured on the actual task, not read off a benchmark from someone else’s data.

## Sources

-   Leviathan, Kalman, Matias, [Fast Inference from Transformers via Speculative Decoding](https://arxiv.org/abs/2211.17192), Google, ICML 2023: 2x to 3x speedup on T5-XXL, same output distribution; Section 2.3: acceptance by the ratio of target to draft probability; Theorem 3.8: speedup as a function of acceptance rate, draft length, and draft run time relative to the target model; Section 3.4 and Discussion: more arithmetic operations in total, no gain without spare compute.
-   Liu et al., [Optimizing Speculative Decoding for Serving Large Language Models Using Goodput](https://arxiv.org/abs/2406.14066), 2024: under high load, speculative decoding lengthens response time instead of shortening it (Section 3, Figure 1).
-   Chen et al., [Accelerating Large Language Model Decoding with Speculative Sampling](https://arxiv.org/abs/2302.01318), DeepMind, 2023: 2x to 2.5x speedup on Chinchilla 70B at batch size 1 in a distributed setup, 4-billion-parameter draft model trained on the same data, distribution-preserving rejection sampling; Section 4.1: with few proposed tokens the check stays memory-bandwidth bound and takes barely longer; Section 6.1: identical outputs are not to be expected because of random-number handling and numerics.
-   Hong et al., [Training Domain Draft Models for Speculative Decoding: Best Practices and Insights](https://arxiv.org/abs/2503.07807), 2025: the acceptance rate of a general draft model drops sharply on domain-specific queries against a domain-adapted target model, Table 1: biology from 60.7 to 37.5 percent.

* * *

*Part of the series [AI Engineering Explained](https://bitvaria.com/en/ai-engineering-begriffe). Related: [Why the Same AI Request Has Two Different Speeds](https://bitvaria.com/en/prefill-decode) (Prefill/Decode), [Why a Model Gets More Expensive the Longer It Talks](https://bitvaria.com/en/kv-cache) (KV Cache), and [Why a Finished Answer Still Has to Wait](https://bitvaria.com/en/continuous-batching) (Continuous Batching).*

-   [ki](https://bitvaria.com/en/tag/ki/)
-   [ai-engineering](https://bitvaria.com/en/tag/ai-engineering/)
-   [speculative-decoding](https://bitvaria.com/en/tag/speculative-decoding/)
-   [efficiency](https://bitvaria.com/en/tag/efficiency/)
-   [llm](https://bitvaria.com/en/tag/llm/)

Share:

[Back to Blog](https://bitvaria.com/en/blog/)