Why an Answer Arrives Slower Than the GPU Allows
For every single token, the GPU re-reads the entire set of model weights just to compute that one token. Speculative decoding lets a small model guess ahead and saves exactly that wait.
For every single token, the GPU re-reads the entire set of model weights just to compute that one token. Speculative decoding lets a small model guess ahead and saves exactly that wait.