Speculative Decoding
Generating several tokens with a fast model and verifying them with the real one.
Overview
Speculative decoding uses a small, cheap model to propose several tokens and the large target model to verify them in a single pass, keeping only the proposals the target model would have produced. It solves the problem that generating one token at a time leaves the GPU idle: the cost is dominated by memory bandwidth rather than arithmetic, so verifying several tokens at once is nearly free. It exists because it speeds up inference without changing the output distribution.
Technology Graph
How Speculative Decoding connects to the rest of the stack.
Uses and depends on
- KV Cache technique Infrastructure Speculative Decoding uses KV Cache High ✓ source
Applied to
- Inference concept Infrastructure Speculative Decoding used for Inference High ✓ source