Speculative Decoding
Generating several tokens with a fast model and verifying them with the real one.
Aperçu
Speculative decoding uses a small, cheap model to propose several tokens and the large target model to verify them in a single pass, keeping only the proposals the target model would have produced. It solves the problem that generating one token at a time leaves the GPU idle: the cost is dominated by memory bandwidth rather than arithmetic, so verifying several tokens at once is nearly free. It exists because it speeds up inference without changing the output distribution.
Graphe des technologies
Comment Speculative Decoding se rattache au reste de la pile.
Utilise et dépend de
- KV Cache technique Infrastructure Speculative Decoding uses KV Cache High ✓ source
Appliqué à
- Inference concept Infrastructure Speculative Decoding used for Inference High ✓ source