Speculative Decoding
Generating several tokens with a fast model and verifying them with the real one.
Resumen
Speculative decoding uses a small, cheap model to propose several tokens and the large target model to verify them in a single pass, keeping only the proposals the target model would have produced. It solves the problem that generating one token at a time leaves the GPU idle: the cost is dominated by memory bandwidth rather than arithmetic, so verifying several tokens at once is nearly free. It exists because it speeds up inference without changing the output distribution.
Grafo de tecnologías
Cómo se conecta Speculative Decoding con el resto del stack.
Usa y depende de
- KV Cache technique Infrastructure Speculative Decoding uses KV Cache High ✓ source
Aplicado a
- Inference concept Infrastructure Speculative Decoding used for Inference High ✓ source