Speculative Decoding
Generating several tokens with a fast model and verifying them with the real one.
Überblick
Speculative decoding uses a small, cheap model to propose several tokens and the large target model to verify them in a single pass, keeping only the proposals the target model would have produced. It solves the problem that generating one token at a time leaves the GPU idle: the cost is dominated by memory bandwidth rather than arithmetic, so verifying several tokens at once is nearly free. It exists because it speeds up inference without changing the output distribution.
Technologiegraph
Wie Speculative Decoding mit dem übrigen Stack zusammenhängt.
Nutzt und hängt ab von
- KV Cache technique Infrastructure Speculative Decoding uses KV Cache High ✓ source
Angewendet auf
- Inference concept Infrastructure Speculative Decoding used for Inference High ✓ source