Speculative Decoding
Generating several tokens with a fast model and verifying them with the real one.
Обзор
Speculative decoding uses a small, cheap model to propose several tokens and the large target model to verify them in a single pass, keeping only the proposals the target model would have produced. It solves the problem that generating one token at a time leaves the GPU idle: the cost is dominated by memory bandwidth rather than arithmetic, so verifying several tokens at once is nearly free. It exists because it speeds up inference without changing the output distribution.
Граф технологий
Как Speculative Decoding связана с остальным стеком.
Использует и зависит от
- KV Cache technique Infrastructure Speculative Decoding uses KV Cache High ✓ source
Применяется к
- Inference concept Infrastructure Speculative Decoding used for Inference High ✓ source