KV Cache
Cached attention keys and values that make generating one token at a time affordable.
Aperçu
A KV cache stores the attention keys and values already computed for earlier tokens, so generating the next token does not recompute them. It solves the cost of autoregressive generation, which would otherwise repeat work proportional to the whole sequence for every new token. It exists because that recomputation dominates inference, and it is what makes the memory footprint of a long conversation grow with its length.
Graphe des technologies
Comment KV Cache se rattache au reste de la pile.
Utilise et dépend de
- Prompt Caching technique Infrastructure Prompt Caching based on KV Cache High ✓ source
Utilisé par
- SGLang tool Infrastructure KV Cache used by SGLang High ✓ source
- Speculative Decoding technique Techniques KV Cache used by Speculative Decoding High ✓ source
- Text Generation Inference tool Infrastructure KV Cache used by Text Generation Inference High ✓ source
- vLLM tool Infrastructure KV Cache used by vLLM High ✓ source