KV Cache
Cached attention keys and values that make generating one token at a time affordable.
Visão geral
A KV cache stores the attention keys and values already computed for earlier tokens, so generating the next token does not recompute them. It solves the cost of autoregressive generation, which would otherwise repeat work proportional to the whole sequence for every new token. It exists because that recomputation dominates inference, and it is what makes the memory footprint of a long conversation grow with its length.
Grafo de tecnologias
Como KV Cache se liga ao resto do stack.
Usa e depende de
- Prompt Caching technique Infrastructure Prompt Caching based on KV Cache High ✓ source
Usado por
- SGLang tool Infrastructure KV Cache used by SGLang High ✓ source
- Speculative Decoding technique Techniques KV Cache used by Speculative Decoding High ✓ source
- Text Generation Inference tool Infrastructure KV Cache used by Text Generation Inference High ✓ source
- vLLM tool Infrastructure KV Cache used by vLLM High ✓ source