KV Cache
Cached attention keys and values that make generating one token at a time affordable.
Überblick
A KV cache stores the attention keys and values already computed for earlier tokens, so generating the next token does not recompute them. It solves the cost of autoregressive generation, which would otherwise repeat work proportional to the whole sequence for every new token. It exists because that recomputation dominates inference, and it is what makes the memory footprint of a long conversation grow with its length.
Technologiegraph
Wie KV Cache mit dem übrigen Stack zusammenhängt.
Nutzt und hängt ab von
- Prompt Caching technique Infrastructure Prompt Caching based on KV Cache High ✓ source
Genutzt von
- SGLang tool Infrastructure KV Cache used by SGLang High ✓ source
- Speculative Decoding technique Techniques KV Cache used by Speculative Decoding High ✓ source
- Text Generation Inference tool Infrastructure KV Cache used by Text Generation Inference High ✓ source
- vLLM tool Infrastructure KV Cache used by vLLM High ✓ source