KV Cache
Cached attention keys and values that make generating one token at a time affordable.
Overview
A KV cache stores the attention keys and values already computed for earlier tokens, so generating the next token does not recompute them. It solves the cost of autoregressive generation, which would otherwise repeat work proportional to the whole sequence for every new token. It exists because that recomputation dominates inference, and it is what makes the memory footprint of a long conversation grow with its length.
Technology Graph
How KV Cache connects to the rest of the stack.
Uses and depends on
- Prompt Caching technique Infrastructure Prompt Caching based on KV Cache High ✓ source
Used by
- SGLang tool Infrastructure KV Cache used by SGLang High ✓ source
- Speculative Decoding technique Techniques KV Cache used by Speculative Decoding High ✓ source
- Text Generation Inference tool Infrastructure KV Cache used by Text Generation Inference High ✓ source
- vLLM tool Infrastructure KV Cache used by vLLM High ✓ source