KV Cache
Cached attention keys and values that make generating one token at a time affordable.
Обзор
A KV cache stores the attention keys and values already computed for earlier tokens, so generating the next token does not recompute them. It solves the cost of autoregressive generation, which would otherwise repeat work proportional to the whole sequence for every new token. It exists because that recomputation dominates inference, and it is what makes the memory footprint of a long conversation grow with its length.
Граф технологий
Как KV Cache связана с остальным стеком.
Использует и зависит от
- Prompt Caching technique Infrastructure Prompt Caching based on KV Cache High ✓ source
Используется в
- SGLang tool Infrastructure KV Cache used by SGLang High ✓ source
- Speculative Decoding technique Techniques KV Cache used by Speculative Decoding High ✓ source
- Text Generation Inference tool Infrastructure KV Cache used by Text Generation Inference High ✓ source
- vLLM tool Infrastructure KV Cache used by vLLM High ✓ source