vLLM
High-throughput inference server built around paged attention and continuous batching.
Обзор
vLLM serves language models with two optimisations that matter more than the model itself: paged attention, which stops the KV cache from fragmenting, and continuous batching, which keeps the GPU busy as requests arrive and finish. It solves the throughput problem that makes naive serving uneconomical, where the GPU idles waiting for the slowest sequence in a batch. It exists because serving cost, not model quality, is what decides whether an application is viable.
Граф технологий
Как vLLM связана с остальным стеком.
Использует и зависит от
Применяется к
Инструменты
ПО, которое реализует vLLM или интегрируется с ним.