vLLM
High-throughput inference server built around paged attention and continuous batching.
Visão geral
vLLM serves language models with two optimisations that matter more than the model itself: paged attention, which stops the KV cache from fragmenting, and continuous batching, which keeps the GPU busy as requests arrive and finish. It solves the throughput problem that makes naive serving uneconomical, where the GPU idles waiting for the slowest sequence in a batch. It exists because serving cost, not model quality, is what decides whether an application is viable.
Grafo de tecnologias
Como vLLM se liga ao resto do stack.
Usa e depende de
Aplicado a
Ferramentas
Software que implementa ou se integra com vLLM.