vLLM
High-throughput inference server built around paged attention and continuous batching.
Resumen
vLLM serves language models with two optimisations that matter more than the model itself: paged attention, which stops the KV cache from fragmenting, and continuous batching, which keeps the GPU busy as requests arrive and finish. It solves the throughput problem that makes naive serving uneconomical, where the GPU idles waiting for the slowest sequence in a batch. It exists because serving cost, not model quality, is what decides whether an application is viable.
Grafo de tecnologías
Cómo se conecta vLLM con el resto del stack.
Usa y depende de
Aplicado a
Herramientas
Software que implementa o se integra con vLLM.