vLLM
High-throughput inference server built around paged attention and continuous batching.
Überblick
vLLM serves language models with two optimisations that matter more than the model itself: paged attention, which stops the KV cache from fragmenting, and continuous batching, which keeps the GPU busy as requests arrive and finish. It solves the throughput problem that makes naive serving uneconomical, where the GPU idles waiting for the slowest sequence in a batch. It exists because serving cost, not model quality, is what decides whether an application is viable.
Technologiegraph
Wie vLLM mit dem übrigen Stack zusammenhängt.
Nutzt und hängt ab von
Angewendet auf
Werkzeuge
Software, die vLLM implementiert oder integriert.