vLLM

High-throughput inference server built around paged attention and continuous batching.

tool Infrastructure Apache-2.0 7 relationships

Aperçu

vLLM serves language models with two optimisations that matter more than the model itself: paged attention, which stops the KV cache from fragmenting, and continuous batching, which keeps the GPU busy as requests arrive and finish. It solves the throughput problem that makes naive serving uneconomical, where the GPU idles waiting for the slowest sequence in a batch. It exists because serving cost, not model quality, is what decides whether an application is viable.

Graphe des technologies

Comment vLLM se rattache au reste de la pile.

Utilise et dépend de

KV Cache technique Infrastructure vLLM uses KV Cache High ✓ source
GPU hardware Hardware vLLM depends on GPU High ✓ source

Appliqué à

LLM concept Models vLLM used for LLM High ✓ source
Llama model Models vLLM used for Llama High ✓ source
Mistral model Models vLLM used for Mistral High ✓ source
Qwen model Models vLLM used for Qwen High ✓ source

Outils

Logiciels qui implémentent vLLM ou s'y intègrent.

Infrastructure

Position de la technologie

AI
↓
Infrastructure
↓
tool
↓
vLLM

Technologies liées

GPU hardware
KV Cache technique
LLM concept
Llama model
Mistral model
Qwen model
Ray framework

Ressources officielles

Dernière mise à jour: 2026-10-08
Dernière vérification: 2026-10-08

Sources citées

Chaque relation de cette page porte sa source.

docs.vllm.ai 3 claims
arxiv.org 2 claims
docs.ray.io 1 claim
docs.vllm.ai 1 claim