vLLM

High-throughput inference server built around paged attention and continuous batching.

tool Infrastructure Apache-2.0 7 relationships

Overview

vLLM serves language models with two optimisations that matter more than the model itself: paged attention, which stops the KV cache from fragmenting, and continuous batching, which keeps the GPU busy as requests arrive and finish. It solves the throughput problem that makes naive serving uneconomical, where the GPU idles waiting for the slowest sequence in a batch. It exists because serving cost, not model quality, is what decides whether an application is viable.

Technology Graph

How vLLM connects to the rest of the stack.

Uses and depends on

KV Cache technique Infrastructure vLLM uses KV Cache High ✓ source
GPU hardware Hardware vLLM depends on GPU High ✓ source

Applied to

LLM concept Models vLLM used for LLM High ✓ source
Llama model Models vLLM used for Llama High ✓ source
Mistral model Models vLLM used for Mistral High ✓ source
Qwen model Models vLLM used for Qwen High ✓ source

Tools

Software that implements or integrates with vLLM.

Infrastructure

Technology Position

AI
↓
Infrastructure
↓
tool
↓
vLLM

Related Technologies

GPU hardware
KV Cache technique
LLM concept
Llama model
Mistral model
Qwen model
Ray framework

Official resources

Last updated: 2026-10-08
Last verified: 2026-10-08

Sources cited

Every relationship on this page carries a citation.

docs.vllm.ai 3 claims
arxiv.org 2 claims
docs.ray.io 1 claim
docs.vllm.ai 1 claim