SGLang
Serving engine built around reusing shared prompt prefixes.
Visão geral
SGLang serves language models with a runtime that lets a program describe its decoding once and reuse it, so common prompt prefixes, constrained output and multi-call workflows are handled by the engine rather than re-sent for every request. It solves the cost of repeated prefill: when many requests share a long prefix, recomputing it dominates. It exists because agent and structured-output workloads have exactly that shape, and the serving engine is where the saving has to happen.
Grafo de tecnologias
Como SGLang se liga ao resto do stack.
Usa e depende de
- KV Cache technique Infrastructure SGLang uses KV Cache High ✓ source
Aplicado a
- Inference concept Infrastructure SGLang used for Inference High ✓ source
Ferramentas
Software que implementa ou se integra com SGLang.