SGLang
Serving engine built around reusing shared prompt prefixes.
Aperçu
SGLang serves language models with a runtime that lets a program describe its decoding once and reuse it, so common prompt prefixes, constrained output and multi-call workflows are handled by the engine rather than re-sent for every request. It solves the cost of repeated prefill: when many requests share a long prefix, recomputing it dominates. It exists because agent and structured-output workloads have exactly that shape, and the serving engine is where the saving has to happen.
Graphe des technologies
Comment SGLang se rattache au reste de la pile.
Utilise et dépend de
- KV Cache technique Infrastructure SGLang uses KV Cache High ✓ source
Appliqué à
- Inference concept Infrastructure SGLang used for Inference High ✓ source
Outils
Logiciels qui implémentent SGLang ou s'y intègrent.