SGLang
Serving engine built around reusing shared prompt prefixes.
Überblick
SGLang serves language models with a runtime that lets a program describe its decoding once and reuse it, so common prompt prefixes, constrained output and multi-call workflows are handled by the engine rather than re-sent for every request. It solves the cost of repeated prefill: when many requests share a long prefix, recomputing it dominates. It exists because agent and structured-output workloads have exactly that shape, and the serving engine is where the saving has to happen.
Technologiegraph
Wie SGLang mit dem übrigen Stack zusammenhängt.
Nutzt und hängt ab von
- KV Cache technique Infrastructure SGLang uses KV Cache High ✓ source
Angewendet auf
- Inference concept Infrastructure SGLang used for Inference High ✓ source
Werkzeuge
Software, die SGLang implementiert oder integriert.