SGLang
Serving engine built around reusing shared prompt prefixes.
Resumen
SGLang serves language models with a runtime that lets a program describe its decoding once and reuse it, so common prompt prefixes, constrained output and multi-call workflows are handled by the engine rather than re-sent for every request. It solves the cost of repeated prefill: when many requests share a long prefix, recomputing it dominates. It exists because agent and structured-output workloads have exactly that shape, and the serving engine is where the saving has to happen.
Grafo de tecnologías
Cómo se conecta SGLang con el resto del stack.
Usa y depende de
- KV Cache technique Infrastructure SGLang uses KV Cache High ✓ source
Aplicado a
- Inference concept Infrastructure SGLang used for Inference High ✓ source
Herramientas
Software que implementa o se integra con SGLang.