Text Generation Inference
Hugging Face's serving stack for transformer models.
Überblick
Text Generation Inference serves models from the Hugging Face ecosystem with continuous batching, token streaming and quantisation built in. It solves the gap between having a model on the Hub and running it as a service, where the pieces needed are the same every time and are easy to get subtly wrong. It exists as the reference serving path for that ecosystem, which matters because most open-weight models are published there first.
Technologiegraph
Wie Text Generation Inference mit dem übrigen Stack zusammenhängt.
Nutzt und hängt ab von
- KV Cache technique Infrastructure Text Generation Inference uses KV Cache High ✓ source
Angewendet auf
- Inference concept Infrastructure Text Generation Inference used for Inference High ✓ source