Inference
Running a trained model to produce output, as opposed to training it.
Überblick
Inference is the act of running a trained model on an input. It solves nothing by itself; it is the other half of the lifecycle, and its economics are different from training in every respect. It exists as a distinct concern because training is a one-off bounded by budget while inference is continuous and bounded by latency and cost per request, so the optimisations that matter - batching, caching, quantisation - are not the same ones.
Technologiegraph
Wie Inference mit dem übrigen Stack zusammenhängt.
Angewendet auf
- Distillation technique Techniques Distillation used for Inference High ✓ source
- FP8 concept Algorithms FP8 used for Inference High ✓ source
- FlashAttention technique Techniques FlashAttention used for Inference High ✓ source
- GGUF infrastructure Infrastructure GGUF used for Inference High ✓ source
- KV Cache technique Infrastructure KV Cache used for Inference High ✓ source
- MI300X hardware Hardware MI300X used for Inference High ✓ source
- ONNX infrastructure Infrastructure ONNX used for Inference High ✓ source
- Ollama tool Infrastructure Ollama used for Inference High ✓ source
- Prompt Caching technique Infrastructure Prompt Caching used for Inference High ✓ source
- Quantization technique Techniques Quantization used for Inference High ✓ source
- Ray framework Infrastructure Ray used for Inference High ✓ source
- SGLang tool Infrastructure SGLang used for Inference High ✓ source
- Safetensors infrastructure Infrastructure Safetensors used for Inference High ✓ source
- Speculative Decoding technique Techniques Speculative Decoding used for Inference High ✓ source
- Text Generation Inference tool Infrastructure Text Generation Inference used for Inference High ✓ source
- Triton Inference Server tool Infrastructure Triton Inference Server used for Inference High ✓ source
- llama.cpp tool Infrastructure llama.cpp used for Inference High ✓ source