Обзор

15 technologies shown

• vLLM tool High-throughput inference server built around paged attention and continuous batching.
• Ollama tool Runs open-weight models locally behind a simple command and HTTP API.
• Inference concept Running a trained model to produce output, as opposed to training it.
• Prompt Caching technique Reusing the computed state of an unchanged prompt prefix.
• Tokenizer tool Component that converts text to tokens and back.
• Ray framework Distributed execution framework for scaling Python workloads.
• Safetensors infrastructure File format for model weights designed to avoid arbitrary code execution.
• llama.cpp tool CPU-first inference for quantised models, in portable C++.
• ONNX infrastructure Interchange format for models, so a trained model runs on another runtime.
• GGUF infrastructure Quantised model format designed for fast loading and CPU inference.
• SGLang tool Serving engine built around reusing shared prompt prefixes.
• Triton Inference Server tool NVIDIA's server for deploying models from several frameworks behind one API.
• TensorRT-LLM tool NVIDIA's compiler and runtime for optimised LLM inference on its GPUs.
• Text Generation Inference tool Hugging Face's serving stack for transformer models.
• KV Cache technique Cached attention keys and values that make generating one token at a time affordable.