TensorRT-LLM
NVIDIA's compiler and runtime for optimised LLM inference on its GPUs.
Resumen
TensorRT-LLM compiles a model into an optimised engine for a specific NVIDIA GPU, applying kernel fusion, quantisation and in-flight batching. It solves the gap between a model that runs and a model that runs fast enough to be worth serving, which on NVIDIA hardware comes down to kernels tuned for that exact chip. It exists because the fastest path is hardware-specific, and the cost is that the engine must be rebuilt for each GPU and model combination.
Grafo de tecnologías
Cómo se conecta TensorRT-LLM con el resto del stack.
Usa y depende de
- Tensor Core hardware Hardware TensorRT-LLM uses Tensor Core High ✓ source
- NVIDIA CUDA platform Hardware TensorRT-LLM depends on NVIDIA CUDA High ✓ source