TensorRT-LLM
NVIDIA's compiler and runtime for optimised LLM inference on its GPUs.
Overview
TensorRT-LLM compiles a model into an optimised engine for a specific NVIDIA GPU, applying kernel fusion, quantisation and in-flight batching. It solves the gap between a model that runs and a model that runs fast enough to be worth serving, which on NVIDIA hardware comes down to kernels tuned for that exact chip. It exists because the fastest path is hardware-specific, and the cost is that the engine must be rebuilt for each GPU and model combination.
Technology Graph
How TensorRT-LLM connects to the rest of the stack.
Uses and depends on
- Tensor Core hardware Hardware TensorRT-LLM uses Tensor Core High ✓ source
- NVIDIA CUDA platform Hardware TensorRT-LLM depends on NVIDIA CUDA High ✓ source