TensorRT-LLM
NVIDIA's compiler and runtime for optimised LLM inference on its GPUs.
Überblick
TensorRT-LLM compiles a model into an optimised engine for a specific NVIDIA GPU, applying kernel fusion, quantisation and in-flight batching. It solves the gap between a model that runs and a model that runs fast enough to be worth serving, which on NVIDIA hardware comes down to kernels tuned for that exact chip. It exists because the fastest path is hardware-specific, and the cost is that the engine must be rebuilt for each GPU and model combination.
Technologiegraph
Wie TensorRT-LLM mit dem übrigen Stack zusammenhängt.
Nutzt und hängt ab von
- Tensor Core hardware Hardware TensorRT-LLM uses Tensor Core High ✓ source
- NVIDIA CUDA platform Hardware TensorRT-LLM depends on NVIDIA CUDA High ✓ source