TensorRT-LLM
NVIDIA's compiler and runtime for optimised LLM inference on its GPUs.
Обзор
TensorRT-LLM compiles a model into an optimised engine for a specific NVIDIA GPU, applying kernel fusion, quantisation and in-flight batching. It solves the gap between a model that runs and a model that runs fast enough to be worth serving, which on NVIDIA hardware comes down to kernels tuned for that exact chip. It exists because the fastest path is hardware-specific, and the cost is that the engine must be rebuilt for each GPU and model combination.
Граф технологий
Как TensorRT-LLM связана с остальным стеком.
Использует и зависит от
- Tensor Core hardware Hardware TensorRT-LLM uses Tensor Core High ✓ source
- NVIDIA CUDA platform Hardware TensorRT-LLM depends on NVIDIA CUDA High ✓ source