TensorRT-LLM

NVIDIA's compiler and runtime for optimised LLM inference on its GPUs.

tool Infrastructure 5 relationships

Aperçu

TensorRT-LLM compiles a model into an optimised engine for a specific NVIDIA GPU, applying kernel fusion, quantisation and in-flight batching. It solves the gap between a model that runs and a model that runs fast enough to be worth serving, which on NVIDIA hardware comes down to kernels tuned for that exact chip. It exists because the fastest path is hardware-specific, and the cost is that the engine must be rebuilt for each GPU and model combination.

Graphe des technologies

Comment TensorRT-LLM se rattache au reste de la pile.

Utilise et dépend de

Tensor Core hardware Hardware TensorRT-LLM uses Tensor Core High ✓ source
NVIDIA CUDA platform Hardware TensorRT-LLM depends on NVIDIA CUDA High ✓ source

Appliqué à

LLM concept Models TensorRT-LLM used for LLM Medium ✓ source
Llama model Models TensorRT-LLM used for Llama High ✓ source
Qwen model Models TensorRT-LLM used for Qwen High ✓ source

Position de la technologie

AI
↓
Infrastructure
↓
tool
↓
TensorRT-LLM

Technologies liées

Llama model
NVIDIA CUDA platform
Qwen model
Tensor Core hardware
LLM concept

Ressources officielles

Dernière mise à jour: 2026-10-08
Dernière vérification: 2026-10-08

Sources citées

Chaque relation de cette page porte sa source.