TensorRT-LLM

Overview#

TensorRT-LLM is NVIDIA's official inference compiler and runtime for large language models. It sits on top of the long-standing TensorRT graph compiler and adds three things: a Python API specialised for Transformer blocks (attention, RMSNorm, RoPE, GQA, MoE), an extensive library of pre-built model definitions covering most production-relevant open weights, and a C++ batch manager that handles in-flight batching, paged KV cache, beam search and speculative decoding. The aim is to take a HuggingFace or NeMo checkpoint and emit a serialised engine that exploits every architectural feature the underlying NVIDIA silicon offers — FP8 on Hopper, FP4 and second-generation Transformer Engine on Blackwell, NVLink-aware collectives, tensor cores, MIG slices.

The crucial conceptual distinction from vLLM, SGLang or TGI is that TensorRT-LLM is not a server. It is a compiler that produces engine binaries plus a runtime that loads them. Production deployments almost always wrap that runtime in NVIDIA Triton Inference Server using the `tensorrtllm_backend`, which provides the HTTP and gRPC API surface, request queuing, model versioning, multi-instance support and Prometheus metrics. Separating compile and serve is the design choice that buys the last 30-40 percent of performance — kernel selection, autotuning and memory layout can all assume fixed shapes — and pays for it with operational complexity.

Where vLLM optimises for breadth and developer ergonomics (one Python command, every new model on day one), TensorRT-LLM optimises for the absolute floor of latency at the highest sustainable throughput on NVIDIA silicon. The library is Apache 2.0 but is tightly coupled to the CUDA toolchain (CUDA 12.4+, cuDNN 9.x, TensorRT 10.x, NCCL 2.21+) and is not portable to AMD ROCm, Intel Gaudi or AWS Neuron. By mid-2026 it ships as a Python wheel, an NGC container (`nvcr.io/nvidia/tritonserver:24.10-trtllm-python-py3`) and a stand-alone C++ runtime; the release cadence is roughly monthly with new model architectures lagging vLLM by one to four weeks. Yobibyte exposes TensorRT-LLM as an opt-in performance variant — Yobitel customers stand up a workload first on the vLLM default and promote stable production endpoints to a TensorRT-LLM-backed engine through the managed workspace, never touching `trtllm-build` or Triton config files directly.

This entry documents the production surface: the build CLI and Python API, the compilation pipeline and runtime mechanics, the parallelism strategies, the Triton deployment shape, the limits and quotas, the observability hooks, and the sizing, cost and migration models you need to operate TensorRT-LLM at scale on Yobitel and beyond. This entry helps you stand up TensorRT-LLM for production LLM serving with the right flags, sizing and operational practices — whether you are operating raw upstream on your own NVIDIA fleet or consuming TensorRT-LLM as the Yobibyte opt-in performance variant.

Quick start#

The example below takes Llama 3.1 70B from a HuggingFace checkpoint to an FP8 TensorRT engine sharded across 4x H100 SXM5, then serves it behind Triton with the TensorRT-LLM backend. Stage 1 converts the HF checkpoint into a TensorRT-LLM checkpoint with the desired tensor-parallel layout. Stage 2 runs `trtllm-build` to emit one engine binary per GPU rank with batch size, sequence length and precision baked in. Stage 3 packages the engines into a Triton model repository and launches `tritonserver`. Stage 4 hits the OpenAI-compatible endpoint exposed by the bundled `openai_frontend`.

# 0. Install TensorRT-LLM (or use the NGC container)
pip install "tensorrt-llm==0.14.0" --extra-index-url https://pypi.nvidia.com

# 1. Convert HuggingFace checkpoint to TRT-LLM checkpoint (TP=4, FP8)
python examples/llama/convert_checkpoint.py \
    --model_dir ./llama-3.1-70b-instruct-hf \
    --output_dir ./ckpt/llama3-70b-fp8-tp4 \
    --dtype bfloat16 \
    --tp_size 4 \
    --use_fp8 \
    --calib_dataset cnn_dailymail \
    --calib_size 512

# 2. Build per-rank engines with shapes baked in
trtllm-build \
    --checkpoint_dir ./ckpt/llama3-70b-fp8-tp4 \
    --output_dir ./engines/llama3-70b-fp8-tp4 \
    --gemm_plugin fp8 \
    --use_fp8_context_fmha enable \
    --use_paged_context_fmha enable \
    --paged_kv_cache enable \
    --remove_input_padding enable \
    --use_fused_mlp enable \
    --max_batch_size 64 \
    --max_input_len 16384 \
    --max_output_len 2048 \
    --max_num_tokens 16384 \
    --kv_cache_free_gpu_mem_fraction 0.92 \
    --workers 4

# 3. Stage into a Triton model repository and launch
cp -r ./engines/llama3-70b-fp8-tp4/* triton_models/llama3_70b/1/
tritonserver --model-repository=./triton_models \
    --grpc-port=8001 --http-port=8000 --metrics-port=8002

# 4. Hit the OpenAI-compatible frontend
curl http://localhost:8000/v2/models/ensemble/generate \
    -H "Content-Type: application/json" \
    -d '{
      "text_input": "Summarise FlashAttention-3 in 2 lines.",
      "max_tokens": 128,
      "temperature": 0.0
    }'

How it works#

A TensorRT-LLM workflow has three stages, executed once per (model, precision, layout, target GPU) tuple. First, a HuggingFace or NeMo checkpoint is converted into a TensorRT-LLM checkpoint via the model-specific `convert_checkpoint.py` — a sharded directory of weights plus a `config.json` that records vocabulary size, hidden dimension, number of layers, GQA group count, RoPE parameters and the parallelism layout. Second, `trtllm-build` consumes that checkpoint and a set of build-time hyperparameters (max batch, max input length, max output length, beam width, quantisation mode, plugin selection) and emits one serialised engine binary per GPU rank. Third, the engines are loaded into the TensorRT-LLM runtime — typically inside a Triton model instance — which exposes the model behind a request queue managed by the C++ batch manager.

At runtime the batch manager implements in-flight batching: every iteration, it admits new requests from the queue if the paged KV cache has free blocks, advances all running sequences by one decoded token, evicts completed sequences, and triggers the next forward pass. Forward execution uses FlashAttention-3 on Hopper and Blackwell (via the `paged_context_fmha` plugin), with a paged-KV variant that gathers K and V from the block pool through per-sequence block tables. The KV pool size is computed at engine load from `kv_cache_free_gpu_mem_fraction` x (device memory - weights - activation working set), divided by block size in tokens.

Tensor parallelism shards each weight matrix across GPUs within an NVLink island, using a custom AllReduce kernel optimised for small message sizes typical of decode steps (the upstream NCCL collective adds latency that matters when each step lasts a few hundred microseconds). Pipeline parallelism splits layers into stages across nodes, tolerating lower interconnect bandwidth. Expert parallelism handles MoE architectures by partitioning experts across ranks. The three can be composed: a DeepSeek-V3 671B deployment might run TP=8 + PP=2 + EP=8 across two H100 SXM5 nodes connected by 400 Gb/s InfiniBand.

Speculative decoding integrates at the batch-manager layer. The runtime supports three families: draft-target speculation (a smaller model proposes k tokens, the target verifies in one parallel forward), Medusa heads (additional decoder heads attached to the target predict multiple future tokens), and EAGLE-2 (a learned draft tree that adapts to acceptance rate). On chat workloads with Llama 3 8B drafting for 70B, end-to-end latency typically drops 1.6-2.4x at unchanged output quality.

• In-flight batching — NVIDIA's term for continuous batching; admits and evicts sequences between iterations to keep tensor cores busy.
• Paged KV cache — block-structured KV memory with prefix sharing, equivalent in spirit to vLLM's PagedAttention; enabled with `--paged_kv_cache enable`.
• FP8 and FP4 quantisation — leverages Hopper Transformer Engine and Blackwell FP4 cores via per-tensor scaling factors learned during a calibration pass.
• Speculative decoding — supports draft-target, Medusa heads, EAGLE-2 and lookahead decoding, configured at build time and tuned at runtime.
• Tensor, pipeline and expert parallelism — partitions large models across NVLink islands (TP), nodes (PP) and MoE expert sets (EP).
• Custom AllReduce — a latency-optimised collective for small tensor-parallel groups within an NVLink island; bypasses NCCL for sub-microsecond reductions.
• Plugin system — fused kernels (`gemm_plugin`, `gpt_attention_plugin`, `rmsnorm_plugin`, `lookup_plugin`) compiled into the engine, selected per architecture.

Reference and specifications#

Every long-lived TensorRT-LLM deployment is parameterised by the `trtllm-build` command line and the runtime config passed into the batch manager. The table below is the canonical reference for the build CLI surface as of TensorRT-LLM v0.14 (June 2026). Most flags accept `enable` / `disable` literals rather than booleans; that is intentional and mirrors the TensorRT plugin naming convention. Flags not listed here are either internal tuning knobs that the defaults handle correctly or experimental features documented in the upstream reference.

Workload patterns#

Three workload shapes cover the bulk of TensorRT-LLM production deployments: high-throughput chat on a 70B-class model with in-flight batching, long-context summarisation with paged context attention, and ultra-low-latency endpoints accelerated by EAGLE-2 speculative decoding. Each has its own preferred build configuration and Triton runtime policy. These are also the three promotion paths Yobibyte automates when a customer upgrades a workload from the vLLM default to the TensorRT-LLM opt-in variant — the build matrix, engine validation against the vLLM baseline and rollout behind the same OpenAI-compatible URL are what a team running raw upstream on Kubernetes signs up to operate themselves.

Pattern A — Llama 3 70B FP8 TP=4 chat endpoint with in-flight batching. Targets p50 TTFT under 180 ms and 6,000+ sustained output tok/s on a single 4-GPU H100 SXM5 node. Pattern B — long-context endpoint with 128K context window using paged context FMHA; trades peak throughput for the ability to process 100K-token prompts without blowing the activation budget. Pattern C — speculative decoding via EAGLE-2 for chat workloads where p99 inter-token latency matters more than raw throughput; pairs a small EAGLE head with the target 70B engine.

# A — Llama 3 70B FP8 TP=4, in-flight batching, chat-shaped
trtllm-build \
    --checkpoint_dir ./ckpt/llama3-70b-fp8-tp4 \
    --output_dir ./engines/llama3-70b-chat \
    --gemm_plugin fp8 \
    --use_fp8_context_fmha enable \
    --paged_kv_cache enable \
    --use_inflight_batching enable \
    --remove_input_padding enable \
    --use_fused_mlp enable \
    --max_batch_size 128 \
    --max_input_len 8192 \
    --max_output_len 2048 \
    --max_num_tokens 16384 \
    --kv_cache_free_gpu_mem_fraction 0.92 \
    --workers 4

# B — 128K long-context with paged context FMHA (H200 preferred)
trtllm-build \
    --checkpoint_dir ./ckpt/llama3-70b-fp8-tp4 \
    --output_dir ./engines/llama3-70b-128k \
    --gemm_plugin fp8 \
    --use_fp8_context_fmha enable \
    --use_paged_context_fmha enable \
    --paged_kv_cache enable \
    --multi_block_mode enable \
    --max_batch_size 16 \
    --max_input_len 131072 \
    --max_output_len 4096 \
    --max_num_tokens 8192 \
    --kv_cache_free_gpu_mem_fraction 0.95

# C — EAGLE-2 speculative decoding for low p99 latency
# C1. Build the EAGLE-2 draft head separately
python examples/eagle/convert_checkpoint.py \
    --model_dir ./llama-3-eagle-head \
    --output_dir ./ckpt/llama3-70b-eagle \
    --tp_size 4

# C2. Build the target engine with EAGLE support
trtllm-build \
    --checkpoint_dir ./ckpt/llama3-70b-fp8-tp4 \
    --output_dir ./engines/llama3-70b-eagle-target \
    --gemm_plugin fp8 \
    --use_fp8_context_fmha enable \
    --paged_kv_cache enable \
    --use_inflight_batching enable \
    --speculative_decoding_mode eagle \
    --max_draft_len 5 \
    --max_batch_size 64 \
    --max_input_len 4096 \
    --max_output_len 1024

Sizing and capacity planning#

TensorRT-LLM throughput is bounded first by KV-cache memory, then by tensor-core FLOPs, then by NVLink AllReduce bandwidth at TP > 4. The planning model below assumes Llama-family architectures with grouped-query attention, FP8 weights and KV, and the canonical chat / RAG / long-context / batch mix at 4K input / 256 output. Tokens-per-second figures are mid-range observed values from InferenceBench v3 sustained runs; treat them as planning anchors, not contractual.

Engine builds are the operational cost most teams under-estimate. A 70B FP8 TP=4 engine takes 15-25 minutes to build on the same hardware it serves; long-context variants can take 45 minutes. CI pipelines that maintain a heterogeneous fleet (H100 SXM5, H200, B200) need a build matrix per (model, precision, TP, max_input, max_output, GPU SM, TRT-LLM version). Plan for the build farm capacity, the registry storage, and the rebuild churn that follows every CUDA / driver upgrade.

Limits and quotas#

TensorRT-LLM enforces hard limits at the engine boundary (baked in at build time) and soft limits at the runtime batch manager (configurable per Triton model instance). Operational ceilings (GPU memory, NCCL groups, file descriptors, /dev/shm) come from the host OS and CUDA runtime, identical to any other CUDA workload.

Observability#

TensorRT-LLM does not expose Prometheus metrics directly; observability flows through the Triton Inference Server hosting the engine. Triton's `/metrics` endpoint emits standard request counters, latency histograms and GPU utilisation per model instance, with the `tensorrtllm_backend` adding TRT-LLM-specific gauges for KV cache utilisation, paused requests, and active in-flight batch size. For deeper kernel-level analysis, NVTX markers embedded in the runtime allow Nsight Systems to capture per-iteration traces showing prefill, attention, MLP and AllReduce phases in microsecond detail.

The metrics worth alerting on in production are: time-to-first-token p95, inter-token latency p95, KV cache utilisation, paused request count (sequences evicted from the running batch), Triton inference queue duration, and GPU SM occupancy from DCGM. The following Prometheus rules cover the common failure modes for a TRT-LLM deployment behind Triton.

• nv_inference_request_duration_us — Triton total request latency including queueing.
• nv_inference_queue_duration_us — time the request spent waiting before scheduling.
• nv_trt_llm_kv_cache_block_fraction — fraction of paged KV blocks in use; 0.95+ means capacity headroom is gone.
• nv_trt_llm_paused_requests — sequences evicted from the running batch under KV pressure; non-zero is a capacity smell.
• nv_trt_llm_active_request_count — current in-flight batch size; should track max_batch_size at steady-state load.
• nv_trt_llm_time_to_first_token_ms — prefill latency; correlate with prompt-length histogram.
• nv_trt_llm_inter_token_latency_ms — decode latency; should approach 1 / theoretical tok/s when batch is full.
• DCGM_FI_DEV_SM_OCCUPANCY and DCGM_FI_DEV_GPU_UTIL — pair with TRT-LLM metrics to distinguish compute vs memory vs idle bottlenecks.

# Prometheus rules for a TensorRT-LLM deployment behind Triton
groups:
  - name: trtllm-sla
    interval: 30s
    rules:
      - alert: TRTLLMHighTimeToFirstToken
        expr: histogram_quantile(0.95,
                sum by (le, model) (
                  rate(nv_trt_llm_time_to_first_token_ms_bucket[5m]))) > 400
        for: 5m
        labels: { severity: warning, team: inference }
        annotations:
          summary: "TRT-LLM TTFT p95 above 400ms on {{ $labels.model }}"

      - alert: TRTLLMKVCachePressure
        expr: nv_trt_llm_kv_cache_block_fraction > 0.95
        for: 10m
        labels: { severity: warning }
        annotations:
          summary: "Paged KV cache >95 percent full — pauses imminent"

      - alert: TRTLLMPausedRequestsSpike
        expr: increase(nv_trt_llm_paused_requests[5m]) > 10
        for: 5m
        labels: { severity: critical }
        annotations:
          summary: "Pause rate climbing — capacity insufficient or runaway request"

      - alert: TRTLLMQueueBuildup
        expr: histogram_quantile(0.95,
                sum by (le, model) (
                  rate(nv_inference_queue_duration_us_bucket[5m]))) > 500000
        for: 10m
        labels: { severity: warning }
        annotations:
          summary: "Triton queue p95 above 500ms — under-provisioned or stuck batch"

      - alert: TRTLLMSMUnderutilised
        expr: avg_over_time(DCGM_FI_DEV_SM_OCCUPANCY[5m]) < 0.30
              and rate(nv_inference_request_success[5m]) > 0
        for: 15m
        labels: { severity: info }
        annotations:
          summary: "GPU SM occupancy under 30 percent — investigate pipeline bubble or kernel selection"

Cost and FinOps#

TensorRT-LLM cost economics are dominated by three levers: GPU rental rate (identical to vLLM on the same hardware), achieved tokens-per-second-per-GPU (typically 1.4-1.8x vLLM at matched latency), and engine-build amortisation. The 40 percent lower $/M tokens versus vLLM holds for stable production models where the engine survives weeks or months; for rapidly rotating models the build-and-validate cost erodes the saving. The table below uses Yobitel UK list pricing (June 2026) and InferenceBench v3 throughput anchors at 4K input / 256 output.

• TensorRT-LLM wins versus vLLM on $/M tokens for any model that lives in production unchanged for more than two weeks. Below that horizon, vLLM's build-free deployment loop typically wins on total cost of ownership.
• FP8 weights + FP8 KV + FP8 context FMHA is the highest $/M-tokens lever on Hopper; BF16 is roughly 1.7x more expensive at the same SLO.
• Engine-build cost on a 70B FP8 TP=4 model is ~25 minutes on the same hardware that serves it. Build once, amortise across the engine lifetime; do not rebuild for trivial config changes.
• FOCUS-conformant billing exports from Yobitel include `inference_engine=tensorrt-llm`, `model_name` and `engine_build_id` resource tags so $/M tokens can be sliced by tenant, model and engine generation.
• Spot capacity is harder to operate with TRT-LLM than with vLLM: engine load takes 30-90 seconds on a fresh GPU, so pre-emption costs more wall-clock time. Reserve a small on-demand floor for spot-pre-empted traffic.

Security and compliance#

TensorRT-LLM inherits Triton Inference Server's security surface: bearer-token or mTLS auth at the gateway (Envoy, NGINX, AWS ALB), per-tenant quotas enforced upstream of Triton, and network isolation following the standard pattern — inference pods have no public-internet egress, engines are pulled from a private artefact registry, and per-replica NetworkPolicy locks ingress to the gateway service account. Engine binaries are immutable artefacts; signing them at build time and verifying signatures at load time is the recommended supply-chain control.

Hardware-rooted confidentiality flows through NVIDIA Confidential Compute on H100, H200 and B200 — the same TEE mode that protects training workloads protects TRT-LLM inference. Weights stay encrypted in GPU memory; the host kernel and even the hypervisor cannot read them. Yobitel sovereign tenancies in London-1 and Frankfurt-1 enable Confidential Compute by default for regulated workloads; the latency cost is roughly 3-5 percent versus non-confidential mode at matched throughput.

Regulatory implications are model- and data-specific. For UK public-sector workloads the canonical path is Yobitel sovereign tenancies operating under NCSC Cloud Security Principles, G-Cloud 14 lot definitions and the OFFICIAL handling caveat; the engine itself is a control-plane component on those tenancies. For EU GDPR, the engine processes prompt and completion data only in volatile GPU memory and the on-disk engine binary; ensure any logged inputs are masked at the Triton frontend. For US HIPAA workloads, run inside a BAA-covered VPC and disable request logging at the Triton layer; for FedRAMP-equivalent profiles, pin to the FIPS-validated CUDA build and use NIAP-approved cipher suites at the ingress.

Migration and alternatives#

Most production migrations to TensorRT-LLM come from one of three origins: vLLM (chasing the last 30-40 percent of throughput on a stabilised model), raw HuggingFace `transformers.generate()` (often a 10-15x uplift), or a hosted SaaS API (when sovereignty, custom models or unit-cost economics demand self-hosting). The migration effort is dominated not by code but by engine-build CI, kernel autotuning per SKU, and validation that the compiled engine produces output equivalent to the source checkpoint within tolerance.

If you are already running vLLM on Kubernetes with the OpenAI-compatible API surface, the migration target is a Triton deployment with the OpenAI frontend enabled. The code block below shows the equivalent deployments side by side; you can roll TRT-LLM behind the same `Service` and shift traffic at the gateway once the engine build pipeline is green.

# TRT-LLM behind Triton on Kubernetes with NVIDIA GPU Operator
kubectl apply -f - <<'YAML'
apiVersion: apps/v1
kind: Deployment
metadata: { name: llama3-70b-trtllm }
spec:
  replicas: 2
  selector: { matchLabels: { app: llama3-70b-trtllm } }
  template:
    metadata: { labels: { app: llama3-70b-trtllm } }
    spec:
      containers:
        - name: triton
          image: nvcr.io/nvidia/tritonserver:24.10-trtllm-python-py3
          args:
            - "tritonserver"
            - "--model-repository=/models"
            - "--grpc-port=8001"
            - "--http-port=8000"
            - "--metrics-port=8002"
          resources:
            limits: { nvidia.com/gpu: 4 }
          ports:
            - { containerPort: 8000, name: http }
            - { containerPort: 8001, name: grpc }
            - { containerPort: 8002, name: metrics }
          volumeMounts:
            - { name: engines, mountPath: /models }
            - { name: dshm, mountPath: /dev/shm }
      volumes:
        - name: engines
          persistentVolumeClaim: { claimName: llama3-70b-engines }
        - name: dshm
          emptyDir: { medium: Memory, sizeLimit: 8Gi }
YAML

# Equivalent vLLM deployment for the same model (for migration comparison)
# vllm serve meta-llama/Meta-Llama-3.1-70B-Instruct \
#     --tensor-parallel-size 4 --quantization fp8 --max-model-len 32768 \
#     --enable-prefix-caching --enable-chunked-prefill --port 8000

# Equivalent on AWS (bare p5 with NGC TRT-LLM container)
aws ec2 run-instances \
    --instance-type p5.48xlarge \
    --image-id "$(aws ec2 describe-images --owners amazon \
        --filters 'Name=name,Values=Deep Learning OSS Nvidia Driver AMI GPU PyTorch*' \
        --query 'sort_by(Images,&CreationDate)[-1].ImageId' --output text)" \
    --user-data "$(cat <<'EOF'
#!/bin/bash
docker run --gpus all -p 8000:8000 -p 8001:8001 -p 8002:8002 \
    -v /opt/engines:/models \
    nvcr.io/nvidia/tritonserver:24.10-trtllm-python-py3 \
    tritonserver --model-repository=/models
EOF
)"

Troubleshooting#

The error table below covers the failure modes that account for roughly 80 percent of production TensorRT-LLM incidents observed on Yobitel-operated fleets and the upstream community tracker. Each row maps an observable symptom to the underlying mechanism and the minimum-viable fix; for build-time errors the fix usually means a fresh `trtllm-build` run rather than a runtime config change.

Where this fits in the Yobitel stack#

TensorRT-LLM is the opt-in low-latency variant inside Yobibyte, Yobitel's AI-native platform. Models in the Yobibyte catalogue land first as vLLM-backed endpoints — the fast path to a working API — and a workload can be promoted to a TensorRT-LLM variant once the model, precision and shape have stabilised in production. The promotion runs through Yobibyte's managed engine-build pipeline: customers point at the model and target SLO, the platform compiles the engine on the right SKU, validates output equivalence against the vLLM baseline, and rolls the new engine behind the same OpenAI-compatible URL. No build flags, registries or Triton config to manage by hand.

Omniscient Compute scores every TensorRT-LLM engine release continuously against vLLM and SGLang on InferenceBench across H100 SXM5, H200, B200 and (on the Yobitel-AMD partnership) MI300X tenancies — a strictly NVIDIA-only result on the MI300X line. The benchmark mix covers chat (4K input / 256 output), RAG (16K shared prefix / 512 output), long context (96K input / 4K output) and offline batch. Customers see live $/M tokens, p50 / p95 latency and sustained tok/s for every supported model in their region, so the choice between vLLM and TensorRT-LLM becomes a numbers-driven trade-off rather than a vendor pitch.

For UK and EU sovereign workloads, TensorRT-LLM runs on Yobitel London-1 and Frankfurt-1 inside tenancies aligned to NCSC Cloud Security Principles, G-Cloud 14 lot definitions and the OFFICIAL handling caveat, with NVIDIA Confidential Compute enabled by default. The combination of a transparent open-source engine, sovereign hardware, hardware-rooted weight confidentiality and continuous published benchmarks is what lets Yobitel customers run sub-200 ms LLM endpoints in country without ceding either visibility or control to a hosted SaaS API.