Optimizing Large Language Model Inference Latency with vLLM and TensorRT-LLM

Inference dominates the lifetime cost of a production LLM. Two engines lead the open ecosystem: vLLM, with PagedAttention and continuous batching, and TensorRT-LLM, with fused kernels and aggressive quantization on NVIDIA hardware. Choosing and tuning them correctly can cut latency and cost by multiples.

Understand your latency budget

Separate time-to-first-token (TTFT), dominated by prefill compute, from inter-token latency (ITL), dominated by memory bandwidth during decode. Chat UX is sensitive to TTFT; batch summarization cares about total throughput.

vLLM configuration

python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --tensor-parallel-size 2 \
  --gpu-memory-utilization 0.92 \
  --max-num-seqs 256 \
  --enable-prefix-caching \
  --quantization fp8

Key levers

• Prefix caching: reuse KV for shared system prompts — big wins for RAG templates.
• max-num-seqs: higher increases throughput but can raise ITL; tune against SLOs.
• FP8 / AWQ quantization: roughly halves memory, enabling larger batches.

TensorRT-LLM for peak performance

TensorRT-LLM compiles the model into an optimized engine with in-flight batching and fused attention. Build times are longer and engines are hardware-specific, so use it for stable, high-volume models. Serve via Triton's TensorRT-LLM backend for production features like metrics and model ensembles.

Speculative decoding

A small draft model proposes several tokens that the target model verifies in one forward pass. When acceptance rates are high — common for code and structured output — decode speed improves substantially with no change in output distribution.

Measure, then autoscale

Benchmark with realistic prompt and output length distributions. Autoscale on queue depth or pending tokens rather than GPU utilization, which saturates early and hides queueing latency.