Triton Inference Server on Kubernetes: Multi-Model Serving
vLLM is great for LLMs. Triton is what you reach for when you need to serve 20 different models on one GPU cluster.
vLLM has one job. Serve transformer language models with PagedAttention and continuous batching. It does that job better than anything else.
Triton has a different job. Serve any model, in any framework, on any hardware, from a single endpoint. PyTorch, TensorFlow, ONNX, TensorRT, Python code, OpenVINO, even custom backends. All from the same server.
That flexibility is the point. And the reason most teams end up running Triton alongside vLLM rather than instead of it.
This article covers what Triton actually does well, how to run it on Kubernetes, and when to pick it over vLLM or KServe.
What Triton Is
Triton Inference Server is an open source model serving runtime from NVIDIA. It started as TensorRT Inference Server in 2018. The rename came in 2020 when the project expanded beyond TensorRT.
At the core, Triton is a process that loads models from a directory and serves them over HTTP and gRPC. That is the whole shape of it.
Where Triton earns its complexity is in what it does once the model is loaded.
Dynamic batching. Triton collects individual inference requests and groups them into batches on the fly. A classifier getting 100 requests per second with 10ms between them gets served as 10 batches of 10, not 100 batches of 1. Throughput goes up 5 to 10x.
Concurrent model execution. Multiple models run on the same GPU in parallel. A single Triton pod can serve a ResNet classifier, a BERT embedder, and a CLIP image encoder simultaneously. They share the GPU.
Model ensembles. A single inference request can trigger a pipeline of multiple models. Preprocessing, main model, postprocessing, all chained together server side. The client sees one request and one response.
Framework agnostic. The backend system lets Triton load PyTorch, ONNX, TensorRT, TensorFlow SavedModel, Python, and custom C++ models from the same server. You do not need three different inference servers for three different model types.
The Model Repository
Triton loads models from a directory structure called the model repository. The layout matters because it encodes versioning, configuration, and backend selection.
model-repository/
├── resnet50/
│ ├── 1/
│ │ └── model.onnx
│ ├── 2/
│ │ └── model.onnx
│ └── config.pbtxt
├── bert-embedder/
│ ├── 1/
│ │ └── model.pt
│ └── config.pbtxt
└── clip-ensemble/
├── 1/
└── config.pbtxt
Each model gets a directory. Each version gets a numbered subdirectory. The config.pbtxt tells Triton what backend to use, what the input and output tensors look like, and how to batch requests.
