If you are running 1.28 and the current version is 1.32, you are 4 versions behind. That is 3 sequential upgrades to catch up. Each one can break things.

Most teams put off upgrades because they are scared. The upgrade process is poorly documented for production environments. The official docs cover the happy path. They do not cover what happens when something goes wrong mid-upgrade.

This article covers the full upgrade strategy: planning, pre-flight checks, the upgrade itself, and what to do when things break.

Understanding the Upgrade Path

Kubernetes does not support skipping minor versions. You must upgrade one minor version at a time.

1.28 → 1.29 → 1.30 → 1.31 → 1.32

You cannot go directly from 1.28 to 1.32. Each hop requires a full upgrade cycle: control plane first, then worker nodes.

Within each minor version, you can jump patch versions freely. Going from 1.30.2 to 1.30.8 is safe and does not require the full upgrade procedure. Just update the kubelet and kubectl binaries.

The support window:

Version    Released     End of Support
1.30       Apr 2024     Jun 2025
1.31       Aug 2024     Oct 2025
1.32       Dec 2024     Feb 2026
1.33       Apr 2025     Jun 2026

If you are more than 2 versions behind the current release, prioritize upgrading. You are running on borrowed time.

Pre-Upgrade Checklist

Run these checks before every upgrade. Do not skip them. They catch 90% of upgrade failures before they happen.

Check 1: Read the changelog for breaking changes

Every Kubernetes release removes deprecated APIs, changes default behaviors, and sometimes breaks addons.

# Check for deprecated APIs in your cluster
# Install kubent (kube-no-trouble)
kubectl krew install deprecations

# Or use the standalone tool
kubent

kubent scans your cluster for resources using deprecated or removed APIs. If it finds any, update those resources BEFORE upgrading. An API that was deprecated in 1.30 might be removed in 1.32. Your manifests will fail to apply after the upgrade.

Check 2: Verify addon compatibility

Your CNI plugin (Calico, Cilium, Flannel), CSI drivers, ingress controller, and cert-manager all have Kubernetes version requirements. Check each one against the target version.

# Check current versions of critical addons
kubectl get pods -n kube-system -o custom-columns=\
  NAME:.metadata.name,\
  IMAGE:.spec.containers[0].image

If your CNI plugin does not support the target Kubernetes version, upgrade the CNI first.

Check 3: Back up etcd

This is non-negotiable. If the upgrade fails catastrophically, the etcd backup is your recovery path.

etcdctl snapshot save /var/backups/etcd/pre-upgrade-$(date +%Y%m%d).db \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

# Verify the snapshot
etcdctl snapshot status /var/backups/etcd/pre-upgrade-*.db --write-out=table

Copy the backup off the control plane node. If the node dies during upgrade, the backup on the node is useless.

Check 4: Verify PodDisruptionBudgets

PDBs control how many pods can be unavailable during node drains. If a PDB prevents draining, the upgrade stalls.

# List all PDBs
kubectl get pdb --all-namespaces

# Check for PDBs that might block drains
kubectl get pdb --all-namespaces -o jsonpath='{range .items[*]}{.metadata.namespace}/{.metadata.name}: minAvailable={.spec.minAvailable} maxUnavailable={.spec.maxUnavailable}{"\n"}{end}'

A PDB with minAvailable equal to the current replica count blocks all drains. Either increase replicas or temporarily relax the PDB during upgrades.

Check 5: Dry run the upgrade

# On the first control plane node
sudo kubeadm upgrade plan

This shows exactly what will be upgraded and flags any issues. If it reports errors, fix them before proceeding.

Upgrading the Control Plane

The control plane upgrades one node at a time. Never upgrade all control plane nodes simultaneously.

Step 1: Upgrade kubeadm on the first control plane node

# Update the package repository
sudo apt-get update

# Install the target version of kubeadm
sudo apt-get install -y kubeadm=1.31.0-1.1

# Verify
kubeadm version

Step 2: Apply the upgrade

# On the FIRST control plane node only
sudo kubeadm upgrade apply v1.31.0

This upgrades the API server, controller manager, scheduler, and kube-proxy on this node. etcd is upgraded if it is managed by kubeadm (stacked topology).

Expected output:

[upgrade/successful] SUCCESS! Your cluster was upgraded to "v1.31.0". Enjoy!

If it fails, do NOT proceed. Check the error. Common failures: etcd health check fails (fix etcd first), certificate issues (renew with kubeadm certs renew all), or insufficient disk space.

Step 3: Upgrade kubelet and kubectl on the first node

sudo apt-get install -y kubelet=1.31.0-1.1 kubectl=1.31.0-1.1
sudo systemctl daemon-reload
sudo systemctl restart kubelet

Step 4: Verify the first node

kubectl get nodes
# The upgraded node should show v1.31.0
# Other nodes still show the old version - this is expected

Step 5: Upgrade remaining control plane nodes

On each additional control plane node:

sudo apt-get install -y kubeadm=1.31.0-1.1

# Note: use "upgrade node" not "upgrade apply" for subsequent nodes
sudo kubeadm upgrade node

sudo apt-get install -y kubelet=1.31.0-1.1 kubectl=1.31.0-1.1
sudo systemctl daemon-reload
sudo systemctl restart kubelet

Wait for each node to show Ready before moving to the next.

Upgrading Worker Nodes

Worker nodes upgrade one at a time (or in batches if you have capacity).

Step 1: Drain the node

kubectl drain <node-name> \
  --ignore-daemonsets \
  --delete-emptydir-data \
  --grace-period=120 \
  --timeout=300s

This evicts all pods from the node. DaemonSets stay (they run on every node). Pods with emptyDir volumes lose their data.

Step 2: Upgrade kubeadm, kubelet, kubectl

# SSH into the worker node
sudo apt-get update
sudo apt-get install -y kubeadm=1.31.0-1.1
sudo kubeadm upgrade node
sudo apt-get install -y kubelet=1.31.0-1.1 kubectl=1.31.0-1.1
sudo systemctl daemon-reload
sudo systemctl restart kubelet

Step 3: Uncordon the node

kubectl uncordon <node-name>

The node is now schedulable again. Pods will be scheduled back onto it.

Step 4: Verify and move to the next node

kubectl get nodes
# Upgraded node shows v1.31.0 and Ready status

Repeat for each worker node. If you have GPU nodes, upgrade them last. GPU pods take longer to reschedule because of model loading times.

Handling GPU Nodes During Upgrades

GPU nodes need special attention:

The GPU Operator must be compatible with the target Kubernetes version. Check the NVIDIA GPU Operator compatibility matrix before upgrading.

vLLM pods take 1 to 30 minutes to restart (model loading). Plan for this downtime per node. If you have a PVC-backed model cache, restart is faster (1 to 3 minutes instead of 30).

Drain GPU nodes with a longer grace period:

kubectl drain gpu-node-1 \
  --ignore-daemonsets \
  --delete-emptydir-data \
  --grace-period=300 \
  --timeout=600s

After uncordoning, verify the GPU Operator components restart correctly:

kubectl get pods -n gpu-operator -o wide | grep gpu-node-1

All GPU Operator pods on that node should be Running before proceeding to the next GPU node.

Post-Upgrade Validation

Run these checks after all nodes are upgraded:

# 1. All nodes on the new version and Ready
kubectl get nodes

# 2. All system pods healthy
kubectl get pods -n kube-system

# 3. etcd cluster healthy
etcdctl endpoint health --cluster

# 4. DNS working
kubectl run dns-test --image=busybox:1.36 --rm -it --restart=Never -- nslookup kubernetes.default

# 5. Create and delete a test resource
kubectl create configmap upgrade-test --from-literal=version=1.31
kubectl delete configmap upgrade-test

# 6. Check for pods in bad states
kubectl get pods --all-namespaces --field-selector status.phase!=Running,status.phase!=Succeeded

# 7. Verify GPU workloads (if applicable)
kubectl get pods -n inference

Rollback Strategy

If the upgrade fails mid-way:

For control plane: restore from the etcd backup taken in the pre-upgrade checklist. This rolls back the cluster state to before the upgrade.

For worker nodes: the failed node can be drained and reimaged with the old version. Other nodes continue running normally.

The key: always have the etcd backup. Without it, there is no rollback. The pre-upgrade etcd snapshot is your safety net.

The Bottom Line

Kubernetes upgrades are not optional. The support window is 14 months. After that, you are running unpatched software in production.

The process: back up etcd, check deprecated APIs, verify addon compatibility, upgrade control plane one node at a time, upgrade workers one node at a time, validate everything.

Do not skip the pre-flight checks. Do not upgrade all nodes at once. Do not skip the etcd backup. These three mistakes cause 90% of upgrade failures.

Next week: Autoscaling Inference Workloads: HPA and KEDA for GPU Pods.

If you are running self-managed Kubernetes clusters, I cover operations, upgrades, and GPU infrastructure every week. Subscribe at kubenatives.com.

Network policies fix this. They are firewall rules for pod-to-pod traffic. But most teams implement them wrong. They add a restrictive policy, DNS breaks silently, and they spend hours debugging before removing the policy and giving up.

This article covers how to implement network policies correctly. Starting with the one rule that prevents 90% of the problems.

The Default Behavior

With no network policies, Kubernetes networking is fully open. Every pod can reach every other pod on any port. Every pod can reach external services. There are no restrictions.

The moment you create a NetworkPolicy in a namespace, the behavior changes. Pods selected by the policy are now restricted. Traffic not explicitly allowed by a policy is denied.

This is the part that catches people. Adding one policy does not just restrict what that policy covers. It implicitly denies everything else for the selected pods.

# This policy allows port 8080 ingress from the frontend namespace.
# But it ALSO denies all other ingress to these pods.
# AND it denies all egress from these pods (if policyTypes includes Egress).
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-frontend
  namespace: backend
spec:
  podSelector:
    matchLabels:
      app: api-server
  policyTypes:
  - Ingress
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          name: frontend
    ports:
    - protocol: TCP
      port: 8080

Rule Zero: Allow DNS First

This is the single most important rule. Before you create any other network policy, deploy this one in every namespace:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns
  namespace: backend
spec:
  podSelector: {}
  policyTypes:
  - Egress
  egress:
  - to:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: kube-system
    ports:
    - protocol: UDP
      port: 53
    - protocol: TCP
      port: 53

Without this, pods lose DNS resolution the moment you add any egress policy. The failure is silent. No error message. No rejection. Queries are dropped and the application waits for a timeout.

This is the #1 cause of “network policies broke everything.” Deploy the DNS egress rule first. In every namespace. Before anything else.

Building a Zero-Trust Network Step by Step

The safest approach is to start with a default deny policy and then explicitly allow what you need. Here is the order:

Step 1: Default deny all traffic

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: backend
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  - Egress

This blocks all ingress and egress for every pod in the namespace. Nothing can communicate. This is intentionally extreme. You will add allow rules next.

Step 2: Allow DNS egress (Rule Zero)

Deploy the DNS policy from above. Pods can now resolve names but cannot reach anything else.

Step 3: Allow inter-service communication

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: api-server-ingress
  namespace: backend
spec:
  podSelector:
    matchLabels:
      app: api-server
  policyTypes:
  - Ingress
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          name: frontend
      podSelector:
        matchLabels:
          app: web
    ports:
    - protocol: TCP
      port: 8080

This allows the web pod in the frontend namespace to reach the api-server pod in the backend namespace on port 8080. Nothing else can reach the api-server.

Step 4: Allow database access

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: database-ingress
  namespace: backend
spec:
  podSelector:
    matchLabels:
      app: postgres
  policyTypes:
  - Ingress
  ingress:
  - from:
    - podSelector:
        matchLabels:
          app: api-server
    ports:
    - protocol: TCP
      port: 5432

Only the api-server in the same namespace can reach postgres on port 5432. The frontend cannot reach the database directly. A compromised frontend pod cannot access your data.

Step 5: Allow external egress for pods that need it

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: api-server-egress
  namespace: backend
spec:
  podSelector:
    matchLabels:
      app: api-server
  policyTypes:
  - Egress
  egress:
  # DNS
  - to:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: kube-system
    ports:
    - protocol: UDP
      port: 53
    - protocol: TCP
      port: 53
  # Database
  - to:
    - podSelector:
        matchLabels:
          app: postgres
    ports:
    - protocol: TCP
      port: 5432
  # External APIs (HTTPS)
  - to:
    - ipBlock:
        cidr: 0.0.0.0/0
        except:
        - 10.0.0.0/8
        - 172.16.0.0/12
        - 192.168.0.0/16
    ports:
    - protocol: TCP
      port: 443

The api-server can reach DNS, the database, and external HTTPS endpoints. It cannot reach anything else inside the cluster. The ipBlock with except clauses blocks access to other internal services while allowing external API calls.

Common Mistakes

Mistake 1: Forgetting DNS egress. The #1 cause of “network policies broke everything.” Always deploy the DNS allow rule first.

Mistake 2: Using podSelector without namespaceSelector. A podSelector alone only matches pods in the same namespace. To allow traffic from another namespace, you must include a namespaceSelector.

# WRONG: only matches pods in the SAME namespace
- from:
  - podSelector:
      matchLabels:
        app: web

# RIGHT: matches pods in the frontend namespace
- from:
  - namespaceSelector:
      matchLabels:
        name: frontend
    podSelector:
      matchLabels:
        app: web

Mistake 3: AND vs OR logic. When namespaceSelector and podSelector are in the same from entry (same YAML block), they are AND logic. Both must match. When they are separate entries (separate list items), they are OR logic. Either can match.

# AND: must be in frontend namespace AND have app=web label
- from:
  - namespaceSelector:
      matchLabels:
        name: frontend
    podSelector:
      matchLabels:
        app: web

# OR: anything in frontend namespace OR anything with app=web label
- from:
  - namespaceSelector:
      matchLabels:
        name: frontend
  - podSelector:
      matchLabels:
        app: web

The difference is one hyphen. One wrong indent and your policy allows far more traffic than intended. This is the most dangerous mistake in network policies.

Mistake 4: Not labeling namespaces. NetworkPolicies select namespaces by label, not by name. If your namespace does not have a label, the namespaceSelector cannot match it.

# Label your namespaces
kubectl label namespace frontend name=frontend
kubectl label namespace backend name=backend

Kubernetes 1.21+ automatically adds the label kubernetes.io/metadata.name to every namespace. Use that for reliability.

Mistake 5: Forgetting monitoring and logging egress. Your pods need to reach Prometheus (for scraping) and your log aggregator. If you block egress without allowing these, you lose observability.

Testing Network Policies

Never deploy network policies blind. Test them first.

# Deploy a test pod
kubectl run nettest --image=busybox:1.36 -n backend --rm -it --restart=Never -- sh

# Test DNS
nslookup kubernetes.default

# Test service connectivity
wget -qO- --timeout=3 http://api-server:8080/health

# Test external connectivity
wget -qO- --timeout=3 https://httpbin.org/get

Run these tests before and after applying each policy. If something breaks, you know exactly which policy caused it.

The Debug Checklist

When a pod cannot reach a service after network policies are applied:

1. Check if DNS works: nslookup kubernetes.default from inside the pod.

2. If DNS fails: the DNS egress rule is missing.

3. If DNS works but the service is unreachable: the ingress policy on the destination does not allow traffic from the source pod or namespace.

4. Check namespace labels: kubectl get namespace <ns> --show-labels.

5. Check pod labels: kubectl get pod <pod> --show-labels.

6. Verify the policy is selecting the right pods: kubectl get networkpolicies -n <ns> -o yaml.

The Bottom Line

Network policies are simple in concept and dangerous in practice. The implicit deny behavior catches everyone. The AND vs OR selector logic catches even experienced engineers.

Start with Rule Zero (allow DNS). Add default deny. Then explicitly allow each communication path. Test after every policy. Do not deploy all policies at once.

Five policies can secure a namespace. One missing DNS rule can break it.

Next week: Kubernetes Upgrade Strategy: kubeadm Cluster Upgrades Without Downtime.

If you are running production Kubernetes clusters, I cover networking, GPU infrastructure, and operations every week. Subscribe at kubenatives.com.

• Setting up GPU node isolation for the first time

• Adding a new GPU tier to an existing cluster

• Configuring per-team GPU quotas

• Setting up priority classes for GPU workloads

File 1: gpu-node-taints.sh

Apply taints to GPU nodes. Run once per node or set at the node pool level.

#!/bin/bash
# gpu-node-taints.sh
# Apply taints to GPU nodes for workload isolation

set -euo pipefail

echo "=== Tainting Production GPU Nodes (Tier 1) ==="
for node in $(kubectl get nodes -l gpu-tier=production -o jsonpath='{.items[*].metadata.name}'); do
  kubectl taint nodes $node nvidia.com/gpu=present:NoSchedule --overwrite
  kubectl label nodes $node gpu-tier=production --overwrite
  echo "  Tainted: $node"
done

echo ""
echo "=== Tainting Development GPU Nodes (Tier 2) ==="
for node in $(kubectl get nodes -l gpu-tier=development -o jsonpath='{.items[*].metadata.name}'); do
  kubectl taint nodes $node nvidia.com/gpu=present:NoSchedule --overwrite
  kubectl label nodes $node gpu-tier=development --overwrite
  echo "  Tainted: $node"
done

echo ""
echo "=== Verification ==="
echo "Production GPU nodes:"
kubectl get nodes -l gpu-tier=production -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints
echo ""
echo "Development GPU nodes:"
kubectl get nodes -l gpu-tier=development -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints

File 2: priority-classes.yaml

# gpu-priority-classes.yaml
# Three tiers of GPU workload priority

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: gpu-production
value: 1000000
globalDefault: false
preemptionPolicy: PreemptLowerPriority
description: |
  Production GPU inference workloads.
  Highest priority. Will preempt development and batch workloads.
---
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: gpu-development
value: 100000
globalDefault: false
preemptionPolicy: PreemptLowerPriority
description: |
  Development GPU workloads (notebooks, experiments).
  Preempted by production. Will preempt batch workloads.
---
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: gpu-batch
value: 10000
globalDefault: false
preemptionPolicy: Never
description: |
  Batch GPU jobs (training, data processing).
  Lowest priority. Will NOT preempt other workloads.
  Waits for available GPUs.

kubectl apply -f priority-classes.yaml

Read more

Without any configuration, the Kubernetes scheduler treats them all the same. It sees available CPU and memory. It does not distinguish between a $200/month CPU node and a $30K/year GPU node.

A basic nginx pod with 100m CPU and 128Mi memory can land on your H100 node. The scheduler found resources. It scheduled the pod. It did exactly what it was designed to do.

This article covers how to stop that from happening. And how to build a multi-tier GPU cluster where the right workloads land on the right hardware every time.

The Problem: GPUs as Shared Resources

By default, any pod can schedule on any node that has enough CPU and memory. GPU nodes have CPU and memory in addition to GPUs. Non-GPU workloads see the CPU and memory and schedule there.

The result: GPU nodes run a mix of GPU workloads and random CPU workloads. The CPU workloads consume memory and CPU that GPU workloads need. And you are paying GPU pricing for pods that do not use GPUs.

# Check what is running on your GPU nodes
kubectl get pods --all-namespaces -o wide --field-selector spec.nodeName=gpu-node-1

If you see system pods, monitoring agents, log collectors, and random application pods alongside your vLLM deployment, your GPU nodes are not isolated.

Taints: Keep Non-GPU Workloads Off GPU Nodes

A taint on a node tells the scheduler: “Do not place pods here unless they explicitly tolerate this taint.”

# Add a taint to all GPU nodes
kubectl taint nodes gpu-node-1 nvidia.com/gpu=present:NoSchedule
kubectl taint nodes gpu-node-2 nvidia.com/gpu=present:NoSchedule

After this, no pod can schedule on these nodes unless it has a matching toleration.

On managed Kubernetes (EKS, GKE, AKS), you set taints at the node pool level. Every node in the pool gets the taint automatically:

# GKE example: GPU node pool with taint
gcloud container node-pools create gpu-pool \
  --cluster=my-cluster \
  --machine-type=a2-highgpu-1g \
  --accelerator=type=nvidia-tesla-a100,count=1 \
  --num-nodes=3 \
  --node-taints=nvidia.com/gpu=present:NoSchedule

Important: The NVIDIA GPU Operator automatically adds the taint nvidia.com/gpu=present:NoSchedule when it detects GPU hardware. If you are using the GPU Operator, the taints are already there. Check with:

kubectl get nodes -l nvidia.com/gpu.present=true \
  -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.taints}{"\n"}{end}'

Tolerations: Allow GPU Workloads to Schedule

Your GPU pods need a toleration that matches the taint:

spec:
  tolerations:
  - key: nvidia.com/gpu
    operator: Exists
    effect: NoSchedule
  containers:
  - name: vllm
    resources:
      limits:
        nvidia.com/gpu: "1"

The operator: Exists means “tolerate this taint regardless of value.” This is simpler than matching a specific value and works for any GPU taint.

What about system pods? DaemonSets like the GPU Operator, node exporter, and log collectors need to run on GPU nodes too. They should also have the toleration. The GPU Operator DaemonSets already include it. Your monitoring stack may need it added manually.

Node Affinity: Target Specific GPU Types

Taints prevent the wrong pods from landing on GPU nodes. But in a mixed GPU cluster (A100s and T4s), you also need to ensure the right pods land on the right GPU type.

GPU Feature Discovery (part of the GPU Operator) labels each node with its GPU model:

nvidia.com/gpu.product=NVIDIA-A100-SXM4-80GB
nvidia.com/gpu.product=NVIDIA-T4
nvidia.com/gpu.product=NVIDIA-H100-80GB-HBM3

Use node affinity to target specific GPU types:

spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: nvidia.com/gpu.product
            operator: In
            values:
            - NVIDIA-A100-SXM4-80GB
  tolerations:
  - key: nvidia.com/gpu
    operator: Exists
    effect: NoSchedule
  containers:
  - name: vllm
    resources:
      limits:
        nvidia.com/gpu: "1"

This pod will only schedule on A100 nodes. Even if T4 nodes have available GPUs.

The Multi-Tier GPU Cluster Pattern

Here is the pattern I use for production clusters with mixed GPU types:

Each tier has its own taint and label. Workloads use tolerations to enter the GPU tiers and node affinity to target the right tier.

# Production inference: must land on Tier 1
affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
      - matchExpressions:
        - key: gpu-tier
          operator: In
          values:
          - production

# Dev notebook: must land on Tier 2
affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
      - matchExpressions:
        - key: gpu-tier
          operator: In
          values:
          - development

Cost Isolation with Resource Quotas

Taints control where pods run. Resource quotas control how much each team can consume.

apiVersion: v1
kind: ResourceQuota
metadata:
  name: gpu-quota
  namespace: ml-team-a
spec:
  hard:
    requests.nvidia.com/gpu: "4"
    limits.nvidia.com/gpu: "4"
---
apiVersion: v1
kind: ResourceQuota
metadata:
  name: gpu-quota
  namespace: ml-team-b
spec:
  hard:
    requests.nvidia.com/gpu: "2"
    limits.nvidia.com/gpu: "2"

Team A can use up to 4 GPUs. Team B can use up to 2. Neither team can consume more than their quota regardless of what is available in the cluster.

Combine resource quotas with LimitRanges to prevent individual pods from requesting too many GPUs:

apiVersion: v1
kind: LimitRange
metadata:
  name: gpu-limits
  namespace: ml-team-a
spec:
  limits:
  - type: Container
    max:
      nvidia.com/gpu: "2"
    default:
      nvidia.com/gpu: "1"

No single container in team A’s namespace can request more than 2 GPUs. Default is 1 if not specified.

Priority Classes for GPU Workloads

When GPU capacity is scarce, priority classes determine which pods get GPUs first and which get preempted.

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: gpu-production
value: 1000000
globalDefault: false
description: "Production GPU inference workloads"
---
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: gpu-development
value: 100000
globalDefault: false
description: "Development GPU workloads"
---
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: gpu-batch
value: 10000
globalDefault: false
preemptionPolicy: Never
description: "Batch GPU jobs - do not preempt others"

Production inference gets the highest priority. If a production pod needs a GPU and all are allocated, Kubernetes preempts a development or batch pod to make room.

The preemptionPolicy: Never on the batch class means batch jobs will wait for GPUs but will never kick out other workloads.

# Use in pod spec
spec:
  priorityClassName: gpu-production

Common Mistakes

Mistake 1: No taints on GPU nodes. Random CPU workloads consume resources on expensive GPU hardware. Always taint GPU nodes.

Mistake 2: Forgetting tolerations on GPU Operator DaemonSets. If you add custom taints, the GPU Operator pods need matching tolerations. Otherwise the operator cannot run on GPU nodes, which means GPUs are never registered.

Mistake 3: Using nodeSelector instead of nodeAffinity. nodeSelector is simpler but less flexible. You cannot express “schedule on A100 OR H100” with nodeSelector. nodeAffinity supports multiple values and complex expressions.

Mistake 4: No resource quotas. Without quotas, one team can consume all GPUs in the cluster. This is fine with 2 engineers. It is chaos with 10 teams.

Mistake 5: Same priority for all GPU workloads. When capacity is tight, production inference and a Jupyter notebook have the same priority. The notebook should yield to production. Use priority classes.

The Bottom Line

GPU nodes are expensive. Treat them like expensive resources.

Taints keep non-GPU workloads off GPU hardware. Tolerations allow GPU workloads in. Node affinity targets specific GPU types. Resource quotas cap per-team consumption. Priority classes ensure production wins when capacity is scarce.

Five mechanisms. Together they turn a cluster where anything runs anywhere into a multi-tier platform where the right workloads land on the right hardware at the right priority.

Next week: Network Policies in Practice: When Your Pods Cannot Talk to Each Other.

If you are building GPU infrastructure on Kubernetes, I cover scheduling, model serving, and production operations every week. Subscribe at kubenatives.com.

Then someone asks: “How do we roll out a new model version without downtime?” Or: “How do we know if the model’s responses are getting worse?” Or: “Can we test GPT-4o alongside Llama 3 and route traffic based on the use case?”

These are not model serving questions. These are LLMOps questions. And most teams have no framework for answering them.

This article covers the 6 patterns that turn a model deployment into a production LLM system.

Pattern 1: Model Versioning and Rollouts

In traditional software, you version your code. In LLMOps, you version three things: the model weights, the serving configuration, and the prompt templates. All three change independently. All three affect output quality.

The problem. You upgrade from Llama 3.1 8B to Llama 3.1 70B. The model is better. But your prompts were tuned for the 8B version. The 70B version interprets the system prompt differently. Response quality drops even though the model improved.

The pattern. Version the model and the prompt together as a single deployment unit. A “model version” is not just the weights. It is the weights plus the serving config plus the prompt template.

On Kubernetes, this maps to a Deployment revision. Each revision locks in:

# Version 1: Llama 3.1 8B + prompt v1
containers:
- name: vllm
  image: vllm/vllm-openai:v0.6.0
  args:
  - --model
  - meta-llama/Llama-3.1-8B-Instruct
  env:
  - name: SYSTEM_PROMPT_VERSION
    value: "v1"

# Version 2: Llama 3.1 70B + prompt v2
containers:
- name: vllm
  image: vllm/vllm-openai:v0.6.0
  args:
  - --model
  - meta-llama/Llama-3.1-70B-Instruct
  - --tensor-parallel-size
  - "2"
  env:
  - name: SYSTEM_PROMPT_VERSION
    value: "v2"

Rollout strategy. Never switch 100% of traffic at once. Use a canary deployment. Route 5% of traffic to the new version. Monitor quality metrics for 24 hours. If quality holds, increase to 25%, then 50%, then 100%.

KServe handles this natively with traffic splitting on InferenceService revisions. Without KServe, use Istio VirtualService or a gateway that supports weighted routing.

Pattern 2: Prompt Management

Prompts are configuration, not code. But most teams hardcode them in application source files. This means changing a prompt requires a code deployment, a build pipeline, a review cycle, and a rollout.

The pattern. Store prompts in ConfigMaps or an external prompt store. The serving layer reads the prompt at request time, not at build time.

apiVersion: v1
kind: ConfigMap
metadata:
  name: prompt-templates
  namespace: inference
data:
  system-v1.txt: |
    You are a helpful assistant for DevOps engineers.
    Answer questions about Kubernetes, Docker, and cloud infrastructure.
    Be concise. Use code examples when relevant.
  system-v2.txt: |
    You are a senior infrastructure engineer assistant.
    Provide production-ready advice with specific commands.
    Always mention potential risks and rollback steps.

Mount the ConfigMap into the application pod. The application reads the prompt file at request time. To update a prompt, update the ConfigMap. The pods pick up the change without restarting (if using subPath mounts, a restart is needed).

Why this matters. Prompt iteration is fast. Model deployment is slow (minutes to load weights). Decoupling prompts from deployments means you can iterate on prompts in seconds without touching the model.

Pattern 3: LLM Gateway and Routing

Most production systems do not use a single model. They use different models for different tasks. A small fast model for classification. A large model for generation. A specialized model for code.

The pattern. An LLM gateway sits between your application and the model backends. It handles routing, fallback, rate limiting, and load balancing across multiple models.

The gateway routes requests based on metadata in the request: model name, use case tag, user tier, or custom headers. If the primary model is overloaded or down, the gateway falls back to an alternative.

On Kubernetes, LiteLLM is the most common open source LLM gateway. It provides an OpenAI compatible API that proxies to multiple backends.

# LiteLLM config
model_list:
  - model_name: "generation"
    litellm_params:
      model: "hosted_vllm/meta-llama/Llama-3.1-70B-Instruct"
      api_base: "http://vllm-70b.inference:8000/v1"
  - model_name: "generation"
    litellm_params:
      model: "gpt-4o"
      api_key: "os.environ/OPENAI_API_KEY"
    # Fallback: if vLLM is down, route to OpenAI

Why this matters. Without a gateway, your application has hardcoded model endpoints. Switching models requires application changes. With a gateway, you change the routing config. The application never knows.

Pattern 4: Quality Observability

GPU metrics tell you if the model is running. They do not tell you if it is running well.

TTFT, throughput, and cache utilization are infrastructure metrics. They measure serving performance. But a model can serve fast responses that are completely wrong.

The pattern. Add a quality observability layer that tracks response characteristics over time.

Metrics to track:

Response length distribution. A sudden drop in average response length can indicate the model is generating truncated or degenerate responses. Plot a histogram of response token counts. Alert if the distribution shifts.

Refusal rate. How often the model refuses to answer (returns “I cannot help with that” or similar). A spike in refusals after a prompt change indicates the guardrails are too aggressive.

Latency per output token. Not just TTFT. Measure the time per token during decoding. If this increases without load changes, the model may be struggling with certain prompt patterns.

User feedback signals. Thumbs up/down, regenerate clicks, copy events. These are noisy individually but powerful in aggregate. A drop in positive signals after a model change is a quality regression.

# Custom metrics to export alongside vLLM metrics
- name: llm_response_tokens_total
  type: histogram
  help: Distribution of response lengths in tokens
  buckets: [10, 50, 100, 200, 500, 1000, 2000]

- name: llm_refusal_total
  type: counter
  help: Number of refusal responses detected

- name: llm_user_feedback
  type: counter
  labels: [feedback_type]
  help: User feedback signals (positive, negative, regenerate)

Why this matters. You cannot improve what you do not measure. Infrastructure metrics tell you the system is running. Quality metrics tell you it is working.

Pattern 5: Guardrails

Production LLMs need input and output filtering. Without guardrails, a user can prompt to inject the model to ignore its system instructions. Or the model can generate harmful, incorrect, or off-topic content.

The pattern. Two layers of filtering. Input guardrails before the model processes the request. Output guardrails after the model generates a response.

Request → Input Filter → Model → Output Filter → Response

Input filtering catches prompt injection attempts, PII in user messages, and requests that are outside the model’s intended scope.

Output filtering catches harmful content, PII in responses, and responses that contradict known facts (hallucination detection).

On Kubernetes, guardrails run as a sidecar container or a separate microservice in the request path. Running them as a sidecar keeps latency low (no network hop). Running them as a separate service allows independent scaling and updates.

The latency tradeoff. Every guardrail adds latency. Input filtering adds 10 to 50ms. Output filtering can add 50 to 200ms for content classification. For interactive chat applications, this matters. For batch processing, it does not.

The pattern for production: Run lightweight keyword and regex filters in the request path (low latency). Run heavier ML-based content classifiers asynchronously. Flag problematic responses for review rather than blocking them in real time.

Pattern 6: Cost Attribution

GPU infrastructure is expensive. When multiple teams share a model serving platform, you need to know who is using what and how much it costs.

The pattern. Tag every request with a team or project identifier. Aggregate GPU-seconds and token counts per tag. Charge back to teams based on usage.

On Kubernetes, this maps to namespace-level resource quotas for GPU allocation and request-level tagging for usage tracking.

Request headers:
  X-Team: search-team
  X-Project: product-search
  X-Budget-Code: eng-2024-q3

The LLM gateway logs these headers alongside token counts and GPU time. A billing pipeline aggregates usage per team per day.

search-team: 2.4M input tokens, 800K output tokens, 48 GPU-hours
recommendation-team: 1.1M input tokens, 400K output tokens, 22 GPU-hours

Why this matters. Without cost attribution, GPU spend is a shared cost that nobody owns. With attribution, teams optimize their own usage. The team generating 1M tokens per day will find ways to cache, batch, or use smaller models when they see their bill.

Putting It All Together

The 6 patterns form a stack:

Model serving is the foundation. You already have this from the vLLM and Triton articles.

Start with model versioning and prompt management. These are the minimum for operating in production. Without them, every change is a risky manual process.

Add the LLM gateway when you serve multiple models or need fallback routing.

Add guardrails when you serve external users or handle sensitive data.

Add quality observability and cost attribution as the system matures and multiple teams start using it.

Do not build all 6 on day one. Start with the bottom of the stack and work up.

The Bottom Line

Deploying a model is the easy part. The hard part is versioning it, routing traffic to it, knowing if it is working well, keeping it safe, and understanding what it costs.

These 6 patterns are not theoretical. They are the operational layer that turns a vLLM deployment into a production LLM system. Every team running LLMs in production eventually builds all of them. The question is whether you build them intentionally or discover the need at 3 AM.

Start with model versioning and prompt management. Add the rest as your system grows.

Next week: vLLM Model Loading Strategies: PVCs, Init Containers, and Shared Storage.

If you are building LLM infrastructure on Kubernetes, I cover the intersection of GPU infrastructure, model serving, and production operations every week. Subscribe at kubenatives.com.

• Setting up CoreDNS for a new cluster

• Debugging intermittent DNS failures

• Enabling DNS query logging temporarily

• Optimizing DNS performance with caching

Template 1: Production CoreDNS ConfigMap

This replaces the default CoreDNS Corefile with production-ready settings.

apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns
  namespace: kube-system
data:
  Corefile: |
    .:53 {
        errors
        health {
            lameduck 5s
        }
        ready
        kubernetes cluster.local in-addr.arpa ip6.arpa {
            pods insecure
            fallthrough in-addr.arpa ip6.arpa
            ttl 30
        }
        prometheus :9153
        forward . /etc/resolv.conf {
            max_concurrent 1000
        }
        cache 30 {
            success 9984 30
            denial 9984 5
        }
        loop
        reload
        loadbalance
    }

Read more

You spend an hour checking network policies, firewall rules, and pod security settings. Then someone runs nslookup from inside the pod, and DNS does not resolve.

It was DNS. It is always DNS.

But “it is DNS” is not a diagnosis. There are exactly 5 causes of DNS failures in Kubernetes. This article covers all of them with the exact commands to identify each one.

How Kubernetes DNS Works

Before debugging, you need to understand the path a DNS query takes inside a cluster.

When a pod makes a DNS request, here is what happens:

The application calls getaddrinfo() or a similar resolver function. The resolver reads /etc/resolv.conf inside the container. That file points to the CoreDNS Service IP (typically 10.96.0.10). The query goes to CoreDNS. CoreDNS looks up the answer and returns it.

Share

The /etc/resolv.conf inside every pod looks like this:

nameserver 10.96.0.10
search default.svc.cluster.local svc.cluster.local cluster.local
options ndots:5

Three lines. Each one matters. Each one can break.

nameserver is the CoreDNS ClusterIP. If CoreDNS is down, no DNS works.

search is the list of domains Kubernetes appends to short names. When you call my-service, Kubernetes actually tries my-service.default.svc.cluster.local first, then my-service.svc.cluster.local, then my-service.cluster.local, then the bare name.

ndots:5 is the setting that causes the most confusion and the most wasted time in production. More on this below.

Cause 1: CoreDNS Pods Are Not Running

The simplest cause. If CoreDNS is down, nothing resolves.

# Check CoreDNS pods
kubectl get pods -n kube-system -l k8s-app=kube-dns

Every pod should be Running. If any pod is in CrashLoopBackOff, Pending, or Error, that is your DNS problem.

Common reasons CoreDNS pods fail:

CoreDNS ConfigMap has a syntax error. Someone edited the Corefile and introduced a typo. CoreDNS cannot start with an invalid configuration.

# Check the Corefile
kubectl get configmap coredns -n kube-system -o yaml

Resource limits are too low. On large clusters, CoreDNS needs more CPU and memory than the defaults. If it is OOMKilled, DNS fails intermittently under load.

# Check for OOMKilled events
kubectl describe pods -n kube-system -l k8s-app=kube-dns | grep -A5 "Last State"

The node running CoreDNS is unhealthy. CoreDNS runs as a Deployment (usually 2 replicas). If both land on the same node and that node has issues, DNS fails cluster-wide.

The fix: Ensure CoreDNS replicas are spread across nodes with pod anti-affinity. Most managed K8s providers do this by default. Self-managed clusters often miss it.

Cause 2: The ndots Problem

This is the most common DNS performance issue in Kubernetes. It does not cause DNS to fail. It causes DNS to be slow.

The ndots:5 setting in /etc/resolv.conf tells the resolver: “If the name has fewer than 5 dots, append the search domains before trying the name as-is.”

When your application calls api.stripe.com, the resolver counts the dots. Two dots. Fewer than 5. So it tries:

1. api.stripe.com.default.svc.cluster.local  → NXDOMAIN
2. api.stripe.com.svc.cluster.local          → NXDOMAIN
3. api.stripe.com.cluster.local              → NXDOMAIN
4. api.stripe.com                            → SUCCESS

Four DNS queries for one lookup. The first three always fail. Each failure takes time. On a busy cluster with thousands of pods, this multiplies into millions of unnecessary DNS queries per hour.

The 5-second timeout. Each failed query has a timeout. With the default timeout of 5 seconds and multiple search domains, a single external DNS lookup can take 15 to 20 seconds in the worst case. This is the “5-second timeout” that shows up in application latency and makes engineers think the network is slow.

The fix:

Option 1: Use fully qualified domain names (FQDNs) with a trailing dot. api.stripe.com. (note the dot at the end) tells the resolver “this is a complete name, do not append search domains.” The resolver sends one query instead of four.

Option 2: Lower ndots in your pod spec for pods that make many external DNS calls:

spec:
  dnsConfig:
    options:
    - name: ndots
      value: "2"

With ndots:2, names with 2 or more dots (like api.stripe.com) are tried as-is first. Internal service names (like my-service) still get the search domain treatment because they have 0 dots.

Option 3: Use a node-level DNS cache (NodeLocal DNSCache). This caches responses locally on each node, eliminating the network hop to CoreDNS for repeated queries. It also handles negative caching, so the failed search domain lookups resolve instantly from cache.

# Check if NodeLocal DNSCache is running
kubectl get pods -n kube-system -l k8s-app=node-local-dns

Cause 3: Service Has No Endpoints

DNS resolves the Service name correctly. But the Service has no healthy backends. The connection still fails.

This looks like a DNS problem because curl my-service:8080 times out. But DNS is working fine. The Service just has nothing to route to.

# Check if the Service has endpoints
kubectl get endpoints my-service

# Expected: at least one IP:port listed
# If empty: no pods match the Service selector

If the endpoints list is empty:

The Service selector does not match any pod labels. This is the most common cause. A typo in the selector or the pod labels.

# Compare Service selector with pod labels
kubectl get svc my-service -o jsonpath='{.spec.selector}'
kubectl get pods -l app=my-service

The pods exist but are not Ready. If the readiness probe is failing, Kubernetes removes the pod from the endpoints list. The pod is running but not receiving traffic.

# Check pod readiness
kubectl get pods -l app=my-service -o wide
# Look for 0/1 in the READY column

Cause 4: DNS Policy Misconfiguration

Every pod has a dnsPolicy setting. The default is ClusterFirst, which means “use CoreDNS for everything.” But if someone sets it to the wrong value, DNS breaks.

The four DNS policies:

ClusterFirst (default): Uses CoreDNS. Internal names resolve to cluster services. External names get forwarded to upstream DNS. This is what you want 99% of the time.

Default: Uses the node’s DNS configuration, not CoreDNS. Internal service names do not resolve. This is almost never what you want in a cluster.

None: No DNS configuration at all. You must provide everything in dnsConfig. Used for very specific edge cases.

ClusterFirstWithHostNet: For pods running with hostNetwork: true. Uses CoreDNS but falls back to the node’s DNS if CoreDNS does not respond.

The most common mistake: setting dnsPolicy: Default, thinking it means “use the default Kubernetes DNS.” It does not. It means “use the node’s DNS, skip CoreDNS entirely.” Internal service names stop resolving.

# Check a pod's DNS policy
kubectl get pod my-pod -o jsonpath='{.spec.dnsPolicy}'

If a pod can resolve external names (google.com) but not internal names (my-service.default.svc.cluster.local), check the DNS policy first. It is probably set to Default instead of ClusterFirst.

Cause 5: Network Policy Blocking DNS

If you have NetworkPolicies in your cluster, they might block DNS traffic. CoreDNS runs on port 53 (UDP and TCP). If your network policy does not explicitly allow egress to port 53, DNS queries are silently dropped.

# Check for network policies in the pod's namespace
kubectl get networkpolicies -n <namespace>

If network policies exist, verify they allow DNS egress:

# NetworkPolicy that allows DNS
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns
spec:
  podSelector: {}
  policyTypes:
  - Egress
  egress:
  - to:
    - namespaceSelector: {}
    ports:
    - protocol: UDP
      port: 53
    - protocol: TCP
      port: 53

The tricky part: DNS failures from network policies are silent. The query is dropped. The application waits for a timeout. There is no error message saying “blocked by policy.” It just looks like DNS is slow or unresponsive.

How to test: Exec into the pod and run a manual DNS query:

kubectl exec -it my-pod -- nslookup kubernetes.default

If this times out but CoreDNS pods are healthy, a network policy is likely blocking the traffic.

The 3-Minute Debug Script

Run this script when DNS is broken. It checks all 5 causes in order.

#!/bin/bash
echo "=============================="
echo "Kubernetes DNS Debug"
echo "=============================="

echo ""
echo "=== 1. CoreDNS Pod Status ==="
kubectl get pods -n kube-system -l k8s-app=kube-dns -o wide

echo ""
echo "=== 2. CoreDNS Service ==="
kubectl get svc -n kube-system kube-dns

echo ""
echo "=== 3. CoreDNS Endpoints ==="
kubectl get endpoints -n kube-system kube-dns

echo ""
echo "=== 4. CoreDNS ConfigMap ==="
kubectl get configmap coredns -n kube-system -o jsonpath='{.data.Corefile}' 2>/dev/null
echo ""

echo ""
echo "=== 5. DNS Resolution Test ==="
kubectl run dns-test --image=busybox:1.36 --rm -it --restart=Never -- \
  sh -c "nslookup kubernetes.default && echo 'Internal DNS: OK' || echo 'Internal DNS: FAILED'"

echo ""
echo "=== 6. External DNS Test ==="
kubectl run dns-test-ext --image=busybox:1.36 --rm -it --restart=Never -- \
  sh -c "nslookup google.com && echo 'External DNS: OK' || echo 'External DNS: FAILED'"

echo ""
echo "=== 7. Network Policies ==="
kubectl get networkpolicies --all-namespaces --no-headers 2>/dev/null | wc -l
echo "network policies found in the cluster"

echo ""
echo "=============================="
echo "Debug complete."
echo "=============================="

Reading the results:

Internal DNS fails + External DNS fails = CoreDNS is down or unreachable. Check cause 1 and cause 5.

Internal DNS works + External DNS fails = CoreDNS upstream forwarding is broken. Check the Corefile forward directive.

Both work but application is slow = The ndots problem. Check cause 2.

DNS works from debug pod but not from application pod = DNS policy or network policy issue specific to that pod. Check causes 4 and 5.

The resolv.conf Cheat Sheet

Every DNS issue starts with what is in /etc/resolv.conf inside the pod:

kubectl exec my-pod -- cat /etc/resolv.conf

What to look for:

The nameserver should be the CoreDNS ClusterIP (usually 10.96.0.10). If it is a different IP, check the pod’s DNS policy.

The search domains should include <namespace>.svc.cluster.local. If they are missing, the pod cannot resolve short service names.

The ndots value controls how many dots trigger the search domain behavior. Default is 5. Lower it if external DNS is slow.

The Bottom Line

Five causes. Five debug steps. The script checks all of them in 3 minutes.

When DNS breaks: check CoreDNS pods first. If they are healthy, check endpoints. If endpoints exist, check ndots. If ndots is fine, check DNS policy. If the policy is correct, check network policies.

Do not start with tcpdump. Do not start with Wireshark. Start with the 5 causes in order. The answer is almost always in the first three.

Next week: LLMOps on Kubernetes: Patterns for Running LLMs in Production.

If you are running production Kubernetes clusters, I cover control plane internals, GPU infrastructure, and debugging every week. Subscribe at kubenatives.com.

Today I’m launching DevOpsBeast, a course platform for senior DevOps engineers preparing for interviews at FAANG and other top tier tech companies. It teaches the design first reasoning that real interviews test: Kubernetes architecture, GPU infrastructure, LLM operations, security, performance tuning, and identity systems.

This has been quietly building in the background for the past few months, and I want to walk you through what’s inside, why I built it, and what’s coming next.

Why I Built This

Most DevOps interview prep is broken.

You either get tutorial videos that teach you tools without context, or you get LeetCode-style problem sets that have nothing to do with what interviewers actually ask. Neither prepares you for the moment when a senior engineer says, “Your cluster is slow. What do you do?”

That question doesn’t have the right answer. It has the right approach. And the difference between candidates who get hired at FAANG-level companies and candidates who don’t is whether they can demonstrate that approach in 45 minutes of high-pressure design conversation.

So I built DevOpsBeast.

It’s a course platform focused on one specific thing: teaching the design-first reasoning that senior DevOps interviews actually test.

What’s Inside

Every course follows the same format. Realistic scenarios, architecture design, technical reasoning, trade-off analysis, and the actual interview questions that come from these scenarios at companies like Atlassian, Netflix, Stripe, and the FAANG names you’d expect.

Here’s what’s live or in active development:

Production GPU Infrastructure on Kubernetes. Running GPU workloads at scale. Driver management, MIG, time slicing, multi tenancy, cost optimization, and the architecture decisions that separate a working setup from a production one. This is the course that’s most relevant if you’re moving into AI/ML platform work.

LLM Operations for MLOps Engineers. 30 essential LLM concepts taught through the lens of operating them at scale. Inference serving, RAG architectures, agent infrastructure, hallucination detection, cost engineering. Not how to train models. How to deploy, monitor, and defend them in production.

Kubernetes Security. 40 lessons covering the full attack surface and defense layers. The Kubernetes API, RBAC, STRIDE threat modeling, Pod Security Admission, network policies, runtime detection, supply chain security, incident response. Every lesson starts with a breach scenario and walks through both the attack and the defense.

Kubernetes Performance Optimization. The course I wish I’d had when I first hit “the cluster is slow” and didn’t know where to look. Control plane tuning, etcd performance, scheduler throughput, resource right sizing, CPU throttling, network and storage performance, autoscaling deep dives, and dedicated optimization for EKS, GKE, and AKS.

Free Courses (Linux Fundamentals plus Networking Fundamentals). These are the prerequisites for everything else. Free, no email required, available right now.

Why This Matters Now

DevOps interviews have shifted in the last two years.

Companies aren’t asking “do you know how to write a Dockerfile” anymore. They’re asking “design a multi tenant Kubernetes platform that supports 200 teams with proper isolation, cost attribution, and security boundaries.” They’re asking “your inference latency is 3 seconds and the team needs it under 500ms, diagnose and fix.” They’re asking “we just had a security breach in our CI/CD pipeline, walk me through your incident response.”

These are senior staff and principal level questions. And the engineers who answer them well aren’t the ones who memorized more tools. They’re the ones who can reason through novel scenarios using frameworks they’ve internalized.

That’s what I’m trying to teach.

The Companion Resources

I’ve also been writing a blog at devopsbeast.com/blog covering debugging scenarios that don’t fit neatly into a course. The latest one is about Kubernetes certificate expiry, the silent killer that takes down production at 2 AM with no warning. If you’ve never been hit by it, bookmark that post before you do.

And I’m posting interview-related content on LinkedIn most days. Real questions, model answers, common misconceptions. If that’s useful, follow me there.

What’s Next

Over the next few months, I’ll be filling in lessons, adding new courses (security engineering deep dives, observability for distributed systems, platform engineering interview prep), and publishing more of these debugging blog posts.

If you’re preparing for a senior DevOps role or just want to deepen your design-first thinking, head over to devopsbeast.com and explore.

Thanks for reading. As always, hit reply if you have questions, feedback, or specific topics you want me to cover.

Sharon

P.S. The free courses (Linux Fundamentals, Networking Fundamentals) are genuinely free and don’t require an email signup. They’re meant to be useful on their own, even if you never look at the paid stuff. Start there if you want to see how I teach.

Read devopsbeast blog here

0/12 nodes are available: 12 Insufficient nvidia.com/gpu.

You have 12 GPU nodes. nvidia-smi works on all of them. The GPUs are physically there.

So why does Kubernetes think there are no GPUs available?

There are exactly 7 reasons this happens. This article covers all of them in order of likelihood. Work through them top to bottom. You will find the root cause in under 5 minutes

Reason 1: All GPUs Are Already Allocated

This is the most common cause. And the most misunderstood.

Kubernetes treats GPUs as integers. When a pod requests nvidia.com/gpu: 1, it gets an entire physical GPU. There is no fractional allocation. A pod using 8GB of an 80GB A100 still consumes 1 full GPU from the allocatable pool.

Check the actual allocation:

# Show allocatable vs allocated GPUs on each node
kubectl describe nodes | grep -A5 "Allocated resources" | grep -B1 "nvidia.com/gpu"

If allocated equals allocatable on every node, you do not have a scheduling bug. You have a capacity problem.

The fix:

Add more GPU nodes. Or enable GPU sharing (MIG, Time-Slicing, or MPS) to run multiple workloads per physical GPU. We covered all three sharing strategies in detail in the MIG vs Time-Slicing vs MPS article.

Quick capacity check:

# Total GPUs in the cluster
kubectl get nodes -l nvidia.com/gpu.present=true \
  -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.allocatable.nvidia\.com/gpu}{"\n"}{end}'

Reason 2: The GPU Operator Is Not Healthy

This is the second most common cause. The NVIDIA GPU Operator has 8 components that form a dependency chain. If any component fails, GPUs do not show up as allocatable resources.

# Check GPU Operator pod status
kubectl get pods -n gpu-operator

Every pod should be Running or Completed. If any pod is in CrashLoopBackOff, Init, or Error, that is your problem.

The dependency chain: NFD → Driver → Container Toolkit → Device Plugin → GFD → DCGM → MIG Manager → Validator.

The first unhealthy pod in this chain is your root cause. Everything below it is a symptom.

Common failures:

Driver pod crashing. The nouveau kernel module conflicts with the NVIDIA driver. Or the driver container cannot compile kernel modules for your host kernel version. On managed Kubernetes (EKS, GKE, AKS), the platform pre-installs drivers. Set driver.enabled=false in the GPU Operator ClusterPolicy.

Device plugin not running. It depends on the container toolkit. If the toolkit did not configure the runtime correctly, the device plugin cannot register GPUs. Fix the toolkit first.

Validator stuck in Init:0/4. Do not debug the validator. It is reporting that something upstream failed. Look up the chain.

We covered all 8 components in detail in the NVIDIA GPU Operator article.

Reason 3: Node Labels Are Missing

The GPU Operator uses node labels to decide where to deploy its components. If the labels are missing, the operator has no targets.

# Check for NVIDIA PCI device labels (set by NFD)
kubectl get nodes -l feature.node.kubernetes.io/pci-10de.present=true

If this returns nothing, Node Feature Discovery is not running. Without it, the GPU Operator does not know which nodes have GPUs.

# Check NFD pods
kubectl get pods -n gpu-operator -l app.kubernetes.io/component=worker

Also check for GPU Feature Discovery labels:

# Check GPU-specific labels
kubectl get node <gpu-node> -o json | \
  jq '.metadata.labels | with_entries(select(.key | startswith("nvidia.com")))'

If you see no nvidia.com/gpu.product or nvidia.com/gpu.count labels, GFD is not running or not healthy.

Reason 4: Taints and Tolerations Mismatch

GPU nodes often have taints to prevent non-GPU workloads from being scheduled on them. If your GPU pod does not have the matching toleration, the scheduler rejects it.

# Check taints on GPU nodes
kubectl get nodes -l nvidia.com/gpu.present=true \
  -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.taints}{"\n"}{end}'

Common GPU node taints:

nvidia.com/gpu=present:NoSchedule

Your pod spec needs the matching toleration:

tolerations:
- key: nvidia.com/gpu
  operator: Exists
  effect: NoSchedule

The describe pod output will tell you if taints are the issue:

kubectl describe pod <pending-pod>
# Look for: "0/12 nodes are available: 12 node(s) had untolerated taint"

If you see “untolerated taint” in the Events section, add the toleration to your pod spec.

Reason 5: Node Affinity Mismatch

If your pod requests a specific GPU type using node affinity, and no nodes match, the pod stays Pending.

# Example: pod requires H100 but cluster only has A100s
affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
      - matchExpressions:
        - key: nvidia.com/gpu.product
          operator: In
          values:
          - NVIDIA-H100-80GB-HBM3

Check what GPU types actually exist in your cluster:

# List all GPU types across nodes
kubectl get nodes -l nvidia.com/gpu.present=true \
  -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.metadata.labels.nvidia\.com/gpu\.product}{"\n"}{end}'

The describe pod output will show:

0/12 nodes are available: 12 node(s) didn't match Pod's node affinity/selector

The fix: Either change the node affinity to match your actual GPU types, or add nodes with the requested GPU type.

Reason 6: Resource Requests Exceed Node Capacity

Your pod requests more GPUs than any single node has. Or the pod requests a combination of GPU, CPU, and memory that no node can satisfy.

# Check your pod's resource requests
kubectl get pod <pending-pod> -o jsonpath='{.spec.containers[*].resources}'

# Check what's available on each GPU node
kubectl describe nodes | grep -A15 "Allocated resources"

Common scenarios:

A pod requests nvidia.com/gpu: 4 but your nodes only have 2 GPUs each. No single node can satisfy the request.

A pod requests nvidia.com/gpu: 1 and memory: 256Gi but GPU nodes only have 128Gi of RAM. The GPU is available but the memory is not.

A pod with tensor parallelism requests 8 GPUs. You have 8 GPUs across 4 nodes (2 each). Tensor parallelism requires all GPUs on the same node. No node has 8.

The fix: Reduce the resource requests, add larger nodes, or use a different parallelism strategy.

Reason 7: MIG Configuration Mismatch

If MIG is enabled on your GPUs, the resource names change. Instead of nvidia.com/gpu, MIG instances are advertised as specific profile resources:

nvidia.com/mig-1g.10gb
nvidia.com/mig-2g.20gb
nvidia.com/mig-3g.40gb
nvidia.com/mig-7g.80gb

A pod requesting nvidia.com/gpu: 1 will not match a MIG-enabled node. The node no longer advertises nvidia.com/gpu. It advertises the MIG profile resources instead.

# Check what GPU resources the node advertises
kubectl describe node <gpu-node> | grep nvidia

If you see nvidia.com/mig-* resources instead of nvidia.com/gpu, your pod needs to request the specific MIG profile:

resources:
  limits:
    nvidia.com/mig-3g.40gb: 1

The describe pod output is not always clear about this. It will say “Insufficient nvidia.com/gpu” even though the real issue is that the resource name has changed because MIG is enabled.

The 5-Minute Debug Script

Save this script. Run it first every time a GPU pod is Pending.

#!/bin/bash
echo "=============================="
echo "GPU Pod Pending Debug Script"
echo "=============================="

echo ""
echo "=== 1. Pending GPU Pods ==="
kubectl get pods --all-namespaces --field-selector=status.phase=Pending \
  -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,NODE:.spec.nodeName | head -20

echo ""
echo "=== 2. GPU Operator Health ==="
kubectl get pods -n gpu-operator --no-headers | awk '{print $1, $3}' | grep -v Running | grep -v Completed
NOT_RUNNING=$(kubectl get pods -n gpu-operator --no-headers | awk '{print $3}' | grep -v Running | grep -v Completed | wc -l)
if [ "$NOT_RUNNING" -eq 0 ]; then
  echo "All GPU Operator pods healthy."
else
  echo "WARNING: $NOT_RUNNING GPU Operator pods are NOT healthy."
fi

echo ""
echo "=== 3. GPU Node Labels ==="
kubectl get nodes -l feature.node.kubernetes.io/pci-10de.present=true \
  -o custom-columns=NAME:.metadata.name,GPU:.metadata.labels.nvidia\\.com/gpu\\.product,COUNT:.metadata.labels.nvidia\\.com/gpu\\.count 2>/dev/null
NODE_COUNT=$(kubectl get nodes -l feature.node.kubernetes.io/pci-10de.present=true --no-headers 2>/dev/null | wc -l)
if [ "$NODE_COUNT" -eq 0 ]; then
  echo "WARNING: No nodes with GPU labels found. NFD may not be running."
fi

echo ""
echo "=== 4. GPU Allocation ==="
for node in $(kubectl get nodes -l feature.node.kubernetes.io/pci-10de.present=true -o name 2>/dev/null); do
  echo "--- $node ---"
  kubectl describe $node | grep -A3 "nvidia.com"
done

echo ""
echo "=== 5. GPU Node Taints ==="
kubectl get nodes -l feature.node.kubernetes.io/pci-10de.present=true \
  -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints 2>/dev/null

echo ""
echo "=== 6. Pending Pod Events ==="
PENDING_POD=$(kubectl get pods --all-namespaces --field-selector=status.phase=Pending -o jsonpath='{.items[0].metadata.name}' 2>/dev/null)
PENDING_NS=$(kubectl get pods --all-namespaces --field-selector=status.phase=Pending -o jsonpath='{.items[0].metadata.namespace}' 2>/dev/null)
if [ -n "$PENDING_POD" ]; then
  echo "Events for $PENDING_NS/$PENDING_POD:"
  kubectl describe pod $PENDING_POD -n $PENDING_NS | tail -20
else
  echo "No pending pods found."
fi

echo ""
echo "=============================="
echo "Debug complete."
echo "=============================="

This script checks all 7 reasons in order. In 30 seconds you know whether the problem is capacity, the GPU Operator, labels, taints, affinity, resources, or MIG configuration.

The Decision Tree

When a GPU pod is Pending, follow this exact order:

Step 1. Read the Events section in kubectl describe pod. It tells you the scheduler’s exact reason for rejection.

Step 2. If “Insufficient nvidia.com/gpu”: check allocation. Are all GPUs already in use?

Step 3. If GPUs show 0 allocatable: check the GPU Operator. kubectl get pods -n gpu-operator.

Step 4. If GPU Operator pods are healthy but GPUs are not allocatable: check node labels. Is NFD running?

Step 5. If “untolerated taint”: add the toleration to your pod spec.

Step 6. If “node affinity/selector”: check what GPU types actually exist vs what the pod requests.

Step 7. If MIG is enabled: check that the pod requests the MIG profile resource, not nvidia.com/gpu.

Start at Step 1. The Events section narrows the search immediately. Do not skip it.

The Bottom Line

GPU pods get stuck in Pending for 7 reasons. 6 of them are configuration issues, not hardware problems.

Read the Events section first. Run the debug script second. The root cause is almost always visible within 30 seconds.

The hardest part is not finding the problem. It is resisting the urge to blame the scheduler when the answer is sitting in kubectl describe pod.

Next week: Kubernetes DNS Troubleshooting: CoreDNS, ndots, and the 5-Second Timeout.

If you are building GPU infrastructure on Kubernetes, I cover this intersection every week. Subscribe at kubenatives.com.

It is not about redundancy. You could have redundancy with 2 nodes. It is about quorum. And quorum is the reason your cluster stays consistent when things fail.

This article explains why 3, what happens with 2, 4, and 5, and the exact failure scenarios you need to plan for.

The Quorum Formula

etcd uses the Raft consensus algorithm. Every write must be acknowledged by a majority of members before it is committed.

The formula:

Quorum = (N / 2) + 1

3 nodes → quorum = 2 → tolerates 1 failure
5 nodes → quorum = 3 → tolerates 2 failures
7 nodes → quorum = 4 → tolerates 3 failures

The general rule: a cluster of N nodes can tolerate (N - 1) / 2 failures.

This is why 3 is the minimum for HA. With 3 nodes and a quorum of 2, you can lose 1 node and the cluster keeps accepting writes. With 2 nodes, the quorum is also 2. Lose 1 and you lose quorum. The cluster goes read only.

2 nodes is worse than 1 node for write availability. With 1 node, there is no consensus requirement. Writes always succeed (until that node dies). With 2 nodes, both must be healthy for writes to succeed. You added hardware but reduced availability.

What Happens When You Lose Quorum

When etcd loses quorum, it enters a read only state. The API server can still read existing state (pods, services, configurations). But it cannot write.

This means:

No new pods can be created. No existing pods can be modified. No Deployments can be scaled. No ConfigMaps can be updated. No new nodes can join. No Secrets can be created or rotated.

Existing workloads continue running. The kubelet on each node keeps running its containers. Health checks continue. But nothing can change.

If a running pod crashes during a quorum loss, it will not be restarted by a controller because the controller cannot write the new pod spec to etcd. The kubelet will try to restart the container locally based on the restartPolicy, but the Deployment controller cannot create a replacement.

This is why quorum loss is a critical incident. The cluster looks alive but is frozen.

Why 3, Not 4

4 nodes seems like an improvement over 3. More hardware, more redundancy. But the math tells a different story.

4 nodes tolerates the same number of failures as 3. You added a node but gained zero additional fault tolerance. You did add more write latency though, because every write now needs 3 acknowledgments instead of 2.

This is why production clusters use odd numbers: 3, 5, or 7. Even numbers add cost and latency without improving fault tolerance.

When 5 makes sense. If losing a single node keeps you up at night because maintenance windows overlap with failures, go to 5. A 5 node cluster tolerates 2 simultaneous failures. This means you can take 1 node down for maintenance and still survive an unexpected failure.

For most production clusters under 500 nodes, 3 is the right answer. The cost and operational complexity of 5 etcd nodes is only justified when the blast radius of quorum loss is extremely high.

Split-Brain: The Scenario Everyone Fears

Split-brain happens when a network partition divides the cluster into two groups, and both groups think they are the active cluster.

In traditional systems without consensus, this is catastrophic. Both sides accept writes. When the partition heals, you have conflicting state and no way to automatically reconcile.

Raft prevents this by design. Here is what actually happens with 3 nodes:

Scenario: Network partition isolates Node A from Nodes B and C.

Node A is alone. It has 1 out of 3 members. It cannot form a quorum (needs 2). It stops accepting writes. It becomes read-only.

Nodes B and C have 2 out of 3 members. They form a quorum. They elect a new leader (if A was the leader). Writing continues normally.

When the partition heals, Node A rejoins and catches up on all the writes it missed. No conflicting state. No data loss.

The key insight: Raft makes split-brain impossible as long as you have an odd number of nodes. The minority side always fails to reach a quorum. The majority side always succeeds. There is never ambiguity about which side is authoritative.

With an even number (4 nodes), a network partition could create a 2-2 split. Neither side has a quorum. Both sides go read-only. This is another reason odd numbers are better.

The 3 Node Architecture

The load balancer distributes API requests across all 3 API servers. The API servers are stateless. Anyone can handle any request.

etcd runs on all 3 nodes (stacked topology). One etcd member is the leader. Writes go to the leader and are replicated to followers.

The scheduler and controller manager use leader election. Only one instance is active at a time. If the active instance dies, another takes over within seconds.

The load balancer is critical. Without it, kubectl and the kubelets point at a single API server IP. If that node goes down, nothing can reach the control plane even though 2 healthy nodes are still running.

Use a Layer 4 (TCP) load balancer. Do not use Layer 7 (HTTP). The API server handles its own TLS. Health check endpoint: /healthz on port 6443.

Stacked vs External in HA Context

In the architecture above, etcd runs on the same nodes as the API server. This is stacked topology.

External topology separates etcd onto its own dedicated nodes:

External topology means 6 nodes instead of 3. More cost. More complexity. But etcd gets a dedicated disk and CPU. No resource contention with the API server.

Which one for HA? Both provide quorum protection. The difference is performance under load, not availability.

Stacked is fine for clusters with fewer than 200 nodes. Simple to set up. Kubeadm supports it natively.

External becomes necessary when etcd disk latency degrades because the API server is consuming the same I/O bandwidth. We covered this in detail in the Stacked vs External etcd article.

Failure Scenarios: What Actually Happens

Scenario 1: One node goes down (expected)

Quorum is maintained (2 of 3). A new etcd leader is elected if the failed node was the leader. The scheduler and controller manager fail over if they were active on the failed node. API requests continue through the load balancer to the remaining 2 nodes.

Impact: Brief spike in API latency during leader election (typically under 5 seconds). No service disruption.

Scenario 2: Two nodes go down simultaneously

Quorum is lost (1 of 3). etcd becomes read only. The API server can read state but cannot write. No new pods, deployments, or changes.

Existing workloads keep running. The kubelet on worker nodes continues managing its containers. But nothing can be updated or replaced.

Recovery: Bring at least 1 node back online to restore quorum. etcd will automatically re-form consensus.

Scenario 3: etcd data corruption on one node

The corrupted member falls behind. etcd detects the inconsistency through Raft log verification. The member stops participating in consensus.

Recovery: Remove the corrupted member from the cluster. Provision a new node. Add it as a new etcd member. It will automatically replicate data from the healthy members.

# Remove the bad member
etcdctl member remove MEMBER_ID

# Add a new member
etcdctl member add new-node --peer-urls=https://NEW_IP:2380

# Start etcd on the new node with the --initial-cluster-state=existing flag

Scenario 4: Disk fills up on one node

etcd performance degrades as the disk fills. Write latency increases. If the etcd database hits its storage quota, that member triggers a NOSPACE alarm.

If only one member hits NOSPACE, the cluster continues (2 of 3 are healthy). But you should act immediately because the remaining members are likely on the same trajectory.

Recovery: Follow the NOSPACE runbook (compact, defrag, disarm alarm). Then investigate why disk usage grew and fix the root cause.

Scenario 5: Control plane node scheduled for maintenance

Drain the node’s workloads (if it also runs worker pods). The other 2 nodes maintain quorum. Perform maintenance. Bring the node back.

Important: Never take 2 nodes down for maintenance simultaneously. With 3 nodes, losing 2 means quorum loss. Always verify the first node is healthy and has rejoined the etcd cluster before starting maintenance on the second.

# Verify cluster health before maintenance
etcdctl endpoint health --cluster
etcdctl endpoint status --write-out=table

The Health Checks You Need

Daily automated check

#!/bin/bash
# etcd-health-check.sh

CERTS="--cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key"

echo "=== Cluster Health ==="
ETCDCTL_API=3 etcdctl endpoint health --cluster $CERTS

echo ""
echo "=== Member Status ==="
ETCDCTL_API=3 etcdctl endpoint status --cluster --write-out=table $CERTS

echo ""
echo "=== Active Alarms ==="
ETCDCTL_API=3 etcdctl alarm list $CERTS

Prometheus alerts for quorum

groups:
- name: etcd-quorum
  rules:
  - alert: EtcdMemberDown
    expr: up{job="etcd"} == 0
    for: 2m
    labels:
      severity: critical
    annotations:
      summary: "etcd member {{ $labels.instance }} is down"
      description: "With 3 members, losing 1 means you are one failure from quorum loss."

  - alert: EtcdInsufficientMembers
    expr: count(up{job="etcd"} == 1) < 2
    for: 1m
    labels:
      severity: page
    annotations:
      summary: "etcd cluster has lost quorum"
      description: "Fewer than 2 etcd members are healthy. Cluster is read-only."

The Bottom Line

The number 3 is not arbitrary. It is the minimum required for distributed consensus to work with fault tolerance.

3 nodes, quorum of 2, tolerates 1 failure. Add a load balancer in front. Use odd numbers. Never take 2 nodes down at the same time.

If you understand quorum, you understand why your cluster survives node failures. If you do not, you will learn the hard way at 3 AM.

Next week: Why Your GPU Pods Are Pending: Debugging Kubernetes GPU Scheduling.

If you are running production Kubernetes clusters, I cover control plane internals, GPU infrastructure, and model serving every week. Subscribe at kubenatives.com.

A multi-tenant LLM inference platform serving roughly a dozen production workloads on H100 nodes. Standard setup: vLLM behind KServe, KServe behind Istio, autoscaling driven by request rate and queue depth.

The platform had been stable for months. Then it started OOMing.

Not constantly. Not predictably. Just sometimes, in the early morning hours UTC, one of the vLLM pods would go OOMKilled. Exit code 137. Always between 2 AM and 4 AM. Never the same pod twice in a row. Service recovered within 60 seconds because the deployment had multiple replicas, but the on-call alert woke up an engineer every time.

For the first week we did what every team does. We assumed it was traffic. We pulled the request rate metrics. The 3 AM window was the lowest-traffic period of the day. Less than 5% of peak QPS.

That was the first signal that this was not a normal OOM.

The Easy Explanations That Were Wrong

A week of investigation eliminated the obvious causes one by one.

Hypothesis 1: A specific request was blowing up memory.

We grabbed the request logs from the 5 minutes before each OOM event. No common pattern. Different models, different prompt sizes, different output lengths. Some events had no requests in flight at all when the OOM fired. Ruled out.

Hypothesis 2: Memory leak accumulating until breakpoint.

We graphed container_memory_working_set_bytes for each pod over 24 hours. Memory was stable. Pods that had been running for 30 hours had the same memory footprint as pods that had been running for 2. There was no slow growth. The OOM happened from a stable baseline.

Hypothesis 3: Bad model image with a regression.

Rolled back to the previous vLLM version. OOMs continued at the same rate. Rolled forward. Same. The vLLM version was not the variable.

Hypothesis 4: Noisy neighbor on the host.

Checked DCGM metrics for all GPUs on the same physical node during OOM events. No correlated GPU memory pressure or compute contention. The OOMing pod was on a node where every other GPU workload was idle or low utilization.

Hypothesis 5: Resource limit set too low.

This is where most teams stop. They raise the memory limit by 25% and call it solved. We tried it. The OOMs moved later in the night, then resumed at the new threshold a few days later. Higher limit, same problem, slower recurrence.

That last one was the breakthrough, even though we did not know it yet. The fact that raising the limit only delayed the OOM rather than preventing it meant something was actively growing memory. We just could not see what.

Read more

Pods are stuck. Deployments are failing. API requests are slow. Users are complaining.

You open a terminal and start running commands. kubectl get pods. kubectl describe pod. kubectl logs. You scroll through the output looking for something that stands out.

Twenty minutes later, you’re deep in a rabbit hole, debugging a network policy that has nothing to do with the actual problem.

This is how most engineers debug Kubernetes. Randomly. They start with whatever command comes to mind first and hope to stumble on the root cause.

There is a better way. A systematic framework that works for every Kubernetes problem. It starts at the top of the stack and works down through five layers. Each layer has specific symptoms, specific commands, and a clear signal indicating whether to stay at that layer or move to the next.

The Five Layer Model

Every Kubernetes problem lives at one of five layers. The layers are ordered from most common to least common. Start at Layer 1 and work down. Most problems resolve in the first two layers.

Layer 1: Application. The container itself is broken. Bad config, missing env vars, crashed process, OOM.

Layer 2: Pod Scheduling. The pod can’t get placed on a node. Resource limits, taints, affinity rules, node capacity.

Layer 3: Networking. The pod is running, but can’t communicate. DNS failures, service misconfig, network policies, and ingress issues.

Layer 4: Cluster Infrastructure. The control plane is degraded. etcd performance, API server latency, scheduler delays, and certificate expiry.

Layer 5: Node and Hardware. The underlying node is unhealthy. Disk pressure, memory pressure, kubelet issues, and GPU driver failures.

The framework works because Kubernetes problems almost always manifest at the application layer first. A pod crashes. A deployment doesn’t roll out. A request times out. The root cause might be at any layer, but the symptoms always show up at the top.

Layer 1: Application Debugging

This is where 60% of production issues live. The container is doing something wrong. Before blaming Kubernetes, check the application.

The first three commands

Run these in order for any pod that isn’t healthy:

# 1. What is the pod doing right now?
kubectl get pod <pod-name> -o wide

# 2. What happened to it?
kubectl describe pod <pod-name>

# 3. What is the application saying?
kubectl logs <pod-name> --tail=100

The get pod output tells you the current state. Is it Running, Pending, CrashLoopBackOff, Error, or ImagePullBackOff? Each state points to a different problem.

The describe pod output tells you the history. Look at the Events section at the bottom. Read it from bottom to top. The first event is usually the trigger.

The logs output tells you what the application thinks is happening. If the container crashed, use --previous to see the last run’s logs before the crash.

kubectl logs <pod-name> --previous --tail=100

CrashLoopBackOff

This is the most common pod failure. The container starts, crashes, restarts, crashes again. Kubernetes backs off the restart interval exponentially.

The root cause is almost always in the application logs. Check:

# See the exit code
kubectl get pod <pod-name> -o jsonpath='{.status.containerStatuses[0].lastState.terminated.exitCode}'

Exit code 1 means the application crashed on its own. Check logs for the error.

Exit code 137 means Kubernetes killed the container. It ran out of memory (OOMKilled). Check:

kubectl describe pod <pod-name> | grep -i oom

If it was OOMKilled, the fix is either increasing the memory limit or fixing the memory leak in the application.

Exit code 143 means the container received SIGTERM. Kubernetes asked it to stop gracefully. This happens during rollouts, scaling, or node drains.

ImagePullBackOff

The container image can’t be downloaded. Check:

kubectl describe pod <pod-name> | grep -A5 "Events"

Common causes: wrong image name, wrong tag, private registry without image pull secrets, or the registry is down.

# Check if image pull secrets are configured
kubectl get pod <pod-name> -o jsonpath='{.spec.imagePullSecrets}'

Readiness and Liveness Probes

A pod is Running but not receiving traffic. The readiness probe is failing.

# Check probe configuration and recent failures
kubectl describe pod <pod-name> | grep -A10 "Readiness\|Liveness"

Common mistake: the readiness probe checks an endpoint that takes 30 seconds to respond, but the timeout is set to 1 second. The pod is healthy but Kubernetes thinks it isn’t.

The signal to move to Layer 2

If kubectl describe pod shows the pod is Pending (not Running, not CrashLoopBackOff), the problem isn’t the application. The pod hasn’t been scheduled yet. Move to Layer 2.

Layer 2: Pod Scheduling

The pod exists but it’s stuck in Pending. Kubernetes can’t find a node to run it on.

The diagnostic command

kubectl describe pod <pod-name> | grep -A20 "Events"

The Events section tells you exactly why the scheduler rejected the pod. The message will say something like:

0/12 nodes are available: 6 Insufficient cpu, 4 node(s) had taint, 2 node(s) didn't match pod affinity.

Read this carefully. It tells you how many nodes exist, how many were filtered, and why each one was rejected.

Insufficient resources

# Check available resources across all nodes
kubectl top nodes

# Check a specific node's allocation
kubectl describe node <node-name> | grep -A15 "Allocated resources"

Compare the pod’s resource requests against what’s available. If the pod requests 4 CPU and 16Gi memory, but no node has that much free, the pod stays Pending.

The fix is either reducing the pod’s resource requests, adding more nodes, or cleaning up unused workloads to free resources.

Taints and tolerations

Nodes can have taints that repel pods. The pod needs a matching toleration to land on a tainted node. GPU nodes almost always have taints.

# Check node taints
kubectl describe node <node-name> | grep -A3 "Taints"

# Check pod tolerations
kubectl get pod <pod-name> -o jsonpath='{.spec.tolerations}' | jq .

If the node has a taint and the pod doesn’t have a matching toleration, the scheduler will skip that node.

Node selectors and affinity

# Check what the pod requires
kubectl get pod <pod-name> -o jsonpath='{.spec.nodeSelector}' | jq .
kubectl get pod <pod-name> -o jsonpath='{.spec.affinity}' | jq .

# Check what nodes have
kubectl get nodes --show-labels | grep <expected-label>

If the pod requires gpu-type=a100 but no node has that label, the pod stays Pending forever.

PersistentVolumeClaim binding

kubectl get pvc -n <namespace>

If the PVC status is Pending, the pod can’t start because its storage isn’t ready. Check the PVC events:

kubectl describe pvc <pvc-name> -n <namespace> | grep -A10 "Events"

The signal to move to Layer 3

If the pod is Running but the service isn’t working (requests fail, connections time out, DNS doesn’t resolve), the problem is networking. Move to Layer 3.

Layer 3: Networking

The pod is running. The application is healthy. But traffic isn’t reaching it. Or it can’t reach other services.

Service connectivity

First, verify the service exists and has endpoints:

# Check the service
kubectl get svc <service-name> -n <namespace>

# Check if the service has endpoints (pods backing it)
kubectl get endpoints <service-name> -n <namespace>

If endpoints shows zero addresses, the service selector doesn’t match any running pods. Compare the service selector with the pod labels:

# Service selector
kubectl get svc <service-name> -o jsonpath='{.spec.selector}'

# Pod labels
kubectl get pods -n <namespace> --show-labels

DNS resolution

The most common networking issue in Kubernetes. The pod can’t resolve service names.

# Test DNS from inside a pod
kubectl exec -it <pod-name> -- nslookup <service-name>
kubectl exec -it <pod-name> -- nslookup <service-name>.<namespace>.svc.cluster.local

If DNS fails, check CoreDNS:

# Is CoreDNS running?
kubectl get pods -n kube-system -l k8s-app=kube-dns

# CoreDNS logs
kubectl logs -n kube-system -l k8s-app=kube-dns --tail=50

A common cause of slow DNS is the ndots setting. By default, Kubernetes adds ndots:5 to resolv.conf, which means any name with fewer than 5 dots gets appended with search domains before the actual lookup. A simple lookup for api.example.com generates 4 failed queries before the real one succeeds.

The fix:

spec:
  dnsConfig:
    options:
      - name: ndots
        value: "2"

Network policies

If you have network policies in your cluster, they might be blocking traffic between pods.

# List network policies in the namespace
kubectl get networkpolicies -n <namespace>

# Describe a specific policy
kubectl describe networkpolicy <policy-name> -n <namespace>

A missing egress rule means the pod can’t make outbound connections. A missing ingress rule means nothing can connect to the pod. An empty pod selector {} applies to all pods in the namespace.

Testing connectivity

# Test pod to pod connectivity
kubectl exec -it <pod-a> -- curl -v http://<pod-b-ip>:<port>

# Test pod to service connectivity
kubectl exec -it <pod-a> -- curl -v http://<service-name>:<port>

# Test pod to external connectivity
kubectl exec -it <pod-a> -- curl -v https://httpbin.org/get

The signal to move to Layer 4

If all pods are slow (not just one service), if kubectl itself is slow, or if you see etcdserver: request timed out in logs, the problem is the control plane. Move to Layer 4.

Layer 4: Cluster Infrastructure

The control plane is degraded. This affects everything in the cluster, not just one application.

Symptoms

kubectl commands take 5+ seconds. Deployments don’t roll out. Pod creation is delayed. Controller reconciliation falls behind. Events show etcdserver: request timed out.

API server health

# Check API server response time
time kubectl get nodes

# Check API server metrics (if accessible)
kubectl get --raw /metrics | grep apiserver_request_duration_seconds

# Check API server logs
kubectl logs -n kube-system kube-apiserver-<node> --tail=50

If the API server is slow, the cause is almost always etcd. The API server is stateless. etcd is not.

etcd health

# Quick health check
etcdctl endpoint health --cluster \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

# Detailed status
etcdctl endpoint status --write-out=table --cluster \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

Check the metrics that predict etcd failures:

etcd_disk_wal_fsync_duration_seconds p99 above 10ms means disk latency. etcd_mvcc_db_total_size_in_bytes approaching the quota means NOSPACE is coming. etcd_server_leader_changes_seen_total above 1 per hour means instability.

We covered all five etcd failure modes in detail in our etcd debugging guide.

Certificate expiry

kubeadm certs check-expiration

If certificates expire, everything breaks at once. Existing pods keep running from kubelet cache. But nothing new can be created, updated, or deleted.

Scheduler health

# Check scheduler logs
kubectl logs -n kube-system kube-scheduler-<node> --tail=30

# Check if scheduler is falling behind
kubectl get --raw /metrics | grep scheduler_scheduling_attempt_duration_seconds

The signal to move to Layer 5

If specific nodes show problems (NotReady status, high resource usage, kubelet errors) but the control plane is healthy, the issue is at the node level. Move to Layer 5.

Layer 5: Node and Hardware

Individual nodes are unhealthy. This only affects pods running on those specific nodes.

Node status

# Check all node statuses
kubectl get nodes

# Look for conditions on a specific node
kubectl describe node <node-name> | grep -A10 "Conditions"

The Conditions section shows:

MemoryPressure: the node is running out of RAM. DiskPressure: the node is running out of disk. PIDPressure: the node has too many processes. Ready: False means the kubelet can’t communicate with the API server.

Kubelet health

# Check kubelet status on the node
systemctl status kubelet

# Kubelet logs
journalctl -u kubelet --tail=50

Common kubelet issues: certificate expired, container runtime not responding, disk full on the node.

GPU specific issues

For GPU nodes, check the GPU Operator components:

# Are all GPU Operator pods running?
kubectl get pods -n gpu-operator -o wide

# Can the node see GPUs?
kubectl describe node <gpu-node> | grep nvidia.com/gpu

# Check nvidia-smi on the node
kubectl debug node/<gpu-node> -it --image=nvidia/cuda:12.0-base -- nvidia-smi

If nvidia-smi fails, the GPU driver isn’t loaded. Check the driver container in the GPU Operator.

We covered the full GPU Operator debugging path in our GPU Operator article.

Disk pressure

# Check disk usage on the node
kubectl debug node/<node> -it --image=ubuntu -- df -h

# Check container image storage
kubectl debug node/<node> -it --image=ubuntu -- du -sh /var/lib/containerd

Old container images and unused layers accumulate over time. Kubernetes garbage collection should handle this, but sometimes it falls behind.

The Quick Reference Checklist

When something breaks in production, run through this sequence:

1. kubectl get pods -n <namespace>
   → What state are the affected pods in?

2. If CrashLoopBackOff or Error:
   → kubectl logs <pod> --previous --tail=100
   → Layer 1: Application issue

3. If Pending:
   → kubectl describe pod <pod> (read Events)
   → Layer 2: Scheduling issue

4. If Running but not working:
   → kubectl exec <pod> -- curl <service>
   → kubectl exec <pod> -- nslookup <service>
   → Layer 3: Networking issue

5. If everything is slow:
   → time kubectl get nodes
   → etcdctl endpoint health --cluster
   → Layer 4: Control plane issue

6. If specific node problems:
   → kubectl describe node <node> (check Conditions)
   → systemctl status kubelet
   → Layer 5: Node/hardware issue

This sequence takes 2 minutes. It eliminates 80% of possible causes and points you at the right layer immediately. No more guessing.

The Debugging Mindset

Three rules that make debugging faster:

Rule 1: Read the Events. Every kubectl describe output has an Events section. Read it. From bottom to top. The events tell you what Kubernetes already knows about the problem. Most engineers skip this and start guessing.

Rule 2: Check one layer at a time. Don’t jump between application logs, network policies, and etcd metrics in the same debugging session. Start at Layer 1. If the evidence points to a different layer, move there deliberately. Randomized debugging wastes time.

Rule 3: Reproduce before you fix. If you can’t reproduce the problem on demand, you don’t understand it yet. A fix applied without understanding the root cause is just a workaround that will break again later.

What This Framework Connects To

This article is the anchor for production debugging at KubeNatives. Every specific debugging guide links back here:

Our etcd debugging guide covers Layer 4 in depth: the 5 ways etcd breaks and the metrics that predict each failure.

Our GPU Operator article covers Layer 5 for GPU nodes: the 8 components and the initialization dependency chain.

Our DNS troubleshooting guide (coming soon) will cover Layer 3 in depth: CoreDNS, ndots, and the 5 second timeout problem.

Each supporting article gives you the deep dive for a specific problem. This framework tells you which article to reach for.

Next week: Deploying vLLM on Kubernetes: From Single Pod to Production.

If you’re running production Kubernetes, I cover control plane operations, GPU infrastructure, and model serving every week. Subscribe at kubenatives.com.

Symptom

Your vLLM pod restarted during normal traffic. Users saw 503 errors for the duration of the restart. The pod eventually came back but might OOM again on the next large request.

Signals you are in this runbook:

$ kubectl get pod vllm-0
NAME      READY   STATUS      RESTARTS   AGE
vllm-0    1/1     Running     3          2h

$ kubectl describe pod vllm-0 | grep -A3 "Last State"
    Last State:     Terminated
      Reason:       OOMKilled
      Exit Code:    137

Exit code 137 means the container received SIGKILL from the kernel OOM killer. Not from a crash. Not from vLLM code. The kernel decided the container used too much memory and killed it.

Quick Triage: Is This GPU Memory or Host Memory?

This is the first branch. vLLM has two memory failure modes and they need different fixes.

Check pod events:

kubectl describe pod vllm-0 | grep -A2 -i "oom\|killed"

If you see “Memory cgroup out of memory” in kubelet events: This is host memory OOM. The container exceeded its resources.limits.memory. Jump to Procedure A.

If you see “CUDA out of memory” or “torch.cuda.OutOfMemoryError” in vLLM logs: This is GPU memory OOM. The model tried to allocate more VRAM than available on the device. Jump to Procedure B.

If you see both or cannot tell: Pull the last 200 lines of logs from the previous container:

kubectl logs vllm-0 --previous --tail=200 | grep -iE "oom|memory|cuda|killed"

Look for the first memory related error. That is the trigger. Everything after is cascade.

Procedure A: Host Memory OOM (exit 137, kernel killed the container)

What happened: the container exceeded resources.limits.memory. Kubernetes killed it.

Root causes, ranked by frequency:

1. Memory limit set too low for the model size (most common)

2. Prefix caching or KV cache overflow into host memory via swap or CPU offload

3. Memory leak in vLLM (rare, usually requires version upgrade)

Step 1: Confirm the limit violation

# What was the memory limit?
kubectl get pod vllm-0 -o jsonpath='{.spec.containers[0].resources.limits.memory}'
# Example output: 32Gi

# What did it actually use before death?
kubectl top pod vllm-0 --containers 2>/dev/null || echo "metrics-server needed"

If limits are 32Gi and a 70B model needs host memory to mirror the weights during load, you will hit the limit on startup.

Read more

Most weeks you get a technical deep dive from me on Fridays. Today is different.

I want to put a workshop on your radar that I think is worth your Saturday.

Internal Developer Platforms have been the dominant platform engineering conversation for two years now. Most teams I talk to are either building one badly, buying one they do not fully understand, or avoiding the topic because they have seen too many failed platform projects.

The pattern is consistent. Teams start with a portal (usually Backstage) and work backwards into the underlying platform. That order is wrong. It is why so many IDPs end up as another bottleneck instead of a force multiplier.

Ajay Chankramath runs Platformetrics and previously led Platform Engineering at Thoughtworks. He is running a two day workshop on April 25 and 26 on building an AI powered IDP from scratch. I asked him a few questions on the stuff most teams get wrong.

When is a team actually ready to build an IDP?

Ajay: When you can name your top three developer friction points based on data, not gut feeling. If you have not watched a developer go through onboarding end to end, you are not ready to build the platform. Do not start building a platform just because you learned about a solution. Start when you truly understand the problems.

How do IDP patterns need to evolve for AI and ML workloads?

Ajay: AI workloads break three assumptions baked into the standard IDP: resource primitives, lifecycle, and failure modes.

IDPs need to treat GPU pools as first class resources with their own abstractions. They need to build golden paths for ML workflows, not just microservices. They need to integrate model registries and experiment trackers into the service catalog. And they need observability for inference latency, confidence scores, and data drift.

The standard Backstage style IDP was not designed for workloads that can fail by giving confident wrong answers for weeks.

What will engineers walk away understanding?

Ajay: How the layers connect to each other.

You can learn about each tool from its documentation. This workshop teaches what happens when a developer submits a service request in the portal, which triggers a golden path scaffolder, which provisions a namespace with RBAC and quotas, which applies policies via OPA, which is monitored by an SLO driven alerting stack, which feeds into an AI powered alert correlator.

That end to end chain, from portal click to production insight, is the platform.

Workshop details

Building an AI Powered Internal Developer Platform from Scratch

Saturday April 25 and Sunday April 26, 2026 11 AM to 3 PM ET each day 4 PM to 8 PM UK / 8:30 PM to 12:30 AM IST / 7 PM to 11 PM Gulf

Hosted by Deep Engineering by Packt.

What’s included:

Live hands on sessions with Ajay across two days. Working code for AI platform features that runs locally without API keys. A 30 to 60 minute one on one Platform Journey consultation with Ajay. Certificate of Completion plus a Credly digital badge you can add to LinkedIn.

Refunds available up to 3 days before the event. Seats are limited.

Register here

Why I am sharing this

I am selective about what I put in front of this list.

Ajay’s answer to the AI workloads question landed for me because it names a real gap in how most teams are thinking about ML platforms today. GPU pools as first class resources. Model registries in the service catalog. Observability that covers data drift, not just p99 latency. Most IDPs I have seen do none of this.

If you are on a platform team, a DevOps team going through an AI transformation, or an SRE figuring out how to support ML workloads, this workshop will save you months of trial and error.

Disclosure: This is a paid partnership with Deep Engineering by Packt. I only promote things I would send to a friend.

Regular Friday content this week covers the production Kubernetes debugging framework I use on our clusters. More on that in a few days.

Sharon

For LLM inference, the same proxy introduces problems that do not exist in typical microservice architectures. Long lived streaming connections, large response bodies, and GPU sensitive latency make Istio defaults a bad fit.

Your vLLM pods are not broken. Your model is not broken. Istio is working exactly as designed. The design just does not match inference workloads.

This article covers the 5 most common Istio issues with inference pipelines and how to fix each one.

Issue 1: Sidecar Injection on GPU Pods

By default, Istio injects a sidecar proxy into every pod in labeled namespaces. GPU pods get a sidecar too. The sidecar consumes CPU and memory that could go to the inference workload.

The sidecar itself is not the problem. The problem is the sidecar default resource requests. 100m CPU and 128Mi memory, per pod. On a GPU node where every CPU core matters for tokenization and request handling, this overhead adds up across pods.

Fix options:

Option 1: Disable sidecar injection for inference pods.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm
spec:
  template:
    metadata:
      annotations:
        sidecar.istio.io/inject: "false"

If your inference pods do not need mTLS to the model clients, skip the sidecar. You keep Istio everywhere else in the cluster. The GPU pods run clean.

Option 2: Keep the sidecar but tune it.

annotations:
  sidecar.istio.io/proxyCPU: "50m"
  sidecar.istio.io/proxyMemory: "64Mi"

Lower the sidecar resource requests if you still want mTLS. Most inference sidecars do not need 100m CPU.

Issue 2: Streaming Responses Terminated Early

vLLM supports token streaming over HTTP. The client opens a connection, sends a prompt, and receives tokens as they generate. A long generation might take 30 to 60 seconds.

Istio default timeouts kill these connections before generation finishes.

The culprit is usually the Envoy idle timeout. For a VirtualService, the default is 15 seconds of no activity. Streaming LLM output sends tokens intermittently. Between tokens, the connection sits idle. 15 seconds later, Envoy closes the stream.

The fix:

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: vllm
spec:
  hosts:
  - vllm.inference.svc.cluster.local
  http:
  - route:
    - destination:
        host: vllm
    timeout: 300s

Set the timeout to cover your longest expected generation. 5 minutes is safe for most workloads. Longer if you serve 70B models or reasoning models with multi minute thinking phases.

Also check the connection level idle timeout in the DestinationRule. The default there is 1 hour, which is fine, but some teams override it and forget.

Issue 3: Connection Pool Limits Starving the Inference Service

Istio DestinationRule defaults limit the number of concurrent connections and pending requests. For microservices, this protects against cascading failures. For inference, it starves the service.

Default settings to watch:

connectionPool:
  tcp:
    maxConnections: 100
  http:
    http1MaxPendingRequests: 1024
    http2MaxRequests: 1024

Under heavy inference traffic, you hit the connection limit before you hit the GPU limit. Requests queue outside the pod. Users see 503 errors. GPU utilization looks fine. Your instinct is to scale up replicas. That does not help. The ceiling is in Istio, not in vLLM.

The fix:

apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: vllm
spec:
  host: vllm
  trafficPolicy:
    connectionPool:
      tcp:
        maxConnections: 1000
      http:
        http1MaxPendingRequests: 10000
        http2MaxRequests: 10000

Raise the limits significantly for inference services. The actual bottleneck should be GPU throughput, not proxy accounting.

Issue 4: Envoy Buffer Limits on Large Response Bodies

A single inference response can be hundreds of kilobytes. A long context completion or a structured output with a large JSON schema can push past a megabyte.

Envoy has a default buffer limit of 1 MiB per request or response. Larger bodies get truncated or rejected. The client sees a partial response or a 500 error.

The fix:

Set the buffer size on the Envoy filter.

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: increase-buffer-limit
spec:
  configPatches:
  - applyTo: NETWORK_FILTER
    match:
      listener:
        filterChain:
          filter:
            name: "envoy.filters.network.http_connection_manager"
    patch:
      operation: MERGE
      value:
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
          max_request_headers_kb: 96
          stream_idle_timeout: 300s

For large responses specifically, configure the per route buffer size or disable buffering on the inference route. Streaming already avoids buffering the full body. If you are using streaming, this issue does not apply. If you are not, switch to streaming before you fight Envoy buffers.

Issue 5: mTLS Handshake on Cold Pods

Istio enforces mTLS between pods by default. Every connection starts with a certificate exchange. Normally this adds 5 to 15ms to the first request.

For inference pods, the first request already carries significant overhead. vLLM compiles CUDA graphs on the first inference call. The cold start penalty can be 2 to 10 seconds depending on the model. Add the mTLS handshake on top and the user sees a 12 second response on the first call.

The handshake itself is cheap per request. The problem is that warmup probes, readiness checks, and synthetic traffic often do not exercise the mTLS path. Your first real user request pays for the handshake and for the cold model at the same time.

The fix:

Pre warm the pod with a real inference request during startup. A postStart hook that sends a short prompt through the sidecar forces the certificate exchange and the CUDA graph compile before the pod is marked ready.

lifecycle:
  postStart:
    exec:
      command:
      - /bin/sh
      - -c
      - |
        sleep 30 && \
        curl -X POST http://localhost:8000/v1/completions \
          -H "Content-Type: application/json" \
          -d '{"model":"meta-llama/Llama-3.1-8B-Instruct","prompt":"warmup","max_tokens":1}'

Combine this with a readiness probe that waits for the warmup to complete. New users never hit a cold pod.

When to Use Istio vs When to Skip It

The honest answer: most inference platforms do not need Istio.

vLLM talks to a model store and a load balancer. That is 2 connections. NetworkPolicies handle isolation. DNS handles service discovery. Prometheus handles observability. You get 90% of what Istio provides, at zero proxy overhead, with 10% of the operational complexity.

Use Istio when:

Compliance requires mTLS between all services (SOC 2, HIPAA, PCI). You need canary deployments with traffic splitting between model versions. You need detailed per request observability beyond Prometheus metrics. You have 50 plus services and need centralized traffic management.

Skip Istio when:

Your inference pipeline has fewer than 20 services. Your team does not have Istio operational experience. Streaming latency is critical and any buffering overhead matters. Your security boundary is the namespace, not the pod.

The simplest debug step: temporarily remove the sidecar with sidecar.istio.io/inject: "false" and test. If inference works without Istio, the problem is Istio configuration. Add the sidecar back and fix the specific issue.

The Bottom Line

Istio is not broken. It is doing exactly what it was designed to do. The design assumes short lived HTTP requests between stateless microservices. Inference workloads violate every assumption in that design.

The 5 issues in this article cover 90% of Istio inference problems in production. Sidecar overhead. Streaming timeouts. Connection pool limits. Buffer sizes. Cold start handshakes.

Fix them once and document the pattern. Every new inference service in your cluster inherits the right configuration. Nobody spends a Saturday chasing 30 second latency that turned out to be a default timeout.

The service mesh is a tool. Not a requirement.

Next week: A/B Testing LLM Models in Production with Kubernetes.

If you are running production Kubernetes clusters, I cover control plane internals, GPU infrastructure, and model serving every week. Subscribe at kubenatives.com.

Kubernetes gave it an entire A100 with 80GB. The device plugin reported the GPU as fully allocated. Your next pod is stuck in Pending because the scheduler sees zero GPUs available.

This is the fundamental problem with GPU scheduling in Kubernetes. The default device plugin treats GPUs as indivisible integers. One GPU, one pod. No sharing. No fractional allocation. No memory awareness.

We covered why this happens in our GPU scheduling deep dive. This article goes deeper on the three strategies that fix it.

Multi-Instance GPU (MIG). Time-Slicing. Multi-Process Service (MPS).

Each one works at a different level of the stack. Each one provides different isolation guarantees. Each one is the right choice for different workloads.

What the Default Device Plugin Actually Does

The NVIDIA device plugin runs as a DaemonSet on every GPU node. It discovers the physical GPUs, registers them with the kubelet as extended resources (nvidia.com/gpu), and assigns them to pods.

The key limitation is that extended resources in Kubernetes only support integers. You can request nvidia.com/gpu: 1 or nvidia.com/gpu: 2. You cannot request nvidia.com/gpu: 0.5. Fractional GPUs do not exist at the scheduler level.

When a pod requests 1 GPU, the device plugin assigns the entire physical GPU. All memory. All compute cores. All memory bandwidth. Nobody else can use that GPU until the pod releases it.

For a 70B model using 75GB of an 80GB A100, this makes sense. For a 7B model using 8GB, you just wasted $25K worth of GPU capacity.

The three sharing strategies all make a single physical GPU appear as multiple resources to the device plugin. But they do it at completely different layers.

MIG: Hardware Level Partitioning

Multi-Instance GPU is built into the GPU silicon itself. It is available on NVIDIA Ampere (A100, A30) and Hopper (H100, H200) architectures.

MIG physically partitions a GPU into up to seven independent instances. Each instance gets its own dedicated Streaming Multiprocessors, memory controllers, L2 cache, and VRAM allocation.

How it works in Kubernetes

When MIG is enabled, the GPU Operator’s MIG Manager creates instances based on a profile you configure. Each instance appears as a separate resource to the device plugin.

Instead of advertising nvidia.com/gpu: 1, the node advertises resources like:

nvidia.com/mig-1g.5gb: 7    # Seven 1g.5gb instances
nvidia.com/mig-2g.10gb: 3   # Three 2g.10gb instances
nvidia.com/mig-3g.20gb: 2   # Two 3g.20gb instances

Pods request a specific MIG profile:

resources:
  limits:
    nvidia.com/mig-1g.5gb: 1

The scheduler treats each MIG instance as a separate resource. A pod on a 1g.5gb instance can only access the memory and compute allocated to that instance. It cannot see or affect other instances on the same physical GPU.

What MIG gives you

True hardware isolation. Each MIG instance has its own memory controller and L2 cache. A pod on instance A cannot access the memory of instance B. If a process on instance A crashes, instance B is completely unaffected. This is the same isolation you get from physically separate GPUs.

Predictable performance. Each instance has dedicated compute and memory bandwidth. The performance of one instance does not degrade when other instances are under load. You can make SLA guarantees per instance.

Error isolation. A GPU fault in one instance does not affect other instances. For production serving where uptime matters, this is significant.

What MIG costs you

Limited GPU support. MIG only works on A100, A30, H100, H200, and H800 GPUs. If you run T4s, V100s, or A10Gs, MIG is not an option.

Fixed partition sizes. You cannot create arbitrary MIG profiles. Each GPU model supports a specific set of predefined profiles. On an A100 40GB, you choose from 1g.5gb, 2g.10gb, 3g.20gb, 4g.20gb, and 7g.40gb. You pick from a menu. You do not define custom sizes.

Reconfiguration requires draining. Changing the MIG profile requires stopping all workloads on that GPU first. You cannot dynamically repartition under load. Plan your profiles ahead of time and match them to your workload sizes.

Maximum 7 instances. Even on the largest GPUs, you can only create up to 7 MIG instances. If you need to share a GPU among 10 or 20 lightweight workloads, MIG alone is not enough.

When to use MIG

Production inference serving where you need SLA guarantees per model. Multi-tenant environments where different teams share GPU node pools. Any scenario where memory isolation is a hard requirement.

Time-Slicing: Software Level Multiplexing

Time-Slicing is the simplest GPU sharing strategy. It makes a single GPU appear as multiple “replicas” to the device plugin. The GPU’s compute time is shared among all pods through CUDA’s context switching mechanism.

How it works in Kubernetes

You configure a ConfigMap that tells the device plugin how many replicas to create per GPU:

apiVersion: v1
kind: ConfigMap
metadata:
  name: time-slicing-config
  namespace: gpu-operator
data:
  any: |-
    version: v1
    sharing:
      timeSlicing:
        resources:
          - name: nvidia.com/gpu
            replicas: 4

After applying this and labeling your nodes, a node with 1 physical GPU advertises nvidia.com/gpu: 4. The scheduler sees 4 available GPUs. It can place up to 4 pods. Each pod thinks it has a dedicated GPU. In reality they all share the same physical hardware.

The GPU switches between the pods’ CUDA contexts, giving each one a “time slice” of the compute resources. This is similar to how a CPU time slices between processes.

What Time-Slicing gives you

Works on any NVIDIA GPU. T4, V100, A10G, A100, H100. Any GPU the device plugin supports. No hardware generation requirements.

Zero workload changes. Your pods do not need to know they are sharing. They request nvidia.com/gpu: 1 exactly like they would for an exclusive GPU. The sharing is transparent.

Configurable oversubscription. You decide how many replicas per GPU. 4 replicas, 8 replicas, 10 replicas. Whatever makes sense for your workload density.

What Time-Slicing costs you

No memory isolation. This is the big one. All pods sharing a GPU have access to the full GPU memory. There are no limits on how much VRAM each pod can allocate.

If one pod allocates 70GB of VRAM on an 80GB GPU, the other three pods will OOM when they try to allocate even a small amount.

You can set 4 replicas. But there is no mechanism to say “each replica gets 20GB.” The pods are on the honor system. Pods do not have honor.

No fault isolation. A CUDA error in one pod can affect all other pods sharing the same GPU. One misbehaving workload can take down three others.

No performance guarantees. When multiple pods actively use the GPU, they share compute time equally. Four active pods each get roughly 25% of the compute throughput. A pod’s performance degrades proportionally to the number of active neighbors.

Context switching overhead. The GPU saves and restores state when switching between CUDA contexts. For workloads with large GPU memory footprints, this overhead can be significant.

When to use Time-Slicing

Development and testing environments where isolation does not matter. Lightweight inference workloads where each model uses a small fraction of GPU memory. Older GPU hardware (T4, V100) where MIG is not available. Teams that want the simplest possible path to GPU sharing.

MPS: CUDA Level Concurrent Execution

Multi-Process Service is a CUDA feature that allows multiple processes to execute on the GPU simultaneously. Not by taking turns like Time-Slicing. By actually running CUDA kernels from different processes in parallel on different Streaming Multiprocessors.

How it works in Kubernetes

MPS requires running an MPS daemon on each GPU node. The NVIDIA device plugin supports MPS as a sharing mode:

apiVersion: v1
kind: ConfigMap
metadata:
  name: mps-config
  namespace: gpu-operator
data:
  any: |-
    version: v1
    sharing:
      mps:
        resources:
          - name: nvidia.com/gpu
            replicas: 4

Like Time-Slicing, this makes one GPU appear as 4 resources. But the execution model is fundamentally different.

With Time-Slicing, only one CUDA context is active at a time. The GPU switches between them.

With MPS, multiple CUDA contexts run concurrently. The MPS server mediates access to the GPU’s Streaming Multiprocessors. Kernels from different processes execute in parallel.

What MPS gives you

True concurrent execution. Multiple pods run CUDA kernels on the GPU at the same time. For workloads that do not fully utilize the GPU’s compute capacity, this means significantly higher aggregate throughput compared to Time-Slicing.

Reduced context switching overhead. Processes run concurrently rather than sequentially. No context switch penalty. The GPU does not need to save and restore state between processes.

Compute partitioning (partial). You can limit the percentage of Streaming Multiprocessors available to each MPS client using CUDA_MPS_ACTIVE_THREAD_PERCENTAGE. This gives you some control over compute allocation.

Memory limits. MPS supports per-client memory limits through CUDA_MPS_PINNED_DEVICE_MEM_LIMIT. You can cap how much GPU memory each client can allocate. This provides some memory protection that Time-Slicing lacks entirely.

What MPS costs you

No memory isolation. Despite supporting memory limits, MPS does not provide hardware-level memory isolation. Processes share the same memory space. A rogue process can potentially read or corrupt another process’s GPU memory. The memory limits are enforced at the CUDA API level, not the hardware level.

Single user assumption. MPS was designed for single-user environments where all processes are trusted. In multi-tenant Kubernetes environments, this assumption may not hold.

Incompatible with MIG. You cannot use MPS inside MIG instances as of current GPU Operator versions. It is one or the other.

Error propagation. A fatal CUDA error from one MPS client terminates the MPS server. This kills all other clients sharing that GPU. One bad deployment takes down every model on that GPU. This is worse than Time-Slicing. Time-Slicing causes intermittent interference. MPS causes immediate total failure.

When to use MPS

High throughput inference with multiple small models where concurrent execution improves aggregate throughput. Workloads from a single team where all processes are trusted. Scenarios where Time-Slicing’s sequential execution is a throughput bottleneck.

The Decision Framework

Start with the isolation requirement.

If you need memory isolation and SLA guarantees per workload, the answer is MIG. No other option provides hardware-level isolation. If your workloads run on A100 or H100 GPUs and isolation matters, MIG is the only correct choice.

If you do not need isolation (dev/test, single-team workloads, lightweight inference), you can choose between Time-Slicing and MPS.

Then consider your GPU hardware.

MIG requires Ampere or Hopper GPUs. If you run older hardware (T4, V100) or mid-range GPUs (A10G, L4), MIG is not available. Your options are Time-Slicing or MPS.

Then consider your workload pattern.

Bursty workloads (high utilization for short periods, then idle) work well with Time-Slicing. The sequential execution does not matter because the pods rarely compete for compute at the same time.

Continuously active workloads (always doing inference, always using GPU compute) benefit from MPS. Kernels run in parallel rather than sequentially, which gives better aggregate throughput.

The hybrid approach.

For production H100/A100 clusters, you can combine MIG with Time-Slicing. Create MIG instances for hardware isolation. Then apply Time-Slicing within each MIG instance for additional density.

Example: partition an A100 into two 3g.20gb MIG instances. Apply 2x Time-Slicing on each instance. You now have 4 “GPU slots.” Each one has 20GB of isolated memory. Pairs share via Time-Slicing. This is the best of both worlds for many inference workloads.

Kubernetes Resource Comparison

Here is what each strategy looks like from the scheduler’s perspective:

Default (no sharing):

# Node advertises:
nvidia.com/gpu: 1

# Pod requests:
nvidia.com/gpu: 1
# Gets entire physical GPU

MIG:

# Node advertises:
nvidia.com/mig-1g.5gb: 7

# Pod requests:
nvidia.com/mig-1g.5gb: 1
# Gets isolated MIG instance with 5GB VRAM

Time-Slicing (4 replicas):

# Node advertises:
nvidia.com/gpu: 4   # Oversubscribed from 1 physical GPU

# Pod requests:
nvidia.com/gpu: 1
# Gets shared access, no memory limit

MPS (4 replicas):

# Node advertises:
nvidia.com/gpu: 4   # Oversubscribed from 1 physical GPU

# Pod requests:
nvidia.com/gpu: 1
# Gets concurrent access via MPS server

Time-Slicing and MPS look identical from the scheduler’s perspective. The difference is entirely in the runtime behavior. The scheduler does not know whether it is assigning an exclusive GPU, a MIG instance, a time slice, or an MPS client.

This is both elegant (transparent to workloads) and dangerous (no visibility into actual resource guarantees).

Common Mistakes

Mistake 1: Using Time-Slicing for production inference without memory limits. You set 4 replicas on an 80GB A100. Three pods use 15GB each. The fourth pod deploys a larger model that allocates 40GB. One of the first three pods OOMs on its next request. There is no mechanism to prevent this.

Mistake 2: Choosing MIG profiles that do not match workload sizes. You create seven 1g.5gb instances on an A100. Your smallest model needs 8GB. None of the instances are usable. Plan your MIG profiles around your actual model memory requirements.

Mistake 3: Forgetting that MIG reconfiguration requires draining. You cannot change MIG profiles while workloads are running. Cordon the node. Drain the GPU workloads. Reconfigure. Uncordon. Automate this or you will be doing it manually at 2 AM.

Mistake 4: Ignoring the MPS error propagation risk. One MPS client crash kills the MPS server and all other clients. In production, one bad deployment can take down every model on that GPU. If you use MPS, make sure your workloads are well tested.

Mistake 5: Not monitoring actual GPU utilization after enabling sharing. You enabled 8x Time-Slicing. The node shows 8 “GPUs” allocated. But what is the actual SM utilization? What is the actual memory usage? Without DCGM Exporter metrics, you are flying blind. GPU sharing without GPU monitoring is just organized waste.

The Monitoring You Need

Whatever sharing strategy you choose, you need visibility into what is actually happening on the GPU:

DCGM_FI_DEV_GPU_UTIL          # SM (compute) utilization %
DCGM_FI_DEV_FB_USED           # Framebuffer (VRAM) used in MB
DCGM_FI_DEV_FB_FREE           # Framebuffer free in MB
DCGM_FI_DEV_MEM_COPY_UTIL     # Memory bandwidth utilization %
DCGM_FI_PROF_SM_ACTIVE        # SM active (more granular)

With DCGM Exporter (part of the GPU Operator), these metrics are available in Prometheus. Build a dashboard that shows per-GPU utilization alongside your sharing configuration.

If you set 4x Time-Slicing and actual SM utilization is 95%, you are oversubscribed. If it is 20%, you could go to 8x.

The goal of GPU sharing is not maximum pod count per GPU. It is maximum useful work per GPU dollar.

The Bottom Line

MIG when you need isolation. Time-Slicing when you need simplicity. MPS when you need throughput.

Start with Time-Slicing for dev/test. Graduate to MIG for production. Consider MPS for high-throughput single-team inference workloads. Use the MIG plus Time-Slicing hybrid for the best balance of isolation and density.

Do not pick a sharing strategy without monitoring GPU utilization first. Measure your actual workload memory and compute usage. Then choose the strategy that matches your isolation requirements and hardware capabilities.

Next week: Deploying vLLM on Kubernetes: From Single Pod to Production.

If you manage GPU clusters on Kubernetes, I cover GPU infrastructure, model serving, and production operations every week. Subscribe at kubenatives.com.

The official docs tell you how to install the NVIDIA device plugin. They don’t tell you what happens when the GPU Feature Discovery pod crashes silently and your scheduler stops placing GPU workloads.

They don’t tell you that running etcd on the same nodes as your GPU workloads will create latency spikes that look like application bugs. They don’t tell you that a 7B model on an A100 wastes 90% of a $30K card unless you configure MIG properly.

I learned all of this the hard way. Running H100 clusters in production, debugging at 2 AM, reading NVIDIA docs that assume you already know the answer.

That’s why I built this course.

GPU Infrastructure on Kubernetes is a structured, text based course that covers everything from the NVIDIA GPU Operator internals to production model serving — with the depth that KubeNatives readers expect, plus step by step walkthroughs, exercises, and production checklists.

Here’s what it covers:

The GPU Operator deep dive. All 7 components. What each one does, how they depend on each other, and how to debug when one fails. Most engineers only know about the device plugin. This section covers the other 6 that actually cause your production issues.

GPU partitioning strategies. MIG, time slicing, and MPS explained with real configuration examples. The decision framework for choosing between them. Cost modeling so you can calculate exactly how much you’re wasting with whole GPU allocation.

Scheduling and resource management. How K8s GPU scheduling actually works under the hood. Topology awareness, NUMA alignment, and why pod placement matters for inference latency. The configs that took our p99 from 200ms to 40ms.

Model serving on GPU nodes. vLLM and Triton deployment patterns. Resource requests that actually make sense for inference workloads. Autoscaling GPU workloads without the cold start penalty.

Monitoring and debugging. DCGM metrics that predict failures before they happen. The GPU pod pending decision tree. Memory pressure debugging. Thermal throttling detection.

Production checklists and failure modes. Every section ends with a checklist you can use in your own clusters and a catalog of the failure modes I’ve encountered. These alone will save you dozens of debugging hours.

This isn’t a weekend tutorial. It’s the course I wished existed when I started running GPU infrastructure. Every section is 3 to 4 times deeper than the newsletter articles they’re based on, with exercises and real production scenarios.

The course is live now at devopsbeast.com

If you’ve been reading KubeNatives every week — this is the full picture, structured so you can go from zero GPU experience to confidently running production GPU workloads.

etcdserver: request timed out

This is the moment most engineers realize something they should have known all along: etcd is the most critical component in your Kubernetes cluster, and nobody was watching it.

Every piece of the cluster state lives in etcd. Every pod, every secret, every configmap, every deployment, every service account.

When etcd is slow, the API server is slow. When etcd is down, the cluster is read-only. When etcd loses data, you restore from a backup and hope it’s recent.

This guide covers the five ways etcd breaks in production, the metrics that predict each failure before it happens, and the exact commands to diagnose and fix them.

How etcd Actually Stores Your Cluster

Before debugging etcd, you need to understand what’s inside it.

etcd is a key-value store organized as a flat namespace under /registry. Every Kubernetes resource maps to a key:

/registry/pods/default/nginx-abc123
/registry/deployments/production/api-server
/registry/secrets/kube-system/cluster-admin-token
/registry/configmaps/monitoring/prometheus-config

The value at each key is the full serialized object (protobuf by default, JSON in older clusters). A deployment with 50 replicas doesn’t create 50 keys. It creates one key for the Deployment and 50 keys for the individual Pods.

Every write to etcd creates a new revision. etcd uses Multi-Version Concurrency Control (MVCC), which means it keeps old revisions around until they’re compacted. This is how kubectl --watch works: it reads from a specific revision and streams all changes after it.

The critical implication: etcd’s database grows with every write, even if you’re updating the same key over and over. A deployment that gets updated 1,000 times creates 1,000 revisions of that key. Without compaction, the database grows without bound.

Problem 1: Database Size Growing Out of Control

This is the most common etcd failure in production, and it’s completely preventable.

The symptoms: etcd starts slowly. API server latency creeps up. Eventually, you see the NOSPACE alarm, and writing stops entirely. Your cluster becomes read-only. No new pods, no config changes, no deployments.

Why it happens: etcd’s default storage limit is 2GB (configurable up to 8GB). Every revision takes space. If auto-compaction isn’t configured or isn’t keeping up, the database grows until it hits the limit.

Kubernetes API servers are configured with the default --etcd-compaction-interval=5m, which compacts revisions older than 5 minutes.

But compaction alone doesn’t reclaim disk space. It marks old revisions as free but leaves gaps in the database file. The file doesn’t shrink until you defragment.

The metric that predicts this:

etcd_mvcc_db_total_size_in_bytes

Monitor this. If it’s growing steadily and approaching your --quota-backend-bytes limit, you’re heading for NOSPACE.

Also compare dbSize vs dbSizeInUse:

etcdctl endpoint status --write-out=table \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

If DB SIZE is significantly larger than DB SIZE IN USE (more than 50% difference), fragmentation is the problem. Compaction ran, but defragmentation hasn’t.

The fix:

Step 1: Compact old revisions.

# Get the current revision
rev=$(etcdctl endpoint status --write-out=json \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  | jq -r '.[0].Status.header.revision')

# Compact everything older than current revision
etcdctl compact $rev \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

Step 2: Defragment each member (one at a time, not in parallel).

# Defragment a single member
etcdctl defrag \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

Important: defragmentation blocks reads and writes on that member. Do it one member at a time, starting with followers, and defragment the leader last to avoid triggering an unnecessary leader election. Wait 30 to 60 seconds between members.

Step 3: If the NOSPACE alarm triggered, disarm it after reclaiming space.

etcdctl alarm disarm \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

Prevention: Set up auto-compaction and schedule periodic defragmentation. Most production teams run defragmentation as a weekly CronJob during low traffic windows. The etcd-defrag tool from the etcd community automates the rolling defrag process safely.

Problem 2: Disk Latency Killing Performance

etcd’s performance is directly tied to disk write latency. Every Raft consensus write requires an fsync to the Write Ahead Log (WAL). If that fsync is slow, every API server request that writes to etcd is slow.

The symptoms: API server requests are slow across the board. kubectl apply takes seconds. Controller reconciliation loops are delayed. But etcd isn’t crashing and the database isn’t full.

Why it happens: etcd is running on shared storage, spinning disks, or network attached storage with variable latency. The official recommendation is fsync latency under 10ms. Anything above that and you’ll see degradation. Above 50ms and things start breaking.

The most common version of this: etcd is running on the same nodes as the API server (stacked topology) and sharing the disk with container workloads, logging agents, and monitoring exporters. We covered this tradeoff in detail in our stacked vs external etcd article.

The metric that predicts this:

etcd_disk_wal_fsync_duration_seconds

This is the single most important etcd metric. If the p99 is above 10ms, you have a disk problem. Above 50ms, expect leader elections and cluster instability.

Also watch:

etcd_disk_backend_commit_duration_seconds

This measures how long it takes to commit data to the backend database (boltdb). Healthy clusters show this under 25ms at p99.

The fix:

Short term: Identify what’s competing for disk I/O on the etcd nodes.

# Check disk I/O on etcd nodes
iostat -x 1 5

# Check what processes are doing the most I/O
iotop -o

Long term: Move etcd to dedicated NVMe storage. This is the single biggest performance improvement you can make. When we moved etcd from shared storage to dedicated NVMe in our clusters, API server p99 latency dropped 40%.

If you’re on managed Kubernetes (EKS, GKE, AKS), the cloud provider handles etcd storage. If you’re running self-managed clusters, dedicated SSDs or NVMe for etcd is not optional in production.

Problem 3: Leader Elections and Cluster Instability

etcd uses the Raft consensus protocol. At any given time, one member is the leader and the others are followers. The leader handles all writes and replicates them to followers. If the leader becomes unresponsive, the remaining members elect a new leader.

Occasional leader elections are normal (during upgrades, node maintenance). Frequent leader elections are a sign of trouble.

The symptoms: Intermittent API server timeouts. kubectl commands sometimes work, sometimes hang. Logs show elected leader messages repeatedly.

Why it happens: The most common causes are network partitions between etcd members, disk latency causing the leader to miss heartbeat deadlines, and resource contention (CPU or memory pressure) on etcd nodes.

Raft requires the leader to send heartbeats to followers within a configurable interval (default 100ms). If the leader misses enough heartbeats (default election timeout is 1000ms), followers trigger an election. During the election, the cluster cannot process writes.

The metrics that predict this:

etcd_server_leader_changes_seen_total

More than one leader change per hour indicates instability. More than one per minute is a crisis.

etcd_network_peer_round_trip_time_seconds

This measures the network latency between etcd members. If it’s spiking, network issues are causing the leader to miss heartbeats.

etcd_server_heartbeat_send_failures_total

Rising heartbeat failures mean the leader is having trouble reaching followers.

The fix:

Check the etcd member list and endpoint status to identify which member is the current leader and if any members are unhealthy:

etcdctl member list --write-out=table \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

etcdctl endpoint status --write-out=table --cluster \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

Look at the RAFT TERM column. If it’s much higher than expected for the cluster’s age, you’ve had many elections.

For network issues between members, check the latency between etcd nodes:

# From each etcd node to the others
ping -c 10 <other-etcd-node-ip>

etcd members should be in the same availability zone or, at a minimum, have sub-millisecond network latency between them. Cross-AZ etcd is technically possible, but adds latency to every write.

Problem 4: Slow Reads from Too Many Objects

As your cluster grows, the number of objects in etcd increases. A cluster with 5,000 pods, 2,000 configmaps, 3,000 secrets, and 500 services has tens of thousands of keys. Listing all pods across all namespaces means etcd reads and returns all of those objects.

The symptoms: kubectl get pods --all-namespaces takes 10+ seconds. Controller managers are slow to reconcile. The API server’s LIST requests show high latency.

Why it happens: The API server translates LIST requests into etcd range queries. A range query on /registry/pods/ returns every pod in the cluster. With thousands of pods, that’s megabytes of serialized data that etcd has to read, the API server has to deserialize, and the network has to transfer.

The metric that predicts this:

apiserver_request_duration_seconds{verb="LIST"}

If LIST operations are significantly slower than GET operations, object count is the issue.

Also check the total key count:

etcdctl endpoint status --write-out=json \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  | jq '.[0].Status.dbSize'

The fix:

Clean up unused resources. This sounds obvious, but most clusters accumulate orphaned resources over time:

# Find completed jobs older than 24 hours
kubectl get jobs --all-namespaces \
  --field-selector status.successful=1 \
  -o json | jq -r '.items[] | select(.status.completionTime < (now - 86400 | todate)) | .metadata.name'

# Find orphaned replica sets (old rollouts)
kubectl get rs --all-namespaces \
  -o json | jq -r '.items[] | select(.spec.replicas == 0) | "\(.metadata.namespace)/\(.metadata.name)"'

# Find unused configmaps not referenced by any pod
# (This requires more scripting but is worth the effort on large clusters)

Set ttlSecondsAfterFinished on Jobs so completed jobs clean themselves up. Set revisionHistoryLimit on Deployments (default is 10, consider lowering to 3 for large clusters).

For clusters above 5,000 nodes, consider enabling the API server’s watch cache and pagination to reduce the load on etcd from LIST operations.

Problem 5: Certificate Expiry

etcd uses mutual TLS for all communication: between etcd members (peer certificates) and between the API server and etcd (client certificates). When these certificates expire, etcd stops accepting connections. The API server can no longer read or write cluster state.

The symptoms: Everything breaks at once. All kubectl commands fail. The API server logs show TLS handshake failures. Pods stop being scheduled. Existing pods keep running (kubelet works from cache), but nothing new can be created.

Why it happens: kubeadm-provisioned clusters issue certificates with a 1 year expiry by default. If you don’t renew them before they expire, etcd communication fails.

The metric that predicts this:

There’s no etcd metric for certificate expiry. You need to check the certificates directly:

# Check etcd server certificate expiry
openssl x509 -in /etc/kubernetes/pki/etcd/server.crt -noout -enddate

# Check etcd peer certificate expiry
openssl x509 -in /etc/kubernetes/pki/etcd/peer.crt -noout -enddate

# Check etcd CA certificate expiry
openssl x509 -in /etc/kubernetes/pki/etcd/ca.crt -noout -enddate

# Check all K8s certificates at once (kubeadm)
kubeadm certs check-expiration

The fix:

If certificates haven’t expired yet, renew them:

# Renew all certificates (kubeadm)
kubeadm certs renew all

# Restart control plane components to pick up new certs
systemctl restart kubelet

If certificates have already expired, you need to renew them on each control plane node and restart the static pods. This is one of the most stressful operations in Kubernetes because the cluster is essentially down until it’s fixed.

Prevention: Set a monitoring alert for certificate expiry 30 days before they expire. Add this as a Prometheus alerting rule or a simple cron job that checks openssl x509 -enddate weekly.

The etcd Health Check Runbook

When something feels wrong with the cluster, run this sequence. It covers 90% of etcd issues in under 2 minutes:

#!/bin/bash
# etcd-health-check.sh
# Run this from a control plane node

CERTS="--cacert=/etc/kubernetes/pki/etcd/ca.crt \
       --cert=/etc/kubernetes/pki/etcd/server.crt \
       --key=/etc/kubernetes/pki/etcd/server.key"
EP="--endpoints=https://127.0.0.1:2379"

echo "=== 1. Cluster Health ==="
etcdctl endpoint health --cluster $EP $CERTS

echo ""
echo "=== 2. Member Status ==="
etcdctl endpoint status --write-out=table --cluster $EP $CERTS

echo ""
echo "=== 3. Alarm Status ==="
etcdctl alarm list $EP $CERTS

echo ""
echo "=== 4. Certificate Expiry ==="
echo "Server cert:"
openssl x509 -in /etc/kubernetes/pki/etcd/server.crt -noout -enddate
echo "Peer cert:"
openssl x509 -in /etc/kubernetes/pki/etcd/peer.crt -noout -enddate

echo ""
echo "=== 5. Database Size ==="
etcdctl endpoint status --write-out=json $EP $CERTS \
  | jq '.[0] | {
    dbSize: (.Status.dbSize / 1048576 | floor | tostring + " MB"),
    dbSizeInUse: (.Status.dbSizeInUse / 1048576 | floor | tostring + " MB"),
    fragmentation: (((.Status.dbSize - .Status.dbSizeInUse) / .Status.dbSize * 100) | floor | tostring + "%"),
    leader: .Status.leader,
    raftTerm: .Status.raftTerm
  }'

Save this as etcd-health-check.sh on every control plane node. Run it at the first sign of cluster slowness. Run it weekly as a habit.

The output tells you in 30 seconds whether you have a health problem, size problem, fragmentation problem, certificate problem, or leader stability problem.

The Metrics Dashboard

If you’re running Prometheus, these metrics should be added to your etcd dashboard. Ordered by priority:

Set alerts on the Critical thresholds. These metrics predict etcd failures before they become outages. We use these exact thresholds in our production H100 clusters, and they’ve caught degrading disks, network issues, and runaway compaction before they impacted workloads.

The Bottom Line

etcd doesn’t crash dramatically. It degrades slowly. API requests get a little slower. LIST operations take a little longer. Disk usage creeps up. Then one day a write fails and your cluster is read-only.

The five problems covered here account for the vast majority of etcd issues in production:

1. Database size growing out of control → monitor, compact, defragment

2. Disk latency killing performance → dedicated NVMe, isolate I/O

3. Leader elections and instability → check network, check disk, check resources

4. Slow reads from too many objects → clean up, set TTLs, limit revision history

5. Certificate expiry → monitor, automate renewal, alert 30 days before

The health check runbook takes 30 seconds to run and catches all five. Make it a habit.

Paid subscribers: The complete NOSPACE Emergency Recovery Runbook is live

Next week: MIG vs Time-Slicing vs MPS: Which GPU Sharing Strategy and When.

If you’re running production Kubernetes, I cover control plane operations, GPU infrastructure, and model serving every week.

You search “model serving Kubernetes” and get three names: vLLM, Triton Inference Server, and KServe. Every comparison article gives you a feature table and says, “It depends.”

Not helpful when you’re making an architecture decision that you’ll live with for the next two years.

Here’s the core insight that most comparisons miss: these three tools operate at different layers of the stack.

Comparing them side by side is like comparing nginx, Flask, and Kubernetes itself. They can overlap, but they’re fundamentally designed to solve different problems.

Let me explain what each one actually does, where it sits in the architecture, and how to pick the right combination for your workload.

The Three Layers of Model Serving

Before comparing the tools, you need to understand the three layers involved in serving models on Kubernetes:

Layer 1: The Inference Engine. This is the component that actually runs your model. It loads weights into GPU memory, processes input tensors, and generates outputs.

vLLM and Triton’s TensorRT-LLM backend are inference engines. They care about token throughput, memory management, and GPU utilization.

Layer 2: The Inference Server. This wraps the engine in an HTTP/gRPC API, handles request batching, manages model loading and unloading, and exposes health checks.

Triton Inference Server operates at this layer. vLLM also has its own built-in server with an OpenAI-compatible API.

Layer 3: The Orchestration Platform. This manages the Kubernetes resources around your inference workloads: autoscaling, canary deployments, traffic splitting, model versioning, and rollback.

KServe operates at this layer. It doesn’t serve models itself. It orchestrates the things that do.

The confusion in every comparison article comes from mixing these layers. vLLM vs Triton is a Layer 1/2 comparison.

KServe vs either of them is a Layer 2/3 comparison. They’re answering different questions entirely.

vLLM: The LLM Specialist

vLLM is a purpose-built inference engine for large language models. Developed at UC Berkeley, it introduced PagedAttention, a memory management technique that treats GPU memory as virtual memory pages rather than allocating fixed, contiguous blocks per request.

What it does well:

PagedAttention eliminates the memory fragmentation that kills GPU utilization in LLM serving.

Traditional inference servers pre-allocate memory for the maximum sequence length per request. A request that uses 2K tokens still reserves 32K tokens of memory.

vLLM allocates memory in small pages and grows dynamically, which means you can serve 3 to 5x more concurrent requests on the same GPU.

Continuous batching is the other major advantage. Traditional batching waits for a batch to fill before processing.

vLLM processes requests at the iteration level, inserting new requests into the batch as soon as a slot opens. This keeps GPU utilization above 90% even with variable request lengths.

The built-in server exposes an OpenAI-compatible API out of the box. If your application already uses the OpenAI API, you can point it at vLLM with no code changes.

It supports tensor parallelism to split large models across multiple GPUs, speculative decoding to reduce latency, and a wide range of quantization formats, including GPTQ, AWQ, and FP8.

What it doesn’t do:

vLLM is LLM only. It doesn’t support computer vision models, speech recognition models, or traditional ML models such as XGBoost or scikit-learn.

It doesn’t have a model repository, model versioning, or ensemble pipelines. It doesn’t support traffic splitting, canary deployments, or Kubernetes-native autoscaling.

It’s a fast, focused engine that does one thing extremely well: serve LLM inference requests with maximum GPU efficiency.

When to use it: You’re serving one or a few large language models. Your primary concern is token throughput and per-request latency.

You want the fastest path from “model in a registry” to “production inference endpoint.”

Triton Inference Server: The Multi-Framework Platform

Triton is NVIDIA’s general-purpose inference server. It’s designed to serve any model framework (PyTorch, TensorFlow, ONNX, TensorRT, XGBoost, and custom Python backends) through a unified API.

What it does well:

Model diversity is Triton’s superpower. If your organization runs a mix of workloads, including LLMs for chat, a BERT model for embeddings, a ResNet for image classification, and an XGBoost model for fraud detection, Triton serves all of them through the same infrastructure. Same API, same monitoring, same deployment patterns.

The model repository is a feature that matters more than people realize in production. Triton watches a directory (local, S3, or GCS) and automatically loads, unloads, and version manages models.

You deploy a new model version by dropping it in a folder. Triton handles the rest, including graceful transitions from v1 to v2.

Model ensembles let you chain multiple models in a pipeline.

For example: tokenizer → embedding model → reranker.

Each step runs as a separate model in Triton, and the server handles the data passing between them.

This is particularly useful for RAG pipelines where you need embeddings and generation in the same request flow.

Dynamic batching works well for models with fixed output lengths (classification, embeddings). For LLMs specifically, Triton uses the TensorRT-LLM backend or can integrate vLLM as a backend, which gives you PagedAttention and continuous batching through Triton’s enterprise API.

What it doesn’t do:

Triton is more complex to set up than vLLM. The model repository structure, config files, and backend selection add configuration overhead.

For pure LLM workloads, the setup complexity doesn’t justify itself unless you need Triton’s multi-model capabilities.

TensorRT-LLM (Triton’s optimized LLM backend) delivers excellent raw performance but requires model compilation to TensorRT format, which adds a build step and limits flexibility when you need to swap models quickly.

It also doesn’t handle Kubernetes orchestration. Triton is a server, not a platform. You still need to manage Deployments, Services, HPAs, and rollout strategies yourself.

When to use it: You’re serving multiple model types across frameworks. You need a unified inference API for your platform team. You’re already invested in the NVIDIA ecosystem and want maximum hardware optimization.

KServe: The Kubernetes Orchestration Layer

KServe is fundamentally different from vLLM and Triton. It’s a Kubernetes Custom Resource Definition (CRD) that manages the lifecycle of inference workloads.

As of late 2025, it’s a CNCF incubating project, which signals long-term community support and ecosystem integration.

What it does well:

KServe treats model serving as a Kubernetes native problem. You define an InferenceService, and KServe creates the Deployment, Service, HPA, and optionally the Knative serving resources. A simple deployment looks like this:

apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: llama-service
spec:
  predictor:
    model:
      modelFormat:
        name: huggingface
      resources:
        limits:
          nvidia.com/gpu: "1"
      storageUri: "hf://meta-llama/Llama-3.1-8B-Instruct"

That single resource handles everything: pulling the model, starting the serving runtime, configuring the GPU resources, setting up the endpoint, and enabling autoscaling.

Traffic management is where KServe shines for production workflows. You can run canary deployments with percentage-based traffic splitting between model versions.

You can A/B test model versions by routing a percentage of traffic to a new revision while monitoring performance before cutting over.

Autoscaling is built in through both Knative (scaling to zero based on request count) and KEDA integration (scaling based on custom metrics such as vLLM’s pending request queue or GPU utilization from DCGM).

For LLM workloads with bursty traffic patterns, this matters because you’re not paying for idle GPUs during low traffic periods.

The runtime pluggability is a critical design choice. KServe doesn’t serve models itself. It supports multiple serving runtimes, including vLLM, Triton, Hugging Face TGI, and custom runtimes.

This means you can use vLLM as the engine for LLM workloads and Triton for everything else, all managed through the same KServe InferenceService API.

What it doesn’t do:

KServe adds infrastructure complexity. It requires Knative or a Kubernetes Gateway API implementation, Istio or another service mesh (optional but recommended), and cert-manager. The installation footprint is significant compared to deploying vLLM directly.

It also adds latency. The routing layer (Istio/Knative) adds 1-3ms per request. For latency-sensitive applications where every millisecond matters, this overhead needs to be measured against the operational benefits.

For small teams serving a single model, KServe is overkill. The operational overhead of maintaining the KServe stack doesn’t justify itself until you have multiple models, multiple teams, or deployment patterns that require traffic management.

When to use it: You’re running multiple models across teams. You need canary deployments, traffic splitting, or the ability to scale to zero. You want a platform abstraction that decouples model developers from Kubernetes operations.

The Decision Framework

Here’s how I think about this decision for production workloads:

Start with your workload type.

If you’re only serving LLMs (chat, completion, RAG generation), start with vLLM. It gives you the best performance per GPU dollar with the least configuration overhead. Deploy it as a Kubernetes Deployment with an HPA, and you’re running in production.

If you’re serving a mix of model types (LLMs, embeddings, vision, and traditional ML), Triton is the right foundation.

The model repository and unified API eliminate the operational burden of maintaining separate infrastructure for each model type.

Then decide if you need orchestration.

If you’re deploying one or two models and your team manages Kubernetes directly, skip KServe.

Write your Deployments, Services, and HPAs by hand. The added abstraction isn’t worth the infrastructure cost.

If you’re running a model serving platform for multiple teams, need canary deployments between model versions, or want to scale to zero to manage GPU costs, add KServe on top. Use vLLM or Triton as the serving runtime underneath.

The combination that works for most teams:

For LLM-focused teams: vLLM as the engine, deployed directly as a Kubernetes Deployment. Add KServe when you outgrow manual deployments.

For platform teams serving diverse models: Triton as the inference server for everything, with KServe as the orchestration layer for lifecycle management.

For the hybrid case (LLMs plus other models): vLLM for LLM workloads, Triton for everything else, KServe orchestrating both through the same InferenceService API.

The Kubernetes Resource Comparison

Here’s what each tool actually creates when you deploy it:

vLLM standalone:

# You create and manage:
- Deployment (vLLM container + model config)
- Service (ClusterIP or LoadBalancer)
- HPA (custom metrics or resource based)
- PVC (for model storage, optional)
- ConfigMap (for vLLM args)

Triton standalone:

# You create and manage:
- Deployment (Triton container + model repo mount)
- Service (gRPC + HTTP ports)
- HPA (custom metrics)
- PVC or S3 config (model repository)
- ConfigMap (per model config.pbtxt files)

KServe with vLLM runtime:

# You create:
- InferenceService (single resource)

# KServe creates and manages:
- Deployment
- Service
- HPA or Knative autoscaler
- Virtual Service (traffic routing)
- Revision tracking

The tradeoff is clear. Direct deployment gives you full control but more YAML to manage. KServe gives you less YAML but adds infrastructure dependencies.

Performance Characteristics

These numbers aren’t benchmarks. They’re directional characteristics to understand the performance profile of each tool.

vLLM optimizes for token throughput. PagedAttention and continuous batching typically achieve 3 to 5x higher throughput than naive PyTorch serving for LLM workloads.

Latency is optimized at the engine level with speculative decoding and chunked prefill.

Triton with TensorRT-LLM can match or exceed vLLM’s raw throughput by optimizing the model graph for specific GPU architectures.

The tradeoff is compilation time and reduced flexibility. With the vLLM backend, Triton inherits vLLM’s performance characteristics plus a small overhead from the Triton serving layer.

KServe adds routing overhead (1-3ms through the ingress/service mesh layer). This is negligible for most LLM workloads, where generation takes hundreds of milliseconds to seconds.

The autoscaling behavior (especially scale-to-zero with Knative) can add a cold-start latency of 30 seconds or more as GPU pods initialize and load models.

For latency-sensitive applications, measure the full stack. Inference engine performance matters most, but routing, autoscaling cold starts, and model loading time all contribute to the end-user experience.

The Hybrid Architecture

The architecture I recommend for most production ML platforms looks like this:

vLLM handles the LLM workloads where PagedAttention and continuous batching matter most. Triton handles everything else through its multi-framework model repository.

KServe sits on top, providing a unified InferenceService API, traffic management, and autoscaling for all of them.

Each engine is matched to the GPU tier that makes economic sense. LLMs get the H100s. Embedding models get A100s. Vision models get T4s.

The GPU scheduling and node pool configuration (taints, tolerations, node affinity) ensure workloads land on the right hardware.

This connects directly to our GPU scheduling article, where we covered how device plugins, MIG, and time-slicing control which workloads get which GPUs.

Common Mistakes

Mistake 1: Starting with KServe for a single model. If you’re serving one LLM, a Deployment plus Service plus HPA is 40 lines of YAML.

KServe adds Knative, Istio, cert-manager, and the KServe controller. That’s a lot of infrastructure for one model.

Mistake 2: Using Triton for LLM-only workloads. Triton’s strengths are multi-framework support and the model repository.

If you’re only serving LLMs, vLLM gives you better performance with less configuration. Don’t add complexity you don’t need.

Mistake 3: Ignoring the runtime layer in KServe. KServe is only as good as the runtime underneath. Deploying KServe with a default Hugging Face runtime when you should be using vLLM means you’re getting KServe’s orchestration benefits while leaving 3 to 5x throughput on the table.

Mistake 4: Treating Triton and vLLM as competitors. They’re increasingly complementary. Triton can use vLLM as a backend, providing PagedAttention via Triton’s enterprise API.

The official Triton vLLM backend is actively maintained and production-ready.

Mistake 5: Not measuring cold start latency. Scaling KServe to zero sounds great for GPU cost savings.

But if your model takes 45 seconds to load onto a GPU, the first request after scale-up gets a 45-second latency spike. Measure this before enabling scale to zero in production.

Quick Reference

The Bottom Line

Don’t pick one. Understand what layer each tool operates at, and combine them based on your workload.

If you’re serving LLMs on Kubernetes, start with vLLM. Get it running, measure your throughput, and understand your GPU utilization.

Add Triton when you need to serve non-LLM models alongside your LLMs. Add KServe when you need platform-level orchestration for multiple models and teams.

The worst decision is over-engineering your first deployment. Start simple. Add complexity when the problem demands it, not before.

Next week: etcd Debugging Guide: When Your Cluster Starts Losing Its Memory.

If you’re building inference infrastructure on Kubernetes, I cover GPU scheduling, model serving, and production operations every week. Subscribe at kubenatives.com.

• vLLM pod killed with OOMKilled (CPU memory)

• vLLM pod crashes with CUDA out of memory (GPU memory)

• vLLM pod exits with no clear error but restarts repeatedly

• Performance degradation before eventual crash

Step 0: Identify Which OOM You Have

There are two types. They have different causes and different fixes.

# Check pod status
kubectl describe pod <vllm-pod> -n <namespace>

CPU OOM (OOMKilled):

State:          Terminated
  Reason:       OOMKilled
  Exit Code:    137

This means the container exceeded its Kubernetes memory limit. The kubelet killed it.

GPU OOM (CUDA out of memory):

State:          Terminated
  Reason:       Error
  Exit Code:    1

Check the logs:

kubectl logs <vllm-pod> -n <namespace> --previous

Look for:

torch.cuda.OutOfMemoryError: CUDA out of memory.

or

RuntimeError: NCCL error: out of memory

This means the model or KV cache exceeded available GPU VRAM.

Part 1: CPU OOM (OOMKilled / Exit Code 137)

Cause 1: Memory limit set too low

vLLM needs CPU memory for model loading, tokenization, request handling, and internal buffers. This is in ADDITION to GPU memory.

# Check current memory limits
kubectl get pod <vllm-pod> -o jsonpath='{.spec.containers[0].resources}'

The fix: Increase the memory limit. Rule of thumb:

8B model:   memory limit = 16-24 Gi
13B model:  memory limit = 24-32 Gi
70B model:  memory limit = 48-64 Gi

resources:
  requests:
    memory: 48Gi    # For 70B model
    cpu: "8"
    nvidia.com/gpu: "2"
  limits:
    memory: 64Gi    # 30% headroom over request
    nvidia.com/gpu: "2"
    # Do NOT set CPU limits (causes throttling)

Important: Do NOT set CPU limits on vLLM pods. CPU limits cause throttling which slows tokenization and request handling. Set CPU requests (for scheduling) but leave limits unset.

Read more