Multi-node GPU workloads are different. A Llama 70B model split across 2 nodes with 4 GPUs each requires all-reduce operations at every training step or tensor-parallel inference call. That is hundreds of gigabytes per second of cross-node traffic.

The default Kubernetes network stack uses kernel TCP over a CNI like Calico or Cilium. Maximum practical throughput: 10 to 25 Gbps. Round-trip latency: 100 to 500 microseconds.

For distributed GPU training, those numbers are 10 to 100x too slow. The GPUs finish their math and wait on the network. Your $30K per hour H100 cluster becomes a $30K per hour network stall.

This article covers what RDMA and InfiniBand actually are, why they matter for GPU workloads, and how to integrate them with Kubernetes.

What RDMA Actually Does

RDMA stands for Remote Direct Memory Access. The name describes the mechanic exactly. One machine reads or writes the memory of another machine directly, without involving either machine’s CPU or operating system kernel.

In a normal TCP send, data goes from application memory, into kernel buffers, through the TCP stack, across the network card, into the other machine’s network card, back up through its kernel, into its application memory. Every hop copies data and switches context.

RDMA skips all of that. The application pins a memory region, hands a pointer to the NIC, and the NIC does DMA directly. The remote NIC receives and writes straight into the target memory region. No kernel involvement on either side.

The result: 200+ Gbps throughput, single digit microsecond latency, and near zero CPU usage on both ends.

For GPU workloads, there is a second layer. GPUDirect RDMA lets the NIC read from and write to GPU memory directly, bypassing host CPU memory entirely. The GPU sends data. The NIC picks it up from GPU memory. It lands in GPU memory on the other side. CPUs and host RAM are not in the path.

This is how NCCL (NVIDIA Collective Communications Library) achieves the kind of throughput distributed training needs.

InfiniBand vs RoCE: The Two Ways to Get RDMA

RDMA is a protocol. You need hardware that supports it. Two options exist in practice.

InfiniBand. A dedicated network technology designed from the start for RDMA. Purpose built switches, cables, and adapters. NVIDIA Quantum-2 InfiniBand delivers 400 Gbps per port. You run a completely separate network alongside your Ethernet for RDMA traffic.

Pros: best performance, lowest latency, deterministic behavior under load. Cons: dedicated hardware, expensive switches, separate cabling, separate management plane.

RoCE (RDMA over Converged Ethernet). Runs RDMA over standard Ethernet hardware. Specifically RoCE v2, which runs over IP and can be routed. Requires Ethernet switches that support priority flow control (PFC) and explicit congestion notification (ECN).

Pros: uses existing Ethernet skills, single network fabric, easier to integrate with existing infrastructure. Cons: performance depends heavily on switch configuration, more sensitive to network congestion, harder to debug.

For greenfield GPU clusters at real scale (256+ GPUs), InfiniBand remains the default. For smaller clusters or shared general purpose infrastructure, RoCE v2 is the pragmatic choice.

Most cloud providers expose RDMA via RoCE. AWS EFA, Azure InfiniBand (rebranded RoCE in some SKUs), GCP compact placement with RoCE. You rarely touch native InfiniBand unless you run bare metal.

The Problem with Standard Kubernetes Pod Networking

A standard Kubernetes pod gets one network interface, typically eth0, configured by the CNI plugin. The interface runs over kernel TCP. Bandwidth is capped by the CNI overlay and the kernel stack.

GPU workloads need three things the default setup does not provide.

Multiple network interfaces. Pods need the standard eth0 for Kubernetes traffic (API server, service discovery, application calls) plus additional high speed interfaces for RDMA traffic. The CNI only gives you one.

Host network level access to the RDMA device. The NIC exposes RDMA resources (queue pairs, protection domains, memory regions) through a device file like /dev/infiniband/uverbs0. Pods need this device mounted and the right capabilities to use it.

CPU pinning and NUMA awareness. RDMA performance depends on the memory being allocated on the same NUMA node as the NIC. GPU workloads already need NUMA affinity to the GPU. The GPU, the NIC, and the pinned memory all need to be on the same NUMA node for peak throughput.

Default Kubernetes ignores all of this. The CNI gives you one interface. Devices are not mounted. NUMA is invisible to the scheduler.

The Stack That Actually Works

Production RDMA on Kubernetes requires four components working together.

Multus CNI. A meta plugin that lets pods have multiple network interfaces. Multus delegates the primary interface to your normal CNI (Calico, Cilium, whatever) and lets secondary interfaces use different plugins. This is how pods get eth0 for Kubernetes traffic and net1 through net4 for RDMA.

apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: rdma-net
spec:
  config: |
    {
      "cniVersion": "0.3.1",
      "type": "macvlan",
      "master": "ib0",
      "mode": "bridge",
      "ipam": {
        "type": "whereabouts",
        "range": "10.56.0.0/16"
      }
    }

Pods reference this definition via annotation.

metadata:
  annotations:
    k8s.v1.cni.cncf.io/networks: rdma-net

NVIDIA Network Operator. Automates the host setup. Installs the MOFED drivers (Mellanox OpenFabrics Enterprise Distribution), configures NICs, sets up SR-IOV if using virtualized NICs, and deploys the RDMA device plugin. Without this, every node requires manual driver installation.

SR-IOV Device Plugin. Exposes virtual functions (VFs) of the RDMA NIC as Kubernetes extended resources. Pods request RDMA VFs the same way they request GPUs.

resources:
  requests:
    nvidia.com/gpu: 8
    nvidia.com/hostdev: 4
    rdma/shared_ib: 1

Topology Manager. Kubernetes feature (stable since 1.27) that coordinates scheduling decisions across CPU, memory, GPU, and device plugins. Enable it with --topology-manager-policy=single-numa-node to ensure the GPU, the NIC, and the pinned memory all land on the same NUMA node.

# kubelet configuration
topologyManagerPolicy: "single-numa-node"
cpuManagerPolicy: "static"
memoryManagerPolicy: "Static"
reservedMemory:
  - numaNode: 0
    limits:
      memory: 1100Mi

This is where most teams get stuck. The components exist. The integration between them is fragile. Single NUMA policy can cause pods to fail scheduling if the NUMA node is full. Relaxing to best-effort gives scheduling flexibility but loses guaranteed performance.

Validating the Setup

Once the stack is deployed, a pod should see multiple interfaces and RDMA devices.

# Inside the pod
$ ip link show
1: lo: ...
2: eth0: ...                    # Kubernetes pod network
3: net1@if123: ...              # RDMA interface via Multus

$ ibv_devices
    device          	   node GUID
    ------          	----------------
    mlx5_0          	ec0d9a0300xxxxxx

$ ibv_devinfo mlx5_0
hca_id:	mlx5_0
	transport:			InfiniBand (0)
	fw_ver:				28.40.1000
	link_layer:			Ethernet     # or InfiniBand for native IB

Then benchmark. perftest is the standard tool.

# On one pod
$ ib_write_bw -d mlx5_0 --report_gbits

# On another pod
$ ib_write_bw -d mlx5_0 --report_gbits <server_pod_ip>

Healthy numbers: 180 to 200 Gbps on 200 Gbps fabrics. 380+ Gbps on 400 Gbps fabrics. If you see 10 to 25 Gbps, the traffic is falling back to TCP over the default network. The pods are not actually using RDMA.

NCCL Tests as the Real Benchmark

perftest proves RDMA works. NCCL tests prove GPU workloads actually use it.

# Inside a pod with 8 GPUs and RDMA NICs
$ mpirun -np 8 --allow-run-as-root \
  -mca btl_tcp_if_include eth0 \
  -x NCCL_IB_HCA=mlx5_0,mlx5_1,mlx5_2,mlx5_3 \
  -x NCCL_IB_GID_INDEX=3 \
  -x NCCL_DEBUG=INFO \
  all_reduce_perf -b 8 -e 8G -f 2 -g 1

The output shows bus bandwidth. On an 8x H100 node with 4x 400 Gbps NICs, you should see all reduce bandwidth around 350 to 400 GB/s for large messages. If you see 40 GB/s, NCCL is not using RDMA. Check the environment variables, check the NIC assignments, check that the RDMA interfaces are actually exposed to the pod.

When You Need This

Not every GPU workload needs RDMA. Single node training and single node inference do not. The GPUs in a single box talk to each other over NVLink, which is already faster than any network.

RDMA matters when:

Distributed training across nodes. Any model where the training job spans more than one physical machine. The all reduce after each step is the bottleneck. RDMA cuts it by 10 to 100x.

Tensor parallel inference across nodes. Very large models (70B, 405B) that do not fit on a single node split their layers across multiple machines. Every forward pass requires cross node communication. RDMA is table stakes.

Parameter server architectures. Less common now but still used in some recommendation model training.

High bandwidth data loading. Feeding training jobs from a distributed object store at line rate. GPUDirect Storage with RDMA lets GPUs read training data from NVMe arrays without going through CPU memory.

If your GPU workloads fit entirely on single nodes, skip this complexity entirely. Kubernetes default networking is fine. Come back when you need to scale past one machine.

The Bottom Line

RDMA is not optional for multi node GPU workloads. It is the difference between a cluster that computes and a cluster that waits on the network.

The stack is real but workable. Multus for multiple interfaces. NVIDIA Network Operator for host setup. SR-IOV Device Plugin for resource scheduling. Topology Manager for NUMA alignment. Production deployments use all four.

The hardest part is not any single component. It is the integration. A working RDMA cluster has 20 different things configured correctly in the right order. Miss one and traffic silently falls back to TCP. Your cluster computes. Nothing errors. The throughput is just 10x lower than it should be.

Validate with perftest for the fabric. Validate with NCCL tests for the GPU workload. Only trust the setup once both confirm the numbers you expected.

This concludes the first 26 weeks of KubeNatives. The journey has covered control plane internals, GPU infrastructure, model serving, LLMOps patterns, and now the network fabric that holds it all together. Next quarter goes deeper into each pillar with advanced topics including Dynamic Resource Allocation, pod priority for ML workloads, LLM gateway patterns, and multi cluster GPU federation. Thank you for reading.

The wrong way: update the Deployment image, wait for the rollout, monitor Grafana for 10 minutes, call it a success.

The right way: split 5% of production traffic to the new model, compare quality metrics side by side for 48 hours, ramp traffic if quality holds, roll back instantly if it drops.

This article covers the 4 things you need to A/B test LLM models in production on Kubernetes. Traffic splitting. Quality metrics. Shadow traffic. Rollback automation.

Why LLM A/B Testing Is Different

Traditional web service A/B testing compares latency, error rates, and conversion. All of those are server side metrics. You look at Grafana and know the answer in an hour.

LLM A/B testing compares response quality. Which model gave better answers? More helpful? More accurate? Less hallucinated? Those questions cannot be answered by server side metrics alone.

Three things are different about LLM A/B testing:

Quality is subjective. Latency is a number. Response quality needs human evaluation, LLM as judge evaluation, or proxy metrics like thumbs up rate.

The distribution matters. A new model might be better on average but catastrophically worse on 1% of prompts. You cannot see that in aggregate metrics.

Cost is variable. Llama 70B costs more GPU time per request than 8B. You are not just testing quality. You are testing whether the quality improvement justifies the cost increase.

A good A/B testing setup measures all three at once.

Pattern 1: Traffic Splitting with KServe InferenceService

KServe supports traffic splitting natively. You define one InferenceService with multiple revisions. Each revision points to a different model deployment. KServe routes incoming requests across them based on percentage.

apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: llm-router
spec:
  predictor:
    canaryTrafficPercent: 5
    model:
      modelFormat:
        name: vllm
      storageUri: "pvc://models/llama-3.3-70b"
  # Previous revision (llama-3.1-8b) keeps the other 95%

The canaryTrafficPercent: 5 field sends 5% of traffic to the new model. The other 95% still hits the previous revision. You can dial that number without redeploying.

Ramp pattern: 5% → 25% → 50% → 100%. Hold each stage for at least 24 hours. Monitor quality metrics between each step. If any metric degrades, roll back by setting canaryTrafficPercent: 0.

If you are not on KServe, Istio VirtualService supports the same pattern with weight fields across two routes.

Pattern 2: Quality Metrics You Can Actually Measure

“Better quality” is not a metric. You need something you can put in Prometheus and alert on. Four options, ranked by how quickly they return signal.

Thumbs up rate. If your product has a feedback widget, thumbs up divided by total responses. Cheap. Fast. Lags by minutes. Best for consumer products with high traffic.

Response acceptance rate. For code completion or suggestion systems, the percentage of suggestions the user actually accepts. Also cheap, also fast.

LLM as judge. Run a second, stronger model (like Claude or GPT 4o) to grade responses from both A and B. Give it a rubric. Track win rate. Costs money per request but gives signal on quality dimensions users never report on.

Human eval. Most expensive, slowest, highest quality. Sample 50 to 100 responses per model per day and have a human grader score them. Reserve for the final ramp decision before going to 100%.

A good setup uses at least two of these. Thumbs up rate for real time signal. LLM as judge for deeper evaluation. Human eval as the final gate.

Pattern 3: Shadow Traffic for Risk Free Testing

Sometimes you want to test a new model without risking user experience at all. Shadow traffic mirrors real requests to the new model, compares the responses, and throws the new response away. Users never see it.

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: llm-shadow
spec:
  hosts:
  - llm.inference.svc.cluster.local
  http:
  - route:
    - destination:
        host: llama-3-1-8b
        port:
          number: 8000
      weight: 100
    mirror:
      host: llama-3-3-70b
      port:
        number: 8000
    mirrorPercentage:
      value: 10.0

Istio mirror with mirrorPercentage: 10.0 sends a copy of 10% of requests to the new model. The original response from the production model is what the user sees. The shadow response is logged for comparison.

This is the safest way to run any new model. Zero user risk. You gather weeks of comparison data before you send any real traffic.

The catch: shadow traffic doubles GPU cost on the shadowed percentage. If you shadow 100% of traffic, you pay for 2x the compute. Shadow 5 to 10% and it is a reasonable investment for the risk reduction.

Pattern 4: Automated Rollback on Quality Regression

Manual rollback at 3 AM is how you end up with a worse model in production for 6 hours before someone notices. Automate it.

The idea: alert on quality regression, automatically drop canaryTrafficPercent to 0 when the alert fires.

# Prometheus alert
- alert: CanaryQualityRegression
  expr: |
    (
      rate(llm_thumbs_up_total{revision="canary"}[10m]) /
      rate(llm_responses_total{revision="canary"}[10m])
    )
    <
    (
      rate(llm_thumbs_up_total{revision="stable"}[10m]) /
      rate(llm_responses_total{revision="stable"}[10m])
    ) * 0.9
  for: 15m
  annotations:
    summary: "Canary model thumbs up rate is 10%+ below stable"

Pair the alert with a Kubernetes controller that watches Alertmanager and patches the InferenceService when the alert fires. Argo Rollouts has this built in. You define the metric, the threshold, and the rollback action. It does the rest.

The threshold matters. Too tight and every random variation triggers rollback. Too loose and real regressions slip through. Start at 10% below stable for consumer products. 5% for anything customer facing or paid.

Putting It Together: A Production Ramp Plan

Here is what a safe model upgrade looks like in practice. Llama 3.1 8B currently serving 100%. You want to move to Llama 3.3 70B.

Week 1. Shadow. Deploy Llama 3.3 70B alongside 8B. Mirror 10% of traffic. No user impact. Collect LLM as judge scores comparing responses. Look for win rate above 60% on your prompts.

Week 2. Canary 5%. Shadow data looked good. Move 5% of traffic to the new model. Watch thumbs up rate for both. Watch p99 latency (70B is slower than 8B). Watch cost per request.

Week 3. Ramp. Move to 25%, then 50%. Hold each step 24 to 48 hours. Quality metrics must stay within 5% of the stable revision. Automated rollback on regression.

Week 4. Full. 100% to new model. Keep the old revision around for one more week as a one click rollback option. Then retire it.

Total time: 4 weeks. Annoying? Yes. But this is how you ship a model change without breaking production.

What to Not Test A/B

Not everything needs A/B testing. Three cases where you should just deploy:

Bug fixes. If the current model is producing broken output and the new one fixes it, just deploy. A/B testing a fix prolongs the bug.

Security patches. Same logic. The current version has a known issue. Ship the patched version.

Identical model with config changes. Tuning max_tokens, adjusting the system prompt, changing batch size. A/B testing is overkill for changes that are mathematically unlikely to affect response distribution.

Reserve A/B testing for actual model swaps and major prompt rewrites. Everything else is just cautious engineering.

The Bottom Line

LLM A/B testing on Kubernetes is not hard. It is the same traffic splitting patterns you use for microservices, with quality metrics swapped in for error rate.

Four patterns. Traffic splitting via KServe or Istio. Quality metrics you can actually measure (thumbs up, LLM as judge, human eval). Shadow traffic for the highest risk changes. Automated rollback on regression.

Skip these and every model upgrade is a coin flip. Use them and you ship better models with confidence.

The hardest part is not the infrastructure. It is getting product and engineering to agree on what “better” means before you start the test.

Next week: Triton Inference Server on Kubernetes: Multi-Model Serving.

If you are running production Kubernetes clusters, I cover control plane internals, GPU infrastructure, and model serving every week. Subscribe at kubenatives.com.

When the API server slows down, everything slows down. Deployments stall. Pods stay in Pending longer. Controllers fall behind their desired state. The cluster feels sluggish even though CPU and memory on the nodes look fine.

The hard part is knowing when slow is slow. A 200ms p99 on a list operation is great. A 200ms p99 on a GET pod call is a fire.

This article covers the latency numbers that matter, the thresholds that signal trouble, and the 4 most common causes of API server latency spikes.

The Metric That Matters: apiserver_request_duration_seconds

The API server exposes request latency as a Prometheus histogram broken down by verb, resource, and scope. The raw metric name is apiserver_request_duration_seconds.

You never look at the average. You look at p99 by verb and resource.

A single cluster can have hundreds of different verb plus resource combinations. The ones that matter in practice are small in number. Focus on these.

# GET requests on individual objects
histogram_quantile(0.99,
  sum by (le, resource) (
    rate(apiserver_request_duration_seconds_bucket{verb="GET"}[5m])
  )
)

# LIST requests (the ones most likely to be slow)
histogram_quantile(0.99,
  sum by (le, resource) (
    rate(apiserver_request_duration_seconds_bucket{verb="LIST"}[5m])
  )
)

# WATCH requests (should be near zero latency after setup)
histogram_quantile(0.99,
  sum by (le, resource) (
    rate(apiserver_request_duration_seconds_bucket{verb="WATCH"}[5m])
  )
)

Three verbs. Three different expectations. Treating them the same is why most teams get API server alerting wrong.

What Normal Looks Like

The thresholds below are production numbers from healthy clusters in the 100 to 500 node range. Smaller clusters run faster. Larger clusters with heavy CRD usage run slower. Use these as a starting point and adjust for your baseline.

GET on individual objects: p99 under 50ms is healthy. Under 100ms is acceptable. Over 200ms means something is wrong.

LIST on small collections (under 1000 objects): p99 under 200ms is healthy. Under 500ms is acceptable. Over 1s is a problem.

LIST on large collections (pods across all namespaces, all events): p99 under 1s is healthy. Under 3s is acceptable. Over 5s means etcd or pagination is broken.

CREATE and UPDATE: p99 under 100ms is healthy. Under 250ms is acceptable. Over 500ms usually means etcd write latency, not API server CPU.

WATCH: the duration metric is misleading for watches. They are long lived. Look at apiserver_current_inflight_requests and etcd_server_has_leader instead.

Memorize the shape of these numbers. When you hit an incident at 3 AM, you need to know if 400ms on LIST pods is normal or is the alarm.

Cause 1: etcd Write Latency

The API server is stateless. Every write it accepts must be committed to etcd before the API server returns success. If etcd is slow, the API server is slow. No amount of API server scaling fixes this.

The metric to check: etcd_disk_wal_fsync_duration_seconds.

Healthy etcd: p99 under 10ms. Warning: p99 between 10ms and 50ms. Broken: p99 over 50ms.

When fsync hits 100ms, every write operation on the API server gets queued behind a slow disk. p99 API latency explodes.

The fix: dedicated NVMe storage for etcd. Not network attached disks. Not shared storage. Local NVMe on the etcd nodes. This is the single biggest performance improvement available.

If you are on a cloud provider, use the fastest local instance storage option. On AWS that is i4i or im4gn instances. On GCP that is local SSD. On Azure that is Lsv3 series.

Cause 2: List Operations Against Huge Collections

A controller that lists all pods across all namespaces every 30 seconds is a common cause of API server pressure. Each list pulls the full object from etcd through the API server, serializes it, and ships it to the controller.

On a 500 pod cluster, this is fine. On a 50,000 pod cluster, every list takes several seconds and consumes significant API server CPU.

The metrics to watch:

apiserver_request_total broken down by verb, resource, and subresource. Look for verb=”LIST” counts that dominate the total.

apiserver_response_sizes_bucket shows the size of responses. If LIST pods is returning 50MB responses on every call, you have a problem.

The fix: use label selectors, field selectors, and watch with resource version instead of periodic lists. Every Kubernetes controller written in the last 5 years uses informers that watch and cache rather than list. Audit your custom operators and any home grown controllers. The old pattern of periodic list is what kills clusters.

Cause 3: Admission Webhooks

Every create and update request passes through admission webhooks. Policy controllers, sidecar injectors, mutating webhooks for secrets. A slow webhook adds its latency to every matching request.

A 500ms webhook with failurePolicy: Fail is not just slow. It can deadlock cluster operations. If the webhook pod is down, no pods can be created. Including the webhook pod itself.

The metric: apiserver_admission_webhook_request_total and apiserver_admission_webhook_rejection_count.

The better metric for latency: apiserver_admission_webhook_admission_duration_seconds.

Healthy webhooks run under 100ms p99. Anything over 1s is a production risk.

The fix: set timeoutSeconds: 5 on every webhook at most. Use failurePolicy: Ignore for non critical webhooks. Scope webhooks with objectSelector and namespaceSelector so they only fire on the objects that actually need them. Default webhook scope is everything, which means every single create passes through.

Cause 4: Too Many Watches

The API server holds open connections for every watch client. Controllers, kubelet, custom operators, kubectl commands with watch mode. A single kubelet holds multiple watches.

On large clusters, watch count can reach tens of thousands of concurrent connections. Each one consumes memory and a file descriptor.

The metric: apiserver_registered_watchers broken down by resource.

When watch count spikes, the API server runs out of file descriptors. Every new connection fails. Latency explodes on everything.

The fix: set ulimits on the API server to at least 65535. Use watch with resource version and compaction to reduce memory pressure. Audit custom operators for watches that are never closed. Watch leaks are a real problem in handwritten controllers.

The Alert Set That Catches Real Problems

Most teams alert on p99 latency thresholds and nothing else. That catches slow symptoms, not slow causes.

A good alert set has 4 rules:

# 1. API server latency itself
- alert: APIServerHighLatency
  expr: histogram_quantile(0.99,
    sum by (le, verb) (
      rate(apiserver_request_duration_seconds_bucket{verb!="WATCH"}[5m])
    )
  ) > 1
  for: 5m

# 2. etcd disk latency (leading indicator)
- alert: EtcdSlowDisk
  expr: histogram_quantile(0.99,
    rate(etcd_disk_wal_fsync_duration_seconds_bucket[5m])
  ) > 0.05
  for: 5m

# 3. Webhook latency
- alert: SlowAdmissionWebhook
  expr: histogram_quantile(0.99,
    sum by (le, name) (
      rate(apiserver_admission_webhook_admission_duration_seconds_bucket[5m])
    )
  ) > 1
  for: 5m

# 4. Watch count spike
- alert: TooManyWatchers
  expr: sum(apiserver_registered_watchers) > 50000
  for: 15m

Rule 1 tells you there is a problem. Rules 2, 3, and 4 tell you what the problem is. Without the leading indicators, you wake up at 3 AM to a slow cluster and no idea where to look.

The Debugging Flow

When the API server gets slow in production, work through the causes in order of likelihood.

Step 1: check etcd. etcd_disk_wal_fsync_duration_seconds p99. If it is high, the problem is disk. Fix the disk or move etcd to external NVMe nodes.

Step 2: check webhooks. apiserver_admission_webhook_admission_duration_seconds p99 by webhook name. A slow webhook affects every matching operation. Disable or fix the slow one.

Step 3: check watch count. apiserver_registered_watchers by resource. A spike means a controller is leaking watches.

Step 4: check list patterns. apiserver_request_total by verb. If LIST is dominating, find the controller doing it and fix it to use informers.

80% of API server slowness resolves at step 1 or step 2. Do not start by scaling the API server. Scaling hides the symptom and makes the bill worse.

The Bottom Line

API server latency is the health signal for the entire control plane. Everything flows through it, so it sees every problem first.

Know the normal numbers. p99 under 50ms on GET, under 200ms on LIST for small collections, under 1s on LIST for large ones. Alert on etcd disk latency, webhook latency, and watch count. Investigate those before you investigate the API server itself.

And if you take one thing away: etcd on NVMe. The single highest leverage change you can make for API server performance on a busy cluster.

Next week: A/B Testing LLM Models in Production with Kubernetes.

If you are running production Kubernetes clusters, I cover control plane internals, GPU infrastructure, and model serving every week. Subscribe at kubenatives.com.

It does not work for a 20 node GPU cluster. You cannot SSH into 20 nodes every time someone reports slow inference. You need GPU metrics in Prometheus, dashboards in Grafana, and alerts that page you before users notice.

DCGM Exporter is the NVIDIA Data Center GPU Manager running as a DaemonSet on every GPU node. It collects GPU health and performance metrics and exports them in Prometheus format. This article covers which metrics matter, how to set up the monitoring stack, and what to alert on.

What DCGM Exporter Collects

DCGM Exporter runs as part of the GPU Operator (component 6 of 8). It scrapes GPU metrics at a configurable interval and exposes them on a /metrics endpoint.

The metrics fall into 4 categories:

Utilization metrics tell you how busy the GPU is.

DCGM_FI_DEV_GPU_UTIL          GPU compute utilization (%)
DCGM_FI_DEV_MEM_COPY_UTIL     Memory controller utilization (%)
DCGM_FI_DEV_ENC_UTIL          Encoder utilization (%)
DCGM_FI_DEV_DEC_UTIL          Decoder utilization (%)

Memory metrics tell you how much VRAM is in use.

DCGM_FI_DEV_FB_USED           Framebuffer (VRAM) used (MB)
DCGM_FI_DEV_FB_FREE           Framebuffer free (MB)
DCGM_FI_DEV_FB_TOTAL          Framebuffer total (MB)

Temperature and power metrics tell you about thermal and electrical health.

DCGM_FI_DEV_GPU_TEMP          GPU temperature (°C)
DCGM_FI_DEV_MEMORY_TEMP       Memory temperature (°C)
DCGM_FI_DEV_POWER_USAGE       Current power draw (W)
DCGM_FI_DEV_TOTAL_ENERGY      Total energy consumed (mJ)

Error metrics tell you about hardware problems before they cause failures.

DCGM_FI_DEV_ECC_SBE_VOL       Single-bit ECC errors (correctable)
DCGM_FI_DEV_ECC_DBE_VOL       Double-bit ECC errors (uncorrectable)
DCGM_FI_DEV_RETIRED_SBE       Retired pages (single-bit)
DCGM_FI_DEV_RETIRED_DBE       Retired pages (double-bit)
DCGM_FI_DEV_XID_ERRORS        XID error codes

The 6 Metrics That Actually Matter

Most teams monitor GPU utilization and nothing else. That is like monitoring CPU utilization and ignoring memory, disk, and network. Here are the 6 metrics that predict real problems.

Metric 1: GPU Temperature (DCGM_FI_DEV_GPU_TEMP)

GPUs thermal throttle at 83°C (A100) or 80°C (H100). When throttled, the GPU reduces clock speed to prevent damage. Performance drops 20 to 40% with no warning in application logs.

Alert threshold: warning at 78°C, critical at 82°C.

Metric 2: ECC Errors (DCGM_FI_DEV_ECC_DBE_VOL)

Double-bit ECC errors are uncorrectable memory errors. They indicate failing GPU memory. A single double-bit error means the GPU is producing incorrect computation results. Your model outputs may be wrong.

Alert threshold: any double-bit error is critical. Drain the node immediately.

Single-bit errors are correctable and normal in small quantities. Alert if the rate exceeds 10 per hour.

Metric 3: GPU Memory Used (DCGM_FI_DEV_FB_USED)

Track the ratio of used to total framebuffer. If memory usage is consistently above 95%, one spike will cause a CUDA out of memory crash.

Alert threshold: warning at 90%, critical at 95%.

Metric 4: GPU Utilization (DCGM_FI_DEV_GPU_UTIL)

For vLLM workloads, GPU utilization is normally high (80 to 95%). A sudden drop to 0% indicates the serving process crashed. A sustained drop to 30 to 40% indicates the model is not receiving requests (traffic routing issue).

Alert on sudden changes, not absolute values.

Metric 5: Power Usage (DCGM_FI_DEV_POWER_USAGE)

A100 TDP is 400W. H100 TDP is 700W. If power draw is consistently at TDP, the GPU is running at maximum capacity. If it drops to near zero while workloads are running, the GPU may have entered a fault state.

Useful for capacity planning and cost attribution.

Metric 6: XID Errors (DCGM_FI_DEV_XID_ERRORS)

XID errors are NVIDIA driver error codes. Each code indicates a specific type of failure.

XID 13:  Graphics Engine Exception (GPU hang)
XID 31:  GPU memory page fault
XID 43:  GPU stopped processing
XID 48:  Double-bit ECC error
XID 63:  ECC page retirement (row remapping)
XID 79:  GPU fallen off the bus (hardware failure)

XID 79 is the worst. It means the GPU is no longer communicating with the system. The node needs to be drained and the GPU replaced.

Alert threshold: any XID error is at minimum a warning. XID 48, 63, 79 are critical.

Setting Up the Monitoring Stack

If you installed the GPU Operator, DCGM Exporter is already running. Verify:

kubectl get pods -n gpu-operator -l app=nvidia-dcgm-exporter

All pods should be Running with one pod per GPU node.

ServiceMonitor for Prometheus:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: dcgm-exporter
  namespace: gpu-operator
spec:
  selector:
    matchLabels:
      app: nvidia-dcgm-exporter
  endpoints:
  - port: metrics
    interval: 15s
    path: /metrics

Verify metrics are being scraped:

# Check Prometheus targets
kubectl port-forward -n monitoring svc/prometheus 9090:9090

# In the Prometheus UI, check Status → Targets
# Look for dcgm-exporter targets. All should show UP.

Prometheus Alert Rules

groups:
- name: gpu-health
  rules:
  - alert: GpuTemperatureHigh
    expr: DCGM_FI_DEV_GPU_TEMP > 78
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "GPU temperature above 78°C on {{ $labels.instance }}"
      description: "GPU {{ $labels.gpu }} is at {{ $value }}°C. Thermal throttling starts at 83°C."

  - alert: GpuTemperatureCritical
    expr: DCGM_FI_DEV_GPU_TEMP > 82
    for: 2m
    labels:
      severity: critical
    annotations:
      summary: "GPU temperature critical on {{ $labels.instance }}"

  - alert: GpuDoubleBitEcc
    expr: DCGM_FI_DEV_ECC_DBE_VOL > 0
    for: 0m
    labels:
      severity: critical
    annotations:
      summary: "Uncorrectable ECC error on GPU {{ $labels.gpu }}"
      description: "Double-bit ECC error detected. GPU memory is producing incorrect results. Drain the node."

  - alert: GpuMemoryNearFull
    expr: (DCGM_FI_DEV_FB_USED / DCGM_FI_DEV_FB_TOTAL) > 0.95
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "GPU memory above 95% on {{ $labels.gpu }}"

  - alert: GpuUtilizationDrop
    expr: DCGM_FI_DEV_GPU_UTIL < 10 and DCGM_FI_DEV_GPU_UTIL offset 10m > 50
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "GPU utilization dropped suddenly on {{ $labels.gpu }}"
      description: "GPU util was above 50% and is now below 10%. Serving process may have crashed."

  - alert: GpuXidError
    expr: DCGM_FI_DEV_XID_ERRORS > 0
    for: 0m
    labels:
      severity: warning
    annotations:
      summary: "XID error {{ $value }} on GPU {{ $labels.gpu }}"

  - alert: GpuHighSingleBitEcc
    expr: rate(DCGM_FI_DEV_ECC_SBE_VOL[1h]) > 10
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "Elevated single-bit ECC error rate on {{ $labels.gpu }}"

Grafana Dashboard: Essential Panels

Build a dashboard with these panels. Group by node and GPU index.

Row 1: Overview

• GPU utilization per GPU (time series, 0 to 100%)

• GPU memory used vs total per GPU (bar gauge)

• GPU temperature per GPU (time series with 78°C and 82°C threshold lines)

Row 2: Health

• ECC errors (single-bit and double-bit) per GPU (stat panel, instant value)

• XID errors per GPU (table, last 24 hours)

• Power draw per GPU (time series, with TDP line)

Row 3: Cluster Overview

• Total GPUs allocated vs available (stat panel)

• Average GPU utilization across all GPUs (gauge)

• Nodes with GPU temperature above 78°C (table, filtered)

Common Issues DCGM Catches Early

Cooling failure. Temperature creeps up over days. DCGM shows the trend. You schedule maintenance before the GPU throttles.

Failing GPU memory. Single-bit ECC errors increase gradually. Then a double-bit error occurs. DCGM catches the trend. You drain the node before the GPU produces incorrect model outputs.

GPU fallen off the bus. XID 79. The GPU stops responding. DCGM reports the error. Without DCGM, you find out when users report that inference is not working and someone SSHs into the node.

Silent performance degradation. GPU utilization drops from 85% to 40% but no application errors. The serving process is healthy but something changed (model was reloaded, batch size decreased, traffic routing shifted). DCGM’s utilization trend catches the change.

The Bottom Line

nvidia-smi is for debugging one node. DCGM Exporter is for monitoring a cluster. If you run GPU workloads in production, DCGM metrics in Prometheus with alert rules are not optional.

Monitor 6 metrics: temperature, ECC errors, memory usage, utilization, power, and XID errors. Alert on the thresholds above. Build a Grafana dashboard for visibility.

The GPU that fails silently costs you more than the GPU that fails loudly. DCGM makes silent failures visible.

Next week: Service Mesh Debugging on Kubernetes: When Istio Breaks Your Inference Pipeline.

If you are running GPU clusters in production, I cover monitoring, model serving, and infrastructure operations every week. Subscribe at kubenatives.com.

The result: pods stuck in Pending because requests are too high. Or pods crashing with OOMKilled because memory limits are too low. Or GPU nodes running at 30% utilization because every pod over-requests resources.

This article covers how to set requests and limits correctly for inference workloads, training jobs, and mixed GPU environments.

Requests vs Limits: The Basics

Requests are what the scheduler uses to find a node. The scheduler looks at every node and asks: “Does this node have enough unrequested resources to fit this pod?” If no node has enough, the pod stays Pending.

Limits are the ceiling. If the pod tries to use more than the limit, Kubernetes enforces it. For memory, the pod is OOMKilled. For CPU, the pod is throttled.

resources:
  requests:
    cpu: "4"
    memory: 32Gi
    nvidia.com/gpu: "1"
  limits:
    memory: 48Gi
    nvidia.com/gpu: "1"

Critical difference for GPUs: GPU requests and limits must be equal. You cannot request 0.5 GPUs. You cannot set a limit of 2 GPUs with a request of 1. The NVIDIA device plugin requires requests = limits for nvidia.com/gpu. Always.

Why CPU Limits Are Dangerous for Inference

Do not set CPU limits on vLLM or Triton pods. This is counterintuitive but important.

CPU limits cause throttling. When a pod hits its CPU limit, Kubernetes throttles it. The pod is not killed. It is slowed down. For inference workloads, this means slower tokenization, slower request handling, and higher TTFT.

vLLM uses CPU for tokenization, request scheduling, and output buffering. These are bursty workloads. During model loading, CPU usage spikes. During inference, it drops. A static CPU limit penalizes the spikes.

# WRONG: CPU limit causes throttling during tokenization spikes
resources:
  requests:
    cpu: "4"
    memory: 32Gi
    nvidia.com/gpu: "1"
  limits:
    cpu: "8"        # Remove this
    memory: 48Gi
    nvidia.com/gpu: "1"

# RIGHT: CPU request only, no limit
resources:
  requests:
    cpu: "4"
    memory: 32Gi
    nvidia.com/gpu: "1"
  limits:
    memory: 48Gi
    nvidia.com/gpu: "1"
    # No CPU limit

Set CPU requests (for scheduling) but leave CPU limits unset. The pod can burst above its request when the node has spare CPU. No throttling. No latency spikes.

Memory Sizing for Inference Pods

Memory limits are essential. Without them, a memory leak crashes the node instead of just the pod.

The memory a vLLM pod needs:

Model loading buffer:     ~1x model size (temporary, during download/load)
Tokenizer:                200MB to 2GB (depends on vocab size)
Request handling:          ~100MB per concurrent request
vLLM engine overhead:     1-4GB
Python runtime:           500MB to 1GB

Sizing guide:

Model Size     Memory Request     Memory Limit     Headroom
8B FP16        16Gi               24Gi             50%
13B FP16       24Gi               36Gi             50%
70B FP16       48Gi               64Gi             33%
70B INT4       24Gi               36Gi             50%

The request is what the pod typically uses. The limit provides headroom for spikes during model loading, large batch requests, and garbage collection.

Set the limit at 30 to 50% above the request. This gives the pod room to breathe without letting a runaway process consume the entire node.

GPU Resource Rules

GPUs in Kubernetes have unique constraints that CPU and memory do not:

No fractional GPUs. You cannot request 0.5 GPUs. The device plugin treats GPUs as integers. 0 or 1. If you need fractional GPUs, use MIG, Time-Slicing, or MPS.

Requests must equal limits. The NVIDIA device plugin requires nvidia.com/gpu requests and limits to be identical. Setting requests: 1 and limits: 2 causes a validation error.

No overcommit. Unlike CPU (where you can request 1 core and burst to 4), GPUs are exclusively allocated. If you request 1 GPU, that entire physical GPU is reserved for your pod. No other pod can use it, even if your pod only uses 10% of the GPU.

No resource quotas on GPU memory. Kubernetes has no concept of GPU memory. You cannot request “20GB of GPU VRAM.” You request 1 GPU and get the entire device (40GB, 80GB, whatever the hardware has). GPU memory management is the serving framework’s job (vLLM’s gpu-memory-utilization flag).

Common Patterns

Pattern 1: Single GPU inference pod

resources:
  requests:
    cpu: "4"
    memory: 24Gi
    nvidia.com/gpu: "1"
  limits:
    memory: 36Gi
    nvidia.com/gpu: "1"

For an 8B model on a single GPU. No CPU limit. Memory limit 50% above request.

Pattern 2: Multi-GPU inference with tensor parallelism

resources:
  requests:
    cpu: "8"
    memory: 64Gi
    nvidia.com/gpu: "2"
  limits:
    memory: 96Gi
    nvidia.com/gpu: "2"

For a 70B model across 2 GPUs. Higher CPU request because tensor parallelism increases CPU usage for inter-GPU coordination. /dev/shm must also be mounted (16Gi minimum).

Pattern 3: Training job (batch)

resources:
  requests:
    cpu: "16"
    memory: 128Gi
    nvidia.com/gpu: "4"
  limits:
    memory: 160Gi
    nvidia.com/gpu: "4"

Training is more CPU-intensive than inference. Data loading, gradient computation, and checkpointing all use CPU. Higher CPU request is appropriate. Still no CPU limit.

Pattern 4: Jupyter notebook (development)

resources:
  requests:
    cpu: "2"
    memory: 8Gi
    nvidia.com/gpu: "1"
  limits:
    cpu: "4"       # OK for dev: predictable, non-production
    memory: 16Gi
    nvidia.com/gpu: "1"

For development workloads, CPU limits are acceptable. Notebooks are interactive and predictable. The limit prevents a runaway notebook from consuming an entire node. This is the one exception to the “no CPU limits” rule.

Monitoring Resource Usage

Set requests based on actual usage, not guesses. Monitor your pods and adjust.

# Check actual CPU and memory usage
kubectl top pod -n inference

# Check GPU utilization and memory
kubectl exec -it <vllm-pod> -- nvidia-smi

# Check resource requests vs actual usage over time
# Use Prometheus queries:
# container_cpu_usage_seconds_total
# container_memory_usage_bytes

If your pod consistently uses 2 CPU cores but requests 8, you are wasting 6 cores of scheduling capacity. Other pods cannot use those 6 cores even though they are idle.

If your pod’s memory usage is consistently at 95% of the limit, increase the limit. You are one spike away from OOMKilled.

The Right-Sizing Workflow

1. Start with the sizing guide above for initial deployment.

2. Run the workload for 48 hours with production traffic.

3. Check actual usage with kubectl top and Prometheus.

4. Set requests to the p95 of actual usage (covers normal spikes).

5. Set limits to 30 to 50% above requests (covers unusual spikes).

6. Review monthly and adjust as traffic patterns change.

The Bottom Line

For GPU inference pods: set CPU requests but not CPU limits. Set memory requests and limits with 30 to 50% headroom. Set GPU requests equal to limits (required by the device plugin).

The most common mistake is over-requesting CPU and memory. This wastes scheduling capacity and prevents other pods from running on the same node. The second most common mistake is not setting memory limits, which lets a memory leak crash the entire node.

Right-size based on actual usage. Monitor. Adjust. Repeat.

Next week: GPU Monitoring with DCGM Exporter: The Metrics That Matter.

If you are building GPU infrastructure on Kubernetes, I cover resource management, model serving, and production operations every week. Subscribe at kubenatives.com.

At peak (2 PM), all 4 replicas are busy. Queue depth is high. Users are waiting. You need more capacity.

At 3 AM, traffic drops to near zero. All 4 replicas are idle. GPU utilization is under 5%. You are paying $12/hour for GPUs that are doing nothing.

The Horizontal Pod Autoscaler (HPA) solves this. But GPU workloads have unique challenges that make standard HPA configurations fail. This article covers how to autoscale inference pods correctly.

Why Standard HPA Does Not Work for GPU Pods

The default HPA scales on CPU utilization. For GPU inference pods, CPU utilization is meaningless. The real work happens on the GPU. CPU stays low even when the model is fully saturated.

GPU utilization is also a bad scaling metric for vLLM. PagedAttention keeps GPU utilization high by design. A vLLM pod at 85% GPU utilization might be handling 5 requests or 500. The utilization number does not tell you if the pod is overloaded.

The right metrics for scaling inference pods are request-level metrics, not hardware metrics.

The Right Scaling Metrics

Queue depth (vllm:num_requests_waiting). This is the number of requests waiting to be processed. If requests are queuing, the pod is overloaded. Scale up.

# HPA scaling on queue depth
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: vllm-hpa
  namespace: inference
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: vllm-llama3
  minReplicas: 2
  maxReplicas: 8
  metrics:
  - type: Pods
    pods:
      metric:
        name: vllm_num_requests_waiting
      target:
        type: AverageValue
        averageValue: "5"
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 60
      policies:
      - type: Pods
        value: 2
        periodSeconds: 120
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
      - type: Pods
        value: 1
        periodSeconds: 180

When the average queue depth across pods exceeds 5 requests, the HPA adds replicas. When it drops below 5, it removes them.

Time to First Token (TTFT). This measures how long users wait before seeing the first token. If TTFT exceeds your SLO (say 500ms), scale up.

Active requests (vllm:num_requests_running). The number of requests currently being processed. This indicates current load, not queued load. Useful as a secondary metric alongside queue depth.

Setting Up Custom Metrics

The HPA needs to read vLLM metrics through the Kubernetes metrics API. This requires a metrics adapter.

Option 1: Prometheus Adapter

The Prometheus Adapter exposes Prometheus metrics as Kubernetes custom metrics. vLLM exports metrics to Prometheus. The adapter makes them available to the HPA.

# Prometheus Adapter configuration
rules:
- seriesQuery: 'vllm:num_requests_waiting{namespace!="",pod!=""}'
  resources:
    overrides:
      namespace: {resource: "namespace"}
      pod: {resource: "pod"}
  name:
    matches: "^(.*)"
    as: "vllm_num_requests_waiting"
  metricsQuery: 'avg(<<.Series>>{<<.LabelMatchers>>}) by (<<.GroupBy>>)'

Option 2: KEDA

KEDA (Kubernetes Event Driven Autoscaling) is simpler to set up than the Prometheus Adapter. It can query Prometheus directly without a separate adapter.

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: vllm-scaledobject
  namespace: inference
spec:
  scaleTargetRef:
    name: vllm-llama3
  minReplicaCount: 2
  maxReplicaCount: 8
  cooldownPeriod: 300
  pollingInterval: 15
  triggers:
  - type: prometheus
    metadata:
      serverAddress: http://prometheus.monitoring:9090
      metricName: vllm_queue_depth
      query: |
        avg(vllm:num_requests_waiting{namespace="inference",pod=~"vllm-llama3.*"})
      threshold: "5"
      activationThreshold: "1"

KEDA also supports scaling to zero, which the standard HPA does not. For development environments, this means your GPU pods shut down completely when unused.

The Scale-Up Problem: Model Loading Time

When the HPA triggers a scale-up, a new pod starts. The pod needs to load the model before it can serve requests.

For a 70B model without a PVC cache, that is 15 to 30 minutes. During that time, traffic is still hitting the existing (overloaded) pods. The new replica is not helping yet.

This is the fundamental challenge of autoscaling inference workloads. The scale-up latency is measured in minutes, not seconds.

Fixes:

PVC-backed model cache reduces startup to 1 to 3 minutes. Pre-populated shared storage makes it even faster. These are covered in the vLLM Model Loading article.

Startup probes must have a large enough window. If the startup probe fails before the model loads, the pod gets killed and restarted. An infinite restart loop.

startupProbe:
  httpGet:
    path: /health
    port: 8000
  failureThreshold: 120     # 120 * 10s = 20 minutes
  periodSeconds: 10

Proactive scaling. Instead of reacting to high queue depth, scale up before traffic arrives. If your traffic pattern is predictable (peak at 2 PM, low at 3 AM), use a CronJob to adjust minReplicas on a schedule.

# Scale up before peak hours (1:30 PM)
kubectl patch hpa vllm-hpa -n inference \
  -p '{"spec":{"minReplicas":4}}'

# Scale down after peak (10 PM)
kubectl patch hpa vllm-hpa -n inference \
  -p '{"spec":{"minReplicas":2}}'

The Scale-Down Problem: Disrupting Active Requests

Scaling down terminates a pod. If that pod is in the middle of generating a response, the user gets an error.

Fixes:

Set a generous terminationGracePeriodSeconds. This gives the pod time to finish active requests before shutting down.

spec:
  terminationGracePeriodSeconds: 120

The HPA behavior section controls how aggressively it scales down. Use stabilization windows and slow scale-down policies.

behavior:
  scaleDown:
    stabilizationWindowSeconds: 300    # Wait 5 min of low load
    policies:
    - type: Pods
      value: 1                         # Remove 1 pod at a time
      periodSeconds: 180               # Every 3 minutes

This means: wait for 5 minutes of sustained low load before scaling down. Then remove 1 pod every 3 minutes. Gradual and safe.

A PodDisruptionBudget ensures at least one pod is always running:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: vllm-pdb
  namespace: inference
spec:
  minAvailable: 1
  selector:
    matchLabels:
      app: vllm-llama3

KEDA vs HPA: When to Use Each

Use standard HPA when: You already have the Prometheus Adapter installed. You do not need scale-to-zero. Your scaling logic is simple (one or two metrics).

Use KEDA when: You want to query Prometheus directly without an adapter. You need scale-to-zero for dev environments. You want to scale on external metrics (SQS queue length, Kafka lag, HTTP request rate from an external load balancer).

For most production inference workloads, KEDA is the simpler choice. It handles the metrics pipeline internally and supports more trigger types.

The Complete Autoscaling Stack

vLLM exports metrics → Prometheus scrapes → KEDA queries →
HPA adjusts replicas → New pod starts → Model loads from PVC →
Pod passes startup probe → Traffic routes to new pod

The full stack requires: vLLM with metrics enabled, a ServiceMonitor for Prometheus scraping, KEDA or Prometheus Adapter, an HPA or ScaledObject, PVC-backed model storage for fast startup, and a PodDisruptionBudget for safe scale-down.

The Bottom Line

Do not scale GPU pods on CPU or GPU utilization. Scale on queue depth. Use KEDA for simplicity or Prometheus Adapter for flexibility.

The hard part is not the autoscaling configuration. It is the model loading time on scale-up. Solve that with PVCs and shared storage first. Then add autoscaling on top.

Scale up fast (within 1 to 3 minutes with cached models). Scale down slowly (stabilization windows and one pod at a time). Never scale to zero in production unless you can tolerate cold start latency.

Next week: Resource Requests and Limits for GPU Workloads.

If you are building inference infrastructure on Kubernetes, I cover model serving, GPU scheduling, and autoscaling every week. Subscribe at kubenatives.com.

If you are running 1.28 and the current version is 1.32, you are 4 versions behind. That is 3 sequential upgrades to catch up. Each one can break things.

Most teams put off upgrades because they are scared. The upgrade process is poorly documented for production environments. The official docs cover the happy path. They do not cover what happens when something goes wrong mid-upgrade.

This article covers the full upgrade strategy: planning, pre-flight checks, the upgrade itself, and what to do when things break.

Understanding the Upgrade Path

Kubernetes does not support skipping minor versions. You must upgrade one minor version at a time.

1.28 → 1.29 → 1.30 → 1.31 → 1.32

You cannot go directly from 1.28 to 1.32. Each hop requires a full upgrade cycle: control plane first, then worker nodes.

Within each minor version, you can jump patch versions freely. Going from 1.30.2 to 1.30.8 is safe and does not require the full upgrade procedure. Just update the kubelet and kubectl binaries.

The support window:

Version    Released     End of Support
1.30       Apr 2024     Jun 2025
1.31       Aug 2024     Oct 2025
1.32       Dec 2024     Feb 2026
1.33       Apr 2025     Jun 2026

If you are more than 2 versions behind the current release, prioritize upgrading. You are running on borrowed time.

Pre-Upgrade Checklist

Run these checks before every upgrade. Do not skip them. They catch 90% of upgrade failures before they happen.

Check 1: Read the changelog for breaking changes

Every Kubernetes release removes deprecated APIs, changes default behaviors, and sometimes breaks addons.

# Check for deprecated APIs in your cluster
# Install kubent (kube-no-trouble)
kubectl krew install deprecations

# Or use the standalone tool
kubent

kubent scans your cluster for resources using deprecated or removed APIs. If it finds any, update those resources BEFORE upgrading. An API that was deprecated in 1.30 might be removed in 1.32. Your manifests will fail to apply after the upgrade.

Check 2: Verify addon compatibility

Your CNI plugin (Calico, Cilium, Flannel), CSI drivers, ingress controller, and cert-manager all have Kubernetes version requirements. Check each one against the target version.

# Check current versions of critical addons
kubectl get pods -n kube-system -o custom-columns=\
  NAME:.metadata.name,\
  IMAGE:.spec.containers[0].image

If your CNI plugin does not support the target Kubernetes version, upgrade the CNI first.

Check 3: Back up etcd

This is non-negotiable. If the upgrade fails catastrophically, the etcd backup is your recovery path.

etcdctl snapshot save /var/backups/etcd/pre-upgrade-$(date +%Y%m%d).db \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

# Verify the snapshot
etcdctl snapshot status /var/backups/etcd/pre-upgrade-*.db --write-out=table

Copy the backup off the control plane node. If the node dies during upgrade, the backup on the node is useless.

Check 4: Verify PodDisruptionBudgets

PDBs control how many pods can be unavailable during node drains. If a PDB prevents draining, the upgrade stalls.

# List all PDBs
kubectl get pdb --all-namespaces

# Check for PDBs that might block drains
kubectl get pdb --all-namespaces -o jsonpath='{range .items[*]}{.metadata.namespace}/{.metadata.name}: minAvailable={.spec.minAvailable} maxUnavailable={.spec.maxUnavailable}{"\n"}{end}'

A PDB with minAvailable equal to the current replica count blocks all drains. Either increase replicas or temporarily relax the PDB during upgrades.

Check 5: Dry run the upgrade

# On the first control plane node
sudo kubeadm upgrade plan

This shows exactly what will be upgraded and flags any issues. If it reports errors, fix them before proceeding.

Upgrading the Control Plane

The control plane upgrades one node at a time. Never upgrade all control plane nodes simultaneously.

Step 1: Upgrade kubeadm on the first control plane node

# Update the package repository
sudo apt-get update

# Install the target version of kubeadm
sudo apt-get install -y kubeadm=1.31.0-1.1

# Verify
kubeadm version

Step 2: Apply the upgrade

# On the FIRST control plane node only
sudo kubeadm upgrade apply v1.31.0

This upgrades the API server, controller manager, scheduler, and kube-proxy on this node. etcd is upgraded if it is managed by kubeadm (stacked topology).

Expected output:

[upgrade/successful] SUCCESS! Your cluster was upgraded to "v1.31.0". Enjoy!

If it fails, do NOT proceed. Check the error. Common failures: etcd health check fails (fix etcd first), certificate issues (renew with kubeadm certs renew all), or insufficient disk space.

Step 3: Upgrade kubelet and kubectl on the first node

sudo apt-get install -y kubelet=1.31.0-1.1 kubectl=1.31.0-1.1
sudo systemctl daemon-reload
sudo systemctl restart kubelet

Step 4: Verify the first node

kubectl get nodes
# The upgraded node should show v1.31.0
# Other nodes still show the old version - this is expected

Step 5: Upgrade remaining control plane nodes

On each additional control plane node:

sudo apt-get install -y kubeadm=1.31.0-1.1

# Note: use "upgrade node" not "upgrade apply" for subsequent nodes
sudo kubeadm upgrade node

sudo apt-get install -y kubelet=1.31.0-1.1 kubectl=1.31.0-1.1
sudo systemctl daemon-reload
sudo systemctl restart kubelet

Wait for each node to show Ready before moving to the next.

Upgrading Worker Nodes

Worker nodes upgrade one at a time (or in batches if you have capacity).

Step 1: Drain the node

kubectl drain <node-name> \
  --ignore-daemonsets \
  --delete-emptydir-data \
  --grace-period=120 \
  --timeout=300s

This evicts all pods from the node. DaemonSets stay (they run on every node). Pods with emptyDir volumes lose their data.

Step 2: Upgrade kubeadm, kubelet, kubectl

# SSH into the worker node
sudo apt-get update
sudo apt-get install -y kubeadm=1.31.0-1.1
sudo kubeadm upgrade node
sudo apt-get install -y kubelet=1.31.0-1.1 kubectl=1.31.0-1.1
sudo systemctl daemon-reload
sudo systemctl restart kubelet

Step 3: Uncordon the node

kubectl uncordon <node-name>

The node is now schedulable again. Pods will be scheduled back onto it.

Step 4: Verify and move to the next node

kubectl get nodes
# Upgraded node shows v1.31.0 and Ready status

Repeat for each worker node. If you have GPU nodes, upgrade them last. GPU pods take longer to reschedule because of model loading times.

Handling GPU Nodes During Upgrades

GPU nodes need special attention:

The GPU Operator must be compatible with the target Kubernetes version. Check the NVIDIA GPU Operator compatibility matrix before upgrading.

vLLM pods take 1 to 30 minutes to restart (model loading). Plan for this downtime per node. If you have a PVC-backed model cache, restart is faster (1 to 3 minutes instead of 30).

Drain GPU nodes with a longer grace period:

kubectl drain gpu-node-1 \
  --ignore-daemonsets \
  --delete-emptydir-data \
  --grace-period=300 \
  --timeout=600s

After uncordoning, verify the GPU Operator components restart correctly:

kubectl get pods -n gpu-operator -o wide | grep gpu-node-1

All GPU Operator pods on that node should be Running before proceeding to the next GPU node.

Post-Upgrade Validation

Run these checks after all nodes are upgraded:

# 1. All nodes on the new version and Ready
kubectl get nodes

# 2. All system pods healthy
kubectl get pods -n kube-system

# 3. etcd cluster healthy
etcdctl endpoint health --cluster

# 4. DNS working
kubectl run dns-test --image=busybox:1.36 --rm -it --restart=Never -- nslookup kubernetes.default

# 5. Create and delete a test resource
kubectl create configmap upgrade-test --from-literal=version=1.31
kubectl delete configmap upgrade-test

# 6. Check for pods in bad states
kubectl get pods --all-namespaces --field-selector status.phase!=Running,status.phase!=Succeeded

# 7. Verify GPU workloads (if applicable)
kubectl get pods -n inference

Rollback Strategy

If the upgrade fails mid-way:

For control plane: restore from the etcd backup taken in the pre-upgrade checklist. This rolls back the cluster state to before the upgrade.

For worker nodes: the failed node can be drained and reimaged with the old version. Other nodes continue running normally.

The key: always have the etcd backup. Without it, there is no rollback. The pre-upgrade etcd snapshot is your safety net.

The Bottom Line

Kubernetes upgrades are not optional. The support window is 14 months. After that, you are running unpatched software in production.

The process: back up etcd, check deprecated APIs, verify addon compatibility, upgrade control plane one node at a time, upgrade workers one node at a time, validate everything.

Do not skip the pre-flight checks. Do not upgrade all nodes at once. Do not skip the etcd backup. These three mistakes cause 90% of upgrade failures.

Next week: Autoscaling Inference Workloads: HPA and KEDA for GPU Pods.

If you are running self-managed Kubernetes clusters, I cover operations, upgrades, and GPU infrastructure every week. Subscribe at kubenatives.com.

Network policies fix this. They are firewall rules for pod-to-pod traffic. But most teams implement them wrong. They add a restrictive policy, DNS breaks silently, and they spend hours debugging before removing the policy and giving up.

This article covers how to implement network policies correctly. Starting with the one rule that prevents 90% of the problems.

The Default Behavior

With no network policies, Kubernetes networking is fully open. Every pod can reach every other pod on any port. Every pod can reach external services. There are no restrictions.

The moment you create a NetworkPolicy in a namespace, the behavior changes. Pods selected by the policy are now restricted. Traffic not explicitly allowed by a policy is denied.

This is the part that catches people. Adding one policy does not just restrict what that policy covers. It implicitly denies everything else for the selected pods.

# This policy allows port 8080 ingress from the frontend namespace.
# But it ALSO denies all other ingress to these pods.
# AND it denies all egress from these pods (if policyTypes includes Egress).
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-frontend
  namespace: backend
spec:
  podSelector:
    matchLabels:
      app: api-server
  policyTypes:
  - Ingress
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          name: frontend
    ports:
    - protocol: TCP
      port: 8080

Rule Zero: Allow DNS First

This is the single most important rule. Before you create any other network policy, deploy this one in every namespace:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns
  namespace: backend
spec:
  podSelector: {}
  policyTypes:
  - Egress
  egress:
  - to:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: kube-system
    ports:
    - protocol: UDP
      port: 53
    - protocol: TCP
      port: 53

Without this, pods lose DNS resolution the moment you add any egress policy. The failure is silent. No error message. No rejection. Queries are dropped and the application waits for a timeout.

This is the #1 cause of “network policies broke everything.” Deploy the DNS egress rule first. In every namespace. Before anything else.

Building a Zero-Trust Network Step by Step

The safest approach is to start with a default deny policy and then explicitly allow what you need. Here is the order:

Step 1: Default deny all traffic

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: backend
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  - Egress

This blocks all ingress and egress for every pod in the namespace. Nothing can communicate. This is intentionally extreme. You will add allow rules next.

Step 2: Allow DNS egress (Rule Zero)

Deploy the DNS policy from above. Pods can now resolve names but cannot reach anything else.

Step 3: Allow inter-service communication

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: api-server-ingress
  namespace: backend
spec:
  podSelector:
    matchLabels:
      app: api-server
  policyTypes:
  - Ingress
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          name: frontend
      podSelector:
        matchLabels:
          app: web
    ports:
    - protocol: TCP
      port: 8080

This allows the web pod in the frontend namespace to reach the api-server pod in the backend namespace on port 8080. Nothing else can reach the api-server.

Step 4: Allow database access

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: database-ingress
  namespace: backend
spec:
  podSelector:
    matchLabels:
      app: postgres
  policyTypes:
  - Ingress
  ingress:
  - from:
    - podSelector:
        matchLabels:
          app: api-server
    ports:
    - protocol: TCP
      port: 5432

Only the api-server in the same namespace can reach postgres on port 5432. The frontend cannot reach the database directly. A compromised frontend pod cannot access your data.

Step 5: Allow external egress for pods that need it

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: api-server-egress
  namespace: backend
spec:
  podSelector:
    matchLabels:
      app: api-server
  policyTypes:
  - Egress
  egress:
  # DNS
  - to:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: kube-system
    ports:
    - protocol: UDP
      port: 53
    - protocol: TCP
      port: 53
  # Database
  - to:
    - podSelector:
        matchLabels:
          app: postgres
    ports:
    - protocol: TCP
      port: 5432
  # External APIs (HTTPS)
  - to:
    - ipBlock:
        cidr: 0.0.0.0/0
        except:
        - 10.0.0.0/8
        - 172.16.0.0/12
        - 192.168.0.0/16
    ports:
    - protocol: TCP
      port: 443

The api-server can reach DNS, the database, and external HTTPS endpoints. It cannot reach anything else inside the cluster. The ipBlock with except clauses blocks access to other internal services while allowing external API calls.

Common Mistakes

Mistake 1: Forgetting DNS egress. The #1 cause of “network policies broke everything.” Always deploy the DNS allow rule first.

Mistake 2: Using podSelector without namespaceSelector. A podSelector alone only matches pods in the same namespace. To allow traffic from another namespace, you must include a namespaceSelector.

# WRONG: only matches pods in the SAME namespace
- from:
  - podSelector:
      matchLabels:
        app: web

# RIGHT: matches pods in the frontend namespace
- from:
  - namespaceSelector:
      matchLabels:
        name: frontend
    podSelector:
      matchLabels:
        app: web

Mistake 3: AND vs OR logic. When namespaceSelector and podSelector are in the same from entry (same YAML block), they are AND logic. Both must match. When they are separate entries (separate list items), they are OR logic. Either can match.

# AND: must be in frontend namespace AND have app=web label
- from:
  - namespaceSelector:
      matchLabels:
        name: frontend
    podSelector:
      matchLabels:
        app: web

# OR: anything in frontend namespace OR anything with app=web label
- from:
  - namespaceSelector:
      matchLabels:
        name: frontend
  - podSelector:
      matchLabels:
        app: web

The difference is one hyphen. One wrong indent and your policy allows far more traffic than intended. This is the most dangerous mistake in network policies.

Mistake 4: Not labeling namespaces. NetworkPolicies select namespaces by label, not by name. If your namespace does not have a label, the namespaceSelector cannot match it.

# Label your namespaces
kubectl label namespace frontend name=frontend
kubectl label namespace backend name=backend

Kubernetes 1.21+ automatically adds the label kubernetes.io/metadata.name to every namespace. Use that for reliability.

Mistake 5: Forgetting monitoring and logging egress. Your pods need to reach Prometheus (for scraping) and your log aggregator. If you block egress without allowing these, you lose observability.

Testing Network Policies

Never deploy network policies blind. Test them first.

# Deploy a test pod
kubectl run nettest --image=busybox:1.36 -n backend --rm -it --restart=Never -- sh

# Test DNS
nslookup kubernetes.default

# Test service connectivity
wget -qO- --timeout=3 http://api-server:8080/health

# Test external connectivity
wget -qO- --timeout=3 https://httpbin.org/get

Run these tests before and after applying each policy. If something breaks, you know exactly which policy caused it.

The Debug Checklist

When a pod cannot reach a service after network policies are applied:

1. Check if DNS works: nslookup kubernetes.default from inside the pod.

2. If DNS fails: the DNS egress rule is missing.

3. If DNS works but the service is unreachable: the ingress policy on the destination does not allow traffic from the source pod or namespace.

4. Check namespace labels: kubectl get namespace <ns> --show-labels.

5. Check pod labels: kubectl get pod <pod> --show-labels.

6. Verify the policy is selecting the right pods: kubectl get networkpolicies -n <ns> -o yaml.

The Bottom Line

Network policies are simple in concept and dangerous in practice. The implicit deny behavior catches everyone. The AND vs OR selector logic catches even experienced engineers.

Start with Rule Zero (allow DNS). Add default deny. Then explicitly allow each communication path. Test after every policy. Do not deploy all policies at once.

Five policies can secure a namespace. One missing DNS rule can break it.

Next week: Kubernetes Upgrade Strategy: kubeadm Cluster Upgrades Without Downtime.

If you are running production Kubernetes clusters, I cover networking, GPU infrastructure, and operations every week. Subscribe at kubenatives.com.

• Setting up GPU node isolation for the first time

• Adding a new GPU tier to an existing cluster

• Configuring per-team GPU quotas

• Setting up priority classes for GPU workloads

File 1: gpu-node-taints.sh

Apply taints to GPU nodes. Run once per node or set at the node pool level.

#!/bin/bash
# gpu-node-taints.sh
# Apply taints to GPU nodes for workload isolation

set -euo pipefail

echo "=== Tainting Production GPU Nodes (Tier 1) ==="
for node in $(kubectl get nodes -l gpu-tier=production -o jsonpath='{.items[*].metadata.name}'); do
  kubectl taint nodes $node nvidia.com/gpu=present:NoSchedule --overwrite
  kubectl label nodes $node gpu-tier=production --overwrite
  echo "  Tainted: $node"
done

echo ""
echo "=== Tainting Development GPU Nodes (Tier 2) ==="
for node in $(kubectl get nodes -l gpu-tier=development -o jsonpath='{.items[*].metadata.name}'); do
  kubectl taint nodes $node nvidia.com/gpu=present:NoSchedule --overwrite
  kubectl label nodes $node gpu-tier=development --overwrite
  echo "  Tainted: $node"
done

echo ""
echo "=== Verification ==="
echo "Production GPU nodes:"
kubectl get nodes -l gpu-tier=production -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints
echo ""
echo "Development GPU nodes:"
kubectl get nodes -l gpu-tier=development -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints

File 2: priority-classes.yaml

# gpu-priority-classes.yaml
# Three tiers of GPU workload priority

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: gpu-production
value: 1000000
globalDefault: false
preemptionPolicy: PreemptLowerPriority
description: |
  Production GPU inference workloads.
  Highest priority. Will preempt development and batch workloads.
---
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: gpu-development
value: 100000
globalDefault: false
preemptionPolicy: PreemptLowerPriority
description: |
  Development GPU workloads (notebooks, experiments).
  Preempted by production. Will preempt batch workloads.
---
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: gpu-batch
value: 10000
globalDefault: false
preemptionPolicy: Never
description: |
  Batch GPU jobs (training, data processing).
  Lowest priority. Will NOT preempt other workloads.
  Waits for available GPUs.

kubectl apply -f priority-classes.yaml

Read more

Without any configuration, the Kubernetes scheduler treats them all the same. It sees available CPU and memory. It does not distinguish between a $200/month CPU node and a $30K/year GPU node.

A basic nginx pod with 100m CPU and 128Mi memory can land on your H100 node. The scheduler found resources. It scheduled the pod. It did exactly what it was designed to do.

This article covers how to stop that from happening. And how to build a multi-tier GPU cluster where the right workloads land on the right hardware every time.

The Problem: GPUs as Shared Resources

By default, any pod can schedule on any node that has enough CPU and memory. GPU nodes have CPU and memory in addition to GPUs. Non-GPU workloads see the CPU and memory and schedule there.

The result: GPU nodes run a mix of GPU workloads and random CPU workloads. The CPU workloads consume memory and CPU that GPU workloads need. And you are paying GPU pricing for pods that do not use GPUs.

# Check what is running on your GPU nodes
kubectl get pods --all-namespaces -o wide --field-selector spec.nodeName=gpu-node-1

If you see system pods, monitoring agents, log collectors, and random application pods alongside your vLLM deployment, your GPU nodes are not isolated.

Taints: Keep Non-GPU Workloads Off GPU Nodes

A taint on a node tells the scheduler: “Do not place pods here unless they explicitly tolerate this taint.”

# Add a taint to all GPU nodes
kubectl taint nodes gpu-node-1 nvidia.com/gpu=present:NoSchedule
kubectl taint nodes gpu-node-2 nvidia.com/gpu=present:NoSchedule

After this, no pod can schedule on these nodes unless it has a matching toleration.

On managed Kubernetes (EKS, GKE, AKS), you set taints at the node pool level. Every node in the pool gets the taint automatically:

# GKE example: GPU node pool with taint
gcloud container node-pools create gpu-pool \
  --cluster=my-cluster \
  --machine-type=a2-highgpu-1g \
  --accelerator=type=nvidia-tesla-a100,count=1 \
  --num-nodes=3 \
  --node-taints=nvidia.com/gpu=present:NoSchedule

Important: The NVIDIA GPU Operator automatically adds the taint nvidia.com/gpu=present:NoSchedule when it detects GPU hardware. If you are using the GPU Operator, the taints are already there. Check with:

kubectl get nodes -l nvidia.com/gpu.present=true \
  -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.taints}{"\n"}{end}'

Tolerations: Allow GPU Workloads to Schedule

Your GPU pods need a toleration that matches the taint:

spec:
  tolerations:
  - key: nvidia.com/gpu
    operator: Exists
    effect: NoSchedule
  containers:
  - name: vllm
    resources:
      limits:
        nvidia.com/gpu: "1"

The operator: Exists means “tolerate this taint regardless of value.” This is simpler than matching a specific value and works for any GPU taint.

What about system pods? DaemonSets like the GPU Operator, node exporter, and log collectors need to run on GPU nodes too. They should also have the toleration. The GPU Operator DaemonSets already include it. Your monitoring stack may need it added manually.

Node Affinity: Target Specific GPU Types

Taints prevent the wrong pods from landing on GPU nodes. But in a mixed GPU cluster (A100s and T4s), you also need to ensure the right pods land on the right GPU type.

GPU Feature Discovery (part of the GPU Operator) labels each node with its GPU model:

nvidia.com/gpu.product=NVIDIA-A100-SXM4-80GB
nvidia.com/gpu.product=NVIDIA-T4
nvidia.com/gpu.product=NVIDIA-H100-80GB-HBM3

Use node affinity to target specific GPU types:

spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: nvidia.com/gpu.product
            operator: In
            values:
            - NVIDIA-A100-SXM4-80GB
  tolerations:
  - key: nvidia.com/gpu
    operator: Exists
    effect: NoSchedule
  containers:
  - name: vllm
    resources:
      limits:
        nvidia.com/gpu: "1"

This pod will only schedule on A100 nodes. Even if T4 nodes have available GPUs.

The Multi-Tier GPU Cluster Pattern

Here is the pattern I use for production clusters with mixed GPU types:

Each tier has its own taint and label. Workloads use tolerations to enter the GPU tiers and node affinity to target the right tier.

# Production inference: must land on Tier 1
affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
      - matchExpressions:
        - key: gpu-tier
          operator: In
          values:
          - production

# Dev notebook: must land on Tier 2
affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
      - matchExpressions:
        - key: gpu-tier
          operator: In
          values:
          - development

Cost Isolation with Resource Quotas

Taints control where pods run. Resource quotas control how much each team can consume.

apiVersion: v1
kind: ResourceQuota
metadata:
  name: gpu-quota
  namespace: ml-team-a
spec:
  hard:
    requests.nvidia.com/gpu: "4"
    limits.nvidia.com/gpu: "4"
---
apiVersion: v1
kind: ResourceQuota
metadata:
  name: gpu-quota
  namespace: ml-team-b
spec:
  hard:
    requests.nvidia.com/gpu: "2"
    limits.nvidia.com/gpu: "2"

Team A can use up to 4 GPUs. Team B can use up to 2. Neither team can consume more than their quota regardless of what is available in the cluster.

Combine resource quotas with LimitRanges to prevent individual pods from requesting too many GPUs:

apiVersion: v1
kind: LimitRange
metadata:
  name: gpu-limits
  namespace: ml-team-a
spec:
  limits:
  - type: Container
    max:
      nvidia.com/gpu: "2"
    default:
      nvidia.com/gpu: "1"

No single container in team A’s namespace can request more than 2 GPUs. Default is 1 if not specified.

Priority Classes for GPU Workloads

When GPU capacity is scarce, priority classes determine which pods get GPUs first and which get preempted.

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: gpu-production
value: 1000000
globalDefault: false
description: "Production GPU inference workloads"
---
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: gpu-development
value: 100000
globalDefault: false
description: "Development GPU workloads"
---
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: gpu-batch
value: 10000
globalDefault: false
preemptionPolicy: Never
description: "Batch GPU jobs - do not preempt others"

Production inference gets the highest priority. If a production pod needs a GPU and all are allocated, Kubernetes preempts a development or batch pod to make room.

The preemptionPolicy: Never on the batch class means batch jobs will wait for GPUs but will never kick out other workloads.

# Use in pod spec
spec:
  priorityClassName: gpu-production

Common Mistakes

Mistake 1: No taints on GPU nodes. Random CPU workloads consume resources on expensive GPU hardware. Always taint GPU nodes.

Mistake 2: Forgetting tolerations on GPU Operator DaemonSets. If you add custom taints, the GPU Operator pods need matching tolerations. Otherwise the operator cannot run on GPU nodes, which means GPUs are never registered.

Mistake 3: Using nodeSelector instead of nodeAffinity. nodeSelector is simpler but less flexible. You cannot express “schedule on A100 OR H100” with nodeSelector. nodeAffinity supports multiple values and complex expressions.

Mistake 4: No resource quotas. Without quotas, one team can consume all GPUs in the cluster. This is fine with 2 engineers. It is chaos with 10 teams.

Mistake 5: Same priority for all GPU workloads. When capacity is tight, production inference and a Jupyter notebook have the same priority. The notebook should yield to production. Use priority classes.

The Bottom Line

GPU nodes are expensive. Treat them like expensive resources.

Taints keep non-GPU workloads off GPU hardware. Tolerations allow GPU workloads in. Node affinity targets specific GPU types. Resource quotas cap per-team consumption. Priority classes ensure production wins when capacity is scarce.

Five mechanisms. Together they turn a cluster where anything runs anywhere into a multi-tier platform where the right workloads land on the right hardware at the right priority.

Next week: Network Policies in Practice: When Your Pods Cannot Talk to Each Other.

If you are building GPU infrastructure on Kubernetes, I cover scheduling, model serving, and production operations every week. Subscribe at kubenatives.com.

Then someone asks: “How do we roll out a new model version without downtime?” Or: “How do we know if the model’s responses are getting worse?” Or: “Can we test GPT-4o alongside Llama 3 and route traffic based on the use case?”

These are not model serving questions. These are LLMOps questions. And most teams have no framework for answering them.

This article covers the 6 patterns that turn a model deployment into a production LLM system.

Pattern 1: Model Versioning and Rollouts

In traditional software, you version your code. In LLMOps, you version three things: the model weights, the serving configuration, and the prompt templates. All three change independently. All three affect output quality.

The problem. You upgrade from Llama 3.1 8B to Llama 3.1 70B. The model is better. But your prompts were tuned for the 8B version. The 70B version interprets the system prompt differently. Response quality drops even though the model improved.

The pattern. Version the model and the prompt together as a single deployment unit. A “model version” is not just the weights. It is the weights plus the serving config plus the prompt template.

On Kubernetes, this maps to a Deployment revision. Each revision locks in:

# Version 1: Llama 3.1 8B + prompt v1
containers:
- name: vllm
  image: vllm/vllm-openai:v0.6.0
  args:
  - --model
  - meta-llama/Llama-3.1-8B-Instruct
  env:
  - name: SYSTEM_PROMPT_VERSION
    value: "v1"

# Version 2: Llama 3.1 70B + prompt v2
containers:
- name: vllm
  image: vllm/vllm-openai:v0.6.0
  args:
  - --model
  - meta-llama/Llama-3.1-70B-Instruct
  - --tensor-parallel-size
  - "2"
  env:
  - name: SYSTEM_PROMPT_VERSION
    value: "v2"

Rollout strategy. Never switch 100% of traffic at once. Use a canary deployment. Route 5% of traffic to the new version. Monitor quality metrics for 24 hours. If quality holds, increase to 25%, then 50%, then 100%.

KServe handles this natively with traffic splitting on InferenceService revisions. Without KServe, use Istio VirtualService or a gateway that supports weighted routing.

Pattern 2: Prompt Management

Prompts are configuration, not code. But most teams hardcode them in application source files. This means changing a prompt requires a code deployment, a build pipeline, a review cycle, and a rollout.

The pattern. Store prompts in ConfigMaps or an external prompt store. The serving layer reads the prompt at request time, not at build time.

apiVersion: v1
kind: ConfigMap
metadata:
  name: prompt-templates
  namespace: inference
data:
  system-v1.txt: |
    You are a helpful assistant for DevOps engineers.
    Answer questions about Kubernetes, Docker, and cloud infrastructure.
    Be concise. Use code examples when relevant.
  system-v2.txt: |
    You are a senior infrastructure engineer assistant.
    Provide production-ready advice with specific commands.
    Always mention potential risks and rollback steps.

Mount the ConfigMap into the application pod. The application reads the prompt file at request time. To update a prompt, update the ConfigMap. The pods pick up the change without restarting (if using subPath mounts, a restart is needed).

Why this matters. Prompt iteration is fast. Model deployment is slow (minutes to load weights). Decoupling prompts from deployments means you can iterate on prompts in seconds without touching the model.

Pattern 3: LLM Gateway and Routing

Most production systems do not use a single model. They use different models for different tasks. A small fast model for classification. A large model for generation. A specialized model for code.

The pattern. An LLM gateway sits between your application and the model backends. It handles routing, fallback, rate limiting, and load balancing across multiple models.

The gateway routes requests based on metadata in the request: model name, use case tag, user tier, or custom headers. If the primary model is overloaded or down, the gateway falls back to an alternative.

On Kubernetes, LiteLLM is the most common open source LLM gateway. It provides an OpenAI compatible API that proxies to multiple backends.

# LiteLLM config
model_list:
  - model_name: "generation"
    litellm_params:
      model: "hosted_vllm/meta-llama/Llama-3.1-70B-Instruct"
      api_base: "http://vllm-70b.inference:8000/v1"
  - model_name: "generation"
    litellm_params:
      model: "gpt-4o"
      api_key: "os.environ/OPENAI_API_KEY"
    # Fallback: if vLLM is down, route to OpenAI

Why this matters. Without a gateway, your application has hardcoded model endpoints. Switching models requires application changes. With a gateway, you change the routing config. The application never knows.

Pattern 4: Quality Observability

GPU metrics tell you if the model is running. They do not tell you if it is running well.

TTFT, throughput, and cache utilization are infrastructure metrics. They measure serving performance. But a model can serve fast responses that are completely wrong.

The pattern. Add a quality observability layer that tracks response characteristics over time.

Metrics to track:

Response length distribution. A sudden drop in average response length can indicate the model is generating truncated or degenerate responses. Plot a histogram of response token counts. Alert if the distribution shifts.

Refusal rate. How often the model refuses to answer (returns “I cannot help with that” or similar). A spike in refusals after a prompt change indicates the guardrails are too aggressive.

Latency per output token. Not just TTFT. Measure the time per token during decoding. If this increases without load changes, the model may be struggling with certain prompt patterns.

User feedback signals. Thumbs up/down, regenerate clicks, copy events. These are noisy individually but powerful in aggregate. A drop in positive signals after a model change is a quality regression.

# Custom metrics to export alongside vLLM metrics
- name: llm_response_tokens_total
  type: histogram
  help: Distribution of response lengths in tokens
  buckets: [10, 50, 100, 200, 500, 1000, 2000]

- name: llm_refusal_total
  type: counter
  help: Number of refusal responses detected

- name: llm_user_feedback
  type: counter
  labels: [feedback_type]
  help: User feedback signals (positive, negative, regenerate)

Why this matters. You cannot improve what you do not measure. Infrastructure metrics tell you the system is running. Quality metrics tell you it is working.

Pattern 5: Guardrails

Production LLMs need input and output filtering. Without guardrails, a user can prompt to inject the model to ignore its system instructions. Or the model can generate harmful, incorrect, or off-topic content.

The pattern. Two layers of filtering. Input guardrails before the model processes the request. Output guardrails after the model generates a response.

Request → Input Filter → Model → Output Filter → Response

Input filtering catches prompt injection attempts, PII in user messages, and requests that are outside the model’s intended scope.

Output filtering catches harmful content, PII in responses, and responses that contradict known facts (hallucination detection).

On Kubernetes, guardrails run as a sidecar container or a separate microservice in the request path. Running them as a sidecar keeps latency low (no network hop). Running them as a separate service allows independent scaling and updates.

The latency tradeoff. Every guardrail adds latency. Input filtering adds 10 to 50ms. Output filtering can add 50 to 200ms for content classification. For interactive chat applications, this matters. For batch processing, it does not.

The pattern for production: Run lightweight keyword and regex filters in the request path (low latency). Run heavier ML-based content classifiers asynchronously. Flag problematic responses for review rather than blocking them in real time.

Pattern 6: Cost Attribution

GPU infrastructure is expensive. When multiple teams share a model serving platform, you need to know who is using what and how much it costs.

The pattern. Tag every request with a team or project identifier. Aggregate GPU-seconds and token counts per tag. Charge back to teams based on usage.

On Kubernetes, this maps to namespace-level resource quotas for GPU allocation and request-level tagging for usage tracking.

Request headers:
  X-Team: search-team
  X-Project: product-search
  X-Budget-Code: eng-2024-q3

The LLM gateway logs these headers alongside token counts and GPU time. A billing pipeline aggregates usage per team per day.

search-team: 2.4M input tokens, 800K output tokens, 48 GPU-hours
recommendation-team: 1.1M input tokens, 400K output tokens, 22 GPU-hours

Why this matters. Without cost attribution, GPU spend is a shared cost that nobody owns. With attribution, teams optimize their own usage. The team generating 1M tokens per day will find ways to cache, batch, or use smaller models when they see their bill.

Putting It All Together

The 6 patterns form a stack:

Model serving is the foundation. You already have this from the vLLM and Triton articles.

Start with model versioning and prompt management. These are the minimum for operating in production. Without them, every change is a risky manual process.

Add the LLM gateway when you serve multiple models or need fallback routing.

Add guardrails when you serve external users or handle sensitive data.

Add quality observability and cost attribution as the system matures and multiple teams start using it.

Do not build all 6 on day one. Start with the bottom of the stack and work up.

The Bottom Line

Deploying a model is the easy part. The hard part is versioning it, routing traffic to it, knowing if it is working well, keeping it safe, and understanding what it costs.

These 6 patterns are not theoretical. They are the operational layer that turns a vLLM deployment into a production LLM system. Every team running LLMs in production eventually builds all of them. The question is whether you build them intentionally or discover the need at 3 AM.

Start with model versioning and prompt management. Add the rest as your system grows.

Next week: vLLM Model Loading Strategies: PVCs, Init Containers, and Shared Storage.

If you are building LLM infrastructure on Kubernetes, I cover the intersection of GPU infrastructure, model serving, and production operations every week. Subscribe at kubenatives.com.

• Setting up CoreDNS for a new cluster

• Debugging intermittent DNS failures

• Enabling DNS query logging temporarily

• Optimizing DNS performance with caching

Template 1: Production CoreDNS ConfigMap

This replaces the default CoreDNS Corefile with production-ready settings.

apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns
  namespace: kube-system
data:
  Corefile: |
    .:53 {
        errors
        health {
            lameduck 5s
        }
        ready
        kubernetes cluster.local in-addr.arpa ip6.arpa {
            pods insecure
            fallthrough in-addr.arpa ip6.arpa
            ttl 30
        }
        prometheus :9153
        forward . /etc/resolv.conf {
            max_concurrent 1000
        }
        cache 30 {
            success 9984 30
            denial 9984 5
        }
        loop
        reload
        loadbalance
    }

Read more

You spend an hour checking network policies, firewall rules, and pod security settings. Then someone runs nslookup from inside the pod, and DNS does not resolve.

It was DNS. It is always DNS.

But “it is DNS” is not a diagnosis. There are exactly 5 causes of DNS failures in Kubernetes. This article covers all of them with the exact commands to identify each one.

How Kubernetes DNS Works

Before debugging, you need to understand the path a DNS query takes inside a cluster.

When a pod makes a DNS request, here is what happens:

The application calls getaddrinfo() or a similar resolver function. The resolver reads /etc/resolv.conf inside the container. That file points to the CoreDNS Service IP (typically 10.96.0.10). The query goes to CoreDNS. CoreDNS looks up the answer and returns it.

Share

The /etc/resolv.conf inside every pod looks like this:

nameserver 10.96.0.10
search default.svc.cluster.local svc.cluster.local cluster.local
options ndots:5

Three lines. Each one matters. Each one can break.

nameserver is the CoreDNS ClusterIP. If CoreDNS is down, no DNS works.

search is the list of domains Kubernetes appends to short names. When you call my-service, Kubernetes actually tries my-service.default.svc.cluster.local first, then my-service.svc.cluster.local, then my-service.cluster.local, then the bare name.

ndots:5 is the setting that causes the most confusion and the most wasted time in production. More on this below.

Cause 1: CoreDNS Pods Are Not Running

The simplest cause. If CoreDNS is down, nothing resolves.

# Check CoreDNS pods
kubectl get pods -n kube-system -l k8s-app=kube-dns

Every pod should be Running. If any pod is in CrashLoopBackOff, Pending, or Error, that is your DNS problem.

Common reasons CoreDNS pods fail:

CoreDNS ConfigMap has a syntax error. Someone edited the Corefile and introduced a typo. CoreDNS cannot start with an invalid configuration.

# Check the Corefile
kubectl get configmap coredns -n kube-system -o yaml

Resource limits are too low. On large clusters, CoreDNS needs more CPU and memory than the defaults. If it is OOMKilled, DNS fails intermittently under load.

# Check for OOMKilled events
kubectl describe pods -n kube-system -l k8s-app=kube-dns | grep -A5 "Last State"

The node running CoreDNS is unhealthy. CoreDNS runs as a Deployment (usually 2 replicas). If both land on the same node and that node has issues, DNS fails cluster-wide.

The fix: Ensure CoreDNS replicas are spread across nodes with pod anti-affinity. Most managed K8s providers do this by default. Self-managed clusters often miss it.

Cause 2: The ndots Problem

This is the most common DNS performance issue in Kubernetes. It does not cause DNS to fail. It causes DNS to be slow.

The ndots:5 setting in /etc/resolv.conf tells the resolver: “If the name has fewer than 5 dots, append the search domains before trying the name as-is.”

When your application calls api.stripe.com, the resolver counts the dots. Two dots. Fewer than 5. So it tries:

1. api.stripe.com.default.svc.cluster.local  → NXDOMAIN
2. api.stripe.com.svc.cluster.local          → NXDOMAIN
3. api.stripe.com.cluster.local              → NXDOMAIN
4. api.stripe.com                            → SUCCESS

Four DNS queries for one lookup. The first three always fail. Each failure takes time. On a busy cluster with thousands of pods, this multiplies into millions of unnecessary DNS queries per hour.

The 5-second timeout. Each failed query has a timeout. With the default timeout of 5 seconds and multiple search domains, a single external DNS lookup can take 15 to 20 seconds in the worst case. This is the “5-second timeout” that shows up in application latency and makes engineers think the network is slow.

The fix:

Option 1: Use fully qualified domain names (FQDNs) with a trailing dot. api.stripe.com. (note the dot at the end) tells the resolver “this is a complete name, do not append search domains.” The resolver sends one query instead of four.

Option 2: Lower ndots in your pod spec for pods that make many external DNS calls:

spec:
  dnsConfig:
    options:
    - name: ndots
      value: "2"

With ndots:2, names with 2 or more dots (like api.stripe.com) are tried as-is first. Internal service names (like my-service) still get the search domain treatment because they have 0 dots.

Option 3: Use a node-level DNS cache (NodeLocal DNSCache). This caches responses locally on each node, eliminating the network hop to CoreDNS for repeated queries. It also handles negative caching, so the failed search domain lookups resolve instantly from cache.

# Check if NodeLocal DNSCache is running
kubectl get pods -n kube-system -l k8s-app=node-local-dns

Cause 3: Service Has No Endpoints

DNS resolves the Service name correctly. But the Service has no healthy backends. The connection still fails.

This looks like a DNS problem because curl my-service:8080 times out. But DNS is working fine. The Service just has nothing to route to.

# Check if the Service has endpoints
kubectl get endpoints my-service

# Expected: at least one IP:port listed
# If empty: no pods match the Service selector

If the endpoints list is empty:

The Service selector does not match any pod labels. This is the most common cause. A typo in the selector or the pod labels.

# Compare Service selector with pod labels
kubectl get svc my-service -o jsonpath='{.spec.selector}'
kubectl get pods -l app=my-service

The pods exist but are not Ready. If the readiness probe is failing, Kubernetes removes the pod from the endpoints list. The pod is running but not receiving traffic.

# Check pod readiness
kubectl get pods -l app=my-service -o wide
# Look for 0/1 in the READY column

Cause 4: DNS Policy Misconfiguration

Every pod has a dnsPolicy setting. The default is ClusterFirst, which means “use CoreDNS for everything.” But if someone sets it to the wrong value, DNS breaks.

The four DNS policies:

ClusterFirst (default): Uses CoreDNS. Internal names resolve to cluster services. External names get forwarded to upstream DNS. This is what you want 99% of the time.

Default: Uses the node’s DNS configuration, not CoreDNS. Internal service names do not resolve. This is almost never what you want in a cluster.

None: No DNS configuration at all. You must provide everything in dnsConfig. Used for very specific edge cases.

ClusterFirstWithHostNet: For pods running with hostNetwork: true. Uses CoreDNS but falls back to the node’s DNS if CoreDNS does not respond.

The most common mistake: setting dnsPolicy: Default, thinking it means “use the default Kubernetes DNS.” It does not. It means “use the node’s DNS, skip CoreDNS entirely.” Internal service names stop resolving.

# Check a pod's DNS policy
kubectl get pod my-pod -o jsonpath='{.spec.dnsPolicy}'

If a pod can resolve external names (google.com) but not internal names (my-service.default.svc.cluster.local), check the DNS policy first. It is probably set to Default instead of ClusterFirst.

Cause 5: Network Policy Blocking DNS

If you have NetworkPolicies in your cluster, they might block DNS traffic. CoreDNS runs on port 53 (UDP and TCP). If your network policy does not explicitly allow egress to port 53, DNS queries are silently dropped.

# Check for network policies in the pod's namespace
kubectl get networkpolicies -n <namespace>

If network policies exist, verify they allow DNS egress:

# NetworkPolicy that allows DNS
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns
spec:
  podSelector: {}
  policyTypes:
  - Egress
  egress:
  - to:
    - namespaceSelector: {}
    ports:
    - protocol: UDP
      port: 53
    - protocol: TCP
      port: 53

The tricky part: DNS failures from network policies are silent. The query is dropped. The application waits for a timeout. There is no error message saying “blocked by policy.” It just looks like DNS is slow or unresponsive.

How to test: Exec into the pod and run a manual DNS query:

kubectl exec -it my-pod -- nslookup kubernetes.default

If this times out but CoreDNS pods are healthy, a network policy is likely blocking the traffic.

The 3-Minute Debug Script

Run this script when DNS is broken. It checks all 5 causes in order.

#!/bin/bash
echo "=============================="
echo "Kubernetes DNS Debug"
echo "=============================="

echo ""
echo "=== 1. CoreDNS Pod Status ==="
kubectl get pods -n kube-system -l k8s-app=kube-dns -o wide

echo ""
echo "=== 2. CoreDNS Service ==="
kubectl get svc -n kube-system kube-dns

echo ""
echo "=== 3. CoreDNS Endpoints ==="
kubectl get endpoints -n kube-system kube-dns

echo ""
echo "=== 4. CoreDNS ConfigMap ==="
kubectl get configmap coredns -n kube-system -o jsonpath='{.data.Corefile}' 2>/dev/null
echo ""

echo ""
echo "=== 5. DNS Resolution Test ==="
kubectl run dns-test --image=busybox:1.36 --rm -it --restart=Never -- \
  sh -c "nslookup kubernetes.default && echo 'Internal DNS: OK' || echo 'Internal DNS: FAILED'"

echo ""
echo "=== 6. External DNS Test ==="
kubectl run dns-test-ext --image=busybox:1.36 --rm -it --restart=Never -- \
  sh -c "nslookup google.com && echo 'External DNS: OK' || echo 'External DNS: FAILED'"

echo ""
echo "=== 7. Network Policies ==="
kubectl get networkpolicies --all-namespaces --no-headers 2>/dev/null | wc -l
echo "network policies found in the cluster"

echo ""
echo "=============================="
echo "Debug complete."
echo "=============================="

Reading the results:

Internal DNS fails + External DNS fails = CoreDNS is down or unreachable. Check cause 1 and cause 5.

Internal DNS works + External DNS fails = CoreDNS upstream forwarding is broken. Check the Corefile forward directive.

Both work but application is slow = The ndots problem. Check cause 2.

DNS works from debug pod but not from application pod = DNS policy or network policy issue specific to that pod. Check causes 4 and 5.

The resolv.conf Cheat Sheet

Every DNS issue starts with what is in /etc/resolv.conf inside the pod:

kubectl exec my-pod -- cat /etc/resolv.conf

What to look for:

The nameserver should be the CoreDNS ClusterIP (usually 10.96.0.10). If it is a different IP, check the pod’s DNS policy.

The search domains should include <namespace>.svc.cluster.local. If they are missing, the pod cannot resolve short service names.

The ndots value controls how many dots trigger the search domain behavior. Default is 5. Lower it if external DNS is slow.

The Bottom Line

Five causes. Five debug steps. The script checks all of them in 3 minutes.

When DNS breaks: check CoreDNS pods first. If they are healthy, check endpoints. If endpoints exist, check ndots. If ndots is fine, check DNS policy. If the policy is correct, check network policies.

Do not start with tcpdump. Do not start with Wireshark. Start with the 5 causes in order. The answer is almost always in the first three.

Next week: LLMOps on Kubernetes: Patterns for Running LLMs in Production.

If you are running production Kubernetes clusters, I cover control plane internals, GPU infrastructure, and debugging every week. Subscribe at kubenatives.com.

Today I’m launching DevOpsBeast, a course platform for senior DevOps engineers preparing for interviews at FAANG and other top tier tech companies. It teaches the design first reasoning that real interviews test: Kubernetes architecture, GPU infrastructure, LLM operations, security, performance tuning, and identity systems.

This has been quietly building in the background for the past few months, and I want to walk you through what’s inside, why I built it, and what’s coming next.

Why I Built This

Most DevOps interview prep is broken.

You either get tutorial videos that teach you tools without context, or you get LeetCode-style problem sets that have nothing to do with what interviewers actually ask. Neither prepares you for the moment when a senior engineer says, “Your cluster is slow. What do you do?”

That question doesn’t have the right answer. It has the right approach. And the difference between candidates who get hired at FAANG-level companies and candidates who don’t is whether they can demonstrate that approach in 45 minutes of high-pressure design conversation.

So I built DevOpsBeast.

It’s a course platform focused on one specific thing: teaching the design-first reasoning that senior DevOps interviews actually test.

What’s Inside

Every course follows the same format. Realistic scenarios, architecture design, technical reasoning, trade-off analysis, and the actual interview questions that come from these scenarios at companies like Atlassian, Netflix, Stripe, and the FAANG names you’d expect.

Here’s what’s live or in active development:

Production GPU Infrastructure on Kubernetes. Running GPU workloads at scale. Driver management, MIG, time slicing, multi tenancy, cost optimization, and the architecture decisions that separate a working setup from a production one. This is the course that’s most relevant if you’re moving into AI/ML platform work.

LLM Operations for MLOps Engineers. 30 essential LLM concepts taught through the lens of operating them at scale. Inference serving, RAG architectures, agent infrastructure, hallucination detection, cost engineering. Not how to train models. How to deploy, monitor, and defend them in production.

Kubernetes Security. 40 lessons covering the full attack surface and defense layers. The Kubernetes API, RBAC, STRIDE threat modeling, Pod Security Admission, network policies, runtime detection, supply chain security, incident response. Every lesson starts with a breach scenario and walks through both the attack and the defense.

Kubernetes Performance Optimization. The course I wish I’d had when I first hit “the cluster is slow” and didn’t know where to look. Control plane tuning, etcd performance, scheduler throughput, resource right sizing, CPU throttling, network and storage performance, autoscaling deep dives, and dedicated optimization for EKS, GKE, and AKS.

Free Courses (Linux Fundamentals plus Networking Fundamentals). These are the prerequisites for everything else. Free, no email required, available right now.

Why This Matters Now

DevOps interviews have shifted in the last two years.

Companies aren’t asking “do you know how to write a Dockerfile” anymore. They’re asking “design a multi tenant Kubernetes platform that supports 200 teams with proper isolation, cost attribution, and security boundaries.” They’re asking “your inference latency is 3 seconds and the team needs it under 500ms, diagnose and fix.” They’re asking “we just had a security breach in our CI/CD pipeline, walk me through your incident response.”

These are senior staff and principal level questions. And the engineers who answer them well aren’t the ones who memorized more tools. They’re the ones who can reason through novel scenarios using frameworks they’ve internalized.

That’s what I’m trying to teach.

The Companion Resources

I’ve also been writing a blog at devopsbeast.com/blog covering debugging scenarios that don’t fit neatly into a course. The latest one is about Kubernetes certificate expiry, the silent killer that takes down production at 2 AM with no warning. If you’ve never been hit by it, bookmark that post before you do.

And I’m posting interview-related content on LinkedIn most days. Real questions, model answers, common misconceptions. If that’s useful, follow me there.

What’s Next

Over the next few months, I’ll be filling in lessons, adding new courses (security engineering deep dives, observability for distributed systems, platform engineering interview prep), and publishing more of these debugging blog posts.

If you’re preparing for a senior DevOps role or just want to deepen your design-first thinking, head over to devopsbeast.com and explore.

Thanks for reading. As always, hit reply if you have questions, feedback, or specific topics you want me to cover.

Sharon

P.S. The free courses (Linux Fundamentals, Networking Fundamentals) are genuinely free and don’t require an email signup. They’re meant to be useful on their own, even if you never look at the paid stuff. Start there if you want to see how I teach.

Read devopsbeast blog here

0/12 nodes are available: 12 Insufficient nvidia.com/gpu.

You have 12 GPU nodes. nvidia-smi works on all of them. The GPUs are physically there.

So why does Kubernetes think there are no GPUs available?

There are exactly 7 reasons this happens. This article covers all of them in order of likelihood. Work through them top to bottom. You will find the root cause in under 5 minutes

Reason 1: All GPUs Are Already Allocated

This is the most common cause. And the most misunderstood.

Kubernetes treats GPUs as integers. When a pod requests nvidia.com/gpu: 1, it gets an entire physical GPU. There is no fractional allocation. A pod using 8GB of an 80GB A100 still consumes 1 full GPU from the allocatable pool.

Check the actual allocation:

# Show allocatable vs allocated GPUs on each node
kubectl describe nodes | grep -A5 "Allocated resources" | grep -B1 "nvidia.com/gpu"

If allocated equals allocatable on every node, you do not have a scheduling bug. You have a capacity problem.

The fix:

Add more GPU nodes. Or enable GPU sharing (MIG, Time-Slicing, or MPS) to run multiple workloads per physical GPU. We covered all three sharing strategies in detail in the MIG vs Time-Slicing vs MPS article.

Quick capacity check:

# Total GPUs in the cluster
kubectl get nodes -l nvidia.com/gpu.present=true \
  -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.allocatable.nvidia\.com/gpu}{"\n"}{end}'

Reason 2: The GPU Operator Is Not Healthy

This is the second most common cause. The NVIDIA GPU Operator has 8 components that form a dependency chain. If any component fails, GPUs do not show up as allocatable resources.

# Check GPU Operator pod status
kubectl get pods -n gpu-operator

Every pod should be Running or Completed. If any pod is in CrashLoopBackOff, Init, or Error, that is your problem.

The dependency chain: NFD → Driver → Container Toolkit → Device Plugin → GFD → DCGM → MIG Manager → Validator.

The first unhealthy pod in this chain is your root cause. Everything below it is a symptom.

Common failures:

Driver pod crashing. The nouveau kernel module conflicts with the NVIDIA driver. Or the driver container cannot compile kernel modules for your host kernel version. On managed Kubernetes (EKS, GKE, AKS), the platform pre-installs drivers. Set driver.enabled=false in the GPU Operator ClusterPolicy.

Device plugin not running. It depends on the container toolkit. If the toolkit did not configure the runtime correctly, the device plugin cannot register GPUs. Fix the toolkit first.

Validator stuck in Init:0/4. Do not debug the validator. It is reporting that something upstream failed. Look up the chain.

We covered all 8 components in detail in the NVIDIA GPU Operator article.

Reason 3: Node Labels Are Missing

The GPU Operator uses node labels to decide where to deploy its components. If the labels are missing, the operator has no targets.

# Check for NVIDIA PCI device labels (set by NFD)
kubectl get nodes -l feature.node.kubernetes.io/pci-10de.present=true

If this returns nothing, Node Feature Discovery is not running. Without it, the GPU Operator does not know which nodes have GPUs.

# Check NFD pods
kubectl get pods -n gpu-operator -l app.kubernetes.io/component=worker

Also check for GPU Feature Discovery labels:

# Check GPU-specific labels
kubectl get node <gpu-node> -o json | \
  jq '.metadata.labels | with_entries(select(.key | startswith("nvidia.com")))'

If you see no nvidia.com/gpu.product or nvidia.com/gpu.count labels, GFD is not running or not healthy.

Reason 4: Taints and Tolerations Mismatch

GPU nodes often have taints to prevent non-GPU workloads from being scheduled on them. If your GPU pod does not have the matching toleration, the scheduler rejects it.

# Check taints on GPU nodes
kubectl get nodes -l nvidia.com/gpu.present=true \
  -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.taints}{"\n"}{end}'

Common GPU node taints:

nvidia.com/gpu=present:NoSchedule

Your pod spec needs the matching toleration:

tolerations:
- key: nvidia.com/gpu
  operator: Exists
  effect: NoSchedule

The describe pod output will tell you if taints are the issue:

kubectl describe pod <pending-pod>
# Look for: "0/12 nodes are available: 12 node(s) had untolerated taint"

If you see “untolerated taint” in the Events section, add the toleration to your pod spec.

Reason 5: Node Affinity Mismatch

If your pod requests a specific GPU type using node affinity, and no nodes match, the pod stays Pending.

# Example: pod requires H100 but cluster only has A100s
affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
      - matchExpressions:
        - key: nvidia.com/gpu.product
          operator: In
          values:
          - NVIDIA-H100-80GB-HBM3

Check what GPU types actually exist in your cluster:

# List all GPU types across nodes
kubectl get nodes -l nvidia.com/gpu.present=true \
  -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.metadata.labels.nvidia\.com/gpu\.product}{"\n"}{end}'

The describe pod output will show:

0/12 nodes are available: 12 node(s) didn't match Pod's node affinity/selector

The fix: Either change the node affinity to match your actual GPU types, or add nodes with the requested GPU type.

Reason 6: Resource Requests Exceed Node Capacity

Your pod requests more GPUs than any single node has. Or the pod requests a combination of GPU, CPU, and memory that no node can satisfy.

# Check your pod's resource requests
kubectl get pod <pending-pod> -o jsonpath='{.spec.containers[*].resources}'

# Check what's available on each GPU node
kubectl describe nodes | grep -A15 "Allocated resources"

Common scenarios:

A pod requests nvidia.com/gpu: 4 but your nodes only have 2 GPUs each. No single node can satisfy the request.

A pod requests nvidia.com/gpu: 1 and memory: 256Gi but GPU nodes only have 128Gi of RAM. The GPU is available but the memory is not.

A pod with tensor parallelism requests 8 GPUs. You have 8 GPUs across 4 nodes (2 each). Tensor parallelism requires all GPUs on the same node. No node has 8.

The fix: Reduce the resource requests, add larger nodes, or use a different parallelism strategy.

Reason 7: MIG Configuration Mismatch

If MIG is enabled on your GPUs, the resource names change. Instead of nvidia.com/gpu, MIG instances are advertised as specific profile resources:

nvidia.com/mig-1g.10gb
nvidia.com/mig-2g.20gb
nvidia.com/mig-3g.40gb
nvidia.com/mig-7g.80gb

A pod requesting nvidia.com/gpu: 1 will not match a MIG-enabled node. The node no longer advertises nvidia.com/gpu. It advertises the MIG profile resources instead.

# Check what GPU resources the node advertises
kubectl describe node <gpu-node> | grep nvidia

If you see nvidia.com/mig-* resources instead of nvidia.com/gpu, your pod needs to request the specific MIG profile:

resources:
  limits:
    nvidia.com/mig-3g.40gb: 1

The describe pod output is not always clear about this. It will say “Insufficient nvidia.com/gpu” even though the real issue is that the resource name has changed because MIG is enabled.

The 5-Minute Debug Script

Save this script. Run it first every time a GPU pod is Pending.

#!/bin/bash
echo "=============================="
echo "GPU Pod Pending Debug Script"
echo "=============================="

echo ""
echo "=== 1. Pending GPU Pods ==="
kubectl get pods --all-namespaces --field-selector=status.phase=Pending \
  -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,NODE:.spec.nodeName | head -20

echo ""
echo "=== 2. GPU Operator Health ==="
kubectl get pods -n gpu-operator --no-headers | awk '{print $1, $3}' | grep -v Running | grep -v Completed
NOT_RUNNING=$(kubectl get pods -n gpu-operator --no-headers | awk '{print $3}' | grep -v Running | grep -v Completed | wc -l)
if [ "$NOT_RUNNING" -eq 0 ]; then
  echo "All GPU Operator pods healthy."
else
  echo "WARNING: $NOT_RUNNING GPU Operator pods are NOT healthy."
fi

echo ""
echo "=== 3. GPU Node Labels ==="
kubectl get nodes -l feature.node.kubernetes.io/pci-10de.present=true \
  -o custom-columns=NAME:.metadata.name,GPU:.metadata.labels.nvidia\\.com/gpu\\.product,COUNT:.metadata.labels.nvidia\\.com/gpu\\.count 2>/dev/null
NODE_COUNT=$(kubectl get nodes -l feature.node.kubernetes.io/pci-10de.present=true --no-headers 2>/dev/null | wc -l)
if [ "$NODE_COUNT" -eq 0 ]; then
  echo "WARNING: No nodes with GPU labels found. NFD may not be running."
fi

echo ""
echo "=== 4. GPU Allocation ==="
for node in $(kubectl get nodes -l feature.node.kubernetes.io/pci-10de.present=true -o name 2>/dev/null); do
  echo "--- $node ---"
  kubectl describe $node | grep -A3 "nvidia.com"
done

echo ""
echo "=== 5. GPU Node Taints ==="
kubectl get nodes -l feature.node.kubernetes.io/pci-10de.present=true \
  -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints 2>/dev/null

echo ""
echo "=== 6. Pending Pod Events ==="
PENDING_POD=$(kubectl get pods --all-namespaces --field-selector=status.phase=Pending -o jsonpath='{.items[0].metadata.name}' 2>/dev/null)
PENDING_NS=$(kubectl get pods --all-namespaces --field-selector=status.phase=Pending -o jsonpath='{.items[0].metadata.namespace}' 2>/dev/null)
if [ -n "$PENDING_POD" ]; then
  echo "Events for $PENDING_NS/$PENDING_POD:"
  kubectl describe pod $PENDING_POD -n $PENDING_NS | tail -20
else
  echo "No pending pods found."
fi

echo ""
echo "=============================="
echo "Debug complete."
echo "=============================="

This script checks all 7 reasons in order. In 30 seconds you know whether the problem is capacity, the GPU Operator, labels, taints, affinity, resources, or MIG configuration.

The Decision Tree

When a GPU pod is Pending, follow this exact order:

Step 1. Read the Events section in kubectl describe pod. It tells you the scheduler’s exact reason for rejection.

Step 2. If “Insufficient nvidia.com/gpu”: check allocation. Are all GPUs already in use?

Step 3. If GPUs show 0 allocatable: check the GPU Operator. kubectl get pods -n gpu-operator.

Step 4. If GPU Operator pods are healthy but GPUs are not allocatable: check node labels. Is NFD running?

Step 5. If “untolerated taint”: add the toleration to your pod spec.

Step 6. If “node affinity/selector”: check what GPU types actually exist vs what the pod requests.

Step 7. If MIG is enabled: check that the pod requests the MIG profile resource, not nvidia.com/gpu.

Start at Step 1. The Events section narrows the search immediately. Do not skip it.

The Bottom Line

GPU pods get stuck in Pending for 7 reasons. 6 of them are configuration issues, not hardware problems.

Read the Events section first. Run the debug script second. The root cause is almost always visible within 30 seconds.

The hardest part is not finding the problem. It is resisting the urge to blame the scheduler when the answer is sitting in kubectl describe pod.

Next week: Kubernetes DNS Troubleshooting: CoreDNS, ndots, and the 5-Second Timeout.

If you are building GPU infrastructure on Kubernetes, I cover this intersection every week. Subscribe at kubenatives.com.

It is not about redundancy. You could have redundancy with 2 nodes. It is about quorum. And quorum is the reason your cluster stays consistent when things fail.

This article explains why 3, what happens with 2, 4, and 5, and the exact failure scenarios you need to plan for.

The Quorum Formula

etcd uses the Raft consensus algorithm. Every write must be acknowledged by a majority of members before it is committed.

The formula:

Quorum = (N / 2) + 1

3 nodes → quorum = 2 → tolerates 1 failure
5 nodes → quorum = 3 → tolerates 2 failures
7 nodes → quorum = 4 → tolerates 3 failures

The general rule: a cluster of N nodes can tolerate (N - 1) / 2 failures.

This is why 3 is the minimum for HA. With 3 nodes and a quorum of 2, you can lose 1 node and the cluster keeps accepting writes. With 2 nodes, the quorum is also 2. Lose 1 and you lose quorum. The cluster goes read only.

2 nodes is worse than 1 node for write availability. With 1 node, there is no consensus requirement. Writes always succeed (until that node dies). With 2 nodes, both must be healthy for writes to succeed. You added hardware but reduced availability.

What Happens When You Lose Quorum

When etcd loses quorum, it enters a read only state. The API server can still read existing state (pods, services, configurations). But it cannot write.

This means:

No new pods can be created. No existing pods can be modified. No Deployments can be scaled. No ConfigMaps can be updated. No new nodes can join. No Secrets can be created or rotated.

Existing workloads continue running. The kubelet on each node keeps running its containers. Health checks continue. But nothing can change.

If a running pod crashes during a quorum loss, it will not be restarted by a controller because the controller cannot write the new pod spec to etcd. The kubelet will try to restart the container locally based on the restartPolicy, but the Deployment controller cannot create a replacement.

This is why quorum loss is a critical incident. The cluster looks alive but is frozen.

Why 3, Not 4

4 nodes seems like an improvement over 3. More hardware, more redundancy. But the math tells a different story.

4 nodes tolerates the same number of failures as 3. You added a node but gained zero additional fault tolerance. You did add more write latency though, because every write now needs 3 acknowledgments instead of 2.

This is why production clusters use odd numbers: 3, 5, or 7. Even numbers add cost and latency without improving fault tolerance.

When 5 makes sense. If losing a single node keeps you up at night because maintenance windows overlap with failures, go to 5. A 5 node cluster tolerates 2 simultaneous failures. This means you can take 1 node down for maintenance and still survive an unexpected failure.

For most production clusters under 500 nodes, 3 is the right answer. The cost and operational complexity of 5 etcd nodes is only justified when the blast radius of quorum loss is extremely high.

Split-Brain: The Scenario Everyone Fears

Split-brain happens when a network partition divides the cluster into two groups, and both groups think they are the active cluster.

In traditional systems without consensus, this is catastrophic. Both sides accept writes. When the partition heals, you have conflicting state and no way to automatically reconcile.

Raft prevents this by design. Here is what actually happens with 3 nodes:

Scenario: Network partition isolates Node A from Nodes B and C.

Node A is alone. It has 1 out of 3 members. It cannot form a quorum (needs 2). It stops accepting writes. It becomes read-only.

Nodes B and C have 2 out of 3 members. They form a quorum. They elect a new leader (if A was the leader). Writing continues normally.

When the partition heals, Node A rejoins and catches up on all the writes it missed. No conflicting state. No data loss.

The key insight: Raft makes split-brain impossible as long as you have an odd number of nodes. The minority side always fails to reach a quorum. The majority side always succeeds. There is never ambiguity about which side is authoritative.

With an even number (4 nodes), a network partition could create a 2-2 split. Neither side has a quorum. Both sides go read-only. This is another reason odd numbers are better.

The 3 Node Architecture

The load balancer distributes API requests across all 3 API servers. The API servers are stateless. Anyone can handle any request.

etcd runs on all 3 nodes (stacked topology). One etcd member is the leader. Writes go to the leader and are replicated to followers.

The scheduler and controller manager use leader election. Only one instance is active at a time. If the active instance dies, another takes over within seconds.

The load balancer is critical. Without it, kubectl and the kubelets point at a single API server IP. If that node goes down, nothing can reach the control plane even though 2 healthy nodes are still running.

Use a Layer 4 (TCP) load balancer. Do not use Layer 7 (HTTP). The API server handles its own TLS. Health check endpoint: /healthz on port 6443.

Stacked vs External in HA Context

In the architecture above, etcd runs on the same nodes as the API server. This is stacked topology.

External topology separates etcd onto its own dedicated nodes:

External topology means 6 nodes instead of 3. More cost. More complexity. But etcd gets a dedicated disk and CPU. No resource contention with the API server.

Which one for HA? Both provide quorum protection. The difference is performance under load, not availability.

Stacked is fine for clusters with fewer than 200 nodes. Simple to set up. Kubeadm supports it natively.

External becomes necessary when etcd disk latency degrades because the API server is consuming the same I/O bandwidth. We covered this in detail in the Stacked vs External etcd article.

Failure Scenarios: What Actually Happens

Scenario 1: One node goes down (expected)

Quorum is maintained (2 of 3). A new etcd leader is elected if the failed node was the leader. The scheduler and controller manager fail over if they were active on the failed node. API requests continue through the load balancer to the remaining 2 nodes.

Impact: Brief spike in API latency during leader election (typically under 5 seconds). No service disruption.

Scenario 2: Two nodes go down simultaneously

Quorum is lost (1 of 3). etcd becomes read only. The API server can read state but cannot write. No new pods, deployments, or changes.

Existing workloads keep running. The kubelet on worker nodes continues managing its containers. But nothing can be updated or replaced.

Recovery: Bring at least 1 node back online to restore quorum. etcd will automatically re-form consensus.

Scenario 3: etcd data corruption on one node

The corrupted member falls behind. etcd detects the inconsistency through Raft log verification. The member stops participating in consensus.

Recovery: Remove the corrupted member from the cluster. Provision a new node. Add it as a new etcd member. It will automatically replicate data from the healthy members.

# Remove the bad member
etcdctl member remove MEMBER_ID

# Add a new member
etcdctl member add new-node --peer-urls=https://NEW_IP:2380

# Start etcd on the new node with the --initial-cluster-state=existing flag

Scenario 4: Disk fills up on one node

etcd performance degrades as the disk fills. Write latency increases. If the etcd database hits its storage quota, that member triggers a NOSPACE alarm.

If only one member hits NOSPACE, the cluster continues (2 of 3 are healthy). But you should act immediately because the remaining members are likely on the same trajectory.

Recovery: Follow the NOSPACE runbook (compact, defrag, disarm alarm). Then investigate why disk usage grew and fix the root cause.

Scenario 5: Control plane node scheduled for maintenance

Drain the node’s workloads (if it also runs worker pods). The other 2 nodes maintain quorum. Perform maintenance. Bring the node back.

Important: Never take 2 nodes down for maintenance simultaneously. With 3 nodes, losing 2 means quorum loss. Always verify the first node is healthy and has rejoined the etcd cluster before starting maintenance on the second.

# Verify cluster health before maintenance
etcdctl endpoint health --cluster
etcdctl endpoint status --write-out=table

The Health Checks You Need

Daily automated check

#!/bin/bash
# etcd-health-check.sh

CERTS="--cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key"

echo "=== Cluster Health ==="
ETCDCTL_API=3 etcdctl endpoint health --cluster $CERTS

echo ""
echo "=== Member Status ==="
ETCDCTL_API=3 etcdctl endpoint status --cluster --write-out=table $CERTS

echo ""
echo "=== Active Alarms ==="
ETCDCTL_API=3 etcdctl alarm list $CERTS

Prometheus alerts for quorum

groups:
- name: etcd-quorum
  rules:
  - alert: EtcdMemberDown
    expr: up{job="etcd"} == 0
    for: 2m
    labels:
      severity: critical
    annotations:
      summary: "etcd member {{ $labels.instance }} is down"
      description: "With 3 members, losing 1 means you are one failure from quorum loss."

  - alert: EtcdInsufficientMembers
    expr: count(up{job="etcd"} == 1) < 2
    for: 1m
    labels:
      severity: page
    annotations:
      summary: "etcd cluster has lost quorum"
      description: "Fewer than 2 etcd members are healthy. Cluster is read-only."

The Bottom Line

The number 3 is not arbitrary. It is the minimum required for distributed consensus to work with fault tolerance.

3 nodes, quorum of 2, tolerates 1 failure. Add a load balancer in front. Use odd numbers. Never take 2 nodes down at the same time.

If you understand quorum, you understand why your cluster survives node failures. If you do not, you will learn the hard way at 3 AM.

Next week: Why Your GPU Pods Are Pending: Debugging Kubernetes GPU Scheduling.

If you are running production Kubernetes clusters, I cover control plane internals, GPU infrastructure, and model serving every week. Subscribe at kubenatives.com.

A multi-tenant LLM inference platform serving roughly a dozen production workloads on H100 nodes. Standard setup: vLLM behind KServe, KServe behind Istio, autoscaling driven by request rate and queue depth.

The platform had been stable for months. Then it started OOMing.

Not constantly. Not predictably. Just sometimes, in the early morning hours UTC, one of the vLLM pods would go OOMKilled. Exit code 137. Always between 2 AM and 4 AM. Never the same pod twice in a row. Service recovered within 60 seconds because the deployment had multiple replicas, but the on-call alert woke up an engineer every time.

For the first week we did what every team does. We assumed it was traffic. We pulled the request rate metrics. The 3 AM window was the lowest-traffic period of the day. Less than 5% of peak QPS.

That was the first signal that this was not a normal OOM.

The Easy Explanations That Were Wrong

A week of investigation eliminated the obvious causes one by one.

Hypothesis 1: A specific request was blowing up memory.

We grabbed the request logs from the 5 minutes before each OOM event. No common pattern. Different models, different prompt sizes, different output lengths. Some events had no requests in flight at all when the OOM fired. Ruled out.

Hypothesis 2: Memory leak accumulating until breakpoint.

We graphed container_memory_working_set_bytes for each pod over 24 hours. Memory was stable. Pods that had been running for 30 hours had the same memory footprint as pods that had been running for 2. There was no slow growth. The OOM happened from a stable baseline.

Hypothesis 3: Bad model image with a regression.

Rolled back to the previous vLLM version. OOMs continued at the same rate. Rolled forward. Same. The vLLM version was not the variable.

Hypothesis 4: Noisy neighbor on the host.

Checked DCGM metrics for all GPUs on the same physical node during OOM events. No correlated GPU memory pressure or compute contention. The OOMing pod was on a node where every other GPU workload was idle or low utilization.

Hypothesis 5: Resource limit set too low.

This is where most teams stop. They raise the memory limit by 25% and call it solved. We tried it. The OOMs moved later in the night, then resumed at the new threshold a few days later. Higher limit, same problem, slower recurrence.

That last one was the breakthrough, even though we did not know it yet. The fact that raising the limit only delayed the OOM rather than preventing it meant something was actively growing memory. We just could not see what.

Read more

Pods are stuck. Deployments are failing. API requests are slow. Users are complaining.

You open a terminal and start running commands. kubectl get pods. kubectl describe pod. kubectl logs. You scroll through the output looking for something that stands out.

Twenty minutes later, you’re deep in a rabbit hole, debugging a network policy that has nothing to do with the actual problem.

This is how most engineers debug Kubernetes. Randomly. They start with whatever command comes to mind first and hope to stumble on the root cause.

There is a better way. A systematic framework that works for every Kubernetes problem. It starts at the top of the stack and works down through five layers. Each layer has specific symptoms, specific commands, and a clear signal indicating whether to stay at that layer or move to the next.

The Five Layer Model

Every Kubernetes problem lives at one of five layers. The layers are ordered from most common to least common. Start at Layer 1 and work down. Most problems resolve in the first two layers.

Layer 1: Application. The container itself is broken. Bad config, missing env vars, crashed process, OOM.

Layer 2: Pod Scheduling. The pod can’t get placed on a node. Resource limits, taints, affinity rules, node capacity.

Layer 3: Networking. The pod is running, but can’t communicate. DNS failures, service misconfig, network policies, and ingress issues.

Layer 4: Cluster Infrastructure. The control plane is degraded. etcd performance, API server latency, scheduler delays, and certificate expiry.

Layer 5: Node and Hardware. The underlying node is unhealthy. Disk pressure, memory pressure, kubelet issues, and GPU driver failures.

The framework works because Kubernetes problems almost always manifest at the application layer first. A pod crashes. A deployment doesn’t roll out. A request times out. The root cause might be at any layer, but the symptoms always show up at the top.

Layer 1: Application Debugging

This is where 60% of production issues live. The container is doing something wrong. Before blaming Kubernetes, check the application.

The first three commands

Run these in order for any pod that isn’t healthy:

# 1. What is the pod doing right now?
kubectl get pod <pod-name> -o wide

# 2. What happened to it?
kubectl describe pod <pod-name>

# 3. What is the application saying?
kubectl logs <pod-name> --tail=100

The get pod output tells you the current state. Is it Running, Pending, CrashLoopBackOff, Error, or ImagePullBackOff? Each state points to a different problem.

The describe pod output tells you the history. Look at the Events section at the bottom. Read it from bottom to top. The first event is usually the trigger.

The logs output tells you what the application thinks is happening. If the container crashed, use --previous to see the last run’s logs before the crash.

kubectl logs <pod-name> --previous --tail=100

CrashLoopBackOff

This is the most common pod failure. The container starts, crashes, restarts, crashes again. Kubernetes backs off the restart interval exponentially.

The root cause is almost always in the application logs. Check:

# See the exit code
kubectl get pod <pod-name> -o jsonpath='{.status.containerStatuses[0].lastState.terminated.exitCode}'

Exit code 1 means the application crashed on its own. Check logs for the error.

Exit code 137 means Kubernetes killed the container. It ran out of memory (OOMKilled). Check:

kubectl describe pod <pod-name> | grep -i oom

If it was OOMKilled, the fix is either increasing the memory limit or fixing the memory leak in the application.

Exit code 143 means the container received SIGTERM. Kubernetes asked it to stop gracefully. This happens during rollouts, scaling, or node drains.

ImagePullBackOff

The container image can’t be downloaded. Check:

kubectl describe pod <pod-name> | grep -A5 "Events"

Common causes: wrong image name, wrong tag, private registry without image pull secrets, or the registry is down.

# Check if image pull secrets are configured
kubectl get pod <pod-name> -o jsonpath='{.spec.imagePullSecrets}'

Readiness and Liveness Probes

A pod is Running but not receiving traffic. The readiness probe is failing.

# Check probe configuration and recent failures
kubectl describe pod <pod-name> | grep -A10 "Readiness\|Liveness"

Common mistake: the readiness probe checks an endpoint that takes 30 seconds to respond, but the timeout is set to 1 second. The pod is healthy but Kubernetes thinks it isn’t.

The signal to move to Layer 2

If kubectl describe pod shows the pod is Pending (not Running, not CrashLoopBackOff), the problem isn’t the application. The pod hasn’t been scheduled yet. Move to Layer 2.

Layer 2: Pod Scheduling

The pod exists but it’s stuck in Pending. Kubernetes can’t find a node to run it on.

The diagnostic command

kubectl describe pod <pod-name> | grep -A20 "Events"

The Events section tells you exactly why the scheduler rejected the pod. The message will say something like:

0/12 nodes are available: 6 Insufficient cpu, 4 node(s) had taint, 2 node(s) didn't match pod affinity.

Read this carefully. It tells you how many nodes exist, how many were filtered, and why each one was rejected.

Insufficient resources

# Check available resources across all nodes
kubectl top nodes

# Check a specific node's allocation
kubectl describe node <node-name> | grep -A15 "Allocated resources"

Compare the pod’s resource requests against what’s available. If the pod requests 4 CPU and 16Gi memory, but no node has that much free, the pod stays Pending.

The fix is either reducing the pod’s resource requests, adding more nodes, or cleaning up unused workloads to free resources.

Taints and tolerations

Nodes can have taints that repel pods. The pod needs a matching toleration to land on a tainted node. GPU nodes almost always have taints.

# Check node taints
kubectl describe node <node-name> | grep -A3 "Taints"

# Check pod tolerations
kubectl get pod <pod-name> -o jsonpath='{.spec.tolerations}' | jq .

If the node has a taint and the pod doesn’t have a matching toleration, the scheduler will skip that node.

Node selectors and affinity

# Check what the pod requires
kubectl get pod <pod-name> -o jsonpath='{.spec.nodeSelector}' | jq .
kubectl get pod <pod-name> -o jsonpath='{.spec.affinity}' | jq .

# Check what nodes have
kubectl get nodes --show-labels | grep <expected-label>

If the pod requires gpu-type=a100 but no node has that label, the pod stays Pending forever.

PersistentVolumeClaim binding

kubectl get pvc -n <namespace>

If the PVC status is Pending, the pod can’t start because its storage isn’t ready. Check the PVC events:

kubectl describe pvc <pvc-name> -n <namespace> | grep -A10 "Events"

The signal to move to Layer 3

If the pod is Running but the service isn’t working (requests fail, connections time out, DNS doesn’t resolve), the problem is networking. Move to Layer 3.

Layer 3: Networking

The pod is running. The application is healthy. But traffic isn’t reaching it. Or it can’t reach other services.

Service connectivity

First, verify the service exists and has endpoints:

# Check the service
kubectl get svc <service-name> -n <namespace>

# Check if the service has endpoints (pods backing it)
kubectl get endpoints <service-name> -n <namespace>

If endpoints shows zero addresses, the service selector doesn’t match any running pods. Compare the service selector with the pod labels:

# Service selector
kubectl get svc <service-name> -o jsonpath='{.spec.selector}'

# Pod labels
kubectl get pods -n <namespace> --show-labels

DNS resolution

The most common networking issue in Kubernetes. The pod can’t resolve service names.

# Test DNS from inside a pod
kubectl exec -it <pod-name> -- nslookup <service-name>
kubectl exec -it <pod-name> -- nslookup <service-name>.<namespace>.svc.cluster.local

If DNS fails, check CoreDNS:

# Is CoreDNS running?
kubectl get pods -n kube-system -l k8s-app=kube-dns

# CoreDNS logs
kubectl logs -n kube-system -l k8s-app=kube-dns --tail=50

A common cause of slow DNS is the ndots setting. By default, Kubernetes adds ndots:5 to resolv.conf, which means any name with fewer than 5 dots gets appended with search domains before the actual lookup. A simple lookup for api.example.com generates 4 failed queries before the real one succeeds.

The fix:

spec:
  dnsConfig:
    options:
      - name: ndots
        value: "2"

Network policies

If you have network policies in your cluster, they might be blocking traffic between pods.

# List network policies in the namespace
kubectl get networkpolicies -n <namespace>

# Describe a specific policy
kubectl describe networkpolicy <policy-name> -n <namespace>

A missing egress rule means the pod can’t make outbound connections. A missing ingress rule means nothing can connect to the pod. An empty pod selector {} applies to all pods in the namespace.

Testing connectivity

# Test pod to pod connectivity
kubectl exec -it <pod-a> -- curl -v http://<pod-b-ip>:<port>

# Test pod to service connectivity
kubectl exec -it <pod-a> -- curl -v http://<service-name>:<port>

# Test pod to external connectivity
kubectl exec -it <pod-a> -- curl -v https://httpbin.org/get

The signal to move to Layer 4

If all pods are slow (not just one service), if kubectl itself is slow, or if you see etcdserver: request timed out in logs, the problem is the control plane. Move to Layer 4.

Layer 4: Cluster Infrastructure

The control plane is degraded. This affects everything in the cluster, not just one application.

Symptoms

kubectl commands take 5+ seconds. Deployments don’t roll out. Pod creation is delayed. Controller reconciliation falls behind. Events show etcdserver: request timed out.

API server health

# Check API server response time
time kubectl get nodes

# Check API server metrics (if accessible)
kubectl get --raw /metrics | grep apiserver_request_duration_seconds

# Check API server logs
kubectl logs -n kube-system kube-apiserver-<node> --tail=50

If the API server is slow, the cause is almost always etcd. The API server is stateless. etcd is not.

etcd health

# Quick health check
etcdctl endpoint health --cluster \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

# Detailed status
etcdctl endpoint status --write-out=table --cluster \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

Check the metrics that predict etcd failures:

etcd_disk_wal_fsync_duration_seconds p99 above 10ms means disk latency. etcd_mvcc_db_total_size_in_bytes approaching the quota means NOSPACE is coming. etcd_server_leader_changes_seen_total above 1 per hour means instability.

We covered all five etcd failure modes in detail in our etcd debugging guide.

Certificate expiry

kubeadm certs check-expiration

If certificates expire, everything breaks at once. Existing pods keep running from kubelet cache. But nothing new can be created, updated, or deleted.

Scheduler health

# Check scheduler logs
kubectl logs -n kube-system kube-scheduler-<node> --tail=30

# Check if scheduler is falling behind
kubectl get --raw /metrics | grep scheduler_scheduling_attempt_duration_seconds

The signal to move to Layer 5

If specific nodes show problems (NotReady status, high resource usage, kubelet errors) but the control plane is healthy, the issue is at the node level. Move to Layer 5.

Layer 5: Node and Hardware

Individual nodes are unhealthy. This only affects pods running on those specific nodes.

Node status

# Check all node statuses
kubectl get nodes

# Look for conditions on a specific node
kubectl describe node <node-name> | grep -A10 "Conditions"

The Conditions section shows:

MemoryPressure: the node is running out of RAM. DiskPressure: the node is running out of disk. PIDPressure: the node has too many processes. Ready: False means the kubelet can’t communicate with the API server.

Kubelet health

# Check kubelet status on the node
systemctl status kubelet

# Kubelet logs
journalctl -u kubelet --tail=50

Common kubelet issues: certificate expired, container runtime not responding, disk full on the node.

GPU specific issues

For GPU nodes, check the GPU Operator components:

# Are all GPU Operator pods running?
kubectl get pods -n gpu-operator -o wide

# Can the node see GPUs?
kubectl describe node <gpu-node> | grep nvidia.com/gpu

# Check nvidia-smi on the node
kubectl debug node/<gpu-node> -it --image=nvidia/cuda:12.0-base -- nvidia-smi

If nvidia-smi fails, the GPU driver isn’t loaded. Check the driver container in the GPU Operator.

We covered the full GPU Operator debugging path in our GPU Operator article.

Disk pressure

# Check disk usage on the node
kubectl debug node/<node> -it --image=ubuntu -- df -h

# Check container image storage
kubectl debug node/<node> -it --image=ubuntu -- du -sh /var/lib/containerd

Old container images and unused layers accumulate over time. Kubernetes garbage collection should handle this, but sometimes it falls behind.

The Quick Reference Checklist

When something breaks in production, run through this sequence:

1. kubectl get pods -n <namespace>
   → What state are the affected pods in?

2. If CrashLoopBackOff or Error:
   → kubectl logs <pod> --previous --tail=100
   → Layer 1: Application issue

3. If Pending:
   → kubectl describe pod <pod> (read Events)
   → Layer 2: Scheduling issue

4. If Running but not working:
   → kubectl exec <pod> -- curl <service>
   → kubectl exec <pod> -- nslookup <service>
   → Layer 3: Networking issue

5. If everything is slow:
   → time kubectl get nodes
   → etcdctl endpoint health --cluster
   → Layer 4: Control plane issue

6. If specific node problems:
   → kubectl describe node <node> (check Conditions)
   → systemctl status kubelet
   → Layer 5: Node/hardware issue

This sequence takes 2 minutes. It eliminates 80% of possible causes and points you at the right layer immediately. No more guessing.

The Debugging Mindset

Three rules that make debugging faster:

Rule 1: Read the Events. Every kubectl describe output has an Events section. Read it. From bottom to top. The events tell you what Kubernetes already knows about the problem. Most engineers skip this and start guessing.

Rule 2: Check one layer at a time. Don’t jump between application logs, network policies, and etcd metrics in the same debugging session. Start at Layer 1. If the evidence points to a different layer, move there deliberately. Randomized debugging wastes time.

Rule 3: Reproduce before you fix. If you can’t reproduce the problem on demand, you don’t understand it yet. A fix applied without understanding the root cause is just a workaround that will break again later.

What This Framework Connects To

This article is the anchor for production debugging at KubeNatives. Every specific debugging guide links back here:

Our etcd debugging guide covers Layer 4 in depth: the 5 ways etcd breaks and the metrics that predict each failure.

Our GPU Operator article covers Layer 5 for GPU nodes: the 8 components and the initialization dependency chain.

Our DNS troubleshooting guide (coming soon) will cover Layer 3 in depth: CoreDNS, ndots, and the 5 second timeout problem.

Each supporting article gives you the deep dive for a specific problem. This framework tells you which article to reach for.

Next week: Deploying vLLM on Kubernetes: From Single Pod to Production.

If you’re running production Kubernetes, I cover control plane operations, GPU infrastructure, and model serving every week. Subscribe at kubenatives.com.

Symptom

Your vLLM pod restarted during normal traffic. Users saw 503 errors for the duration of the restart. The pod eventually came back but might OOM again on the next large request.

Signals you are in this runbook:

$ kubectl get pod vllm-0
NAME      READY   STATUS      RESTARTS   AGE
vllm-0    1/1     Running     3          2h

$ kubectl describe pod vllm-0 | grep -A3 "Last State"
    Last State:     Terminated
      Reason:       OOMKilled
      Exit Code:    137

Exit code 137 means the container received SIGKILL from the kernel OOM killer. Not from a crash. Not from vLLM code. The kernel decided the container used too much memory and killed it.

Quick Triage: Is This GPU Memory or Host Memory?

This is the first branch. vLLM has two memory failure modes and they need different fixes.

Check pod events:

kubectl describe pod vllm-0 | grep -A2 -i "oom\|killed"

If you see “Memory cgroup out of memory” in kubelet events: This is host memory OOM. The container exceeded its resources.limits.memory. Jump to Procedure A.

If you see “CUDA out of memory” or “torch.cuda.OutOfMemoryError” in vLLM logs: This is GPU memory OOM. The model tried to allocate more VRAM than available on the device. Jump to Procedure B.

If you see both or cannot tell: Pull the last 200 lines of logs from the previous container:

kubectl logs vllm-0 --previous --tail=200 | grep -iE "oom|memory|cuda|killed"

Look for the first memory related error. That is the trigger. Everything after is cascade.

Procedure A: Host Memory OOM (exit 137, kernel killed the container)

What happened: the container exceeded resources.limits.memory. Kubernetes killed it.

Root causes, ranked by frequency:

1. Memory limit set too low for the model size (most common)

2. Prefix caching or KV cache overflow into host memory via swap or CPU offload

3. Memory leak in vLLM (rare, usually requires version upgrade)

Step 1: Confirm the limit violation

# What was the memory limit?
kubectl get pod vllm-0 -o jsonpath='{.spec.containers[0].resources.limits.memory}'
# Example output: 32Gi

# What did it actually use before death?
kubectl top pod vllm-0 --containers 2>/dev/null || echo "metrics-server needed"

If limits are 32Gi and a 70B model needs host memory to mirror the weights during load, you will hit the limit on startup.

Read more

Most weeks you get a technical deep dive from me on Fridays. Today is different.

I want to put a workshop on your radar that I think is worth your Saturday.

Internal Developer Platforms have been the dominant platform engineering conversation for two years now. Most teams I talk to are either building one badly, buying one they do not fully understand, or avoiding the topic because they have seen too many failed platform projects.

The pattern is consistent. Teams start with a portal (usually Backstage) and work backwards into the underlying platform. That order is wrong. It is why so many IDPs end up as another bottleneck instead of a force multiplier.

Ajay Chankramath runs Platformetrics and previously led Platform Engineering at Thoughtworks. He is running a two day workshop on April 25 and 26 on building an AI powered IDP from scratch. I asked him a few questions on the stuff most teams get wrong.

When is a team actually ready to build an IDP?

Ajay: When you can name your top three developer friction points based on data, not gut feeling. If you have not watched a developer go through onboarding end to end, you are not ready to build the platform. Do not start building a platform just because you learned about a solution. Start when you truly understand the problems.

How do IDP patterns need to evolve for AI and ML workloads?

Ajay: AI workloads break three assumptions baked into the standard IDP: resource primitives, lifecycle, and failure modes.

IDPs need to treat GPU pools as first class resources with their own abstractions. They need to build golden paths for ML workflows, not just microservices. They need to integrate model registries and experiment trackers into the service catalog. And they need observability for inference latency, confidence scores, and data drift.

The standard Backstage style IDP was not designed for workloads that can fail by giving confident wrong answers for weeks.

What will engineers walk away understanding?

Ajay: How the layers connect to each other.

You can learn about each tool from its documentation. This workshop teaches what happens when a developer submits a service request in the portal, which triggers a golden path scaffolder, which provisions a namespace with RBAC and quotas, which applies policies via OPA, which is monitored by an SLO driven alerting stack, which feeds into an AI powered alert correlator.

That end to end chain, from portal click to production insight, is the platform.

Workshop details

Building an AI Powered Internal Developer Platform from Scratch

Saturday April 25 and Sunday April 26, 2026 11 AM to 3 PM ET each day 4 PM to 8 PM UK / 8:30 PM to 12:30 AM IST / 7 PM to 11 PM Gulf

Hosted by Deep Engineering by Packt.

What’s included:

Live hands on sessions with Ajay across two days. Working code for AI platform features that runs locally without API keys. A 30 to 60 minute one on one Platform Journey consultation with Ajay. Certificate of Completion plus a Credly digital badge you can add to LinkedIn.

Refunds available up to 3 days before the event. Seats are limited.

Register here

Why I am sharing this

I am selective about what I put in front of this list.

Ajay’s answer to the AI workloads question landed for me because it names a real gap in how most teams are thinking about ML platforms today. GPU pools as first class resources. Model registries in the service catalog. Observability that covers data drift, not just p99 latency. Most IDPs I have seen do none of this.

If you are on a platform team, a DevOps team going through an AI transformation, or an SRE figuring out how to support ML workloads, this workshop will save you months of trial and error.

Disclosure: This is a paid partnership with Deep Engineering by Packt. I only promote things I would send to a friend.

Regular Friday content this week covers the production Kubernetes debugging framework I use on our clusters. More on that in a few days.

Sharon