A/B Testing LLM Models in Production with Kubernetes
Swapping Llama 3.1 70B for your old 8B model in one deploy is how you break production. Here is how to do it safely with traffic splitting and real quality metrics.
Your team just finished evaluating a new model. Llama 3.3 70B outperforms your current Llama 3.1 8B on every internal benchmark. The engineering review is done. The cost analysis is done. Time to ship.
The wrong way: update the Deployment image, wait for the rollout, monitor Grafana for 10 minutes, call it a success.
The right way: split 5% of production traffic to the new model, compare quality metrics side by side for 48 hours, ramp traffic if quality holds, roll back instantly if it drops.
This article covers the 4 things you need to A/B test LLM models in production on Kubernetes. Traffic splitting. Quality metrics. Shadow traffic. Rollback automation.
Why LLM A/B Testing Is Different
Traditional web service A/B testing compares latency, error rates, and conversion. All of those are server side metrics. You look at Grafana and know the answer in an hour.
LLM A/B testing compares response quality. Which model gave better answers? More helpful? More accurate? Less hallucinated? Those questions cannot be answered by server side metrics alone.
Three things are different about LLM A/B testing:
Quality is subjective. Latency is a number. Response quality needs human evaluation, LLM as judge evaluation, or proxy metrics like thumbs up rate.
The distribution matters. A new model might be better on average but catastrophically worse on 1% of prompts. You cannot see that in aggregate metrics.
Cost is variable. Llama 70B costs more GPU time per request than 8B. You are not just testing quality. You are testing whether the quality improvement justifies the cost increase.
A good A/B testing setup measures all three at once.
Pattern 1: Traffic Splitting with KServe InferenceService
KServe supports traffic splitting natively. You define one InferenceService with multiple revisions. Each revision points to a different model deployment. KServe routes incoming requests across them based on percentage.
apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
name: llm-router
spec:
predictor:
canaryTrafficPercent: 5
model:
modelFormat:
name: vllm
storageUri: "pvc://models/llama-3.3-70b"
# Previous revision (llama-3.1-8b) keeps the other 95%
The canaryTrafficPercent: 5 field sends 5% of traffic to the new model. The other 95% still hits the previous revision. You can dial that number without redeploying.
Ramp pattern: 5% → 25% → 50% → 100%. Hold each stage for at least 24 hours. Monitor quality metrics between each step. If any metric degrades, roll back by setting canaryTrafficPercent: 0.
If you are not on KServe, Istio VirtualService supports the same pattern with weight fields across two routes.
Pattern 2: Quality Metrics You Can Actually Measure
“Better quality” is not a metric. You need something you can put in Prometheus and alert on. Four options, ranked by how quickly they return signal.
Thumbs up rate. If your product has a feedback widget, thumbs up divided by total responses. Cheap. Fast. Lags by minutes. Best for consumer products with high traffic.
Response acceptance rate. For code completion or suggestion systems, the percentage of suggestions the user actually accepts. Also cheap, also fast.
LLM as judge. Run a second, stronger model (like Claude or GPT 4o) to grade responses from both A and B. Give it a rubric. Track win rate. Costs money per request but gives signal on quality dimensions users never report on.
Human eval. Most expensive, slowest, highest quality. Sample 50 to 100 responses per model per day and have a human grader score them. Reserve for the final ramp decision before going to 100%.
A good setup uses at least two of these. Thumbs up rate for real time signal. LLM as judge for deeper evaluation. Human eval as the final gate.
Pattern 3: Shadow Traffic for Risk Free Testing
Sometimes you want to test a new model without risking user experience at all. Shadow traffic mirrors real requests to the new model, compares the responses, and throws the new response away. Users never see it.
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
name: llm-shadow
spec:
hosts:
- llm.inference.svc.cluster.local
http:
- route:
- destination:
host: llama-3-1-8b
port:
number: 8000
weight: 100
mirror:
host: llama-3-3-70b
port:
number: 8000
mirrorPercentage:
value: 10.0
Istio mirror with mirrorPercentage: 10.0 sends a copy of 10% of requests to the new model. The original response from the production model is what the user sees. The shadow response is logged for comparison.
This is the safest way to run any new model. Zero user risk. You gather weeks of comparison data before you send any real traffic.
The catch: shadow traffic doubles GPU cost on the shadowed percentage. If you shadow 100% of traffic, you pay for 2x the compute. Shadow 5 to 10% and it is a reasonable investment for the risk reduction.
Pattern 4: Automated Rollback on Quality Regression
Manual rollback at 3 AM is how you end up with a worse model in production for 6 hours before someone notices. Automate it.
The idea: alert on quality regression, automatically drop canaryTrafficPercent to 0 when the alert fires.
# Prometheus alert
- alert: CanaryQualityRegression
expr: |
(
rate(llm_thumbs_up_total{revision="canary"}[10m]) /
rate(llm_responses_total{revision="canary"}[10m])
)
<
(
rate(llm_thumbs_up_total{revision="stable"}[10m]) /
rate(llm_responses_total{revision="stable"}[10m])
) * 0.9
for: 15m
annotations:
summary: "Canary model thumbs up rate is 10%+ below stable"
Pair the alert with a Kubernetes controller that watches Alertmanager and patches the InferenceService when the alert fires. Argo Rollouts has this built in. You define the metric, the threshold, and the rollback action. It does the rest.
The threshold matters. Too tight and every random variation triggers rollback. Too loose and real regressions slip through. Start at 10% below stable for consumer products. 5% for anything customer facing or paid.
Putting It Together: A Production Ramp Plan
Here is what a safe model upgrade looks like in practice. Llama 3.1 8B currently serving 100%. You want to move to Llama 3.3 70B.
Week 1. Shadow. Deploy Llama 3.3 70B alongside 8B. Mirror 10% of traffic. No user impact. Collect LLM as judge scores comparing responses. Look for win rate above 60% on your prompts.
Week 2. Canary 5%. Shadow data looked good. Move 5% of traffic to the new model. Watch thumbs up rate for both. Watch p99 latency (70B is slower than 8B). Watch cost per request.
Week 3. Ramp. Move to 25%, then 50%. Hold each step 24 to 48 hours. Quality metrics must stay within 5% of the stable revision. Automated rollback on regression.
Week 4. Full. 100% to new model. Keep the old revision around for one more week as a one click rollback option. Then retire it.
Total time: 4 weeks. Annoying? Yes. But this is how you ship a model change without breaking production.
What to Not Test A/B
Not everything needs A/B testing. Three cases where you should just deploy:
Bug fixes. If the current model is producing broken output and the new one fixes it, just deploy. A/B testing a fix prolongs the bug.
Security patches. Same logic. The current version has a known issue. Ship the patched version.
Identical model with config changes. Tuning max_tokens, adjusting the system prompt, changing batch size. A/B testing is overkill for changes that are mathematically unlikely to affect response distribution.
Reserve A/B testing for actual model swaps and major prompt rewrites. Everything else is just cautious engineering.
The Bottom Line
LLM A/B testing on Kubernetes is not hard. It is the same traffic splitting patterns you use for microservices, with quality metrics swapped in for error rate.
Four patterns. Traffic splitting via KServe or Istio. Quality metrics you can actually measure (thumbs up, LLM as judge, human eval). Shadow traffic for the highest risk changes. Automated rollback on regression.
Skip these and every model upgrade is a coin flip. Use them and you ship better models with confidence.
The hardest part is not the infrastructure. It is getting product and engineering to agree on what “better” means before you start the test.
Next week: Triton Inference Server on Kubernetes: Multi-Model Serving.
If you are running production Kubernetes clusters, I cover control plane internals, GPU infrastructure, and model serving every week. Subscribe at kubenatives.com.