SSL/TLS in Kubernetes: The Complete Mental Model

Why I spent 3 months debugging certificate issues that could have been solved in 3 hours with the right mental framework

It was 2:30 AM on a Tuesday when our payment processing went down. Again.

The error was cryptic: certificate verify failed: unable to get local issuer certificate.

Our monitoring indicated that all services were green, but customers were unable to complete purchases. Revenue was bleeding by the minute.

Sound familiar?

If you're running Kubernetes in production, you've probably been there. SSL/TLS certificate issues are the silent killers of uptime - they work perfectly until they don't, and when they break, they break spectacularly.

After years of wrestling with certificates, I've learned that most SSL/TLS problems in Kubernetes aren't technical problems - they're mental model problems.

Today, I'm sharing the complete mental framework that transformed me from a certificate-fearing engineer to someone who can debug SSL issues in minutes, not hours.

The Mental Model That Changes Everything

Here's what I wish someone had told me on day one:

SSL/TLS in Kubernetes isn't one thing - it's three distinct layers working together:

1. The Certificate Layer - Who you are (identity)

2. The Trust Layer - Who vouches for you (chain of trust)

3. The Termination Layer - Where the encryption happens (architectural decision)

Once you understand these three layers and how they interact, certificate debugging becomes systematic instead of random.

Let me walk you through each one.

Layer 1: The Certificate Layer (Identity)

Think of certificates like government-issued IDs. Just like your driver's license has:

• Your name and photo (subject)

• Expiration date (validity period)

• Issuing authority signature (CA signature)

SSL certificates have:

• Subject/Common Name: What domain this certificate represents

• Subject Alternative Names (SANs): Additional domains it covers

• Validity period: When it's valid (not before/not after dates)

• Public key: Used for encryption

• Signature: Proof it was issued by a trusted authority

The Kubernetes Reality

In Kubernetes, you'll encounter several types of certificates:

1. Application Certificates (Your Apps)

apiVersion: v1
kind: Secret
type: kubernetes.io/tls
metadata:
  name: myapp-tls
data:
  tls.crt: LS0tLS1CRUdJTi... # Base64 encoded certificate
  tls.key: LS0tLS1CRUdJTi... # Base64 encoded private key

2. Internal Kubernetes Certificates (Cluster Communication)

• kubelet → API server

• API server → etcd

• Controller manager → API server

3. Ingress Certificates (External Traffic)

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: myapp-ingress
spec:
  tls:
  - hosts:
    - myapp.company.com
    secretName: myapp-tls

Common Certificate Layer Problems

Problem: certificate is valid for X, not Y Translation: The certificate's subject/SAN doesn't match the hostname you're connecting to.

Example: Your certificate is for api.company.com but you're connecting to api.company.com:8080. Some systems treat these as different hostnames.

Problem: certificate has expired Translation: You're outside the validity period.

Quick Debug Commands:

# Check certificate details
openssl x509 -in cert.pem -text -noout

# Check what certificate a server presents
openssl s_client -connect api.company.com:443 -servername api.company.com

# Check certificate expiry from within a pod
kubectl exec -it <pod> -- openssl s_client -connect internal-service:443

Layer 2: The Trust Layer (Chain of Trust)

This is where most engineers get lost. Here's the simplified mental model:

Trust flows downward in a hierarchy:

• Root CA (Ultimate authority - lives in your OS/browser)

  • Intermediate CA (Delegated authority - signs your certificates)

    • Your Certificate (What you actually use)

Think of it like corporate hierarchy:

• CEO (Root CA) trusts Director (Intermediate CA)

• Director (Intermediate CA) trusts Manager (Your Certificate)

• Therefore, CEO trusts Manager (transitive trust)

The Kubernetes Reality

When your pod tries to connect to an external HTTPS service:

1. Pod gets the certificate chain from the target server

2. Pod checks its trust store (usually /etc/ssl/certs/)

3. Pod validates the chain from your certificate up to a trusted root

Common Trust Layer Problems

Problem: unable to get local issuer certificate Translation: The pod can't find the intermediate CA certificate that signed your certificate.

Problem: certificate signed by unknown authority Translation: The root CA isn't in the pod's trust store.

The Fix: Update your base image or mount custom CA certificates:

apiVersion: v1
kind: ConfigMap
metadata:
  name: custom-ca
data:
  custom-ca.crt: |
    -----BEGIN CERTIFICATE-----
    # Your custom CA certificate here
    -----END CERTIFICATE-----
---
apiVersion: apps/v1
kind: Deployment
spec:
  template:
    spec:
      containers:
      - name: myapp
        volumeMounts:
        - name: custom-ca
          mountPath: /etc/ssl/certs/custom-ca.crt
          subPath: custom-ca.crt
      volumes:
      - name: custom-ca
        configMap:
          name: custom-ca

Pro Tips for Trust Layer

1. Always include the full certificate chain:

# Your tls.crt should contain:
# 1. Your certificate
# 2. Intermediate CA certificate(s)
# 3. (NOT the root CA - it should already be trusted)

cat your-cert.pem intermediate-ca.pem > full-chain.pem

2. Test certificate chains:

# Verify the full chain
openssl verify -CAfile root-ca.pem -untrusted intermediate-ca.pem your-cert.pem

Layer 3: The Termination Layer (Where Encryption Happens)

This is the architectural decision that determines where SSL/TLS encryption gets terminated (decrypted) in your stack.

You have three main options:

Option 1: Ingress Termination (Most Common)

Internet → Load Balancer → Ingress Controller → Pod (HTTP)
         ↑
    TLS terminates here

Pros: Simple, centralized certificate management Cons: Internal traffic is unencrypted

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: myapp
spec:
  tls:
  - hosts:
    - myapp.company.com
    secretName: myapp-tls-secret
  rules:
  - host: myapp.company.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: myapp-service
            port:
              number: 80  # Note: HTTP to backend

Option 2: Pod-Level Termination

Internet → Load Balancer → Ingress Controller → Pod (HTTPS)
                                              ↑
                                        TLS terminates here

Pros: End-to-end encryption Cons: More complex certificate distribution

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: myapp
  annotations:
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
spec:
  rules:
  - host: myapp.company.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: myapp-service
            port:
              number: 443  # HTTPS to backend

Option 3: Service Mesh Termination

Internet → Load Balancer → Ingress → Sidecar → Pod (HTTP)
                                    ↑
                              TLS terminates here

Pros: Automatic mTLS, advanced traffic management Cons: Additional complexity, resource overhead

The Complete Debugging Framework

When SSL/TLS breaks, work through the layers systematically:

Step 1: Verify the Certificate Layer

# What certificate is being presented?
kubectl get secret myapp-tls -o yaml

# Decode and check the certificate
kubectl get secret myapp-tls -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -text -noout

# Check expiry
kubectl get secret myapp-tls -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -dates

Step 2: Verify the Trust Layer

# Check what CAs the pod trusts
kubectl exec -it <pod> -- ls /etc/ssl/certs/

# Test the certificate chain
kubectl exec -it <pod> -- openssl s_client -connect target-service:443 -verify_return_error

Step 3: Verify the Termination Layer

# Check where TLS is terminating
kubectl describe ingress myapp

# Check service endpoints
kubectl get endpoints myapp-service

# Check if backend expects HTTP or HTTPS
kubectl logs -l app=nginx-ingress

Certificate Management Best Practices

1. Use cert-manager for Automation

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: myapp-cert
spec:
  secretName: myapp-tls
  issuerRef:
    name: letsencrypt-prod
    kind: ClusterIssuer
  dnsNames:
  - myapp.company.com
  - api.myapp.company.com

2. Monitor Certificate Expiry

# Prometheus alert rule
- alert: CertificateExpiringSoon
  expr: (x509_cert_not_after - time()) / 86400 < 30
  labels:
    severity: warning
  annotations:
    summary: "Certificate expires in less than 30 days"

3. Test Certificate Rotation

# Force certificate renewal with cert-manager
kubectl annotate certificate myapp-cert cert-manager.io/issue-temporary-certificate=""

Common Gotchas That Will Save You Hours

1. SNI (Server Name Indication) Issues

When multiple certificates are served from the same IP, SNI tells the server which certificate to present.

# Always include -servername when testing
openssl s_client -connect api.company.com:443 -servername api.company.com

2. Internal DNS vs External DNS

Your certificate might be valid for the external domain but not the internal Kubernetes service name.

# This might fail even if external access works
kubectl exec -it pod -- curl https://myapp-service.default.svc.cluster.local

3. Base Image Certificate Trust Stores

Alpine-based images have minimal CA bundles. Consider using distroless or updating CA certificates:

# In your Dockerfile
FROM alpine:latest
RUN apk --no-cache add ca-certificates

The Production-Ready Checklist

Before deploying SSL/TLS in production:

• [ ] Certificate Layer: Valid subject/SANs, future expiry date

• [ ] Trust Layer: Full certificate chain included, custom CAs mounted if needed

• [ ] Termination Layer: Clear understanding of where TLS terminates

• [ ] Monitoring: Certificate expiry alerts configured

• [ ] Automation: cert-manager or equivalent for renewals

• [ ] Testing: End-to-end SSL tests in CI/CD

• [ ] Documentation: Clear runbooks for certificate issues

Wrapping Up

SSL/TLS in Kubernetes doesn't have to be mysterious. With the right mental model - Certificate Layer, Trust Layer, and Termination Layer - you can systematically debug issues instead of randomly trying fixes.

The next time you see certificate verify failed, you'll know exactly which layer to investigate first.

Remember: Most certificate issues aren't about the certificates themselves - they're about understanding how the three layers interact in your specific architecture.

---

What SSL/TLS challenge are you facing right now? Reply and let me know - I might feature your scenario in a future deep-dive.

Found this helpful? Share it with your team. SSL/TLS knowledge is one of those skills that pays dividends every time a certificate issue hits production.

P.S. - Next week, I'll cover "Network Policies: The Kubernetes Security Feature You're Probably Not Using" with hands-on examples of securing pod-to-pod communication.

Thanks for sticking with me through this deep-dive! If this helped you understand better,

hit that ❤️ and subscribe for weekly cloud-native insights.

Want to go deeper? Paid subscribers get exclusive hands-on guides, complete code examples, and early access to everything I publish. Plus access to my complete cloud-native learning vault.