Kubernetes Security Auditing – How to Evaluate Your Cluster Security

Kubenatives Newsletter - Edition #15

As Kubernetes adoption surges in production environments, securing its sprawling attack surface is paramount. From misconfigured RBAC to exploited CRI runtimes, clusters are prime targets for attackers.

In this advanced guide, we’ll dissect the layers of Kubernetes security auditing, providing detailed techniques, tools, and configurations to evaluate and fortify your cluster rigorously.

Let’s dive into the gritty details.

Why Kubernetes Security Audits Matter

Kubernetes’ complexity—spanning control planes, worker nodes, and container runtimes—introduces countless failure points. Default settings often favor usability over security (e.g., permissive RBAC or unencrypted etcd), and the dynamic nature of workloads amplifies risks. A comprehensive audit uncovers:

• Privilege escalation paths via misconfigured RBAC or container capabilities.

• Exposed attack surfaces in the control plane or CNI plugins.

• Compliance gaps against standards like CIS Kubernetes, NIST 800-190, or PCI DSS.

• Zero-day exploitation risks in unpatched components.

This isn’t beginner-level hardening—expect to get your hands dirty with YAML, kernel syscalls, and audit logs.

Step 1: Control Plane Deep Audit

The Kubernetes control plane (API server, controller manager, scheduler, etcd) is the brain of your cluster. Compromise it, and it’s game over. Start here:

API Server Hardening

• Disable anonymous access (--anonymous-auth=false) and enforce OIDC or client certificates (--token-auth-file deprecated since 1.24). Validate with

  kubectl auth can-i --list --as=system:anonymous.

• Use RBAC with least privilege. Audit ClusterRoles with

  kubectl get clusterroles -o yaml | grep -E 'verbs|rules'—look for wildcard verbs (*) or excessive escalate permissions.

• Enable PodSecurity, NodeRestriction, and ValidatingWebhookConfiguration.

  kubectl get validatingwebhookconfigurations.

• Ensure --encryption-provider-config is set for secrets at rest. Test by inspecting etcd directly:

  ETCDCTL_API=3 etcdctl get /registry/secrets.

Etcd Security

• Use --auto-tls=false and --peer-client-cert-auth=true. Verify with

  etcdctl endpoint health --cert.

• Restrict etcd ports (2379/2380) to control plane nodes only via network policies or iptables.

Attack Vector: An unencrypted etcd snapshot can leak every secret in your cluster.

Step 2: Node and Runtime Security

Worker nodes and container runtimes (e.g., containerd, CRI-O) are often overlooked but critical.

Kubelet Audit

• Set --anonymous-auth=false, --authorization-mode=Webhook, and --read-only-port=0. Check via ps aux | grep kubelet.

• Enforce custom Seccomp/AppArmor profiles. Audit with kubectl describe pod <name> | grep -i seccomp.

• Use NoExecute taints to isolate untrusted workloads.

CRI Runtime

• Block privileged containers via PSA (privileged: false). Scan with

  crictl ps -a | grep -i privileged.

• Use runtimeClass with gVisor or Kata Containers for untrusted workloads. Validate with strace on a test pod.

• Run trivy image --severity CRITICAL,HIGH <image> on all base images.

Attack Vector: A compromised container with NET_RAW capability can ARP spoof your CNI.

Step 3: Network Security at Scale

Kubernetes networking is a labyrinth of overlays, service meshes, and policies. Lock it down:

Network Policies

• Implement podSelector: {} with policyTypes: [Ingress, Egress] in every namespace.

• For Calico, enable eBPF mode (bpf_enabled: true) and encrypt VXLAN traffic. For Cilium, use Hubble for flow visibility.

• If using Istio, audit mTLS with istioctl authn tls-check.

External Exposure

• Harden NGINX Ingress with --enable-ssl-passthrough and WAF rules. Check annotations in kubectl get ingress -A -o yaml.

• Restrict LoadBalancer Services with spec.loadBalancerSourceRanges.

Attack Vector: A misconfigured ClusterIP service can expose the API server externally.

Step 4: Vulnerability and Exploit Detection

Static scans aren’t enough—simulate real threats.

Toolset

• Run kube-hunter --remote <cluster-ip> to find exposed endpoints (e.g., kubelet on 10250).

• Deploy Trivy Operator in-cluster for continuous image scanning. Configure with operator.trivy.github.io/scanJobs.

• Deploy Falco with custom rules (e.g., detect exec into a pod: falco -r exec.yaml).

Manual Checks

• Test for Pod Escape:

  kubectl run -it --rm --image=busybox --privileged escape -- sh -c 'nsenter -t 1 -m -u -i -n -p'. If you hit the node, your runtime is vulnerable.

• Use kubectl access-matrix (third-party) to visualize privilege sprawl.

Attack Vector: A CVE in kubelet (e.g., CVE-2021-25741) can allow root filesystem access.

Step 5: Logging and Threat Detection

Audit logs are your forensic lifeline.

Configuration

• Use an advanced audit policy (--audit-policy-file=advanced-audit.yaml) logging requestReceived, responseComplete, and metadata.

• Ship logs to a SIEM (e.g., Loki+Grafana) with fluentd or filebeat.

Runtime Monitoring

• Deploy Falco rules to detect ptrace or chmod +x in containers.

• Use bpftrace to monitor kernel-level anomalies on nodes.

Step 6: Automation and Policy Enforcement

Manual audits don’t scale. Automate:

Tools

• Scan manifests pre-deployment with kubeaudit all -f manifests/.

• Enforce policies with OPA/Gatekeeper. Example:

  • Kind: K8sDenyPrivileged

  • Match: Pod

  • Enforce no privileged containers.

• Use Kyverno to mutate insecure configs (e.g., add securityContext.runAsNonRoot: true).

CI/CD Integration

• Embed trivy, kube-linter, and conftest in your pipelines. Example GitHub Action:

  • name: Scan manifests

  • run: conftest test --policy security.rego manifests/

Key Takeaways for Experts

• Harden the control plane first—etcd and API server are your crown jewels.

• Use runtime isolation (gVisor, Kata) for high-risk workloads.

• Simulate attacks with kube-hunter and Falco to validate defenses.

• Automate everything—OPA and Kyverno are your friends.

This isn’t a one-and-done process. Treat security as a continuous feedback loop, and assume breach until proven otherwise.

Advanced Resources

• CIS Kubernetes Benchmark v1.8.0

• Kube-hunter Documentation

• OPA Gatekeeper Policies

• Falco Custom Rules

Stay Vigilant, Stay Secure

---