Coding Challenge #4

Pod Resource Hog Detector

The Problem

Your cluster is sluggish, and workloads are starving for CPU or memory. The culprit? Pods hogging resources, spiking usage, or leaking memory, choking the nodes.
You need a way to identify pods consuming excessive CPU or memory across your Kubernetes cluster so you can take action—whether it’s scaling, restarting, or alerting the team.

Your Objective

Write a Python script or Golang script using the Kubernetes Python client and Kubernetes Metrics API that:

• Accepts a namespace (default: default)

• Lists all pods in the namespace with their resource usage (CPU and memory)

• For each pod:

  • Fetches CPU and memory usage via the Metrics API

  • Compares usage against thresholds (e.g., CPU > 80% of limit, memory > 90% of limit)

  • Flags pods as "High Usage" if thresholds are exceeded

  • Checks if the pod is in Running state and how long it’s been running

• Prints a concise summary of:

  • Pod name

  • Namespace

  • Status: Running / Other

  • CPU usage (in cores or millicores)

  • Memory usage (in Mi or Gi)

  • Threshold violation (if any)

  • Duration running

Bonus

• Add --all-namespaces support to scan the entire cluster

• Export output to CSV or JSON for reporting

• Add --cpu-threshold and --mem-threshold flags to customize thresholds (as decimals, e.g., 0.8 for 80%)

• Highlight high-usage pods in red, normal usage in green (use ANSI colors)

• Include node-level context: show which node each pod is running on and the node’s total resource pressure

Tips

• Use v1.core_v1_api.list_namespaced_pod() to fetch pods

• Use custom_objects_api.get_namespaced_custom_object() for metrics from the Metrics API (metrics.k8s.io/v1beta1)

• Check status.phase for pod state and status.start_time for duration

• Parse spec.containers[].resources.limits for CPU/memory limits

• Handle cases where limits aren’t set (use node capacity or skip)

• Useful References: Kubernetes Python Client, Metrics API

Why This Is Useful

• Catches resource hogs before they crash your cluster

• Improves workload performance and cost efficiency

• Perfect for DevOps observability dashboards or alerting pipelines

• Can be scheduled as a cronjob to monitor clusters daily

Follow me on X @sharonsahadevan and connect on LinkedIn @sharonsahadevan for more real-world DevOps content.

KubeNatives Newsletter is a reader-supported publication. To get new challenges and deep-dive Kubernetes content, consider subscribing.

---