~/ nmadiraju.com Amazon EKS · CPU throttling
arrow_back Back to nmadiraju
Incident Replay · Compute and scaling

Why Your P99 Latency Spikes on EKS While CPU Sits at 40%

A CPU limit is a quota of CPU time per 100 ms, not a speed. Change the limit, threads and node of api, the service Sam is on call for, and watch one period freeze, then replay Sam's page.

CPU throttling in 60 seconds

  • A CPU limit is a quota of CPU time: limit × 100 ms per 100 ms. Spend it early and every thread is paused (throttled).
  • Threads spend it fast. api's 8 busy threads on 8 vCPUs burn a 1-CPU quota in 12.5 ms, then freeze for 87.5 ms.
  • Average CPU hides it. Frozen time looks idle. Alert on the throttle ratio (throttled ÷ total periods) when it stays above 5 to 10%.
  • More replicas or a bigger node won't fix it. Each Pod keeps the same quota; more cores spend it faster.
  • Keep the request and memory limit; drop the CPU limit. If a cap must stay, set it well above normal use, or give whole cores.

1Try it, then replay the incident

Pod api's CPU limit, turned into a quota and spent within one 100 ms period on an EKS node Left, Sam on call, Prometheus with per-container metrics and a Grafana dashboard. Right, an EKS cluster with node node-1: Pod api with its threads, CPU request and limit; the kubelet, which turns the limit into a quota; and api's cgroup with cpu.max and its throttling counters. The largest panel shows one 100 ms period: the quota draining, one lane per vCPU showing when api's threads run and when they are frozen, and the moment the quota runs out. Sam · on-call Prometheus per-container metrics Grafana api dashboard EKS CLUSTER prod Pod api kubelet turns the limit into a quota api's cgroup (kernel) One 100 ms period · api quota left 0 25 50 75 100 ms running frozen (throttled) no api thread

    2Diagnose it: check yourself

    Check yourself