~/ nmadiraju.com Amazon EKS · DiskPressure eviction
arrow_back Back to nmadiraju
Incident Replay · Resilience

Why DiskPressure Evicted Your Database to Reclaim a Few Gigabytes of Logs

Toggle requests, priorities, limits and the disk layout, and watch who the kubelet evicts; then replay the 3 a.m. page.

DiskPressure eviction in 60 seconds

  • DiskPressure eviction is the kubelet saving its node. Below 10% free disk, the kubelet (each node's agent) evicts Pods at once, with no grace period.
  • Ranking isn't on your side. Pods over their disk request go first, then lower priority, then the biggest overspend. With neither set, the database loses.
  • Fix the host first. Give the kubelet's and container runtime's directories their own volume, as Bottlerocket does, so scratch files can't fill the OS disk.
  • Then insure the workload. A disk request and high priority on the database; a disk limit on leakers, so they're evicted for their own overspend.

1Try it, then replay the incident

A stalled log shipper fills an EKS node's disk and the kubelet ranks Pods for eviction Left, Priya on call and the kubectl events she reads. Right, an EKS cluster with node ip-10-0-3-88: the kubelet with its eviction signal and DiskPressure condition; three Pods, orders-db (Postgres), web and a stalled log shipper, each with its ephemeral-storage usage, request, priority, limit, status and eviction rank; and two EBS volumes, the 20 GiB root and an optional 200 GiB data volume, each with a usage bar and the image GC and eviction lines. Priya · on-call kubectl events EKS CLUSTER prod NODE ip-10-0-3-88 · AL2023 kubelet eviction manager, 10 s loop orders-db · Postgres web logship · stalled EBS root volume · 20 GiB EBS data volume · 200 GiB OS images orders-db web logship evicted free image GC starts (85% used) eviction (90% used)

    2Diagnose it: check yourself

    Check yourself