~/ nmadiraju.com Amazon EKS · Stateful Pods and a dark AZ
arrow_back Back to nmadiraju
Incident Replay · Resilience

When an Availability Zone Goes Dark, Your Stateful Pods Hang: How to Design Around It

Pick what fails, what on-call does and the design, and watch payments-0 recover, hang or risk its data, then replay the outage.

A dark AZ in 60 seconds

  • EBS (Elastic Block Store) is durable but zonal: a volume attaches only in its AZ. When that AZ goesdark, data is safe but stuck.
  • The wait is a guard. A Pod on an unreachable node stays Terminating. Force-delete it and the replacement hangs about 6 minutes on the disk.
  • Fence before you force. Fence = confirm the old node is stopped, then mark it out of service. payments is back a minute after that.
  • Skip the check and you risk the data. A node that's only cut off still writes; forcing the disk away loses writes.
  • Survive a dark AZ with a replica and a volume in each AZ. Creating each volume where its Pod lands (WaitForFirstConsumer) can't move it later.

1Try it, then replay the incident

payments-0, its VolumeAttachment and its EBS volume in a three-AZ EKS cluster when us-east-1a goes dark Left, Riya on call and the checkout clients. Top, the EKS control plane: the API server with node node-a1's status and taints, the StatefulSet payments, the attach/detach controller and the VolumeAttachment for vol-0a1, plus the Amazon EC2 API. Bottom, Region us-east-1 with three AZs, each with two EC2 worker nodes and an EBS volume: payments-0 runs on node-a1 with vol-0a1 in us-east-1a; a replacement can only start on node-a2 in the same AZ; payments-1 and payments-2 run in us-east-1b and 1c in the per-AZ design. Lines from each volume to a node show its attachment: healthy, stale, or forced off a live node. Riya · on-call checkout clients EKS CONTROL PLANE API server StatefulSet attach/detach VolumeAttachment Amazon EC2 API REGION us-east-1 · EKS worker nodes AZ us-east-1a AZ us-east-1b AZ us-east-1c node-a1 node-a2 node-b1 node-b2 node-c1 node-c2 vol-0a1 Amazon EBS gp3 vol-0b1 Amazon EBS gp3 vol-0c1 Amazon EBS gp3 payments-0 payments-0 payments-1 payments-2

    2Diagnose it: check yourself

    Check yourself