~/ nmadiraju.com Amazon EKS · Karpenter Spot storm
arrow_back Back to nmadiraju
Incident Replay · Compute and scaling

The Spot Storm: Tuning Karpenter to Survive a Spot Capacity Crunch

Tune prod's NodePool, buffer and queue, replay the same Spot storm, then follow the incident from one wave to the fix.

The Spot storm in 60 seconds

  • A Spot storm is EC2 reclaiming many Spot (spare, discounted) nodes at once. Worst when all nodes share one pool (one instance type, one zone).
  • Replacements get throttled. EC2 allows 50 CreateFleet launch calls at once, then 5 a second, per account and Region. Past that: RequestLimitExceeded.
  • Breadth is the fix. A NodePool spanning many families, sizes and zones, plus on-demand, spreads nodes over many pools; a crunch takes a slice.
  • Headroom hides the rest. Low-priority pause Pods hold warm room that evicted Pods take in seconds; the interruption queue gives Karpenter the 2-minute warning.
  • Disruption budgets won't help. They only limit voluntary changes such as consolidation (packing Pods onto fewer nodes); Spot interruptions go ahead regardless.

1Try it, then replay the incident

A Spot storm on an EKS cluster: a reclaimed Spot pool, Karpenter's CreateFleet calls against EC2's token bucket, and Pending pods Left, inside AWS Region us-east-1: the EC2 Spot capacity pools, one cell per pool with the reclaimed pool hatched red; the interruption queue (EventBridge rules into SQS); and the EC2 Fleet API with CreateFleet's token bucket gauge. Right, the EKS cluster prod: Karpenter with its NodePool, the Pods card with pause pods and Pending pods, and a grid of an example fleet of 120 Spot nodes colored by what happened to them. Below, tiles for nodes reclaimed, CreateFleet calls, Pending pods and time to recover. AWS REGION us-east-1 · ACCOUNT prod EC2 Spot capacity pools Interruption queue EventBridge rules → SQS EC2 Fleet API · CreateFleet 50 calls at once, then 5 a second per account, per Region EKS CLUSTER prod Karpenter Pods Nodes running2-min warninglostnew Spoton-demandpause pods NODES RECLAIMED CREATEFLEET CALLS PENDING PODS

    2Diagnose it: check yourself

    Check yourself