~/ nmadiraju.com Amazon EKS · Scaling past the VPC CNI limits
arrow_back Back to nmadiraju
Calculator · Networking

Your EKS Nodes Run Out of IPs Before They Run Out of Compute

Pick a node and its settings to see how many Pods fit and what stops the next one, then follow one stuck Pod to the fix.

VPC CNI limits in 60 seconds

  • IPs, not CPU. On EKS, the VPC CNI (the Pod networking plugin) gives every Pod a real VPC address from the node's network cards (ENIs).
  • The ceiling. ENIs × (addresses per ENI − 1) + 2: 29 Pods on an m5.large, however idle its CPU.
  • Find the wall first. Subnet still has free addresses: the node is full. Subnet at 0: you need more address space.
  • Node full: prefixes, then max-pods. Each slot holds 16 addresses, but custom-AMI nodes keep max-pods, kubelet's Pod limit, at 29 until raised (to 110).
  • Subnet empty: custom networking. Pods take addresses from 100.64.0.0/16, costing one ENI of Pods. New clusters can use IPv6 instead.

1Calculate the Pod ceiling

One EKS node's ENIs filling with Pod IPs inside a VPC subnet, with a secondary 100.64 CIDR below Left, the Deployment checkout and its next Pod, Pending. In the middle, a VPC with private subnet subnet-a holding one node: kubelet with its max-pods, aws-node's ipamd, and up to 8 ENI rows whose slots fill with single Pod IPs or /28 prefixes. Below it, the secondary CIDR 100.64.0.0/16 used by custom networking. Right, the VPC console's free IPv4 count, an IPv6 cluster option, and the on-prem network. Deployment flash-sale traffic PRIVATE SUBNET subnet-a 10.0.0.0/20 · ROUTABLE kubelet aws-node ipamd · VPC CNI ENI 1 ENI 2 ENI 3 ENI 4 ENI 5 ENI 6 ENI 7 ENI 8 Pod IP16-IP blockfree slotmax-pods blocksabove the capno subnet IP SECONDARY CIDR · CUSTOM NETWORKING VPC console subnet-a free IPv4 IPv6 cluster new clusters only no Pod address shortage On-prem network sees node-3's own IP can't route to 100.64

    2Diagnose it: check yourself

    Check yourself