~/ nmadiraju.com Amazon EKS · CoreDNS under load
arrow_back Back to nmadiraju
Incident Replay · Networking

Your Compute Scaled Out, Now DNS Is Timing Out

Change checkout's DNS setting, a node cache and CoreDNS's size, and watch one lookup fan out. Then replay the incident from scale-out to fix.

CoreDNS under load in 60 seconds

  • CoreDNS is the cluster's DNS server. After a scale-out it looks too small, but most of the trouble starts before it.
  • ndots:5 turns one lookup into eight. This Pod setting makes api.stripe.com try 3 cluster suffixes before its bare name, each asked twice.
  • A flat 5 s stall is a lost query. The client retries after 5 s. IPv4 and IPv6 queries sent together can lose a packet.
  • Three fixes, in order. ndots:2 cuts the queries; NodeLocal DNSCache answers on the node; CoreDNS replicas grow as nodes join.

1Try it, then replay the incident

One lookup of api.stripe.com from Pod checkout through the node's connection table, NodeLocal DNSCache, CoreDNS and the VPC resolver Left, the queries one lookup of api.stripe.com turns into, each with its A and AAAA result, and Karpenter scaling the nodes out. Center, a VPC with an EKS cluster and node node-7: Pod checkout with its DNS settings, the node's connection table, and NodeLocal DNSCache. Beside the node, CoreDNS with its replica count and CPU, and the proportional autoscaler. Right, the VPC resolver with its limit of 1,024 packets per second per network interface, and the internet. checkout looks up api.stripe.com Karpenter adds nodes VPC 10.0.0.0/16 EKS CLUSTER prod NODE node-7 · 10 Pods Pod checkout · prod DNS server 172.20.0.10 Connection table NodeLocal DNSCache CoreDNS the cluster's DNS server CoreDNS CPU Proportional autoscaler VPC resolver AWS's DNS server 1,024 packets/s per interface Internet api.stripe.com its own DNS

    2Diagnose it: check yourself

    Check yourself