~/ nmadiraju.com Amazon EKS · Asymmetric routing with custom networking
arrow_back Back to nmadiraju
Incident Replay · Networking

When Pods Can't Reach RDS Across a Transit Gateway

Flip the node and network settings, watch the database's reply reach checkout or die, then replay the hang from timeout to fix.

Asymmetric routing in 60 seconds

  • Reverse-path check (rp_filter): in strict mode (1), Linux drops a reply arriving on a different network card than it would answer from. 0 is off.
  • The trap: custom networking always puts Pods on a second card (eth1). The database's reply lands on eth0, so strict mode drops it; checkout hangs.
  • The fix: EKS's network plugin sets eth0 to loose (2: any route back will do); the hardened node image forced strict. Bake loose in.
  • Follow the reply (SYN-ACK), not the opening SYN. Flow Logs show the database answered; tcpdump (packet capture) and dmesg (kernel log) show where it died.
  • Two other traps: skipping SNAT (the address swap) needs a route back to Pods; a firewall needs appliance mode (one firewall both ways).

1Try it, then replay the incident

One SYN from Pod checkout to RDS across a Transit Gateway and an inspection VPC, and its SYN-ACK back Left, Ana on call and VPC Flow Logs. The EKS VPC holds node ip-10-0-4-20 with Pod checkout (100.64.12.34 on eth1), the aws-node VPC CNI, the node kernel's route back, SNAT and reverse-path check, and two network cards: eth0, the primary ENI at 10.0.4.20, and eth1, the secondary ENI that owns the Pod IPs. In the middle, a Transit Gateway. Right, an inspection VPC with one stateful firewall per AZ, and the database VPC with its route table and RDS orders-db at 10.1.20.14 in AZ b. A solid line shows the SYN, a dashed line the SYN-ACK back, and a red cross where the reply is dropped. Ana · on-call Flow Logs RDS ENI, 5432 EKS VPC · 10.0.0.0/16 + 100.64.0.0/16 Pod checkout dial 10.1.20.14:5432 aws-node (VPC CNI) sets eth0 to loose at boot Node kernel eth0 · primary ENI 10.0.4.20 · 10.0.0.0/16 eth1 · secondary ENI EKS subnet route tables 10.1.0.0/16 → tgw-0a1b2 (Transit Gateway) Transit GW 10.1/16 → DB INSPECTION VPC · stateful firewalls Firewall · AZ a Firewall · AZ b DATABASE VPC · 10.1.0.0/16 DB subnet route table 10.0.0.0/16 → tgw-0a1b2 RDS PostgreSQL orders-db 10.1.20.14:5432 · AZ b

    2Diagnose it: check yourself

    Check yourself