~/ nmadiraju.com Amazon EKS · GitOps drift control
arrow_back Back to nmadiraju
Incident Replay · Delivery and operations

Why Argo CD Reverts Your 2 a.m. kubectl Fix in Seconds

Give each change a home and watch what Argo CD fights, then replay Dana's 2 a.m. revert to the fix.

GitOps drift control in 60 seconds

  • Self-heal undoes live edits. Drift is the live cluster differing from Git. With self-heal on, Argo CD puts Git's values back within seconds.
  • Turning it off makes it worse. Edits stick, but Git stops describing what runs, and the next sync wipes them anyway.
  • Give every fast change a home. Passwords to Secrets Manager, flags to a flag store, replicas to the autoscaler, ignored in diff and sync.
  • Keep one break-glass. Suspend auto-sync on the one app, fix it, then backfill Git and turn sync back on.

1Try it, then replay the incident

What Argo CD self-heal reverts on an EKS cluster, and where each fast-changing thing should live instead Left, Dana on call, whose kubectl edits change the cluster live, and Git branch main with checkout's manifests. Center, Argo CD: the API server where break-glass runs, the Application checkout with its sync policy and ignore rule, and the application-controller that compares Git with live and self-heals. Right, EKS cluster prod, namespace checkout: the Deployment, its HPA, External Secrets, the Pods and Secret checkout-db; below, the flag store and AWS Secrets Manager. Tokens for four changes (a password, a flag, replicas, an image) sit in the home each one is given, marked left alone, a catch, or undone. Dana · on-call kubectl edits, live Git · main apps/checkout/prod ARGO CD · NAMESPACE argocd Argo CD API · RBAC argocd app set break-glass: on-call role only Application checkout ignoreDifferences application-controller Git polled ~3 min · live watched EKS CLUSTER prod · NAMESPACE checkout Deployment checkout live state HPA checkout writes spec.replicas External Secrets ExternalSecret checkout-db refreshInterval: 1h Pods checkout read flags at runtime Secret checkout-db mounted by checkout Flag store Secrets Manager prod/checkout/db rotated here Each change ✓ left alone, Git true ! works, with a catch ✗ reverted, or drifting

    2Diagnose it: check yourself

    Check yourself