Design Patterns · Principal Track

Designing Argo CD for a Multi-Account Estate: One Architecture at Five Resolutions

Git is the only write path, the control plane splits on blast radius rather than on cluster count, and read access is treated as a security decision. Verified against Argo CD v3.5.1 · August 2026.

← → to step

The verdict

  1. Split the control plane on blast radius, not on cluster count. One instance per environment tier — production, non-production, and any strong regulatory zone — each managing many clusters. One instance for everything makes a non-prod mistake a production incident; one per cluster multiplies upgrades and dashboards by the size of the estate.
  2. Decide network reach before topology. Hub-and-spoke requires the control plane to open a connection to every managed cluster’s API server. Private endpoints, separate accounts without peering, edge sites and partner clusters all break that quietly. Reach is what picks the topology; scale is the reason usually given and it is the weaker one.
  3. Read access is a security decision. Twice now, read on an Application has been equivalent to reading Secrets across the destination cluster — most recently a critical advisory in May 2026 whose published mitigation, for anyone unable to upgrade, was to tighten read RBAC.
  4. Secrets never pass through the control plane. Resolve them on the destination cluster with an operator. Upstream “strongly cautions” against manifest-generation injection for a concrete reason: generated manifests, secrets included, are cached in plaintext in Redis.
  5. The support window is narrower than most estates plan for. Only the last three minor versions receive fixes. An estate that upgrades annually is unsupported for most of the year, and the 3.x line has carried real breaking changes — an apply-mode change, a cluster version format that breaks version-filtered generators, and a templating engine upgrade that ignores the pinned version field.
  6. Sharding is per cluster, not per application. That is the real ceiling: a single very large cluster cannot be split across controller shards. Past it, the answer is a second instance rather than a bigger one.

How to read the five resolutions

EnvironmentThe estate as it already is — three accounts, clusters owned by another team, an SCM nobody here runs. No solution internals.
RequirementsThe bar this is graded against, as a board. Every non-functional line states how it is verified; every constraint names an owning team.
ConceptualCapabilities stacked by dependency, no product named. Swap Argo CD for another engine and this picture has to survive.
LogicalNamed components and the flows between them — control planes, the reconciler, the repo server, the cache — but no counts, subnets or regions.
PhysicalWhat an SRE stands up: install variant, processor counts, sharding, access entries per account. Values are illustrative and marked as such.
Resolution
—
Flows traffic & connectivity control & config identity & trust telemetry structural, not configured Borders a real container the boundary, or outside it
One Argo CD architecture for a multi-account estate, drawn as a reference topology Engineers, the source-control system and the identity provider sit outside a dashed delivery boundary that spans two clouds. An AWS region holds the platform account: the registered clusters it manages, above a VPC drawn with its subnet tiers in the order traffic crosses them — a public subnet holding the load balancer, a platform subnet holding argocd-server and the ApplicationSet controller, and a private subnet whose availability zone holds the application controller, the repo server and Redis. A GCP region holds a workload account with its cluster API and etcd, an External Secrets operator, a metrics agent, and the workload itself. A private interconnect joins them, carrying only port 443 to the cluster API. The stepper lights the elements and flows each design decision touches. Engineers SCM Identity provider outside the boundary — the design depends on these, it does not own them PROD CONTROL PLANE · 1 OF 3 AWS · PLATFORM ACCOUNT use1a use1b euw1 VPC 10.20.0.0/16 EKS · MANAGEMENT CLUSTER PUBLIC SUBNET L4 load balancer PLATFORM SUBNET argocd-server ApplicationSet PRIVATE SUBNET AVAILABILITY ZONE app controller repo-server Redis Interconnect GCP · WORKLOAD ACCOUNT gcp-a gcp-b stage VPC 10.60.0.0/16 GKE · WORKLOAD CLUSTER CONTROL PLANE ENDPOINT kube API etcd PLATFORM ADD-ONS External Secrets metrics WORKLOAD NODES AVAILABILITY ZONE checkout Service console · gRPC OIDC clone · fetch GetManifests cache patch refresh generates Applications apply · :443 watch · observed state the only writer secret ref → value traffic + connectivity control + config identity + trust source + artifacts telemetry

Decision register — every step on this canvas

#ResolutionDecisionRecommendationRef

Capability → component: how the conceptual tab lands

Conceptual and Requirements deliberately break the shared topology, because a bar and a capability model have no deployment shape. That is what this table owes the reader — and the mapping is many-to-many.

CapabilityWhat implements itWhere it is configured
Declared intentGit repository, Application CRspec.source.repoURL · path · targetRevision
Gated changeProtected branch + policy check — outside Argo CDSCM branch protection
Detect divergenceapplication controller + its cluster cache--status-processors · timeout.reconciliation
Converge or reportsync policy, with a reviewed exception listsyncPolicy.automated{prune,selfHeal} · ignoreDifferences
Prove what is liveoperation state and the resource treestatus.operationState.syncResult.resources
Which cluster, which ringcluster Secrets + ApplicationSet generatorsargocd.argoproj.io/secret-type: cluster
Generated, not copiedApplicationSet matrix / merge generatorsApplicationSet.spec.generators
Who may read and changeArgo CD RBAC + AppProject rolesargocd-rbac-cm · AppProject.spec.destinations

The spine: one concern, five resolutions

Follow the hardest one — how a change actually reaches a cluster — and watch it resolve from a constraint into a command an SRE could run.

ResolutionThe same concern at this levelThe decision captured
EnvironmentClusters live in accounts this team does not own, and no network path to their API servers is guaranteed.Reach is a dated dependency with an owning team, not an assumption
RequirementsNot a component — a bar: deploy only from git, and prove the change landed. Verified continuously, not asserted.Where the functional bar sits, and how each line is tested
ConceptualTwo capabilities: detect divergence, then converge or report. Neither names a mechanism.Converge by default, with a short reviewed exception list — a runtime-mutated resource fights the loop forever
LogicalThe controller pushes to the cluster API; nothing calls in. An agent is the alternative for clusters that cannot be reached.Push, keeping the agent model in view — one-way door
PhysicalAn EKS access entry in each workload account naming the control plane’s role, cluster-wide read and namespace-scoped write.No cross-account IAM role or trust policy is needed; cluster-wide read is not negotiable

Rollout — the order that keeps the doors open

PhaseWhat shipsGate before moving on
1One HA control plane in the platform account, on three nodes. No cluster registered yet.SSO wired and role:admin restricted to a named group — everything after this is auditable
2AppProjects with specific cluster and namespace destinations, before a single Application exists.No project has a wildcard destination
3One non-production cluster registered declaratively, as a secret in git. Manual sync only.An access entry with cluster-wide read and namespace-scoped write, not AmazonEKSClusterAdminPolicy
4Auto-sync on non-production. Self-heal only once drift is understood rather than surprising.A controller kill rehearsed, and running workloads observed to be unaffected
5The production control plane — a separate instance, not a namespace.The repository allow-list differs between the two instances
6The platform baseline via a cluster generator, so a new cluster is self-onboarding.Cluster registration is a reviewed commit, because it is now a mass-deployment trigger

Who owns what

TeamOwnsThe thing they get paged for
PlatformThe control planes, projects, RBAC, generators, upgradesA control plane down, or an upgrade that broke SSO
Application teamsTheir config repository and what it declaresTheir app OutOfSync or Degraded
SecurityRead RBAC scope, the secret store, webhook exposureAn advisory that makes read equivalent to secret access
NetworkReach from the control plane to each cluster APIA registered cluster stuck in Unknown for a real reason
SCM ownersAvailability of git and the identity providerAn outage that stops deploys — but not running workloads

What is most likely to go wrong

FailureWhy it happensWhat blunts it
Read access leaks secretsA critical advisory in May 2026 let a user with read privileges retrieve raw manifests of any resource in the destination cluster, and exposed Secret data for apps using server-side diff.Stay inside the support window; scope read like write; keep secrets out of manifest generation entirely
An upgrade breaks login3.0 changed the Dex subject from sub to federated_claims.user_id, so every existing RBAC policy stopped matching at once.Read the notes for every intermediate minor; rehearse on the non-prod instance first
Helm renders differently after an upgrade3.5 forces Helm 4 and ignores spec.source.helm.version: v3. You cannot pin your way out.Test OCI-over-plain-HTTP repositories before upgrading, not after
Health checks look brokenThe control plane needs cluster-wide read on all resource types even when it writes to two namespaces. A read-restricted install produces errors that look like product bugs.Grant cluster-wide read deliberately and say why in the access request
A cluster registration becomes a mass deploymentA matrix generator over the cluster list creates one Application per app the moment a cluster appears.Narrow project destinations, and treat registration as a reviewed commit
Scaling stops workingSharding is assigned per cluster. One very large cluster cannot be split across controller shards.Size clusters with the shard boundary in mind, or plan the second instance
The webhook is a DoS vectorThe endpoint is unauthenticated by design; three separate high-severity advisories in September 2025 were malformed payloads crashing the parser.Do not expose it without a WAF or an SCM egress allowlist

References