Designing Istio for the Enterprise: One Architecture at Five Resolutions

Every box in istio-system is one Deployment per cluster, and the ×3 is replicas across availability zones — not three gateways. A cluster has one east-west gateway and one istiod, each running three pods. Ingress is the exception: there is one ingress gateway per exposure tier — public, internal and partner — each its own Deployment with its own load balancer, certificates and WAF posture. Still not per namespace and not per application: an application attaches its Gateway or VirtualService configuration to the tier that matches how it is exposed, and inherits that tier's posture rather than running a gateway of its own. The east-west gateway is per cluster but used only when the destination is on another network — and it is the destination's gateway that ends up in the path, not the caller's, because the sidecar dials the target network's entry point directly. Clusters that share a VPC share a network label, so calls between them go direct and never touch a gateway at all. One cluster, with the rest of the estate removed. Two namespaces are meshed and one is not — that is the opt-in model, and it is the single most useful thing to understand about running Istio: the injector webhook selects namespaces by revision label, so nothing is meshed until someone asks. Note also what the mesh does not replace (kube-proxy still does Service VIPs), and that all three gateways are ordinary Deployments in istio-system rather than anything special. What fronts each of them is where the differences actually live. Every gateway is the same thing — a Deployment and a Service — and all that changes is the Service type and the load-balancer scheme. The public and partner tiers get an internet-facing L4 load balancer; the internal tier gets an internal one, reached from the corporate network over Direct Connect or VPN rather than from the internet. The east-west gateway also gets an internal one on :15443, because its callers are peer clusters and never the public. All of them pass TLS through rather than terminating it, which is why the answer is an NLB and not an ALB: the design terminates at Envoy so the gateway can see the client certificate and make the first authorization decision, and an application load balancer that decrypted first would take both away. Where partners are themselves on the same cloud, the same NLB can be published as a PrivateLink endpoint service instead, which removes the internet exposure entirely. There is no egress gateway here, and that is a decision rather than an omission. Istio's documentation states that it cannot securely enforce that egress traffic passes through one — a workload that bypasses its sidecar just leaves — and that something outside the mesh must enforce it. That something is a firewall and a single NAT route, which has to exist either way; adding a gateway on top of it buys a second policy surface, an extra hop and an extra layer of internal mTLS for an outcome the network already delivers. What the mesh keeps is the half only the mesh can do: REGISTRY_ONLY plus a declared ServiceEntry per dependency, so undeclared calls fail closed and every external dependency is a reviewable object. Attribution is not lost either — each sidecar logs its own outbound calls under its own identity, which is the same data, collected from many places instead of one.

Two clusters running the same namespace and the same service name are not two services — they are one, with merged endpoints. That is the defining semantic of multi-cluster, and it is what makes failover work at all: a caller asks for payments and gets every healthy endpoint in the mesh, wherever it runs. It is also the sharpest footgun in the model. Two teams that independently create default/api in different clusters have not created two services; they have silently joined one, and traffic will land on both. Namespace sameness is a naming contract before it is a routing feature. The boundary that decides whether a gateway is involved is the network, not the cloud. Clusters 1 and 2 share a VPC and therefore a topology.istio.io/network value, so istiod hands out pod IPs and the call is ordinary routed traffic. Cluster 3 sits in a second VPC in the same region and the same account — a different network — so istiod substitutes that network's east-west gateway address and the sidecar dials :15443 with the destination in SNI. Note which gateway is in the path: the destination's. The caller does not egress through its own; it dials the target network's entry point directly, so there is one gateway rather than one at each end. Every cluster runs one anyway, because every cluster is somebody else's destination. Locality-preferred routing chooses between the two. Cluster 2's endpoints are local and cheap, so they win while they are healthy; outlier detection ejects them when they are not, and the same call starts taking the gateway hop with no config change and no deploy. Discovery is drawn deliberately across the VPC boundary. istiod in cluster 1 watches cluster 3's Kubernetes API server on :443 exactly as it watches cluster 2's — the network split changes the address it hands back, never whether it knows the endpoint exists.

Discovery is a control-plane relationship; reachability is a data-plane one. They are drawn apart because they fail apart. A cluster learns its peers through a remote secret — a Secret in istio-system holding a kubeconfig for one peer's Kubernetes API server, created by istioctl create-remote-secret and applied like any other object. istiod reads it and watches that API server on :443, merging the peer's Services and EndpointSlices into the registry it already keeps for its own cluster. Two peers are drawn on purpose: the second cloud and an on-premises OpenShift cluster. Neither is a special case — on-prem is another network reached by the same gateway, discovered by the same secret, which is what makes phase four of the rollout an extension rather than a second architecture. Two consequences fall out of the shape. Control planes never talk to each other, so there is no istiod-to-istiod link to firewall, debug or version-match, and a sidecar is only ever configured by the istiod in its own cluster. And every control plane holds a credential for and a watch on every peer, so the relationships grow as N×(N−1): six at three clusters, thirty at six, 2,450 at the fifty in the scale requirement. That is a rotation and audit surface, not just a number. When a watch does break, the endpoints it fed go stale and keep being used — the mesh degrades toward a wrong-but-working picture rather than stopping, which is the trade multi-primary is bought for. Reachability begins before any of that, at the name. A caller resolving payments.payments.svc.cluster.local gets an answer only if that Service exists in its own cluster or DNS proxying answers for it; without one of those the call fails at resolution, with nothing in the mesh telemetry to explain it because Envoy never saw a connection. Only once a name resolves does the network label decide the address istiod hands back: a pod IP inside the same network, that network's east-west gateway outside it.

Every requirement states its own test. A bar with no way to verify it is a preference, and preferences do not survive a delivery argument. The constraints along the bottom are listed with an owning team on purpose — a dependency with no owner is an assumption in disguise, and it is the failure mode that shows up in month five rather than week one.

This tab is not the topology. A capability model has its own decomposition: capabilities layered by dependency, with no locations, no products and no counts. Governance decides what may be declared; control delivers that intent to the enforcement points; enforcement applies it on every call; and all of it is decided against a foundation of workload identity. Swap Istio for another mesh and this picture should barely change — that is the test it exists to pass. The mapping into named components is in the table below.

The dashed teal envelope is the prod mesh — one of three. It is a single trust domain spanning both estates, which is why a workload in either can prove who it is to the other. Two things make that envelope real rather than decorative: every workload inside it chains to one root CA, and every control plane inside it holds a remote secret for every other, so they share one registry. Networks nest inside a mesh rather than the other way round — crossing a network costs a gateway hop, while crossing a mesh costs trust-bundle federation, which is why there are three of these and not thirty. The other two meshes (7 and 8) sit below it with their own trust domains: the regulated one federates with prod for three exported services, the non-prod one is isolated on purpose. The envelope carries no number because it is not a component — it is the boundary the numbered components sit inside, and trust, config and consumers all arrive from outside it. It is a scope, not a perimeter. Nothing is deployed on that line and nothing is filtered at it; a workload inside the mesh can still reach anything the network permits. What the boundary decides is what comes for free: inside it, one workload can authenticate to another with no per-edge setup, a service name means one service wherever it runs, and a policy naming a principal means the same thing in every cluster. It also decides blast radius — one policy vocabulary is also one place a mistake reaches — which is the actual reason non-prod is a separate mesh rather than a namespace with rules on it. Domains A1, A4, E2.

1. Consumers & the ingress edge — who arrives from outside, and the one sanctioned way in and out. Domains D, E8.
2. Root of trust — where workload identity is anchored and how far it reaches. Domain E; one-way doors.
3. Config governance — how a change to the mesh gets reviewed, promoted, and reverted. Domains F, J, M.
4. Estate A — a cloud fleet: its control plane, data plane, gateways and local trust anchor. Domains A, B, C.
5. Cross-location connectivity — what carries service-to-service traffic between estates. Domains A3, D.
6. Estate B — the second cloud fleet, deliberately drawn identically. Domains A, B, C, K.
7. Non-prod mesh — its own trust domain, and deliberately not federated to prod. Domains A1, A4.
8. Regulated mesh — its own trust domain, joined to prod by trust-bundle exchange for three exported services. Domains A1, A4, E9.
9. Observability — the signals every estate emits and where they land. Domain H.
10. Scale & cost envelope — the read-out the design is measured against, not a component. Domains I, L.

Capability → component: how the conceptual tab lands

Because tab 2 has its own decomposition, it needs an explicit bridge into the named components. Note how little of this is one-to-one: a single capability lands in several components, and one component — the gateway — carries three different capabilities at once. That many-to-many mapping is the reason a capability model cannot just be the topology with the labels changed.

The spine: one concern, five resolutions

Progressive resolution only works if you can follow a single concern all the way down. Cross-cluster connectivity is the clearest thread — note that it is not a block at all in two of the five resolutions, and at the conceptual layer it becomes two capabilities, which is the mapping above doing its job:

Every major concern in the design should thread like that. If one only appears in a single tab, either it is genuinely scoped to that depth or a resolution is incomplete.

Decision register — every step on this canvas

Rollout — the order that keeps the doors open

The sequence matters as much as the design. The one-way doors are decided first, before anything is installed, because every later phase assumes them.

Who owns what

The platform team runs the paved road; app teams drive on it. The split below is what makes the guardrails enforceable rather than advisory — R/A is responsible and accountable, C is consulted.

What is most likely to go wrong

References

1. Istio — Deployment models (single vs multi-cluster, network models, control-plane models, tenancy)
2. Istio — Install multicluster (multi-primary, primary-remote)
3. Istio — Security (mTLS, PeerAuthentication, AuthorizationPolicy)
4. Istio — Plug in CA certificates (per-cluster intermediate)
5. Istio — Performance and scalability (config scoping, sizing)
6. Istio — Canary upgrades with revisions
7. Istio — Ambient mesh
8. SPIFFE — trust domains and federation
9. cert-manager — istio-csr
10. Kubernetes Gateway API