Seventy YAML files, and the change that has to touch ten of them

The estate, before any tooling

Two services: checkout calls payments. Both run in three environments, and production is not one cluster:

That is five clusters. Each service needs the same set of objects in each of them. Not an exotic set — the boring one every service ends up with:

k8s/
  checkout/
    deployment.yaml
    service.yaml
    serviceaccount.yaml
    configmap.yaml
    hpa.yaml
    poddisruptionbudget.yaml
    networkpolicy.yaml
  payments/
    ... the same seven

Seven objects × two services × five clusters.

That number alone is not the problem. Seventy files that were each meaningfully different would be fine — that would just be a large system. The problem is what is actually inside them.

What is actually inside them

Here is checkout’s Deployment in staging.

# k8s/checkout/stage-use1/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: checkout
  namespace: shop
  labels:
    app.kubernetes.io/name: checkout
    app.kubernetes.io/part-of: shop
spec:
  replicas: 2
  selector:
    matchLabels:
      app.kubernetes.io/name: checkout
  template:
    metadata:
      labels:
        app.kubernetes.io/name: checkout
    spec:
      serviceAccountName: checkout
      containers:
        - name: checkout
          image: 111122223333.dkr.ecr.us-east-1.amazonaws.com/checkout:1.14.2
          ports:
            - containerPort: 8080
          env:
            - name: PAYMENTS_URL
              value: http://payments.shop.svc.cluster.local:8080
            - name: LOG_LEVEL
              value: info
          resources:
            requests: { cpu: 100m, memory: 192Mi }
            limits:   { memory: 384Mi }
          readinessProbe:
            httpGet: { path: /healthz, port: 8080 }

And here is the same file for production in us-east-1. Read it looking for the differences, not the content.

# k8s/checkout/prod-use1/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: checkout
  namespace: shop
  labels:
    app.kubernetes.io/name: checkout
    app.kubernetes.io/part-of: shop
spec:
  replicas: 6
  selector:
    matchLabels:
      app.kubernetes.io/name: checkout
  template:
    metadata:
      labels:
        app.kubernetes.io/name: checkout
    spec:
      serviceAccountName: checkout
      containers:
        - name: checkout
          image: 111122223333.dkr.ecr.us-east-1.amazonaws.com/checkout:1.14.1
          ports:
            - containerPort: 8080
          env:
            - name: PAYMENTS_URL
              value: http://payments.shop.svc.cluster.local:8080
            - name: LOG_LEVEL
              value: warn
          resources:
            requests: { cpu: 500m, memory: 512Mi }
            limits:   { memory: 1Gi }
          readinessProbe:
            httpGet: { path: /healthz, port: 8080 }

Forty-one lines. Six of them differ.

Six values out of forty-one lines, and one of those six is an accident: production is on 1.14.1 while staging is on 1.14.2. Is that a deliberate hold, or did somebody forget to promote? The file cannot tell you. Nothing in this layout distinguishes “intentionally different” from “nobody got round to it”.

That is the first real cost, and it is not verbosity. It is that the deltas are invisible. The thing you care about — how does prod differ from staging — is buried in thirty-five lines of things that are identical and always will be.

Now change one thing

A platform decision lands: every workload must spread across availability zones. It is four lines.

      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: ScheduleAnyway
          labelSelector:
            matchLabels:
              app.kubernetes.io/name: checkout

Where does it go? Into every Deployment. That is ten files — two services × five clusters — and note the last line: the selector has to name the right service in each one. Paste it into payments with checkout still in the selector and you have written a constraint that silently matches nothing.

Ten edits, one of which is not a copy—paste. And this is the easy kind of change. Consider the ones that follow:

The two things everyone tries first

1 · Substitute variables with sed or envsubst

One file, placeholders, a wrapper script. It works on the first afternoon and then it does this:

# deployment.tmpl.yaml
          replicas: ${REPLICAS}
          image: ${IMAGE}
          env:
            - name: LOG_LEVEL
              value: ${LOG_LEVEL}

$ REPLICAS=6 IMAGE=... LOG_LEVEL=warn envsubst < deployment.tmpl.yaml | kubectl apply -f -

Then someone leaves LOG_LEVEL unset. envsubst does not complain — it substitutes the empty string, and you apply:

            - name: LOG_LEVEL
              value: 

Which is valid YAML. It is a null value where a string was expected, and the failure surfaces wherever the application parses it. The tool has no idea it just did something wrong, because it does not know it is producing YAML — it is doing string replacement on a text file. Give it a value containing a colon and you will find that out the hard way.

2 · Kustomize

This one is a real answer, and it is worth being straight about that. Kustomize takes a base and applies overlays — no templating, no placeholders, just structured patches:

# overlays/prod-use1/kustomization.yaml
resources:
  - ../../base
patches:
  - path: replicas.yaml
    target: { kind: Deployment, name: checkout }

The deltas become explicit and reviewable, which is exactly what the copy-paste layout lost. If your problem is only “the same manifests differ slightly per environment”, this may be all you need, and reaching past it for something heavier is not automatically the right call.

What it does not give you is the other half of the problem — the half that has nothing to do with YAML:

• No unit to name or version. There is no artifact called “checkout 1.14.2” that you can publish, pin, or hand to another team.
• No inverse. kubectl apply has no undo. Rolling back means finding the previous commit and applying that, and hoping nothing was applied out of band in between.
• No record of what is deployed. The cluster knows its current state. It does not know that this state came from revision 7, or what revision 6 looked like.

The same estate as a chart

A chart is a directory with a required Chart.yaml, a values.yaml of defaults, and a templates/ directory of Go templates. The documentation is specific about the shape:

charts/service/
  Chart.yaml          # required
  values.yaml         # default configuration values
  values.schema.json  # optional JSON Schema for values
  templates/
    deployment.yaml
    service.yaml
    serviceaccount.yaml
    configmap.yaml
    hpa.yaml
    poddisruptionbudget.yaml
    networkpolicy.yaml
    _helpers.tpl
    NOTES.txt

# charts/service/Chart.yaml
apiVersion: v2        # required — v2 means "needs at least Helm 3"
name: service         # required
version: 0.4.0        # required — the version of the CHART
appVersion: "1.14.2"  # optional — the version of the app it installs
description: The standard shape for a shop service

apiVersion, name and version are the only required fields.

Note the two versions. version is the chart — the packaging. appVersion is the software inside it. They move independently, and that separation is the first thing the copy-paste layout could not express at all.

One template, instead of ten Deployments

# charts/service/templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ .Release.Name }}
  namespace: {{ .Release.Namespace }}
  labels:
    {{- include "service.labels" . | nindent 4 }}
spec:
  replicas: {{ .Values.replicaCount }}
  selector:
    matchLabels:
      {{- include "service.selectorLabels" . | nindent 6 }}
  template:
    metadata:
      labels:
        {{- include "service.selectorLabels" . | nindent 8 }}
    spec:
      serviceAccountName: {{ .Release.Name }}
      {{- with .Values.topologySpreadConstraints }}
      topologySpreadConstraints:
        {{- toYaml . | nindent 8 }}
      {{- end }}
      containers:
        - name: {{ .Chart.Name }}
          image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
          ports:
            - containerPort: {{ .Values.service.port }}
          env:
            {{- range $k, $v := .Values.env }}
            - name: {{ $k }}
              value: {{ $v | quote }}
            {{- end }}
          resources:
            {{- toYaml .Values.resources | nindent 12 }}
          readinessProbe:
            httpGet:
              path: {{ .Values.probePath }}
              port: {{ .Values.service.port }}

Four things in there are worth pointing at, because they are the ones that make the difference:

• .Release.Name and .Release.Namespace are built in. You do not pass them; Helm supplies them from the command you ran. The same chart installed twice under different release names produces two independent sets of objects.
• include "service.selectorLabels" is defined once in _helpers.tpl. The selector bug from earlier — pasting checkout’s selector into payments — is now structurally impossible, because there is one definition and it derives from the release.
• {{- with .Values.topologySpreadConstraints }} means the block appears only when the value is set. The four-line platform change is now one edit, in one file, and every environment that wants it opts in.
• | quote and | toYaml exist because Helm knows it is emitting YAML. The envsubst failure above cannot happen: value: {{ $v | quote }} produces value: "", not a bare empty scalar.

And the deltas become the whole file

Here is the entire difference between production in us-east-1 and everything else. Not a forty-one line file with six changed lines — six lines.

# envs/prod-use1.yaml
replicaCount: 6
image:
  tag: "1.14.1"          # deliberately held back — see CHANGE-4471
env:
  LOG_LEVEL: warn
resources:
  requests: { cpu: 500m, memory: 512Mi }
  limits:   { memory: 1Gi }
topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: ScheduleAnyway

# envs/dev.yaml
replicaCount: 1
env:
  LOG_LEVEL: debug

Dev is three lines, because dev differs from the default in exactly two ways.

Which value wins

Four sources, and the order is documented rather than emergent — “in order of specificity”, each overriding the one above:

$ helm upgrade --install checkout charts/service \
    --namespace shop --create-namespace \
    -f envs/prod-use1.yaml \
    --set image.tag=1.14.3

--set wins over the file, the file wins over the chart defaults. Which is exactly the shape you want: the chart says what a service is, the environment file says how this one differs, and the pipeline says which build.

One sharp edge worth knowing before it bites: values merge, they do not replace. If the chart default has a httpGet probe and your environment needs an exec probe, setting exec gives you a probe with both, which is invalid. The documented escape is to delete the default explicitly:

readinessProbe:
  httpGet: null          # removes the key from the merged values
  exec:
    command: [/bin/grpc_health_probe, -addr=:8080]

“override the value of the key to be null, in which case Helm will remove the key from the overridden values merge.”

See it before you ship it

Templating that you cannot inspect is worse than no templating. You can always render locally, without a cluster and without installing anything:

$ helm template checkout charts/service -f envs/prod-use1.yaml | head -30
$ helm template checkout charts/service -f envs/prod-use1.yaml \
    | kubectl apply --dry-run=server -f -

$ diff <(helm template checkout charts/service -f envs/staging.yaml) \
        <(helm template checkout charts/service -f envs/prod-use1.yaml)

That last command answers “how does prod differ from staging” in one line — the question the seventy-file layout could not answer at all.

The half that is not about YAML

Everything so far is deduplication, and Kustomize does that too. This is the part that is structurally different, and it is the reason the comparison is not a tie.

A Helm install creates a release: a named, numbered, stored thing. The templates can see it — Release.Revision is documented as “the revision number for this release. On install, this is 1, and it is incremented with each upgrade and rollback.”

$ helm history checkout -n shop
REVISION  UPDATED       STATUS      CHART          APP VERSION  DESCRIPTION
5         Aug 12 09:14  superseded  service-0.3.2  1.14.0       Upgrade complete
6         Aug 18 11:02  superseded  service-0.4.0  1.14.1       Upgrade complete
7         Aug 19 08:41  deployed    service-0.4.0  1.14.3       Upgrade complete

$ helm rollback checkout 6 -n shop

That is the thing kubectl apply has no answer for. Not “re-apply the old commit and hope” — a stored previous state, and one command to return to it. And because Release.IsInstall and Release.IsUpgrade are available to templates, a chart can behave differently on first install than on upgrade — seed a database once, skip it thereafter.

What is current, as of August 2026

This matters more than usual right now, because the version you are probably running is going out of support.

The server-side apply behaviour is the subtle one. Upgrading the CLI does not migrate your existing releases — they keep the apply method they were created with. So a fleet upgraded from Helm 3 runs in a mixed state, and the change in behaviour arrives whenever a release is next recreated rather than when you upgrade. Worth knowing before you are debugging it.

Where Helm is genuinely the wrong tool

Two sections of enthusiasm earn one of honesty.

• It templates text, not YAML. Go templates do not understand the document they are producing. That is why nindent is everywhere, and why a misplaced {{- can turn a valid chart into a parse error at install time. helm template is not optional practice; it is how you find out.
• A chart with forty conditionals is a program. Once values.yaml has grown a hundred keys and the templates are branching on all of them, you have written software with no tests in a language nobody chose. A values.schema.json helps; splitting the chart helps more.
• If you only need overlays, use overlays. One team, one cluster, a handful of manifests that differ in two fields — Kustomize is less machinery and does the job. The case for Helm gets stronger with every environment, cluster and consumer you add, and it is weakest at one of each.
• Distribution is where it wins hardest. If the chart is going to be consumed by somebody who did not write it — another team, another company, a public repository — the versioned, parameterised, self-describing package is the whole point, and nothing else in the ecosystem replaces it.

The arithmetic, once more

The first row is the one people quote. The last three are the ones that matter at three in the morning.