~/tech-with-ugur

GitOps That Survives Bad Days: Flux, Helm, encrypted secrets, and self-healing rollbacks

2026-08-19 devopscybersecurity

Run the companion lab

Every GitOps quickstart tells the same story: commit a manifest, watch a controller apply it, marvel that you never ran kubectl apply. That story is true, and it is also not the point. The happy path is the easy 90% — the part that decides whether you actually trust GitOps is what the system does when a deploy is wrong. When the new image tag doesn’t exist. When a database migration hook wedges halfway. When the retry budget runs out and the controller simply stops. When someone “fixes production” by hand at 2am.

Those failure modes are exactly the parts the tutorials skip, which is why teams meet them for the first time in production, and why the fixes circulate as folklore in GitHub issue threads instead of documentation. So this post does it the other way around: build a complete Flux + Helm loop on a laptop — your own git server included, no cloud account, no GitHub repo — and then deliberately break it, five times, watching the actual recovery mechanics each time.

Everything here comes from a runnable lab. Each failure is a numbered script you can execute yourself, and a single e2e.sh drives all five scenarios end to end and exits non-zero if any promise in this post doesn’t hold.

A GitOps loop that fits on a laptop

The stack is three pieces, all local: a Gitea container as the git remote (with a web UI, so you can watch commits land the way you would on a real forge), a kind cluster running Flux’s controllers, and a small Node.js + Postgres app deployed through a Helm chart. Git is the only place you’re allowed to make a lasting change; everything else is a consequence.

One ./scripts/up.sh builds it all from nothing. The interesting part is the wiring that points Flux at the local Gitea — two small manifests, applied right after flux install.

labs/lab-flux-helm-gitops/infra/flux-sync.yaml:

apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
  name: flux-system
  namespace: flux-system
spec:
  interval: 30s
  ref:
    branch: main
  url: http://gitea:3000/labowner/fleet.git
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: apps
  namespace: flux-system
spec:
  interval: 1m
  path: ./apps
  prune: true
  sourceRef:
    kind: GitRepository
    name: flux-system
  decryption:
    provider: sops
    secretRef:
      name: sops-age
  # HelmRelease health is asserted by the scenarios; keeping wait off
  # stops broken-upgrade demos from flapping this Kustomization.
  wait: false

The GitRepository polls the fleet repo every 30 seconds; the Kustomization applies whatever it finds under apps/ — a namespace, the Postgres manifests, an encrypted secret we’ll get to shortly, and a HelmRelease that hands the app chart to helm-controller. Hold on to that decryption block; it’s the hero of the next section.

One honesty note before moving on. The official happy path here would be flux bootstrap git, but it wants a single git URL that both the Flux CLI on your host and the cluster’s source-controller can reach identically — and on Docker for Mac that’s simply not true for a local Gitea: the host reaches it at localhost:3000 while pods need the Docker-network address. The lab does the equivalent manual bootstrap instead — flux install, the sync manifests above, and a CoreDNS hosts entry that maps gitea to the container’s IP — which turns out to be more instructive anyway, because you see every part that bootstrap would have hidden.

When up.sh finishes you get:

GitOps loop is up:
  Gitea UI:   http://localhost:3000 (labowner / password in tmp/gitea-password)
  App:        http://localhost:8080/api/messages (token in tmp/api-token)
  Watch Flux: flux get helmreleases -n demo --watch

Now let’s break it.

Failure 1: the secret that wants to live in Git

GitOps has an obvious tension at its core: the repo is supposed to hold the entire desired state of the cluster, but some of that state is secret. Base64 in a Secret manifest is not encryption; committing it hands your API tokens to everyone with read access to the repo, forever, in history. This is the security backbone of the whole setup, and it’s where most real-world GitOps deployments quietly cut corners.

The current answer, and the one Flux’s own docs now recommend for new setups, is SOPS with age: encrypt the secret’s values (not the whole file) before committing, and let kustomize-controller decrypt at apply time with a private key that lives only in the cluster. Here is the entire encryption step from the bring-up script — note that the plaintext values are generated at runtime and the intermediate plaintext file survives for exactly three lines:

labs/lab-flux-helm-gitops/scripts/up.sh:

cat > tmp/secret.plain.yaml <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: demo-secrets
  namespace: demo
type: Opaque
stringData:
  apiToken: $(cat tmp/api-token)
  pgPassword: $(cat tmp/pg-password)
EOF
sops --encrypt --age "${AGE_RECIPIENT}" \
  --encrypted-regex '^(data|stringData)$' \
  tmp/secret.plain.yaml > tmp/fleet/apps/secret.enc.yaml
rm tmp/secret.plain.yaml

The --encrypted-regex flag is what keeps the file a valid, diffable Kubernetes manifest: kind, metadata, and structure stay readable, and only the values under stringData become ciphertext. The decryption.provider: sops block you saw in the Kustomization closes the loop: kustomize-controller notices SOPS metadata in an applied manifest and decrypts it with the age key from the in-cluster sops-age secret — plaintext exists only inside the cluster, never in git.

The first scenario script proves the property end to end rather than asserting it. What git stores, verbatim from the repo:

--> What git actually stores (sops metadata, ciphertext values):
    apiToken: ENC[AES256_GCM,data:gR7pze0K8y7w+BHm2aFAJkEO+zr6AgUPO3CIdvizj3s=,iv:...
sops:
    age:
          recipient: age1n2wfcyk2d7dj899ah4g2kpl5zk5mevys5dhjll8vdjgtx7xfzghs78t29h
OK: the real token appears nowhere in git
--> The cluster Secret exists (kustomize-controller decrypted it with the age key):
demo-secrets
--> A request without the token is rejected:
OK: 401
--> A request with the decrypted token succeeds:
{
  "version": 1,
  "messages": [ { "id": 1, "body": "hello from schema v1" } ]
}

The script greps the encrypted file for the real token and fails the run if it ever appears — and then proves the decrypted value works by authenticating to the app with it. Not “the secret got deployed”: the decrypted secret is what’s serving traffic.

Failure 2: the migration that must run first

Upgrades that touch a database schema have an ordering constraint that a plain Deployment rollout cannot express: the migration must complete before the new code sees the database. Helm’s answer is hooks, and under Flux they work the same way — helm-controller shells out to the Helm SDK, hooks included. The lab’s chart ships a real migration Job as a pre-upgrade hook:

labs/lab-flux-helm-gitops/config-repo/charts/demo-app/templates/migrate-job.yaml:

apiVersion: batch/v1
kind: Job
metadata:
  # Version-qualified so each upgrade gets a fresh, findable Job object.
  name: demo-app-migrate-{{ .Values.migrateTo }}-{{ .Values.image.tag }}
  labels:
    app.kubernetes.io/name: demo-app-migrate
  annotations:
    helm.sh/hook: pre-install,pre-upgrade
    helm.sh/hook-weight: "0"
    helm.sh/hook-delete-policy: before-hook-creation
spec:
  backoffLimit: 3
  # Below the HelmRelease timeout (2m) so a wedged hook fails the release
  # inside a single attempt window instead of racing it.
  activeDeadlineSeconds: 110
  template:
    metadata:
      labels:
        app.kubernetes.io/name: demo-app-migrate
    spec:
      restartPolicy: Never
      containers:
        - name: migrate
          image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
          imagePullPolicy: {{ .Values.image.pullPolicy }}
          command: ["npx", "tsx", "src/migrate.ts"]
          # ...

Three details in there do the actual work. helm.sh/hook: pre-install, pre-upgrade makes Helm run the Job — and wait for it — before touching the Deployment at all. helm.sh/hook-delete-policy: before-hook-creation means each upgrade deletes the previous hook Job before creating the new one, so failed Jobs stick around for debugging but never collide. And activeDeadlineSeconds: 110 is deliberately below the HelmRelease’s 2-minute per-attempt timeout: a wedged migration fails inside one attempt window instead of racing the release timeout — pick these two numbers independently and you get flaky, ambiguous failures.

The migration itself is ordinary application code — versioned SQL steps, each applied in a transaction and recorded in a schema_migrations table. The v2 step the upgrade will run:

labs/lab-flux-helm-gitops/app/src/db/migrations.ts:

  {
    id: 2,
    name: "add-author",
    sql: `
      ALTER TABLE messages ADD COLUMN IF NOT EXISTS author TEXT NOT NULL DEFAULT 'anonymous';
      INSERT INTO messages (body, author)
      SELECT 'hello from schema v2', 'flux'
      WHERE NOT EXISTS (SELECT 1 FROM messages WHERE author = 'flux');
    `,
  },

Scenario 2 pushes the v2 release to the fleet repo and then does something most demos skip: it proves the ordering instead of narrating it, by comparing Kubernetes’ own timestamps for the hook Job and the new pod:

labs/lab-flux-helm-gitops/scripts/scenario-2-migration.sh:

job_done="$(kubectl -n demo get job demo-app-migrate-2-v2 -o jsonpath='{.status.completionTime}')"
# Compare against the pod's running.startedAt (not status.startTime): startTime is set
# the moment the pod object is created and can land in the same second as the job's
# completionTime, making a plain string compare flaky. running.startedAt is recorded
# once the container is actually up, which is safely later. Also filter to a pod that
# is actually running and take the most recently started one, since a rolling update
# with replicas=1 can briefly show the outgoing v1 pod alongside the new v2 one.
pod_start="$(kubectl -n demo get pods -l app.kubernetes.io/name=demo-app -o json \
  | jq -r '[.items[] | select(.status.containerStatuses[0].state.running != null)]
      | sort_by(.status.startTime) | last | .status.containerStatuses[0].state.running.startedAt')"
echo "    migration completed: ${job_done}"
echo "    v2 pod started:      ${pod_start}"
[[ "${job_done}" < "${pod_start}" ]] || { echo "FAIL: pod started before the migration finished"; exit 1; }
--> Proving hook ordering: the migration finished BEFORE the v2 pod started
    migration completed: 2026-08-18T22:07:41Z
    v2 pod started:      2026-08-18T22:07:44Z
OK: hook ran first

The migration finished; three seconds later the first v2 container started. If the Job had failed, the Deployment update would never have happened — which is exactly the property the next failure leans on.

Failure 3: the upgrade that can never succeed

With helm on the command line you’d reach for --atomic: roll back automatically if the upgrade fails. In GitOps there is no command line — the upgrade is triggered by a commit, possibly at 3am from a CI pipeline nobody is watching. The equivalent has to live in the declaration itself, and in a HelmRelease it’s the remediation block:

labs/lab-flux-helm-gitops/config-repo/apps/helmrelease.yaml:

apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: demo-app
  namespace: demo
spec:
  interval: 1m
  # Per-attempt budget for install/upgrade, including hooks.
  timeout: 2m
  chart:
    spec:
      chart: ./charts/demo-app
      sourceRef:
        kind: GitRepository
        name: flux-system
        namespace: flux-system
      interval: 1m
      reconcileStrategy: Revision
  # Revert manual kubectl edits to what Helm rendered.
  driftDetection:
    mode: enabled
  install:
    remediation:
      retries: 1
  upgrade:
    remediation:
      # Roll back on failure; after 2 failed retries the release is
      # left in the "upgrade retries exhausted" state on purpose.
      retries: 2
      strategy: rollback
      remediateLastFailure: true
  values:
    image:
      repository: demo-app
      tag: v1
    appVersion: "1"
    migrateTo: "1"

Scenario 3 commits v3 — an image tag that was deliberately never built, so the migration hook’s pod can never be scheduled. The hook stalls, the attempt times out, and upgrade.remediation.strategy: rollback with remediateLastFailure: true tells helm-controller to immediately roll the release back to the last good revision. Helm’s own history tells the story better than any dashboard:

--> Helm's own history tells the story:
REVISION  UPDATED                   STATUS      CHART                        APP VERSION  DESCRIPTION
4         Tue Aug 18 22:15:01 2026  superseded  demo-app-0.1.0+85f61d2fbe4a  1            Upgrade complete
5         Tue Aug 18 22:15:07 2026  failed      demo-app-0.1.0+85f61d2fbe4a  1            Upgrade "demo-app" failed: pre-upgrade hooks failed: failed early due to stalled resources: [Job/demo/demo-app-migrate-2-v3 status: 'Failed']
6         Tue Aug 18 22:17:03 2026  deployed    demo-app-0.1.0+85f61d2fbe4a  1            Rollback to 4

Revision 5 failed in the pre-upgrade hook; revision 6 is the automatic Rollback to 4 (that’s Helm v4’s wording — older Helm versions you’ll meet in the wild phrase it slightly differently, same mechanism). And here is the part worth internalizing: the app never stopped serving v2. Because the failure happened in the hook, Kubernetes never replaced a single running pod — the broken release lost before it ever reached the Deployment. That is what “atomic” means when it’s declared in git instead of typed on a command line.

Failure 4: the stuck state everyone eventually meets

Scenario 3 ends with a warning: Flux will keep retrying v3. The retries: 2 budget means three failed attempts in total, and then comes the state that generates more confused GitHub issues than any other part of Flux’s Helm story — the release that is stuck and will not retry, ever, no matter how many times you mash flux reconcile.

Search for it online and you’ll find it under the name “upgrade retries exhausted”. Worth knowing before you go grepping your own cluster: that message is the older controller wording. On the stack this lab pins (Flux 2.9.4, helm-controller 1.6.3), exhaustion surfaces as a separate Stalled condition with reason RetriesExceeded — the same state, different words, and your alerts should match on the condition, not the folklore string:

--> Waiting for Flux to give up on v3 (retries: 2 means three failed attempts total)...
OK: HelmRelease Stalled condition reports RetriesExceeded
--> The stuck state, verbatim:
    Stalled: Failed to upgrade after 3 attempt(s)
    Ready:   Helm rollback to previous release demo/demo-app.v8 with chart demo-app@0.1.0+85f61d2fbe4a succeeded
--> Note what Flux does now: nothing. No retry loop, no self-recovery.
    Forcing reconciliation of the same broken spec does not help:
OK: still stuck — only a spec change (or suspend/resume) resets the retry budget

Read that Ready message carefully — it says a rollback succeeded. The app is healthy, still serving v2. The release is stuck anyway. This is the counterintuitive core of the state: Stalled is not “the app is down”, it’s “the controller has concluded that retrying this exact spec is pointless and is waiting for you to change something.”

Which is also the recovery. The retry budget resets when the HelmRelease’s generation changes — that is, when the spec actually changes. The scenario proves the negative first (forcing reconciliation of the same broken spec leaves it stuck), then fixes the desired state in git the way you would in real life:

labs/lab-flux-helm-gitops/scripts/scenario-4-recovery.sh:

echo "--> The recovery: fix the desired state in git. Committing working v4."
fleet_fresh_clone
set_release v4 2 2
fleet_push "Deploy demo-app v4"

Push a working v4, the values change bumps the generation, helm-controller immediately tries again, and the release goes Ready. No kubectl delete helmrelease, no manual helm rollback, no surgery — the git push is the unstuck command. And when there is genuinely nothing to change in git (the spec was fine; something external broke and got fixed), the documented equivalent is a suspend/resume pair, which also resets the budget:

flux suspend helmrelease demo-app -n demo && flux resume helmrelease demo-app -n demo

What you should not do is delete the release to make the status go green — that throws away Helm’s revision history and, depending on your chart, the resources under it. The stuck state is annoying precisely because it’s safe: nothing is on fire, and both exits are deliberate and reversible.

Failure 5: the 2am kubectl edit

The last scenario is the shortest and lands the philosophical point. Someone “fixes production” by hand — kubectl scale deployment demo-app --replicas=3, the classic — and then the HelmRelease’s driftDetection.mode: enabled (you saw it in the spec above) does its job:

--> Someone 'fixes production' by hand:
deployment.apps/demo-app scaled
replicas now: 3
--> Asking Flux to reconcile (normally the 1m interval would catch it):
OK: replicas corrected back to 1
--> Git said 1 replica, so it is 1 replica again. The cluster is not the source of truth.

On every reconciliation, helm-controller diffs the live objects against what the current release actually rendered — not just against what it remembers deploying — and patches drift straight back. You can watch both halves in flux events: a DriftDetected event immediately followed by DriftCorrected. The manual edit didn’t survive one reconcile interval, and that’s the whole contract: the cluster is a projection of git, not a database of its own.

What to take back to your real cluster

The lab (up.sh, five scenario scripts, an e2e.sh that runs the entire story and exits non-zero if any of it stops being true) is the proof; these are the portable conclusions:

The full lab, with prerequisites, the honesty notes about where local networking diverges from the cloud path, and the one-shot verification, is in the lab README. Clone it, break all five things, and watch every one of them come back.