Homelab and GitOps

Simplified on purpose: no hostnames, addresses or guest IDs. The real topology is in the repos as Terraform and manifests.

1. 1Edit. Change a manifest in the cluster repo. Run kubectl diff -k against the live cluster to see exactly what will move. Commit explicit paths, never git add -A, and push.
2. 2Gate and build. The app's own CI runs its checks and builds an image tagged main-<timestamp>-<sha>. A bare SHA cannot be ordered; the timestamp is what "newest" means.
3. 3Select and commit. Flux's image reflector scans the registry; the image policy selects the newest sortable tag; the automation controller writes it into the Deployment manifest and commits the bump to main under its own bot name. 80 such commits between 2026-06-21 and 2026-09-21, counted when this page was built.
4. 4Roll out. Flux applies the new manifest. Helm-managed platform pieces are HelmRelease objects; app secrets are SopsSecret objects decrypted in-cluster, so the repo holds nothing in plain text.
5. 5Confirm. Readiness probes gate the rollout. Alertmanager rules and app-level probes (the Kopi Log freshness probe, pvewatch's heartbeat) confirm the result and route anything wrong to Telegram. Out-of-band drift is reverted: a manual kubectl scale was undone within one reconcile, on purpose, to prove it.

A monitor must be simpler and more available than what it watches. NFS mounts that detach after a NAS reboot, container binds that go stale, a VPN container that wedges: these were first "monitored" by workflows inside the cluster, which shared fate with the thing they watched and failed silently for days. Detection and self-heal now run as systemd timers on an always-on, battery-backed controller with SSH to every host. They export a textfile metric per watchdog that Prometheus scrapes, and post an edge-triggered event to n8n, which owns notification, routing and the Telegram credential. Low-level detection never goes back into n8n.

Also decided: no CI kubectl set image steps, ever, because Flux owns the Deployments and reverts that drift (that exact fight once froze a site). Image policies live in the Flux namespace, because a cross-namespace policy marker is silently ignored. Terraform for every guest definition, so a rebuild after a power event is a plan and an apply, not a memory exercise.

• Public source
  homelab-k8s: Flux entry point, HelmReleases, SopsSecrets, image automation. homelab-infra: one Terraform file per guest.
• Automation commits
  80 image-bump commits by the Flux bot, 2026-06-21 to 2026-09-21, in the cluster repo's history. Counted at build time on 2026-09-21.