Skip to content

Designing infrastructure that can rebuild itself

The tools meant to rebuild the platform were running on the platform itself. I split out a small seed environment to break that loop.

Developed 2025–2026 · Independent engineering lab · Details sanitised

The problem

As more of the environment became declarative, the tools used to rebuild it—source control, infrastructure automation and GitOps controllers—were themselves hosted on the platform they were expected to recover.

A total virtualisation-platform rebuild could therefore remove both the infrastructure and its recovery mechanism. Kubernetes also needed a day-zero path before all of its normal DNS, secrets and source-control dependencies were available.

The path through the system

  1. Seed environment
  2. Source of truth
  3. Infrastructure automation
  4. Cluster lifecycle
  5. GitOps control
  6. Workloads

What I did

I modelled the environment as dependency levels and looked for loops in the recovery path. The key loop was simple: the main platform hosted the services whose job was to recreate the main platform.

  • Separated a small, persistent seed environment from higher-order infrastructure.
  • Defined the minimum source, automation and access needed to reconstruct the next layer.
  • Designed an initial GitOps hand-off so the cluster could converge on Git as its source of truth after day zero.
  • Separated cluster-wide platform components from tenant and namespaced applications.
  • Kept break-glass access and bootstrap secrets explicit instead of hiding them behind services that might not exist during recovery.

What changed

The resulting architecture gives disaster recovery a defined starting point outside the principal failure domain. The seed layer remains intentionally small; once it recreates the platform and GitOps control plane, the normal declarative workflow can take over.

The design has since evolved with the wider lab—from earlier lightweight Kubernetes experiments toward Talos and Cluster API on Proxmox—without changing the core recovery principle.

Declarative infrastructure still needs an independently recoverable root. GitOps describes the desired state, but it cannot help when the only systems able to read and apply that state disappeared with the platform.

A bootstrap environment earns its reliability by staying smaller and simpler than what it rebuilds.

Technologies

GitOps · Argo CD · Terraform · Cluster API · Talos Linux · Proxmox · External Secrets (bootstrap design) · Disaster recovery