Skip to content

Diagnosing a tier-0 outage and redesigning its failure domains

A stalled PostgreSQL recovery turned out to be a storage failure underneath it, changing both the recovery plan and the design.

April–May 2026 · Independent engineering lab · Details sanitised

The problem

A node failure left PostgreSQL stalled during data-directory synchronisation on a three-node infrastructure cluster. The database depended on a replicated filesystem mounted through FUSE, and several foundational DNS, identity and certificate services depended in turn on the database.

Database panic messages made a database reset look tempting, while storage errors and an unresponsive filesystem indicated that PostgreSQL was blocked beneath the application layer.

The path through the system

  1. Tier-0 services
  2. PostgreSQL
  3. FUSE mount
  4. Replicated filesystem
  5. Peer nodes
  6. Local storage

What I did

I treated the database message as a symptom and tested the storage dependency first. The combination of a failed peer, filesystem connection errors and a database process waiting during pre-sync was consistent with PostgreSQL waiting on unhealthy I/O rather than an isolated write-ahead-log failure.

  • Set the recovery sequence to stop further database activity and preserve the original data directory before any database-level repair.
  • Defined checks across cluster peer and brick state, healing, filesystem responsiveness, local storage health and kernel I/O evidence.
  • Rejected an immediate write-ahead-log reset because the underlying storage path had not been proven healthy.
  • Selected healthy local storage as the recovery target so database validation would not continue to depend on the failed distributed filesystem.
  • Mapped the circular dependencies among database, DNS, identity, PKI and control-plane services.

What changed

The investigation bounded the fault at the storage layer and avoided recommending a potentially destructive database reset. The chosen recovery path required preserving the data directory and proving a healthy local storage target before database-level repair.

The incident produced a broader architecture direction: stateful tier-0 services should use replication designed for the service, with local storage and independent quorum, instead of assuming that one shared filesystem makes every workload highly available.

Availability is not created by replication alone. If the same storage failure can block a database and the services needed to repair it, the platform has concentrated rather than removed its failure domain.

For PostgreSQL, application-aware replication and node-local storage provide clearer recovery semantics than placing the live data directory on a shared FUSE filesystem.

Technologies

PostgreSQL · GlusterFS · Linux · FUSE · Docker Swarm · Distributed storage · Patroni (design) · Disaster recovery