Skip to content

When a certificate failure was really a firewall-path problem

A certificate request failed inside Kubernetes. The route existed, but a firewall policy elsewhere in the path was blocking it.

July–August 2026 · Independent engineering lab · Details sanitised

The problem

Certificate operations were failing for applications reached through the Kubernetes ingress path. A separate management load-balancer path continued to work, while the relevant service route was present in the firewall's routing table.

The visible symptom sat in the certificate workflow, but the contrast between the working and failing paths suggested that the certificate controller and internal certificate authority were not sufficient explanations on their own.

The path through the system

  1. Application
  2. Ingress
  3. Service VIP
  4. BGP route
  5. Firewall policy
  6. Internal CA

What I did

I followed the complete request path and compared it with the known-good management path. The advertised route confirmed that the firewall knew how to reach the Kubernetes VIP; it did not prove that the relevant policy and session path would permit the traffic.

  • Confirmed that the failure was limited to services behind one ingress and load-balancer path.
  • Verified the Kubernetes service advertisement and the corresponding BGP route at the firewall.
  • Separated reachability, route selection and policy enforcement into distinct checks.
  • Narrowed the fault to firewall policy and session handling between the relevant internal zones.
  • Avoided rotating certificates or changing the internal CA when the evidence did not point there.

What changed

The investigation isolated the failure domain to the firewall path rather than cert-manager, the ingress controller or the certificate authority. That gave the remediation a precise boundary and avoided destabilising otherwise healthy PKI and Kubernetes components.

The diagnostic result also made the service path easier to reason about: a learned route is only one part of connectivity when policy enforcement and stateful sessions sit between the requester and the destination.

An error should not define the boundary of an investigation. Certificate failures can originate in routing and security controls, and successful diagnosis depends on proving each transition in the real traffic path.

For segmented platforms, route validation, policy validation and application-level trust validation should remain separate operational checks.

Technologies

Kubernetes · cert-manager · Step CA · Traefik · MetalLB · BGP · FortiGate · TLS / PKI