Design Partner programme: the first 100 teams get SreNix free. 100 places left.

See the offer and apply →
AI SRE for KubernetesSource-availableBring-your-own-LLMPatent Pending
Book a call
Examples / Stuck certificate request
Fix example · Community · auto-fixed

A stuck certificate request, cleared before it became an outage

cert-manager had a certificate request stuck in a failed state and was not retrying on its own. SreNix found it, cleared it, and confirmed the certificate came back healthy.

What happened

Follow the record on the right, one step at a time.

  1. Detect: a CertificateRequest for the internal API gateway had been stuck for over an hour, past cert-manager's own retry window.
  2. Cause: the request was in a terminal failed state. cert-manager will not automatically retry a request once it reaches that state — it waits for something to clear it.
  3. Policy: deleting a terminal CertificateRequest is one of the five actions on the default closed list. It is safe and reversible: cert-manager simply issues a new one from the owning Certificate.
  4. Fix: the stuck request was deleted. cert-manager created a fresh request within seconds.
  5. Verify: the same check ran again. The certificate reached Ready, and no terminal request remained.
fix record · api-gateway-tlsVERIFIED
03:12:04DETECTCertificateRequest/api-gateway-tls-8k2wq stuck in Failed for 71m
03:12:19CAUSEterminal state · cert-manager will not retry without intervention
03:12:24POLICYdelete-terminal-certificate-request · namespace istio-system · auto-approved
03:12:31FIXdeleted CertificateRequest/api-gateway-tls-8k2wq
03:12:44—cert-manager created CertificateRequest/api-gateway-tls-9n4rx
03:13:58VERIFYCertificate/api-gateway-tls Ready=True · Secret updated · no terminal requests
signed by workload identity · srenix-fixer@prod-eu-1

The rule that allowed it

Deleting a terminal certificate request is deliberately one of the smallest, safest actions in the default catalog — it never touches the certificate's key material or its Secret directly.

# fixers/k8s-limit-raise.policy.yaml
scope:
  namespaces: [payments, checkout]
  resources: [deployment]
bounds:
  memory_multiplier_max: 2.0
  requires_verification: true
  rollback_on_regression: true
approval:
  in_envelope: auto
  out_of_envelope: signed_url
  signer_roles: [sre-oncall]
identity: oidc://srenix-fixer

See another certificate example

An Ingress pointing at the wrong TLS secret, repointed and confirmed.

Read the edge fleets use case

Small clusters, far from anyone on call, where this class of fix matters most.

On-call should be quieter every week

Helm install in 5 minutes. No telemetry exfiltration. No per-investigation surprises.

Talk to usView pricing