What a certificate expiry outage looks like
It is never one error — it is six errors that all mean the same thing.
Internal TLS breaks first. The metrics scrape fails, so dashboards go dark and alerting goes quiet, which hides the real story. Admission webhooks reject workloads cluster-wide: every new pod in every namespace fails to create at once, which looks like a control-plane incident rather than a certificate. Ingress controllers fall over, and browsers get the certificate error that finally tells the customer.
The signature is unrelated things failing in the same minute. A cluster-wide
admission failure with a healthy control plane is a certificate, nine times out of
ten. It is x509: certificate has expired somewhere in the first
twenty minutes of correlating — and correlation is exactly what nobody is
doing at 3am with a pager full of symptoms.
Find the ones expiring before they do
Every certificate in your clusters has a known end date. The first command reads the serving certificate; the second reads what cert-manager itself thinks is coming due.
echo | openssl s_client -connect api.example.com:443 -servername api.example.com 2>/dev/null \
| openssl x509 -noout -dates
kubectl get certificates -A -o wide
kubectl get certificate <name> -n prod -o jsonpath='{.status.conditions}'The list matters more than any single cert: one expiring ingress certificate is an incident; a cluster where nobody can say how many certificates exist is a queue of incidents with the dates already printed. Export both commands’ output to a weekly report and put a 30-day threshold on it.
Why renewal fails quietly
Certificates do not only expire; they expire while everyone believes renewal is automatic. The quiet failure modes:
- 01
Issuer rate limits
Let’s Encrypt limits per registered domain per week. A cluster that renews early because someone’s fix attempt retried the whole path hits the limit, and every renewal behind it queues silently.
- 02
A stale DNS01 zone
The solver credential lost permission, or the zone was renamed during a migration. Issuance fails on new certificates while existing renewals look healthy for weeks.
- 03
An issuer mismatch
Some certificates still reference an old issuer, or a staging issuer that browsers do not trust. They renew forever and never work.
- 04
No alert on the condition
The Certificate’s status condition went False months ago. Nobody reads condition fields; you are supposed to alert on them.
Make the next expiry boring
- Alert at 30, 14 and 7 days — on the certificate object, not on hand-written calendar reminders.
- Verify renewal once per quarter in the same way the expiry would fire: a real TLS handshake against a test hostname, not a dashboard glance.
- One issuer per purpose, referenced through namespaced defaults nobody hand-edits per Certificate.
- Treat every certificate as on-call surface: a cert that expires is an incident, a cert whose renewal nobody can prove is an incident with more lead time.
Srenix checks certificate expiry before anything fails: it reads the same status conditions, correlates an admission failure back to the expired certificate in seconds instead of an hour, and after the renewal re-runs the failed handshake as the proof of fix.