Examples / Container out of memory
Fix exampleContainer keeps running out of memory
The payments service kept restarting. SreNix found why, raised the limit within a safe cap, and checked the restarts stopped.
What happened
Follow the record on the right, one step at a time.
- Detect: the service crashed 8 times in 6 minutes.
- Cause: its memory limit was lower than what it really uses.
- Policy: the rules allow raising memory up to 2× in this namespace.
- Fix: limit raised, service restarted.
- Verify: no restarts for 6 minutes.
fix record · prod-eu-1VERIFIED
10:34:12DETECTCrashLoopBackOff · payments-api · OOMKilled 137
10:34:48CAUSEmemory limit 512Mi vs p99 working set 780Mi
10:35:02POLICYlimit-raise ≤ 2× · namespace payments · auto-approved
10:35:19FIXpatched limits 512Mi → 1Gi, rollout restarted
10:41:30VERIFY0 restarts, error rate 0.02%, 6 min observed
signed by workload identity · srenix-fixer@prod-eu-1
The rule that allowed it
# fixers/k8s-limit-raise.policy.yaml
scope:
namespaces: [payments, checkout]
resources: [deployment]
bounds:
memory_multiplier_max: 2.0
requires_verification: true
rollback_on_regression: true
approval:
in_envelope: auto
out_of_envelope: signed_url
signer_roles: [sre-oncall]
identity: oidc://srenix-fixer
On-call should be quieter every week
Helm install in 5 minutes. No telemetry exfiltration. No per-investigation surprises.