The OOM kill you never see: handling memory bursts without losing a six-hour job

Memory-bursty containers are the ones that hurt most. A batch job, an indexer or a database-like workload climbs toward its memory limit, the kernel kills it with exit code 137, Kubernetes reports OOMKilled, and hours of progress are gone.

The problem

A recent community post described a sidecar that watches a container's memory and asks for more headroom just before the limit, using the in-place pod resize subresource so the container never restarts. The author was candid about the hard parts: how fast a resize actually applies on a loaded node, how to never put the node itself at risk, and the fact that a very sharp burst can still outrun any rescue path. Those hard parts are real, and they apply to every approach to this problem. This article is a proposed design; we have not run it.

Why it keeps happening

• Limits are guessed once and never revisited.
• Autoscalers that read metrics react after the kill, because the metrics pipeline lags the kernel.
• Restarting is the default Kubernetes response, and a restart is the worst outcome for a job that cannot resume.

A practical solution architecture

Solution label: Proposed architecture. A layered design that holds up:

1. Size the baseline from measured history. Use the p95 or p99 working set over several weeks (container_memory_working_set_bytes) instead of the number from the day the chart was written. A vertical autoscaler in recommendation mode is a good source.

2. Give bursts headroom without risking the node. Raise requests together with limits so the scheduler accounts for the borrowed memory, cap the total at a safe fraction of node memory, and give it back when the burst passes. In-place resize makes this possible without a restart on recent Kubernetes versions; check support in your version.

3. Make long jobs resumable. Checkpoint progress so that even a kill costs minutes, not hours.

4. Keep a bounded reactive fix for the kills that still happen. First confirm it is a container-limit kill and not node pressure:

   kubectl get pod <pod> -o jsonpath='{.status.containerStatuses[*].lastState.terminated.reason}'
   kubectl describe node <node>
   kubectl top pod <pod> --containers
   

   The first command should say OOMKilled with exit code 137, and the node should show no MemoryPressure. Compare real usage with the limit, raise the limit by a bounded step, roll it out, and watch for restarts across a full backoff cycle before calling it fixed.

5. Alert on restarts and on the working-set-to-limit ratio, not only on the crash.

6. Treat a leak differently from a burst. If memory climbs forever, raising limits only moves the failure later. Cap how far any automatic change may go.

Raising a memory limit changes production and can pressure the node. Keep every increase bounded, confirm the node has headroom, and prefer a reviewed change for anything beyond a small step.

Where Srenix fits

Srenix is an autonomous AI SRE that runs inside your cluster, and it covers layer 4 and the recurrence class. It is in Beta, and we are working with Design Partners now. It notices the crash loop, reads the exit code, the restart history and the actual memory use, and identifies an OOMKilled container.

It raises the memory limit only within the bounds you set (for example, at most twice the current limit), restarts the rollout, and keeps watching until the pods have stayed up before it marks the finding fixed. The re-check is the same detection that raised the finding, not a guess.

Nothing happens outside your rules: autonomy is off until you turn it on, only the action kinds you list can run, protected namespaces are never touched, a rollback must exist, and confidence has to be at least 0.95. Anything else waits for one tap in Slack. A fix that is later reverted counts as a failure, and a circuit breaker returns that kind of fix to manual approval. Every step lands in a signed, hash-chained audit trail.

Honest limits: Srenix is reactive. It does not prevent the first kill, and a restart still loses in-flight work, so for long jobs that cannot resume, pair it with checkpointing or in-place resize and measured baselines. It does not fix memory leaks in your code. What it removes is the 2am ritual of finding the same OOMKilled pod, editing the same limit and watching the same rollout, again.

Tradeoffs and failure modes

• Sharp bursts can outrun any rescue path, reactive or in-place.
• Headroom costs capacity. Raising requests reduces what else fits on the node.
• Limit increases can hide leaks and move the failure later.
• Checkpointing has a cost in code changes and I/O, but it is the only protection against the first kill.

How to validate

Acceptance criteria are targets, not measured results:

• an OOMKilled test pod is identified as a container-limit kill within the detection window;
• the limit is raised only within the configured bound, and the pods stay up across a full backoff cycle;
• a simulated leak stops at the cap and escalates to a human;
• the audit record shows the before and after values.

Test plan: run a memory-ramp test workload in a staging namespace with a deliberately low limit, observe the kill, apply the bounded fix, then repeat with a leak pattern to confirm the cap holds.

Conclusion

Size from measured history, give bursts bounded headroom, make long jobs resumable, and keep a bounded reactive fix for the rest. To see it handle an OOMKilled workload on your own cluster, you can talk to us at reliora.ai.