The workload is gone, the disk is not: finding orphaned volumes

The workload is gone. The namespace is deleted. The pipeline that created it ran its cleanup and exited zero. And every month the storage bill charges you for the disk that workload was using. Nobody deleted the disk, because on most clusters nobody could say which disk it was.

This is the orphaned-volume leak: the persistent volume outlives the workload, the claim, and often the team that provisioned it. The good news is the cluster still records exactly which volumes are stranded, and it takes two commands to list them. The less good news is that some of those volumes hold data you must not delete.

How the volume outlives the workload

A dynamically provisioned volume inherits its StorageClass's reclaimPolicy, and that policy defaults to Delete. With Delete, removing the claim tears the backing disk down with it, and there is no leak. The leak needs the policy to be Retain. Teams set it deliberately, for databases and other stateful workloads, because Retain keeps the data safe when a claim is deleted. What it also keeps is the disk and its monthly cost.

The mechanics are visible in Kubernetes' own volume lifecycle. A volume enters the Released phase when its claim has been deleted but the cluster has not reclaimed the storage. With Retain, nothing reclaims it, ever, by design. A Released volume cannot bind to a new claim, because it still holds the previous claimant's data. So it sits: not in use, not reusable, not deleted, billing.

Two paths make this normal rather than exceptional:

1. 01

   Retain, then decommission

   A stateful app is removed or migrated. The claim is deleted, the volume goes Released, and the disk keeps its data and its cost line. The correct next step is a decision: reuse it after a manual clean-up, or delete the volume and the storage asset with it. In practice nobody owns the decision, so it defaults to "keep paying".

2. 02

   Resizing recovery left half-finished

   The documented manual recovery for a stuck volume resize tells you to set the volume's policy to Retain, delete the claim, and re-create it against the same volume. If the re-creation never happens, or happens with a new volume, you now have a Retain volume, a stray claim, and possibly two disks.

Find the stranded volumes

The cluster is the source of truth, not the cloud bill. The bill tells you the cost; it will not tell you which volume belonged to which deleted team.

kubectl get pv | grep Released
kubectl get pv -o json | jq -r '.items[]
  | select(.status.phase == "Released")
  | [.metadata.name,
     .spec.storageClassName,
     (.spec.capacity.storage // "?"),
     (.spec.claimRef.namespace // "?")] | @tsv'

Then close the loop in the cloud account, because the volume object and the disk are two different bills. A released AWS EBS volume usually surfaces as an unattached gp3 volume in the same region; a released persistent disk on GCP is an unattached disk. Where the object still exists, the provider's own API names the same asset.

aws ec2 describe-volumes \
  --filters Name=status,Values=available \
  --query 'Volumes[].[VolumeId,Size,VolumeType,Tags]' \
  --output table

One caution before you delete anything, and it is the whole reason this post exists rather than being a one-liner: a Released volume frequently holds the last copy of a database. The Retain policy was chosen precisely because nobody wanted that data thrown away. Treat every finding as a candidate archive until a human confirms otherwise.

Why nobody notices

• Released is a healthy-looking phase. No pod is failing, no probe is failing, no controller is logging errors. There is nothing to alert on unless you alert on the phase itself.
• The cost is slow. One 100 GiB gp3 volume is a line you will not notice on a departmental cloud bill; fifty of them, accumulated over three years of decommissions, is not.
• Ownership evaporates. The volume was created for a namespace, and the namespace, the app and the engineer who requested it are all gone. The volume outlives every human who could answer "is this still needed?"

Make orphans boring

• Run the Released listing weekly, as a report, not as a hunt when the bill spikes.
• Put an age on the finding: a volume released within the last week is probably mid-migration; one released for ninety days is a decision that fell on the floor.
• Give each finding one of three outcomes and write it down: reattach, archive-then-delete, or keep-and-pay-with-a-reason. "Unassigned" is the only wrong answer.
• Back up or image anything you plan to delete, then delete the volume object and the cloud asset, in that order. Deleting only the object leaves the disk; deleting only the disk leaves a broken object.
• Review your StorageClasses once a quarter: every Retain class is a promise that somebody will do the manual reclamation later.

Srenix watches for this shape of leak: it flags volumes that have been Released for more than seven days as one cost-leak finding, and it never deletes them. The reclamation call stays yours, because a released volume is often the last copy of a database.

Sources

The lifecycle, phase definitions and reclaim policies on this page are from the Kubernetes documentation on persistent volumes, including the documented volume-resize recovery procedure. The unattached-volume command matches the available state in the AWS CLI describe-volumes reference. The community thread that prompted this post is on r/kubernetes.