Your S3 bucket's storage bill is far larger than the total size of the objects that ListObjectsV2 returns for it. What is the most likely cause, and how do you fix it permanently?

## The symptom Someone compares the bucket's billed storage against the sum of object sizes from a listing and finds a large, unexplained gap. Sometimes the gap is bigger than the bucket's real contents. Two things produce this in S3, and you should name both before diving in: 1. **Noncurrent object versions** in a versioned bucket — old versions and delete markers are billed but hidden from a default listing. 2. **Incomplete multipart uploads** — stored parts belonging to uploads that were never completed. If the bucket takes large uploads, the second is usually the culprit and is the one people forget entirely. ## Why the parts persist A multipart upload has exactly two terminal states: **completed** or **aborted**. There is no timeout. If a client calls `CreateMultipartUpload`, streams 40 GB of parts, and then the process is killed, the upload simply stays open. S3 keeps every part it received. Those bytes are charged at the storage class the upload was created with, for as long as the upload remains open — which is forever, absent intervention. The reason it hides so well is that the parts are not objects. `ListObjectsV2` reports objects, and no object exists at the key until completion. The parts are reachable only through `ListMultipartUploads` (which uploads are open) and `ListParts` (what a given upload holds). The pathological version is a retry loop. A job uploads a 200 GB file, fails at 90%, and its retry logic starts a *new* multipart upload from scratch rather than resuming the old `UploadId`. Three retries later you are paying for roughly 540 GB of orphaned parts and a 200 GB object, and nothing in the object listing has changed. ## Confirming it ```bash aws s3api list-multipart-uploads --bucket my-bucket # For one open upload, see how much data it is actually holding. aws s3api list-parts --bucket my-bucket --key big.bin --upload-id "$UPLOAD_ID" ``` `ListMultipartUploads` gives you the key, the `UploadId`, the initiator, and the initiation timestamp — the timestamp is what tells you whether an upload is genuinely abandoned or merely slow. `ListParts` gives per-part sizes, which is how you convert "37 open uploads" into a number of gigabytes. For an account-wide view, **S3 Storage Lens** reports incomplete-multipart-upload bytes and object counts per bucket, and it is the right tool when you do not yet know which bucket is leaking. You can abort one immediately: ```bash aws s3api abort-multipart-upload --bucket my-bucket \ --key big.bin --upload-id "$UPLOAD_ID" ``` ## The permanent fix A one-off cleanup script is remediation, not a fix — the leak resumes the next time a job dies. The durable answer is an S3 **lifecycle rule** with the `AbortIncompleteMultipartUpload` action: ```json { "Rules": [{ "ID": "abort-stale-multipart-uploads", "Status": "Enabled", "Filter": { "Prefix": "" }, "AbortIncompleteMultipartUpload": { "DaysAfterInitiation": 7 } }] } ``` S3 then aborts any upload still open more than seven days after it was initiated, discarding its parts and their charges. Applied with `aws s3api put-bucket-lifecycle-configuration`. Choosing `DaysAfterInitiation` is a judgment call: it must be comfortably longer than your slowest legitimate upload, including any resume window your clients rely on. A day is fine for a fast ingest pipeline; a week is a safe general default; anything measured in months defeats the purpose. Note that the action is time-based only — there is no way to say "abort when the initiator's job has failed", because S3 has no idea what your job is doing. ## The other half: fix the client The lifecycle rule caps the damage; it does not stop the leak. Two client-side habits matter: - **Abort on failure.** Wrap the upload so that a non-retryable error calls `AbortMultipartUpload`. The SDK transfer managers generally do this; hand-written loops usually do not. - **Resume, do not restart.** Persist the `UploadId`, and on retry call `ListParts` and send only what is missing. This is faster *and* it stops the leak from multiplying. ## What to say in the interview Name both hidden-storage causes, say why parts are invisible to the object listing, show that you would measure with `ListMultipartUploads` or Storage Lens before acting, and land on the lifecycle rule as the durable control — then add that you would also fix the client's retry path, because the rule bounds the cost rather than eliminating the cause. Finally: put that lifecycle rule on new buckets by default. It costs nothing and it is the single most common piece of S3 hygiene that teams discover only via the bill.