Multipart Upload & Request Performance

## The ceiling that forces the issue A single `PutObject` request to Amazon S3 can carry at most 5 GB of data, and the request body is one HTTP entity: one connection, one shot, no resume. An object created through a multipart upload can be up to 5 TiB. So for anything genuinely large, multipart is not an optimization — it is the only way in. But the size ceiling is the least interesting reason to use it. A 4 GB single `PutObject` that dies at 96% has to start over from byte zero. That is the real problem multipart solves. ## The three-call protocol The API is deliberately small: 1. **`CreateMultipartUpload`** — you name the bucket and key and set the object's metadata (content type, storage class, encryption headers, ACL/ownership settings). S3 returns an **`UploadId`**, an opaque token identifying this in-flight upload. Note that metadata is fixed here, at the *start*, not at completion. 2. **`UploadPart`** — you send one chunk with the `UploadId` and a `PartNumber` between 1 and 10,000. S3 stores the chunk and returns an **ETag** for it. Parts may be uploaded in any order and from any number of machines or threads; the `PartNumber`, not the arrival order, determines position in the final object. Re-uploading the same `PartNumber` overwrites the previous attempt. 3. **`CompleteMultipartUpload`** — you send the ordered list of `{PartNumber, ETag}` pairs. S3 concatenates the parts, and only at that moment does the object become visible at its key. There is a fourth call, **`AbortMultipartUpload`**, which throws away an upload and its parts. `ListMultipartUploads` shows open uploads in a bucket and `ListParts` shows the parts already received for one of them — that pair is how a resuming client works out what it still needs to send. ```bash UP=$(aws s3api create-multipart-upload --bucket my-bucket --key big.bin \ --query UploadId --output text) aws s3api upload-part --bucket my-bucket --key big.bin \ --upload-id "$UP" --part-number 1 --body part1.bin ``` ## What exists while the upload is open This is the detail candidates most often miss. Between `CreateMultipartUpload` and `CompleteMultipartUpload`, the parts are **stored and billed**, but there is **no object**. `GetObject` on the key returns 404 (or the previous version of the object, if one exists — an in-flight multipart upload never partially overwrites what is already there). `ListObjectsV2` does not show the parts either; only `ListMultipartUploads` does. An upload that is never completed and never aborted sits there indefinitely, invisible in the object listing and visible only on the bill. Completion is atomic from a reader's point of view: the object appears whole or not at all. ## Why you use it below 5 GB too - **Parallelism.** One TCP stream rarely fills a fat pipe. Ten concurrent `UploadPart` calls usually do. - **Retry granularity.** A transient network failure costs you one part, not the whole transfer. - **Resumability.** The `UploadId` is durable, so a client can come back after a crash, call `ListParts`, and send only what is missing. - **Unknown length.** You can stream data whose total size you do not know in advance, deciding part by part when to stop. The AWS CLI and the SDK transfer managers switch to multipart automatically above a threshold — the CLI's default `multipart_threshold` is 8 MB — so `aws s3 cp` already does all of this for you. ## The ETag changes shape For a normal `PutObject` of an unencrypted object, the ETag is the MD5 of the content. For a multipart object it is **not**: it is a digest computed over the parts' digests, followed by a dash and the number of parts, like `"a1b2...-137"`. Any integrity check that assumes "ETag equals MD5 of my local file" breaks the moment a file crosses the multipart threshold. If you need end-to-end verification, use S3's checksum support — pass a `ChecksumAlgorithm` such as CRC32C or SHA256 — rather than reverse-engineering the ETag. ## One completion trap `CompleteMultipartUpload` can take a long time for a large object, so S3 keeps the connection alive and may return **HTTP 200 with an error document in the body**. A client that checks only the status code will treat a failed completion as a success. The AWS SDKs parse the body for you; hand-rolled HTTP clients frequently do not. ## Practical shape In interviews, the answer that lands is: three calls plus abort, parts numbered 1–10,000, nothing is visible until complete, the win is parallelism and retry granularity as much as size — and in real code you let the transfer manager drive it.