You are uploading 500 GB objects to Amazon S3 with multipart uploads. What constrains the part size you can choose, and how do you decide on one?

## The three hard limits Amazon S3 constrains multipart uploads in exactly three ways: - **At most 10,000 parts** per upload. - **At least 5 MiB per part**, with one exception: the final part may be any size, including a few bytes. - **At most 5 GiB per part.** Parts do *not* have to be equal in size — that is a common misconception. The rule is only that every part except the last clears 5 MiB. Uniform sizing is a convention of the transfer managers because it makes the arithmetic and the resume logic simple, not an API requirement. If you call `CompleteMultipartUpload` with a non-final part under 5 MiB, S3 rejects the completion with an `EntityTooSmall` error. This is a nasty failure mode because it arrives at the very end, after all the bytes have been transferred. ## The arithmetic that actually decides it The floor is object size divided by 10,000: ``` 500 GB / 10,000 = 50 MB minimum part size 1 TB / 10,000 = 100 MB minimum part size 5 TiB / 10,000 ≈ 537 MiB minimum part size ``` Run the same arithmetic the other way and you see why a fixed small chunk size does not scale: with 8 MB parts, 10,000 parts covers only about 80 GB. Anything larger simply cannot be uploaded at that chunk size. This is exactly why the AWS CLI, which defaults `multipart_chunksize` to 8 MB, silently increases the chunk size for large files rather than failing — and why a hand-rolled uploader that hardcodes 8 MB works fine in testing on small files and then dies in production on a big one. If you are streaming data of unknown total length, you have no divisor. Either pick a part size that can cover your worst case within 10,000 parts, or grow the part size as the upload progresses — legal, since parts need not be uniform. ## Above the floor: the real tradeoff **Bigger parts** - Fewer HTTP requests, so less per-request latency and less request cost. - Better throughput per connection on a clean, fat link. - More data lost and re-sent when a part fails — a failed 500 MB part is a 500 MB retransmit. - More memory: an uploader typically buffers each in-flight part, so **peak memory ≈ part size × concurrency**. Eight threads at 256 MB each is 2 GB of RAM, which is how uploader processes get OOM-killed on small containers. **Smaller parts** - Cheap retries and finer-grained resume. - Better parallelism on a lossy or high-latency link, where you want many small in-flight units. - More requests, so more request charges and more per-request overhead relative to payload. - Risk of hitting the 10,000-part wall. In practice, tens of megabytes — commonly 16 MB to 128 MB — is where most workloads land, subject to the floor. The decision is dominated by two questions: how much memory the uploading process has, and how flaky the network is. ## Concurrency is the other half of the knob Part size alone does not determine throughput; part size times the number of parallel uploads does. On a well-connected host, throughput usually climbs with concurrency until you saturate the NIC or the CPU spent on TLS and checksums. On a small container or a Lambda function with modest memory, concurrency is capped by RAM long before the network is the limit. Tune the pair together, and measure rather than guess — the CLI exposes both as `multipart_chunksize` and `max_concurrent_requests`. ```bash aws configure set default.s3.multipart_chunksize 64MB aws configure set default.s3.max_concurrent_requests 16 ``` ## Practical guidance 1. Compute the floor from the largest object you expect, not the average one. 2. Pick a part size comfortably above that floor so growth does not push you into the wall. 3. Multiply by your intended concurrency and check the result against the uploader's memory limit. 4. Prefer the SDK transfer manager, which enforces the limits and adjusts the chunk size for you, over hand-rolled `UploadPart` loops. 5. If you are uploading from many machines, remember they can share one `UploadId` and each take a disjoint range of part numbers — part size then also determines how evenly you can split the work. The interview answer is the arithmetic plus the tradeoff: 10,000 parts sets the floor, memory and retry cost set the ceiling, and the sweet spot is normally tens of megabytes.