Why should a photo and video sharing app send large uploads straight to the object store instead of through its application servers?
answer
- what app servers are sized for
- minutes per upload, not milliseconds
- bytes cross the network twice
- control plane vs data plane
- short-lived URL for one key
basics
~20 sProxying gigabytes pins app servers to slow byte streams for minutes. Instead the app hands out a short-lived signed upload URL, the phone sends bytes directly to the object store, and app servers handle only small control requests.
solid answer
~50 sAn app server sized for millisecond API calls turns into a bandwidth pipe when every upload streams through it: each 2 GB video from a phone holds a connection, a worker and buffers or temp files for many minutes, the bytes cross the network twice, and proxy or load-balancer timeouts cut long transfers. Scaling the API tier then tracks bytes in flight instead of requests. The standard fix separates the **control plane** from the **data plane**: the client asks the API to start an upload, the API authorizes it, records a `pending` entry and returns a short-lived signed URL scoped to one server-chosen object key; the client uploads straight to the object store, which is built for massively parallel ingest; a completion signal then tells the backend to verify and commit. The cost is that the app can no longer inspect bytes inline, so validation moves to a post-upload scan stage.
code
http · 15 linesPOST /api/uploads HTTP/1.1
Content-Type: application/json
{"fileName": "beach.mp4", "size": 2147483648, "sha256": "9f2c..."}
HTTP/1.1 201 Created
Content-Type: application/json
{"uploadId": "up_81f3", "uploadUrl": "https://objects.example.net/incoming/up_81f3?sig=...", "expiresInSeconds": 900}
PUT /incoming/up_81f3?sig=... HTTP/1.1
Host: objects.example.net
Content-Length: 2147483648
HTTP/1.1 200 OKgo deeper
Recall the core reason: app servers are built for short requests, and proxying gigabytes ties them up for minutes. Then name the fix, a short-lived signed URL that lets the client write to the object store directly.
Walk through the handshake step by step: authorize and record a pending entry, return a scoped URL, upload directly, receive a completion signal, verify, then commit. Mention why bandwidth and timeouts break the proxied design.
Show that you know what moves elsewhere: inline validation becomes a post-upload scan stage, and two stores without a shared transaction force a pending-then-committed order plus a sweeper for abandoned uploads.
Frame it as a capacity and failure-domain decision: separating control and data planes lets the API tier scale on requests and deploy freely, at the price of client complexity and an asynchronous validation pipeline.
## The problem with proxying uploads A typical application server is built to answer **short API requests**: parse a small JSON body, touch a database, respond in tens of milliseconds. A photo and video sharing app, however, receives files measured in gigabytes from phones on cellular or home connections. If every upload is **proxied** (the client sends bytes to the app server, which forwards them to storage), the app tier inherits a completely different workload: - **Long-held connections and workers.** A single upload can take many minutes, pinning a connection, a request thread or event-loop slot, and memory buffers or temporary files for the whole time. - **Double bandwidth.** Every byte travels phone -> app server -> object store, so the app tier pays for ingress and egress of the full payload. - **Timeouts.** Reverse proxies and load balancers usually cap request duration or idle time; a slow multi-gigabyte upload collides with those limits and fails near the end. - **Wrong scaling signal.** Capacity has to be planned for bytes in flight, not for requests per second, so a burst of uploads can starve ordinary API traffic. - **Blast radius.** A deploy or crash of an app server kills every upload it is carrying. ## A rough sizing The numbers below are illustrative, assuming decimal units and a 20 Mbit/s phone uplink. | Quantity | Value | How | |---|---|---| | One 2 GB video | 16,000 Mbit | 2 GB x 8 bits per byte | | Time to upload it | ~800 s (~13 min) | 16,000 Mbit / 20 Mbit/s | | 1,000 concurrent uploads | ~20 Gbit/s inbound | 1,000 x 20 Mbit/s | | Proxied traffic through the app tier | ~40 Gbit/s | 20 in from phones plus 20 out to storage | A fleet sized for API calls is not sized for that, and it should not have to be. ## Control plane vs data plane The fix is to split the flow. The **control plane** (small, authenticated API calls) stays on the app servers; the **data plane** (the bytes) goes straight to an **object store**, a storage service designed to absorb huge numbers of parallel writes. 1. The client calls the API: "I want to upload `beach.mp4`, 2 GiB, with this checksum." 2. The API **authorizes** the user, checks quota and declared size or type limits, chooses the **object key** itself, and writes a `pending` metadata record. 3. The API returns a **short-lived signed URL** (or a set of per-part URLs for a multipart upload) that allows exactly that write for a few minutes. 4. The client sends the bytes directly to the object store using plain HTTP. 5. A **completion signal** arrives: the client calls a `complete` endpoint, the object store emits a notification, or both. 6. The backend verifies that the object exists with the expected size and checksum, then moves the record from `pending` to `committed` and starts processing. ```http POST /api/uploads HTTP/1.1 Content-Type: application/json {"fileName": "beach.mp4", "size": 2147483648} HTTP/1.1 201 Created Content-Type: application/json {"uploadId": "up_81f3", "uploadUrl": "https://objects.example.net/incoming/up_81f3?sig=..."} ``` The app servers now handle a few kilobytes per upload regardless of file size, and they can be deployed or scaled without breaking transfers in progress. ## What the app gives up Bypassing the app tier is not free: - **No inline inspection.** The app never sees the bytes during upload, so it cannot reject a bad file mid-stream. Validation (real file type, size, malware, moderation) moves to a **post-upload scan stage**, and the file stays unpublished until it passes. - **Credential design matters.** The signed URL must be tightly scoped and short-lived; the exact constraints belong to the delegation pattern itself, but the ingest design must ask for them. - **Two sources of truth.** Bytes live in the object store and the record lives in a database, with no shared transaction, so the **commit order** (pending first, committed after verification) and a cleanup job for abandoned uploads become part of the design. - **Client complexity.** The client now talks to two endpoints and should support resumable, multipart transfers for large files. ## Summary Direct-to-storage upload keeps app servers doing what they are good at, authorizing and recording, while the storage layer does what it is good at, moving bytes. Interviewers expect a candidate to name the resource cost of proxying, sketch the grant -> upload -> complete handshake, and acknowledge that validation shifts to after the upload.
- If the phone uploads directly to the object store, how does the backend learn that the upload finished?Two signals are common: the client calls a `complete` endpoint on the API, and the object store emits a completion notification into a queue. Robust designs accept both, treat them as idempotent, and never trust either blindly: the backend reads the object's actual size and checksum from the store before committing. A periodic reconciliation sweep catches uploads whose signal was lost.
- What must the app server still validate when the bytes never pass through it?At grant time: that the user is authorized, has quota, and that the declared size and type are within limits; the server also picks the object key so clients cannot write elsewhere. After the upload: the real stored size, the checksum, the true file type from its leading bytes, and malware and moderation results, all before the file is published.
A restaurant does not route grocery deliveries through the dining room; the manager signs a delivery slip and the truck unloads at the back door.
saying these in an interview costs you the question
- App servers can stream any file size if you just raise the timeouts.
- Direct upload means the backend no longer needs to authorize anything.
- Let the client choose the object key or path it writes to.
- Mark the file available as soon as the signed URL is issued.
- Direct upload removes the need for any server-side validation of the file.