An AWS Lambda function downloads a large object from S3 into /tmp, transforms it, and uploads the result. It passes every test but in production it intermittently fails with "No space left on device", and occasionally hits the function timeout. What is going on, and how would you fix it?

## The two limits in play This scenario is a favourite because it needs two facts at once, and the *intermittent* symptom is the clue that separates a candidate who has operated Lambda from one who has only read the quotas page. **Ephemeral storage.** Every Lambda execution environment gets a writable `/tmp` directory. It defaults to **512 MB** and can be configured up to **10,240 MB** through the function's ephemeral-storage setting. It is the only writable location in the environment; the deployment package under the task root and any layer content under `/opt` are read-only. **Timeout.** A function's timeout defaults to 3 seconds and can be raised to a maximum of **900 seconds — 15 minutes**. That ceiling is not negotiable and no support request lifts it. ## Why the failure is intermittent The crucial property is that `/tmp` is scoped to the **execution environment**, not to the invocation. When Lambda keeps an environment warm and routes a subsequent request to it, that request sees whatever the previous one left on disk. So: - A test run, or a low-traffic period, creates a fresh environment per request. 400 MB of scratch fits in 512 MB every time. Green. - In production one warm environment handles request after request. Two 400 MB files do not fit. The third request into that environment fails with `ENOSPC` while its neighbours, on other environments, succeed. That is the shape of the bug: failures that correlate with traffic and warmth rather than with input, and that cluster on particular environments. Each concurrent environment has its own independent `/tmp`, so this is never contention between concurrent requests — it is accumulation within one. ```python import os, tempfile def lambda_handler(event, context): fd, path = tempfile.mkstemp(dir="/tmp") # unique name per invocation os.close(fd) try: download(event["key"], path) return transform(path) finally: os.remove(path) # always reclaim the space ``` A fixed filename is the other half of the trap: it hides the leak on rewrite but leaves a stale file behind if a later invocation reads it before writing, which produces the far nastier failure — one request silently processing another request's data. ## Fixing it properly **1. Clean up unconditionally.** Delete in a `finally` block (or the language equivalent) so an exception path cannot leak. Use unique names so two invocations on the same environment can never collide. A defensive sweep of `/tmp` during initialization is a reasonable belt-and-braces measure. **2. Don't touch disk at all if you can avoid it.** The strongest fix is architectural: read the object as a stream, transform it in bounded chunks, and write it out through a multipart upload. Peak disk use goes to zero and peak memory becomes a buffer you chose rather than the size of the input. This also removes the coupling between input size and the ephemeral-storage setting entirely — which matters, because inputs grow. **3. Raise ephemeral storage when the work genuinely needs local files.** Some workloads do: a tool that memory-maps a file, a media encoder that seeks randomly, an extraction step that must materialise an archive. Configure it up to 10,240 MB — and note that storage above the included 512 MB is billed for the duration of the invocation, so it is a real cost, not a free safety margin. **4. Give the timeout a real value.** Set it a little above the observed p99 rather than at the maximum: a timeout that is far too generous turns a hung downstream call into 15 minutes of billed duration per request. Read the remaining budget from the context object and decide whether to start another chunk of work rather than being killed mid-write — a timeout kill runs no cleanup code, so it is also how orphaned files appear in the first place. ## When 15 minutes is the actual wall If the transform cannot finish inside 900 seconds no matter how much CPU you buy with the memory setting, the function has outgrown Lambda's execution model, and the honest answer in an interview is to say so rather than to keep tuning: - **Split the work.** Chunk the input by byte range or record range, process each chunk in its own invocation, and combine the results. This is the natural fit when the transform is parallelisable, and it converts one long job into many short ones that also scale out. - **Move it to a container task.** A long-running batch job with no time limit belongs on a container or instance-based compute service, where duration is not capped. A useful framing for the interviewer: Lambda's timeout is not an inconvenience to work around but a statement about what kind of work belongs there. Anything approaching the ceiling is a workload whose growth curve will cross it. ## What good answers include Naming both limits with their real values; explaining environment reuse as the reason the failure is intermittent and test-invisible; proposing cleanup *and* streaming rather than only raising the setting; noting that larger ephemeral storage is billed; and knowing when to stop and move the workload.