skip to content

How can a Python worker use resource.RLIMIT_AS to cap a runaway CSV import?

level: seniorimportance: should knowfreq 28%

answer

  1. Bound the blast radius from inside the process
  2. Address space reserved, not resident memory
  3. Turns runaway growth into a catchable error
  4. Set it in the worker, not the parent
  5. MemoryError, and the handler must not allocate

basics

~20 s

Setting a soft resource.RLIMIT_AS ceiling makes over-large allocations fail as a catchable MemoryError instead of letting the process grow until something outside kills it. Set it in the import worker, well above the steady-state mapped size.

solid answer

~50 s

`resource.RLIMIT_AS` caps the process's virtual address space; an allocation crossing the soft limit fails and CPython raises `MemoryError`, which the importer can catch, log the offending file and locale-dependent format, and move on. Two caveats decide whether it helps. First, it counts **address space**, not resident memory — thread stacks, mapped files and allocator reservations all count in full — so the cap must sit well above the process's steady-state footprint or it fires on unrelated work. Second, a `MemoryError` can surface anywhere, including inside your logging or cleanup path, so the handler must not itself allocate. The stronger design is to run each import in a child process that calls `resource.setrlimit` on itself first: the parent keeps its warm caches and its 45-second cold start, and a bad file costs one child. Treat the limit as a backstop while you fix the parse path to stream.

code

python · 11 lines
python
import resource

soft, hard = resource.getrlimit(resource.RLIMIT_AS)
cap = 8 * 1024**3
try:
    resource.setrlimit(resource.RLIMIT_AS, (cap, hard))
except ValueError as exc:
    print("kernel refused the cap:", exc)
else:
    print("address-space soft limit:", resource.getrlimit(resource.RLIMIT_AS)[0])
    resource.setrlimit(resource.RLIMIT_AS, (soft, hard))

go deeper

for a junior

Know that resource.RLIMIT_AS bounds a process's address space and that crossing it produces MemoryError, an ordinary Python exception you can catch.

for a middle

Explain how to set only the soft half with resource.setrlimit, and why address space counts reservations such as thread stacks and mapped files rather than resident pages.

for a senior

Show the operational judgement: isolate the risky import in a child that limits itself, keep the handler allocation-free, size the cap with headroom, and alert when it trips.

for a principal

Own the policy question of where memory is bounded at all — per process from inside, or by the platform from outside — and what each choice costs in blast radius and in diagnosability.

### The situation A payroll CSV import worker normally streams rows and stays flat in memory. One customer's file carries a locale-dependent number format — a decimal comma, a thousands separator the fast path does not recognise — and the importer falls back to a slower path that materialises the whole file as Python objects before validating it. Memory climbs until the process dies. The worker also pays a 45-second cold start rebuilding caches, so losing the process for one bad file is expensive out of all proportion to the file. `resource.RLIMIT_AS` is the tool for exactly this shape of problem: it turns "grows until something external kills it" into "raises `MemoryError` at a boundary you chose". ### What the limit actually does `resource.RLIMIT_AS` caps the total size of the process's **virtual address space** — every mapping the process holds, whether or not those pages have ever been touched. When an allocation would push the process past the soft limit, the underlying allocation call fails, and CPython converts that failure into `MemoryError`, a normal exception that propagates up the stack like any other. You set it the usual way, reading the pair first so you only tighten the soft value: ```python soft, hard = resource.getrlimit(resource.RLIMIT_AS) resource.setrlimit(resource.RLIMIT_AS, (2 * 1024**3, hard)) # 2 GiB of address space ``` ### Address space is not resident memory This is the caveat that decides whether the limit helps or hurts. `resource.RLIMIT_AS` counts reservations, not usage: thread stacks, memory-mapped files, allocator arenas that reserve a large region and commit it lazily, and any C extension that reserves address space up front all count against it in full. So the number you pick is not "how much RAM this job should use" — it is a looser bound that must sit comfortably above the process's steady-state mapped size, or the process will fail on work that has nothing to do with the runaway import. Measure the real footprint first and leave generous headroom; a cap tuned to the exact working set is a cap that will misfire. ### `MemoryError` is catchable, not safe A `MemoryError` raised by the limit can be caught, and that is the whole point — the importer can abandon the file, record which customer and which format triggered it, and carry on without the 45-second cold start. But it can surface *anywhere*: in the middle of building a dict, inside the logging call you were about to make, inside an `except` block trying to format the error message. Cleanup code that itself allocates may fail. Treat the handler as hostile territory: pre-build the error message, avoid allocating inside it, and be prepared for the process to be in a state you would rather not keep serving from. A C extension that does not check its own allocation results may abort the process outright instead of raising anything. ### Put the limit on the work, not on the service The stronger design is to run the import in a **child process** and set the limit there. The parent keeps its warm caches and its normal ceiling; the child gets the tight `resource.RLIMIT_AS`, does one file, and exits. If it dies badly, the parent sees a non-zero exit status and fails that one import — no cold start, no risk of continuing inside a process that just ran out of address space. Where you set it matters, and Python 3.14 changed the default that decides this: on Unix other than macOS, `multiprocessing` now starts workers with `forkserver` rather than `fork` (macOS and Windows use `spawn`). Limits are inherited at spawn time, so a worker forked from a fork server carries the limits the server had when it was first created — not the ones the parent set afterwards. The reliable pattern is for the worker to call `resource.setrlimit` itself as its first action. ### It is a backstop, not a fix The limit does not make the importer correct. The real repair is on the parse path: stream the file, handle the locale-dependent format explicitly instead of falling back to a whole-file buffer, and bound the batch size so peak memory is a function of the batch, not of the file. `resource.RLIMIT_AS` is what stops an unknown future input from taking the process down while you still have unknown future inputs — a seatbelt, deliberately loose enough never to fire in normal operation, and paired with an alert so that firing is treated as the incident it is.

  • Why is a MemoryError from resource.RLIMIT_AS not a substitute for streaming the file?
    Because it only tells you the job failed, at an arbitrary point, in a process whose state you can no longer fully trust. Rows already applied may need unwinding, and the handler runs in a process that just ran out of address space. The limit stops an unknown input from taking the service down; bounding peak memory by batch size is what makes the importer correct.
  • What does resource.RLIMIT_AS fail to protect against?
    Anything that is not this process's address space: kernel-side memory, memory held by child processes with their own limits, and a platform-level memory ceiling enforced by control groups rather than by rlimits. It also does not bound resident memory — a process can stay under a generous address-space cap while its resident set is far larger than intended, because the two measure different things.
  • How do you pick the number for the soft RLIMIT_AS value?
    Measure the steady-state mapped size of the worker under real load, including thread stacks and any mapped files, then set the cap comfortably above the highest legitimate peak — often a multiple, not a few per cent. A cap tuned to the observed working set will misfire on the first legitimate large job. Alert on the limit firing so a trip is investigated, not absorbed.

It is a circuit breaker, not insulation: sized loosely so it never trips during normal load, and its tripping is an incident to investigate rather than a routine event to swallow.

saying these in an interview costs you the question

  • Treating RLIMIT_AS as a resident-memory cap
  • Sizing the cap to the observed working set with no headroom
  • Allocating freely inside the MemoryError handler
  • Assuming a caught MemoryError leaves the process healthy
  • Setting the limit in the parent and expecting existing workers to get it
  • Using the limit instead of fixing the whole-file buffering

context

resolving links…