skip to content

How do you train a model on a 400 GB file that never fits in memory?

level: seniorimportance: should knowfreq 44%

answer

  1. does the metric still move with more rows?
  2. one chunk in memory at a time
  3. you cannot scale on unread data
  4. hash the categories to fixed width
  5. a sorted file poisons the last chunks

basics

~20 s

Fit out-of-core: stream the file in fixed-size chunks and update a learner that has a per-chunk update rule, so only one chunk sits in memory. First check whether a sample or fewer columns removes the problem.

solid answer

~50 s

First I would try to make the problem go away: fit on a random sample and see whether adding rows still moves the metric, drop columns the model does not use, and narrow the stored types — that often turns 400 GB into something that fits. If it genuinely does not, I go out-of-core: read fixed-size chunks in sequence and apply one update per chunk with a learner that has a partial-update rule. Three things bite. Preprocessing statistics must come from a first pass or from running estimates, because you cannot centre and scale on data you have not read yet. Categorical encodings need a fixed-width scheme such as feature hashing, since building a vocabulary means a second pass. And if the file is sorted, chunk order leaves the model tuned to whatever the tail contained, so shuffle or interleave and hold out a separate validation chunk that never enters an update.

code

python · 15 lines
python
import random

random.seed(0)
rows = [random.gauss(10, 2) for _ in range(10000)]  # pretend this file never fits in RAM

n = 0
mean = 0.0
for start in range(0, len(rows), 500):        # read one chunk at a time
    chunk = rows[start:start + 500]
    for x in chunk:                           # running update, chunk then discarded
        n += 1
        mean += (x - mean) / n

print(round(mean, 6))                         # streamed, one chunk held at a time
print(round(sum(rows) / len(rows), 6))        # same value, whole file in memory

go deeper

for a junior

Know the basic move: read the data in chunks so only a piece is in memory at once, and check first whether a random sample is already enough to train on. Recognize that not every algorithm can be fitted this way.

for a middle

Explain the loop and its requirements: a learner with a per-chunk update rule, preprocessing statistics that cannot be computed from unread data, a fixed-width encoding for categories, and the class list supplied before the first chunk.

for a senior

Demonstrate that you have hit the traps. Talk about a sorted file skewing the model toward its tail, holding out validation rows by a key hash so nothing leaks, and decaying the step size across a pass.

for a principal

Frame it as a cost decision: engineering a streaming pipeline versus sampling, pruning columns or buying memory. Be ready to argue that constraining the model family to updatable ones is a real price, and to say when it is worth paying.

## Before you engineer anything "Out-of-core" is a real technique and also a common way to spend a week on a problem you did not have. Three checks first. **Do you need all the rows?** Fit on a 1% random sample, then 5%, then 25%, and look at whether the validation metric is still improving. On many large logs it flattens early — the file is large because it is long, not because it is rich. If 5% gets you within noise of the full set, the answer is "sample", and everything below is unnecessary. **Do you need all the columns?** A 400 GB clickstream is mostly payload the model never touches. Projecting to the columns the features actually use, before anything else, routinely cuts an order of magnitude. **Do you need those types?** Storing an integer identifier as text, or a proportion at full precision when two decimals decide the split, wastes most of the bytes. Narrowing types and choosing a columnar layout can cut another large factor. Only when a genuinely needed slice still exceeds memory do you go streaming. ## The streaming loop The shape is always the same: open the file, read a chunk you can hold, turn it into features, apply one update, drop it, repeat. The model is the only thing that survives across iterations, so the learner must have an update rule that consumes a chunk — a gradient-updated parameter vector, a set of running counts, a stream-designed tree. If your chosen algorithm has no such rule (a random forest, a plain CART tree), out-of-core fitting is not available to it and you must either sample or switch families. ## Four things that quietly break **1. Preprocessing statistics.** Centring, scaling, quantile bucketing and imputation all need statistics of the whole column, and in the first chunk you have not seen the column. Two honest options: make one cheap pass computing running statistics and nothing else, then a second pass that fits; or use running estimates updated as you go, accepting that the first chunks were transformed with worse estimates than the last. A running mean and variance can be maintained exactly in one pass with an incremental update; there is no reason to hold the column in memory to compute them. **2. Unseen categories and vocabularies.** Building an encoding from the observed categories requires knowing them all, which is a full pass. Feature hashing avoids this: hash the category string into a fixed number of slots, so the feature width is decided in advance and any category — including one that first appears in the final chunk — lands somewhere. You pay in collisions and lose the ability to read a coefficient back to a category. **3. The label set.** For classification, the model must reserve one parameter block per class before it sees the first chunk, and a chunk may well contain only some classes. The class list has to be supplied up front, from metadata rather than discovered from the data. **4. Order.** Files are almost never in random order — they are usually sorted by time, or by user, or grouped by whatever wrote them. Stream a time-sorted file straight through with a fixed step size and the model finishes the pass fitted mostly to the last few chunks. Fixes: pre-shuffle the file once on disk, interleave chunks from several offsets, or shuffle within a large buffer as you read. Decay the step size across the pass so late chunks do not overwrite everything, and make several passes if you can afford them. ## Evaluation under streaming Holding out a validation set means deciding *before* the pass which rows are excluded, and never letting them into an update — a hash of a row key into a holdout bucket does this without a shuffle. The other option is prequential evaluation: for each chunk, predict first, record the error, then update. That gives an error curve over the file for free and never leaks, but the numbers are optimistic-free rather than clean: early chunks are scored by a barely trained model, so the running average understates final quality. ## What to say in an interview Name the escape hatches first (sample, project, narrow types), then the loop, then the three specific traps — statistics, vocabulary, order — and say which family you would pick because it has a partial-update rule. Candidates who jump straight to "stream it in chunks" without asking whether all 400 GB earns its keep are answering the question they wish they had been asked.

  • You must standardize features but can only afford one pass over the file. What do you do?
    Maintain a running count, mean and sum of squared deviations per column and update them as each chunk arrives, standardizing that chunk with the estimates available at the time. The first chunks get worse estimates than the last, which biases their updates slightly; a small decayed step size early, or one cheap statistics-only pass before the fitting pass, removes the problem if a second pass is affordable.
  • Why is feature hashing so common in out-of-core pipelines?
    Because it fixes the feature width in advance. A learned vocabulary needs a full pass to enumerate categories and grows unboundedly as new ones appear, while hashing sends any string — including one first seen in the last chunk — into a preallocated slot. The costs are collisions between unrelated categories and losing the mapping from a coefficient back to a readable feature name.

saying these in an interview costs you the question

  • Jumps to streaming without asking whether a sample suffices
  • Standardizes each chunk by its own local mean and variance
  • Assumes every chunk contains every class label
  • Streams a time-sorted file with a fixed step size
  • Proposes chunked fitting for an algorithm with no update rule

context

resolving links…