skip to content

In gradient-descent fitting, how do full-batch, stochastic and mini-batch updates differ per epoch?

level: juniorimportance: must knowfreq 78%

answer

  1. count the updates in one pass
  2. all rows, one row, or a chunk
  3. cost per update vs updates per pass
  4. same loss, different gradient estimate

basics

~20 s

They differ in how many rows each gradient step averages. Full-batch uses all n rows for one update per epoch; stochastic uses one row, giving n updates; mini-batch averages a chunk of B rows, giving about n/B updates.

solid answer

~50 s

An epoch is one pass over the training data; the three variants differ in how many updates that pass produces. Full-batch averages the gradient over all `n` rows and moves the weights once. Strict stochastic descent computes the gradient from one row and moves `n` times. Mini-batch averages over a chunk of `B` rows and moves about `n/B` times. All three minimise the same loss — for a linear model the squared error, for logistic regression the log loss — and the per-row gradient formula is identical; only the set you average over changes. The tradeoff is cost per update against updates per pass. On a 50-million-row ad-impression file, one full-batch step costs a whole pass before any weight moves, while batches of 4,096 buy roughly 12,000 updates in that same pass, which is what makes the fit tractable.

code

python · 17 lines
python
import random

# 20 points that lie exactly on y = 3x + 2
data = [(i / 20, 3.0 * (i / 20) + 2.0) for i in range(20)]
w, b, lr, batch_size = 0.0, 0.0, 0.3, 4

for epoch in range(300):
    random.shuffle(data)                        # reshuffle before every pass
    for s in range(0, len(data), batch_size):   # 5 mini-batch updates per epoch
        chunk = data[s:s + batch_size]
        err = [(w * x + b) - y for x, y in chunk]
        gw = sum(2 * e * x for e, (x, _) in zip(err, chunk)) / len(chunk)
        gb = sum(2 * e for e in err) / len(chunk)
        w -= lr * gw
        b -= lr * gb

print(round(w, 2), round(b, 2))   # 3.0 2.0

go deeper

for a junior

Be ready to define an epoch as one pass over the data and to say how many updates each variant makes in that pass. Knowing that batch size counts rows, not features or epochs, is the bar here.

for a middle

Explain why the loss being an average over examples is what makes any subset usable, and do the arithmetic out loud: n rows, batch B, so ceil(n/B) updates per epoch and the same total gradient work either way.

for a senior

Show that you pick a batch size from memory limits, throughput and how quickly you need the weights to move, and that you revisit the step size when you change it. Mention reporting epoch-average loss rather than per-batch loss.

for a principal

Own the framing that batch size is a compute-versus-progress decision made against the hardware and the training budget, not a statistical property of the model, and that it must be recorded alongside the step size for a run to be reproducible.

## One update rule, three ways to compute the gradient Fitting a linear or logistic model by gradient descent means repeating a single step: `w <- w - lr * g`, where `g` is the gradient of the training loss with respect to the weights and `lr` is the step size. Everything that separates full-batch, stochastic and mini-batch descent is the answer to one question: **which rows went into `g`?** That choice is available because both losses used here are *averages over examples*. For a linear model fit by squared error, the loss on one row is `(p - y)^2` with `p = w*x + b`, and its gradient contribution is `2*(p - y)*x`. For logistic regression fit by log loss, `loss = -[y*log(p) + (1-y)*log(1-p)]` where `p` is the predicted probability, and the gradient contribution is `(p - y)*x`. In both cases the total training loss is the mean of per-row terms, so its gradient is the mean of per-row gradients. You can average that mean over every row, over one row, or over any chunk in between. ## The three regimes - **Full-batch (batch gradient descent).** `g` is averaged over all `n` rows. One parameter update per pass over the data. - **Stochastic gradient descent (SGD), in the strict sense.** `g` comes from a single row. `n` parameter updates per pass. - **Mini-batch.** `g` is averaged over a chunk of `B` rows. `ceil(n / B)` updates per pass. In everyday speech "SGD" usually means this. An **epoch** is one complete pass over the training set — *not* one update. This is the definition candidates most often get wrong. With 10,000 rows and `B = 500`, one epoch is 20 updates; with `B = 1`, one epoch is 10,000 updates; full-batch, one epoch is one update. ## The cost accounting that decides the choice Per epoch all three variants touch every row exactly once, so the raw arithmetic per epoch is roughly equal. What differs is **how many parameter updates that arithmetic buys**, and **how much work happens between updates**. Take a 50-million-row ad-impression training file. Full-batch descent computes 50 million per-row gradients, averages them, and moves the weights *once*. A 100-epoch budget buys 100 updates — for a model with a few thousand weights, that is nowhere near a converged fit, and every one of those 100 steps costs a full pass over 50 million rows. Now take mini-batches of 4,096: the same single pass yields about 12,200 updates, so the weights are already in a sensible region long before the first epoch ends. That is why mini-batch descent makes the fit tractable at this scale. Pure single-example SGD sits at the other extreme: 50 million updates per epoch, each one moving the weights after looking at a single impression. The arithmetic per update is trivial, but the per-update overhead — reading a row, touching every weight — is paid 50 million times, and a chunk of rows can be processed far more efficiently as a block than one row at a time. It also makes `g` an extremely erratic estimate of the full gradient. Mini-batch wins because it sits between the two: enough rows per update to amortise overhead and to keep `g` a reasonable estimate, few enough that the weights move many times per pass. Memory matters too — only `B` rows need to be resident, so a training file larger than RAM can still be fit chunk by chunk. ## What does *not* change - **The objective is identical.** All three minimise the same training loss over the same data. Batch size does not add a penalty, change the loss function, or make the model "fit only the last batch". - **The gradient stays unbiased.** For a uniformly drawn batch, the batch-averaged gradient is an unbiased estimate of the full-data gradient: its expectation is the full gradient. Smaller batches make the estimate noisier, not systematically wrong. - **The gradient formula is the same.** Nothing about the per-row derivative changes; only the set you average over does. ## Practical notes Batch sizes are conventionally powers of two in the 32–8,192 range, chosen from memory and throughput rather than from statistics. When `n` is not a multiple of `B` the last batch of the epoch is short, which is harmless. Because changing `B` changes how noisy `g` is, a batch-size change usually means revisiting the step size rather than keeping it fixed. And when you report progress, report the loss averaged over an epoch: a per-batch number moves for reasons that have nothing to do with whether the fit is improving. The short version to say out loud: batch size trades **cost per update** against **updates per pass**, full-batch and single-example SGD are the two extremes, and mini-batch is the default because it is the only point on that line that is cheap per update *and* generous with updates.

  • Does mini-batch descent minimise a different objective than full-batch descent?
    No. All three minimise the same training loss over the same data. A batch-averaged gradient is an unbiased estimate of the full-data gradient — its expectation is the full gradient — so a smaller batch makes the estimate noisier, not systematically different. Batch size is a compute knob, not a change to the loss function and not a penalty term.
  • With 2 million rows and a batch size of 512, how many updates does a 10-epoch fit perform?
    About 3,907 updates per epoch (2,000,000 / 512, with a short final batch), so roughly 39,000 updates over 10 epochs. Full-batch descent on the same budget would perform exactly 10 updates. That gap — four orders of magnitude in how often the weights move for the same amount of gradient arithmetic — is the whole argument for mini-batching.
  • Why is single-example stochastic descent rarely used in practice despite making the most updates?
    Each update looks at one row, so the fixed cost of an update — reading the row, touching every weight — is paid `n` times per epoch and dwarfs the arithmetic. A chunk of rows is processed far more efficiently as a block. The single-row gradient is also a very erratic estimate of the full gradient. Mini-batch keeps most of the update frequency while removing both problems.

Full-batch is polling every voter in the country before making one decision; stochastic is deciding again after each doorstep conversation; mini-batch polls a few thousand people, decides, and repeats.

saying these in an interview costs you the question

  • Thinks one epoch always means one gradient update
  • Says mini-batch descent optimises a different loss than full-batch
  • Believes stochastic descent samples random features rather than rows
  • Confuses batch size with the number of epochs
  • Claims full-batch descent is always more accurate because it sees all the data

context

resolving links…