skip to content

Gradient Descent Fitting

3 roadmaps8 questions

How both models get trained when no closed form is practical: batch, stochastic and mini-batch steps on a convex loss. Interviewers ask what makes the fit converge or diverge.

on this pageshow

explore

questions

8

In gradient-descent fitting, how do full-batch, stochastic and mini-batch updates differ per epoch?

level: juniorimportance: must knowfreq 78%

answer

  1. count the updates in one pass
  2. all rows, one row, or a chunk
  3. cost per update vs updates per pass
  4. same loss, different gradient estimate

basics

~20 s

They differ in how many rows each gradient step averages. Full-batch uses all n rows for one update per epoch; stochastic uses one row, giving n updates; mini-batch averages a chunk of B rows, giving about n/B updates.

solid answer

~50 s

An epoch is one pass over the training data; the three variants differ in how many updates that pass produces. Full-batch averages the gradient over all `n` rows and moves the weights once. Strict stochastic descent computes the gradient from one row and moves `n` times. Mini-batch averages over a chunk of `B` rows and moves about `n/B` times. All three minimise the same loss — for a linear model the squared error, for logistic regression the log loss — and the per-row gradient formula is identical; only the set you average over changes. The tradeoff is cost per update against updates per pass. On a 50-million-row ad-impression file, one full-batch step costs a whole pass before any weight moves, while batches of 4,096 buy roughly 12,000 updates in that same pass, which is what makes the fit tractable.

code

python · 17 lines
python
import random

# 20 points that lie exactly on y = 3x + 2
data = [(i / 20, 3.0 * (i / 20) + 2.0) for i in range(20)]
w, b, lr, batch_size = 0.0, 0.0, 0.3, 4

for epoch in range(300):
    random.shuffle(data)                        # reshuffle before every pass
    for s in range(0, len(data), batch_size):   # 5 mini-batch updates per epoch
        chunk = data[s:s + batch_size]
        err = [(w * x + b) - y for x, y in chunk]
        gw = sum(2 * e * x for e, (x, _) in zip(err, chunk)) / len(chunk)
        gb = sum(2 * e for e in err) / len(chunk)
        w -= lr * gw
        b -= lr * gb

print(round(w, 2), round(b, 2))   # 3.0 2.0

go deeper

for a junior

Be ready to define an epoch as one pass over the data and to say how many updates each variant makes in that pass. Knowing that batch size counts rows, not features or epochs, is the bar here.

for a middle

Explain why the loss being an average over examples is what makes any subset usable, and do the arithmetic out loud: n rows, batch B, so ceil(n/B) updates per epoch and the same total gradient work either way.

for a senior

Show that you pick a batch size from memory limits, throughput and how quickly you need the weights to move, and that you revisit the step size when you change it. Mention reporting epoch-average loss rather than per-batch loss.

for a principal

Own the framing that batch size is a compute-versus-progress decision made against the hardware and the training budget, not a statistical property of the model, and that it must be recorded alongside the step size for a run to be reproducible.

## One update rule, three ways to compute the gradient Fitting a linear or logistic model by gradient descent means repeating a single step: `w <- w - lr * g`, where `g` is the gradient of the training loss with respect to the weights and `lr` is the step size. Everything that separates full-batch, stochastic and mini-batch descent is the answer to one question: **which rows went into `g`?** That choice is available because both losses used here are *averages over examples*. For a linear model fit by squared error, the loss on one row is `(p - y)^2` with `p = w*x + b`, and its gradient contribution is `2*(p - y)*x`. For logistic regression fit by log loss, `loss = -[y*log(p) + (1-y)*log(1-p)]` where `p` is the predicted probability, and the gradient contribution is `(p - y)*x`. In both cases the total training loss is the mean of per-row terms, so its gradient is the mean of per-row gradients. You can average that mean over every row, over one row, or over any chunk in between. ## The three regimes - **Full-batch (batch gradient descent).** `g` is averaged over all `n` rows. One parameter update per pass over the data. - **Stochastic gradient descent (SGD), in the strict sense.** `g` comes from a single row. `n` parameter updates per pass. - **Mini-batch.** `g` is averaged over a chunk of `B` rows. `ceil(n / B)` updates per pass. In everyday speech "SGD" usually means this. An **epoch** is one complete pass over the training set — *not* one update. This is the definition candidates most often get wrong. With 10,000 rows and `B = 500`, one epoch is 20 updates; with `B = 1`, one epoch is 10,000 updates; full-batch, one epoch is one update. ## The cost accounting that decides the choice Per epoch all three variants touch every row exactly once, so the raw arithmetic per epoch is roughly equal. What differs is **how many parameter updates that arithmetic buys**, and **how much work happens between updates**. Take a 50-million-row ad-impression training file. Full-batch descent computes 50 million per-row gradients, averages them, and moves the weights *once*. A 100-epoch budget buys 100 updates — for a model with a few thousand weights, that is nowhere near a converged fit, and every one of those 100 steps costs a full pass over 50 million rows. Now take mini-batches of 4,096: the same single pass yields about 12,200 updates, so the weights are already in a sensible region long before the first epoch ends. That is why mini-batch descent makes the fit tractable at this scale. Pure single-example SGD sits at the other extreme: 50 million updates per epoch, each one moving the weights after looking at a single impression. The arithmetic per update is trivial, but the per-update overhead — reading a row, touching every weight — is paid 50 million times, and a chunk of rows can be processed far more efficiently as a block than one row at a time. It also makes `g` an extremely erratic estimate of the full gradient. Mini-batch wins because it sits between the two: enough rows per update to amortise overhead and to keep `g` a reasonable estimate, few enough that the weights move many times per pass. Memory matters too — only `B` rows need to be resident, so a training file larger than RAM can still be fit chunk by chunk. ## What does *not* change - **The objective is identical.** All three minimise the same training loss over the same data. Batch size does not add a penalty, change the loss function, or make the model "fit only the last batch". - **The gradient stays unbiased.** For a uniformly drawn batch, the batch-averaged gradient is an unbiased estimate of the full-data gradient: its expectation is the full gradient. Smaller batches make the estimate noisier, not systematically wrong. - **The gradient formula is the same.** Nothing about the per-row derivative changes; only the set you average over does. ## Practical notes Batch sizes are conventionally powers of two in the 32–8,192 range, chosen from memory and throughput rather than from statistics. When `n` is not a multiple of `B` the last batch of the epoch is short, which is harmless. Because changing `B` changes how noisy `g` is, a batch-size change usually means revisiting the step size rather than keeping it fixed. And when you report progress, report the loss averaged over an epoch: a per-batch number moves for reasons that have nothing to do with whether the fit is improving. The short version to say out loud: batch size trades **cost per update** against **updates per pass**, full-batch and single-example SGD are the two extremes, and mini-batch is the default because it is the only point on that line that is cheap per update *and* generous with updates.

  • Does mini-batch descent minimise a different objective than full-batch descent?
    No. All three minimise the same training loss over the same data. A batch-averaged gradient is an unbiased estimate of the full-data gradient — its expectation is the full gradient — so a smaller batch makes the estimate noisier, not systematically different. Batch size is a compute knob, not a change to the loss function and not a penalty term.
  • With 2 million rows and a batch size of 512, how many updates does a 10-epoch fit perform?
    About 3,907 updates per epoch (2,000,000 / 512, with a short final batch), so roughly 39,000 updates over 10 epochs. Full-batch descent on the same budget would perform exactly 10 updates. That gap — four orders of magnitude in how often the weights move for the same amount of gradient arithmetic — is the whole argument for mini-batching.
  • Why is single-example stochastic descent rarely used in practice despite making the most updates?
    Each update looks at one row, so the fixed cost of an update — reading the row, touching every weight — is paid `n` times per epoch and dwarfs the arithmetic. A chunk of rows is processed far more efficiently as a block. The single-row gradient is also a very erratic estimate of the full gradient. Mini-batch keeps most of the update frequency while removing both problems.

Full-batch is polling every voter in the country before making one decision; stochastic is deciding again after each doorstep conversation; mini-batch polls a few thousand people, decides, and repeats.

saying these in an interview costs you the question

  • Thinks one epoch always means one gradient update
  • Says mini-batch descent optimises a different loss than full-batch
  • Believes stochastic descent samples random features rather than rows
  • Confuses batch size with the number of epochs
  • Claims full-batch descent is always more accurate because it sees all the data

context

resolving links…

open as a page

Why does gradient descent on a logistic regression's log loss reach the same fit from any starting weights?

level: juniorimportance: must knowfreq 62%

basics

~20 s

Log loss is convex in a logistic model's weights, so the error surface has one global minimum and no local minima. Every run that converges lands on the same coefficients, which is why random restarts add nothing.

open as a page

Why do a logistic regression's weights run to infinity when a feature perfectly separates the label?

level: middleimportance: must knowfreq 55%

basics

~20 s

Because a larger weight always lowers the loss. When a feature splits the classes with no overlap, scaling that weight up pushes predictions toward 0 and 1 and log loss toward zero, so no finite maximum-likelihood estimate exists.

open as a page

Why shuffle the training rows before each epoch of stochastic or mini-batch gradient descent?

level: middleimportance: should knowfreq 45%

basics

~20 s

Shuffling makes each batch a random sample of the training set, so its averaged gradient fairly estimates the full-data gradient. Without it, a sorted file gives correlated batches, a sawtooth loss curve, and weights biased toward the pass's last rows.

open as a page

Your training loss becomes NaN by epoch three of a gradient-descent fit — how do you diagnose it?

level: seniorimportance: should knowfreq 52%

basics

~20 s

Log the loss per update, not per epoch. A loss climbing geometrically with weights flipping sign means the step size is too large: cut the learning rate tenfold. A NaN on the very first update points to bad inputs or a broken loss.

open as a page

Your logistic fit's loss drops 1e-9 per epoch while the weight norm doubles — what is happening?

level: seniorimportance: should knowfreq 38%

basics

~20 s

Those two symptoms together are the signature of separated data, not slow learning. The loss is creeping toward zero along a direction with no finite optimum, so the weights grow without bound and extra training will not help.

open as a page

Why is squared-error loss on a logistic model's sigmoid output not convex in the weights?

level: middleimportance: nice to knowfreq 32%

basics

~20 s

Squaring the gap between a label and a sigmoid composes a bowl with an S-curve, producing flat saturated regions and stationary points that are not the global best. Cross-entropy instead collapses to a convex function of the linear score.

open as a page

What stopping criteria end a gradient-descent fit when you watch only the training loss?

level: seniorimportance: nice to knowfreq 36%

basics

~20 s

Three, used together: the full-data gradient norm falling below a relative tolerance, the relative improvement in the epoch-average loss falling below something like 1e-6, and a maximum-epoch budget as a backstop. Record which one fired.

open as a page