How does a tf.GradientTape training step compute and apply gradients in TensorFlow?

## What a tape is `tf.GradientTape` is a recorder. While its context is open, every TensorFlow op executed on a *watched* tensor is appended to a tape, along with enough information to compute that op's derivative. When you later ask for a gradient, TensorFlow walks that recording in reverse and applies the chain rule. The tape auto-watches trainable `tf.Variable` objects that are touched inside the context — which is exactly the set of things a model's weights are — so you rarely have to watch anything explicitly during ordinary training. This is the mechanism behind eager-mode differentiation: there is no static graph you compile in advance and no global backward call. The tape exists for exactly the region of code you wrap. ## The three lines ``` with tf.GradientTape() as tape: logits = model(x, training=True) loss = loss_fn(y, logits) grads = tape.gradient(loss, model.trainable_variables) optimizer.apply_gradients(zip(grads, model.trainable_variables)) ``` **The `with` block** must contain the forward pass *and* the loss. This is the single most common beginner error: computing the loss after the block closes leaves the loss disconnected from the recording, and you get gradients of `None`. **`tape.gradient(target, sources)`** returns a structure matching `sources`. Pass `model.trainable_variables` — not `model.variables`, which also includes non-trainable state such as batch-normalization moving statistics that no optimizer should update. The returned list is positionally aligned with the variable list, which is why `zip` is the idiomatic pairing. **`optimizer.apply_gradients`** takes an iterable of `(gradient, variable)` pairs, applies the optimizer's rule (momentum, Adam moments, weight decay, any gradient clipping configured on the optimizer), increments the optimizer's internal step counter, and assigns the new values. It returns nothing useful; the effect is the mutation of the variables. ## No zero_grad Frameworks that accumulate gradients into the parameter objects require you to clear them before each backward pass. TensorFlow does not: `tape.gradient` returns a fresh list of tensors, and `apply_gradients` consumes it immediately. A candidate who insists TensorFlow needs a gradient-zeroing call is describing a different framework. If you *want* accumulation — for example to simulate a larger batch than fits in memory — you build it yourself: keep a list of `tf.Variable` accumulators, add each micro-batch's gradients into them, apply once every N micro-batches, and reset the accumulators. ## One tape, one backward pass A default tape is non-persistent: it frees its recording as soon as `gradient()` returns, and a second call raises a `RuntimeError`. If you genuinely need several independent gradients from the same forward pass — two losses against different variable subsets, a GAN-style step, or a second-order derivative — construct it as `tf.GradientTape(persistent=True)` and delete the tape when finished so the buffers are released. Nesting two tapes gives higher-order derivatives: the outer tape records the inner tape's gradient computation. ## Speed and correctness details Wrapping the step function in `tf.function` moves it from op-by-op eager execution into a compiled graph, which typically buys a large speedup for small models where Python overhead dominates. Call the model with `training=True` inside the tape so training-time layer behaviour is used. Keep everything in TensorFlow ops: converting an intermediate tensor to NumPy inside the block silently breaks the recorded chain, because NumPy operations are not recorded. Finally, remember the ordering contract. `tape.gradient` does not update anything and `apply_gradients` does not compute anything. Mixing those two responsibilities up in an interview answer — "then I call `tape.gradient` and the weights are updated" — is an immediate tell that the candidate has only ever used `fit()`.