skip to content

In Keras compile(), when must the loss be given from_logits=True?

level: middleimportance: must knowfreq 58%

answer

  1. about what the loss is handed
  2. raw scores versus probabilities
  3. string shorthand hides a default
  4. applied once, never twice
  5. log-sum-exp stability

basics

~20 s

Set from_logits=True whenever the model's last layer emits raw unbounded scores with no softmax or sigmoid activation. The loss then applies the activation internally, which is numerically safer. If the last layer already outputs probabilities, leave it False.

solid answer

~50 s

`from_logits` tells the crossentropy loss what it is being handed. The string shorthands — `"binary_crossentropy"`, `"categorical_crossentropy"`, `"sparse_categorical_crossentropy"` — all resolve to `from_logits=False`, meaning the loss assumes it receives **probabilities** and will take their log directly. If your final `Dense` layer has no activation, those are logits, and taking the log of a raw score is wrong: the reported loss can go negative or `nan`, and the gradients are not the ones you want. The fix is to pass the loss object explicitly, e.g. `keras.losses.SparseCategoricalCrossentropy(from_logits=True)`. The reverse mistake — a softmax output plus `from_logits=True` — does not crash either: the loss softmaxes an already-normalised vector, flattening the distribution so gradients shrink and the model learns slowly. The preferred setup is a linear output layer plus `from_logits=True`, because the fused log-sum-exp path is more numerically stable; apply the activation yourself when you need probabilities from `predict()`.

code

python · 15 lines
python
import keras

# Linear head: no activation on the last layer -> logits
model = keras.Sequential([
    keras.layers.Dense(64, activation="relu"),
    keras.layers.Dense(10),
])
model.compile(
    optimizer="adam",
    loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),
    metrics=["accuracy"],
)

# predict() now returns logits; convert when probabilities are needed
probs = keras.ops.softmax(model.predict(keras.ops.zeros((2, 8)), verbose=0))

go deeper

for a junior

Recall that from_logits=True means the last layer has no softmax or sigmoid, and that the string loss names assume probabilities. Check the final layer's activation before choosing.

for a middle

Explain both mismatch directions and what each looks like — nan or exploding loss one way, quietly compressed gradients the other — and know that you must pass the loss object to change the flag.

for a senior

Argue for a linear head plus from_logits=True on numerical-stability grounds under mixed precision, and account for the consequence at inference: predict() emits scores, so probability-consuming metrics and downstream thresholds need an explicit activation.

for a principal

Treat the head-and-loss pairing as a contract between training and serving: decide once whether the exported model emits logits or probabilities, document it, and make sure calibration, thresholds and monitoring downstream are written against that decision rather than rediscovering it.

## What a logit is here In Keras a **logit** is simply the raw, unbounded output of the final layer before any squashing activation — the value a `Dense(10)` layer with no `activation=` argument produces. A **probability** is what comes out after `softmax` (multiclass, sums to 1) or `sigmoid` (per-label, each in `(0, 1)`). Crossentropy is defined over probabilities, so somewhere between the last layer and the loss value, that activation has to be applied exactly once. `from_logits` is the switch that says where. - `from_logits=False` (the default, and what every string shorthand gives you): "you are handing me probabilities; I will take their log as-is." - `from_logits=True`: "you are handing me raw scores; I will apply the softmax/sigmoid internally, fused with the log." ## The two mismatches **Linear output + `from_logits=False`.** The loss takes the log of numbers that may be negative or far above 1. Keras clips inputs to avoid `log(0)`, so you typically do not get an immediate crash — you get a loss curve that looks broken: values that are implausibly large, that go negative, or that turn into `nan` a few steps in, with accuracy stuck at chance. This is the common version of the bug, because writing `Dense(num_classes)` and `loss="sparse_categorical_crossentropy"` is the natural thing to type. **Softmax output + `from_logits=True`.** The loss softmaxes a vector that already sums to 1. Softmax of a probability vector is a *flatter* probability vector (the values are all within `[0, 1]`, so the exponentials are close together). Nothing raises, the loss decreases, and the model trains — just badly, with a compressed gradient signal that makes convergence slow and the confident-prediction regime unreachable. This is the quieter of the two failures and is worth naming in an interview, because it never announces itself. ## Why the logits path is preferred Computing `log(softmax(z))` in two steps means exponentiating first — and `exp` of a large logit overflows, while `exp` of a very negative one underflows to zero, whose log is `-inf`. Doing it in one fused step lets the implementation subtract the maximum logit first (the log-sum-exp trick) so the intermediate values stay in range. That is why the recommended pattern is a **linear final layer plus `from_logits=True`**, not because it changes the mathematics but because it changes the floating-point behaviour under mixed precision and confident predictions. The cost is that `predict()` then returns logits, not probabilities. If a caller needs probabilities — for a threshold, a calibration step, or a UI — apply `keras.ops.softmax()` or `keras.ops.sigmoid()` to the output, or wrap an activation-only inference model around the trained one. `keras.ops` is Keras 3's backend-agnostic numerics layer, so that code runs unchanged on the TensorFlow, JAX and PyTorch backends. ## Which loss, and the label-shape question next to it `from_logits` is orthogonal to the choice between the sparse and dense crossentropies, and interviewers often ask both in one breath: - `keras.losses.SparseCategoricalCrossentropy` — labels are **integer class indices** of shape `(batch,)`. - `keras.losses.CategoricalCrossentropy` — labels are **one-hot vectors** of shape `(batch, num_classes)`. - `keras.losses.BinaryCrossentropy` — one output unit per label; `from_logits` applies the sigmoid internally. Getting that pair wrong produces a shape error, which is a *loud* failure and therefore much less dangerous than the `from_logits` mismatch. Both are configured in the same `compile()` call. ## Metrics are a separate question Compiling with `metrics=["accuracy"]` on a logits model is fine: accuracy is computed from `argmax`, and `argmax` is invariant to a monotonic squashing like softmax. But metrics that consume probabilities as *values* — `keras.metrics.AUC`, `keras.metrics.Precision` and `keras.metrics.Recall` with a threshold — do care, because a threshold of 0.5 means nothing against a raw score. `keras.metrics.AUC` exposes a `from_logits` argument for this reason. So a model can be losing correctly while its thresholded metrics are nonsense. ## How to check in ten seconds Look at the last layer's activation and at the loss you passed: - `Dense(n)` with no activation → the loss object with `from_logits=True`. - `Dense(n, activation="softmax")` or `"sigmoid"` → `from_logits=False`, so the string shorthand is safe. If you cannot say which of the two your model is, the string shorthand is the one hiding the assumption.

  • What actually goes wrong when a softmax output is paired with from_logits=True?
    Nothing raises. The loss applies softmax to a vector that already sums to one, producing a flatter distribution, so the gradients are compressed and the model converges slowly and never gets confident. It is the quiet version of the mismatch, which is why you check the final layer's activation rather than trusting that the loss is falling.
  • If you train with from_logits=True, what does predict() return?
    Raw logits, not probabilities — the activation lives inside the loss and the loss is not part of inference. Apply `keras.ops.softmax()` or `keras.ops.sigmoid()` to the output, or wrap an inference model that adds the activation layer. Any downstream threshold or calibration must be applied to probabilities, not scores.
  • How does from_logits relate to choosing SparseCategoricalCrossentropy over CategoricalCrossentropy?
    They are independent. `from_logits` describes the *predictions*; sparse-versus-dense describes the *labels* — integer class indices for the sparse form, one-hot vectors for the dense one. Getting the label form wrong raises a shape error immediately, while getting `from_logits` wrong fails silently, which is why the latter is the interview question.
  • Does metrics=["accuracy"] still work correctly on a logits model?
    Yes, because accuracy is derived from `argmax`, and softmax is monotonic so it does not change which index is largest. Threshold-based and probability-valued metrics are different: `keras.metrics.Precision`, `Recall` and `AUC` compare against a threshold like 0.5, which is meaningless on raw scores. `keras.metrics.AUC` takes its own `from_logits` argument.

saying these in an interview costs you the question

  • Thinking from_logits changes the model architecture
  • Assuming a falling loss proves the setting is right
  • Believing the string shorthand defaults to from_logits=True
  • Expecting predict() to return probabilities after logits training
  • Confusing from_logits with the sparse-versus-one-hot label choice

context

resolving links…