What does from_logits=True do in tf.keras.losses.SparseCategoricalCrossentropy?

juniorshould knowfreq 56%

answer

declares what the model's head emits
raw scores versus a distribution
fused softmax is the stable path
mismatch never raises an error
softmax applied twice flattens everything

basics

~20 s

from_logits=True tells the loss that the model's outputs are raw, unnormalized scores, so the loss applies softmax internally in a numerically stable way. The default, False, means the loss assumes the values are already probabilities.

solid answer

~50 s

Keras crossentropy losses can accept either raw scores (logits) or a probability distribution, and from_logits is how you declare which. With from_logits=True the loss fuses the softmax and the log into one numerically stable computation, so very large or very negative scores do not overflow or produce log(0). With the default from_logits=False the loss assumes the inputs already sum to one; it clips them away from zero and takes their log directly. The dangerous part is that a mismatch never raises. If your last layer has activation="softmax" and you also pass from_logits=True, softmax is applied twice: the distribution is flattened, gradients get weak, and the model just trains badly. If your last layer emits logits and you leave from_logits=False, the loss treats raw scores as probabilities and the numbers are meaningless. The recommended setup is a final layer with no activation plus from_logits=True, then apply tf.nn.softmax at inference when you need probabilities.

Keras crossentropy losses can accept either raw scores (logits) or a probability distribution, and `from_logits` is how you declare which. With `from_logits=True` the loss fuses the softmax and the log into one numerically stable computation, so very large or very negative scores do not overflow or produce `log(0)`. With the default `from_logits=False` the loss assumes the inputs already sum to one; it clips them away from zero and takes their log directly. The dangerous part is that a mismatch never raises. If your last layer has `activation="softmax"` and you also pass `from_logits=True`, softmax is applied twice: the distribution is flattened, gradients get weak, and the model just trains badly. If your last layer emits logits and you leave `from_logits=False`, the loss treats raw scores as probabilities and the numbers are meaningless. The recommended setup is a final layer with no activation plus `from_logits=True`, then apply `tf.nn.softmax` at inference when you need probabilities.

code

python · 11 lines

python

import tensorflow as tf

y_true = tf.constant([1, 0])
logits = tf.constant([[0.5, 2.0, -1.0], [3.0, 0.1, 0.2]])

logit_loss = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)
prob_loss = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=False)

print(logit_loss(y_true, logits).numpy())                 # correct
print(prob_loss(y_true, logits).numpy())                  # wrong: raw scores read as probabilities
print(prob_loss(y_true, tf.nn.softmax(logits)).numpy())   # matches the first line

go deeper

for a junior

Know the pairing rule: linear head goes with from_logits=True, softmax head goes with from_logits=False. Say plainly that a mismatch does not raise, it just trains badly.

for a middle

Explain why the fused path is numerically stable — softmax then log can overflow or take log of zero — and describe both failure modes concretely, including what a double softmax does to the gradient signal.

for a senior

Treat it as a review checklist item. When someone reports a model that trains but never learns, check the head activation against the loss flag before tuning anything, and make sure the serving path applies softmax exactly once.

for a principal

Standardize it: one convention across the codebase — logits everywhere in training, normalization only at the serving boundary — so exported models have a single, documented output contract for downstream consumers.

Logits versus probabilities

A classification head is a Dense layer with one unit per class. Its raw output — any real number, positive or negative — is called a logit. Softmax turns a vector of logits into a probability distribution by exponentiating and normalizing. Categorical crossentropy needs the log of the predicted probability of the true class.

That gives two possible contracts for the loss function, and from_logits selects between them:

from_logits=True: "I am handing you raw scores; you do the softmax."
from_logits=False (the default): "I am handing you a probability distribution; just take the log."

Why the fused path is better

Writing softmax and then log separately is numerically fragile. Exponentiating a large logit can overflow to infinity, and a probability that rounds to exactly zero makes log(0) return negative infinity, which turns into NaN as soon as it participates in an average. Passing logits with from_logits=True lets Keras use the algebraically equivalent but stable formulation (the log-sum-exp trick), where the maximum logit is subtracted before exponentiating. That is why the guidance is: leave the final layer linear, and set from_logits=True.

When from_logits=False, Keras defends itself by clipping the incoming values into a small epsilon-bounded range before taking the log. That prevents infinities but does nothing to make wrong inputs meaningful.

The two failure modes

Double softmax. The last layer already has activation="softmax" and you also pass from_logits=True. Keras applies softmax to values that are already in the interval zero to one. Softmax of a near-uniform, already-normalized vector is even flatter, so the model's confident predictions get squashed toward uniform, the gradient signal shrinks and training is slow or plateaus at poor accuracy. Nothing errors, nothing warns; you just get a worse model, which is why this bug survives for weeks in real projects.

Logits read as probabilities. The last layer is linear and you leave the default from_logits=False. Negative logits get clipped up to epsilon, positive ones can exceed one, and the log of that is not a crossentropy of anything. Loss values look strange — often large and barely moving — and accuracy stays near chance.

The diagnostic habit worth building: whenever you see a classification model that "trains but does not learn", check the pairing of the final activation against the loss's from_logits before touching the learning rate.

Which crossentropy

The flag behaves identically across the family. BinaryCrossentropy(from_logits=True) pairs with a single linear output unit and no sigmoid. CategoricalCrossentropy expects one-hot labels; SparseCategoricalCrossentropy expects integer class indices and is the memory-cheaper choice when you have many classes. Only the label format differs between those two — the prediction side and the from_logits semantics are the same.

There are functional equivalents (tf.keras.losses.sparse_categorical_crossentropy and friends) that take the same keyword, so the rule carries over to hand-written losses.

Inference

If you train with logits, the deployed model outputs logits too. argmax over logits gives the same class as argmax over probabilities, so a top-1 classifier needs nothing extra. But if a caller wants calibrated scores you must apply tf.nn.softmax yourself — either in the serving wrapper or by appending a softmax activation layer to an inference-only copy of the model. Forgetting this is how a service starts returning "confidence: 8.4".

The short version for an interview

The flag declares what the model emits. Getting it wrong is silent, costs accuracy rather than raising an exception, and the safe default is a linear head with from_logits=True.

## Logits versus probabilities A classification head is a `Dense` layer with one unit per class. Its raw output — any real number, positive or negative — is called a *logit*. Softmax turns a vector of logits into a probability distribution by exponentiating and normalizing. Categorical crossentropy needs the log of the predicted probability of the true class. That gives two possible contracts for the loss function, and `from_logits` selects between them: - `from_logits=True`: "I am handing you raw scores; you do the softmax." - `from_logits=False` (the default): "I am handing you a probability distribution; just take the log." ## Why the fused path is better Writing softmax and then log separately is numerically fragile. Exponentiating a large logit can overflow to infinity, and a probability that rounds to exactly zero makes `log(0)` return negative infinity, which turns into NaN as soon as it participates in an average. Passing logits with `from_logits=True` lets Keras use the algebraically equivalent but stable formulation (the log-sum-exp trick), where the maximum logit is subtracted before exponentiating. That is why the guidance is: leave the final layer linear, and set `from_logits=True`. When `from_logits=False`, Keras defends itself by clipping the incoming values into a small epsilon-bounded range before taking the log. That prevents infinities but does nothing to make wrong inputs meaningful. ## The two failure modes **Double softmax.** The last layer already has `activation="softmax"` and you also pass `from_logits=True`. Keras applies softmax to values that are already in the interval zero to one. Softmax of a near-uniform, already-normalized vector is even flatter, so the model's confident predictions get squashed toward uniform, the gradient signal shrinks and training is slow or plateaus at poor accuracy. Nothing errors, nothing warns; you just get a worse model, which is why this bug survives for weeks in real projects. **Logits read as probabilities.** The last layer is linear and you leave the default `from_logits=False`. Negative logits get clipped up to epsilon, positive ones can exceed one, and the log of that is not a crossentropy of anything. Loss values look strange — often large and barely moving — and accuracy stays near chance. The diagnostic habit worth building: whenever you see a classification model that "trains but does not learn", check the pairing of the final activation against the loss's `from_logits` before touching the learning rate. ## Which crossentropy The flag behaves identically across the family. `BinaryCrossentropy(from_logits=True)` pairs with a single linear output unit and no sigmoid. `CategoricalCrossentropy` expects one-hot labels; `SparseCategoricalCrossentropy` expects integer class indices and is the memory-cheaper choice when you have many classes. Only the label format differs between those two — the prediction side and the `from_logits` semantics are the same. There are functional equivalents (`tf.keras.losses.sparse_categorical_crossentropy` and friends) that take the same keyword, so the rule carries over to hand-written losses. ## Inference If you train with logits, the deployed model outputs logits too. `argmax` over logits gives the same class as `argmax` over probabilities, so a top-1 classifier needs nothing extra. But if a caller wants calibrated scores you must apply `tf.nn.softmax` yourself — either in the serving wrapper or by appending a softmax activation layer to an inference-only copy of the model. Forgetting this is how a service starts returning "confidence: 8.4". ## The short version for an interview The flag declares what the model emits. Getting it wrong is silent, costs accuracy rather than raising an exception, and the safe default is a linear head with `from_logits=True`.

When would you use CategoricalCrossentropy instead of the sparse version?
CategoricalCrossentropy expects one-hot encoded targets shaped (batch, num_classes); SparseCategoricalCrossentropy expects integer class indices shaped (batch,). Use the sparse one when labels are plain integers — it avoids materializing a wide one-hot matrix, which matters when you have thousands of classes. Use the dense one when your targets are genuinely soft, for example from label smoothing or distillation.
How does this apply to a binary classifier?
The same way: a single output unit with no activation plus `tf.keras.losses.BinaryCrossentropy(from_logits=True)` is the stable pairing, since the loss then fuses sigmoid and log. If the head already has `activation="sigmoid"`, leave the default `from_logits=False`. Passing sigmoid outputs with `from_logits=True` applies sigmoid twice and compresses everything toward 0.5.
If the model outputs logits, how do you serve probabilities?
Apply `tf.nn.softmax` to the model output — either inside a thin serving function, or by building an inference-only model that appends a softmax activation layer on top of the trained one. Do not add softmax to the trained graph and keep training with from_logits=True, or you reintroduce the double-softmax bug.

saying these in an interview costs you the question

Puts softmax on the last layer and also passes from_logits=True
Thinks the flag only affects speed, not correctness
Expects Keras to raise an error when the flag is wrong
Believes from_logits=False is the numerically safer option
Feeds one-hot labels to SparseCategoricalCrossentropy

What does from_logits=True do in tf.keras.losses.SparseCategoricalCrossentropy?

§ answer

basics

solid answer

code

§ go deeper

▾level notes4 levels

▾deep dive~3 min

Logits versus probabilities

Why the fused path is better

The two failure modes

Which crossentropy

Inference

The short version for an interview

▾common follow-ups3 questions

▾red flags5 flags

§ context

▾related questions4 questions

▾related topics2 topics

Logits versus probabilities

Why the fused path is better

The two failure modes

Which crossentropy

Inference

The short version for an interview

answer

go deeper

context