How do you set the weights when a multi-task loss sums a depth error in metres and a segmentation cross-entropy?
answer
- the units decide, not the importance
- compare gradients, not loss values
- normalise first, search second
- total loss cannot rank two weightings
basics
~20 sSet weights so each term contributes comparable gradient, not so the raw numbers match. A depth error in metres dwarfs a cross-entropy in nats, so it dominates by accident of units. Tune the weights on per-task validation metrics, never on total loss.
solid answer
~50 sA weighted sum `L = w_d * L_depth + w_s * L_seg` is only meaningful once the terms are commensurable. Raw magnitudes are set by units and by the loss family: switching depth targets from metres to millimetres inflates that term enormously and silently rebalances the whole model. My first move is to normalise rather than to search — standardise the regression targets, or divide each term by a running estimate of its own magnitude — so every weight starts near one. Then I tune weights against per-task validation metrics, because total loss is not comparable across runs that used different weights. The diagnostic I trust is the size of each term's gradient where it reaches the shared trunk: if one task's gradient norm is orders of magnitude larger, the other terms are acting as faint regularisers rather than as objectives.
go deeper
Be ready to say that a weighted sum mixes terms measured in different units, and that the numerically largest term will drive training unless you rescale or reweight it.
Explain that a weight scales that term's gradient contribution to the shared parameters, and that units, loss family and mean-versus-sum reduction all set the raw scale before any weight is chosen.
Show how you diagnose imbalance in a running job — per-term loss curves and per-task gradient norms at the trunk — and how you select weights on per-task validation metrics rather than on the total.
Own the framing that one weighted sum picks a single point on a trade-off surface, and push the team to agree an exchange rate between tasks up front instead of funding an ever-growing weight sweep.
## The setup A multi-task network has one shared trunk and several task-specific outputs. Training minimises a single scalar, usually a weighted sum: ``` L = w_1 * L_1 + w_2 * L_2 + ... + w_k * L_k ``` A concrete case: a driving-perception trunk that predicts metric depth (an error in metres), lane segmentation (a cross-entropy in nats) and box offsets (a regression term). Left unweighted, the depth term can own the overwhelming majority of the total, and the model trains as a depth model with two decorative extra heads. ## Why the terms are not commensurable Three independent things set a term's raw magnitude, and none of them is importance. - **Units of the target.** Squared error on depth in metres and squared error on depth in millimetres differ by six orders of magnitude for the same physical accuracy. Nothing in the loss knows what a metre is. - **Loss family.** A cross-entropy over a handful of classes sits in a narrow range near ln(number of classes) at initialisation; an unbounded regression error does not. - **Reduction.** Averaging a dense per-pixel loss over hundreds of thousands of pixels and a per-image loss over the batch produces very different scales from identical per-element errors — and the ratio moves when you change crop size or resolution. ## What the weight actually does Gradients are linear in the loss, so `w_i` scales exactly one thing: the size of task i's contribution to the gradient of every shared parameter. The right target for balance is therefore **gradient contribution at the shared parameters**, not equal loss values. Two terms can have identical numerical values and wildly different gradient norms, because the norm depends on the loss's curvature and on how many outputs it touches. This is also why global scale and relative scale behave differently. With plain gradient descent, multiplying the whole sum by a constant is equivalent to scaling the learning rate. With an adaptive optimizer that divides each parameter's update by a running root-mean-square of its own gradients, a global constant nearly cancels out — but the *ratios* between terms never cancel, because they change the direction of the summed gradient. ## A practical recipe 1. **Fix the reduction convention** across tasks first, so weights are not silently absorbing element counts. 2. **Normalise each term** — standardise regression targets, or divide each term by a running (detached) estimate of its own magnitude so all terms enter at order one. Now the weights express priority instead of unit conversion. 3. **Start from equal weights** and confirm on a short run that every term's own validation metric moves at all. 4. **Search a small grid on a log scale** over the one or two ratios that matter. Resist letting this grow into a many-dimensional sweep; if you have six terms, the honest move is to reduce the number of terms or to learn the weights, not to buy more compute. 5. **Log per-term losses and per-task gradient norms** at the trunk throughout training, not just the total. ## Choosing between weightings A single scalarised objective picks **one point on a trade-off surface**. Two weight settings define two different objectives, so their total losses are not comparable — a setting can win on total loss purely by shrinking the term that is expensive to fit. Model selection must therefore be done on per-task validation metrics, with the acceptable exchange rate between tasks stated in advance: how much depth error you will pay for a point of segmentation accuracy is a product decision, not a hyperparameter. A small-weight term deserves an ablation before you trust your intuition about it. Rerun with that weight at zero. If the task's own metric is unchanged, the term was decorative and can go. If other tasks get worse, the term was functioning as a regulariser on the shared representation and its weight should be tuned deliberately rather than left at whatever the sweep produced. ## Things that catch people out - Weights are **not portable**: they encode this dataset's scales, this reduction, these units. Copying them to a new dataset re-introduces the imbalance. - Weights interact with the learning-rate schedule and with weight decay. Changing weights mid-project and keeping the old learning rate is a common source of a run that suddenly will not converge. - Equal loss values are not evidence of balance; they are usually a coincidence of units.
- How do you tell whether a term with a small weight is still doing anything?Watch that term's own validation metric and its gradient norm at the shared parameters, not its share of the total. Then ablate it: rerun with the weight at zero. If the task's metric is unchanged, the term is decorative and can be deleted. If the other tasks get worse, it was acting as a regulariser on the shared representation, and its weight is worth tuning on purpose.
- Why is comparing total validation loss across two runs with different loss weights misleading?Different weights define different objectives, so the two totals are not on the same scale. A run can win on total loss simply by downweighting the term that is hardest to fit, while being worse at everything you ship. Compare per-task metrics under a fixed evaluation protocol, and state up front how much of one task's quality you will trade for another's.
- Does averaging a per-pixel loss over pixels change its effective weight?Yes. Mean reduction divides by the element count, so a dense per-pixel term and a per-image term enter the sum at very different scales even with identical weights, and the ratio shifts again if you change crop size or resolution. Fix the reduction convention across tasks first, then set weights — otherwise the weights are quietly absorbing element counts.
Summing losses in different units is like adding a bill in yen to a bill in dollars: the total is driven by whichever currency prints the bigger numbers, not by which cost matters more.
saying these in an interview costs you the question
- Says the weights should make the raw loss values equal
- Tunes weights by minimising total validation loss
- Treats weights as portable constants across datasets
- Ignores that changing target units silently rescales a term
- Never checks whether a downweighted term still affects anything