How do you set beta in a beta-VAE when you need faithful reconstructions and interpretable factors?

## What beta actually controls A beta-VAE weights the regularisation term: ``` loss = reconstruction + beta * KL( q(z|x) || p(z) ) ``` with beta = 1 recovering the standard VAE. It is tempting to read beta as a quality dial. It is not. It selects an **operating point on a rate–distortion trade-off**: the KL value is the *rate*, the number of nats of input-specific information the code carries, and the reconstruction error is the *distortion*. Sweeping beta traces a curve. Every point on that curve is an optimum of some objective; which one you want depends entirely on what consumes the latent. **Raising beta above one** charges more per nat. The encoder pushes less through, dimensions that were carrying marginal information switch off, and on data generated by a small number of independent factors — the standard synthetic 2-D shapes benchmark, where images vary by position, scale, shape and rotation — the surviving dimensions tend to line up with those factors one at a time. Traversing a single latent dimension then moves one factor while holding the others roughly fixed, which is what people mean by a disentangled code. The cost is reconstruction fidelity, and past some point the code stops being used at all. **Lowering beta below one** buys detail back: sharper reconstructions, more nats in the code, and per-input posteriors that are narrow and scattered. The latent stops resembling the prior in aggregate, so codes drawn from `N(0, I)` decode poorly even though reconstruction looks excellent. ## Why a beta from a paper is worthless on your data Beta is dimensionless only relative to how the reconstruction term is measured. Three things silently rescale it: - **Reduction convention.** Summing squared error over pixels versus averaging over them changes the reconstruction term by a factor equal to the pixel count — often four to six orders of magnitude. The same beta then means something entirely different. - **Data dimensionality and variance.** Higher resolution, more channels, or higher-variance pixels all inflate the reconstruction term relative to a KL measured over a fixed-size latent. - **A learned output variance.** If the decoder's variance is learned rather than fixed, the reconstruction term is effectively divided by it, so the model tunes its own implicit beta while you are tuning the explicit one. The transferable quantity is not beta — it is the **achieved rate in nats per data point**. Report that. Two models with the same rate are comparable in a way that two models with the same beta are not. ## The method **1. Write the acceptance test before you sweep.** Two numbers, agreed in advance: - a maximum acceptable held-out reconstruction error, set by what the downstream consumer tolerates; - a disentanglement measure, which requires ground-truth factors — train a simple probe (a linear or shallow classifier) to predict each known factor from *single* latent dimensions, and report per-factor accuracy plus how many dimensions each factor is spread across. Without ground-truth factors you cannot measure disentanglement, only look at traversals. Say so out loud rather than pretending a number exists. **2. Sweep on a log scale.** Beta over several multiplicative steps, not additive ones. Plot achieved rate against reconstruction error, and the probe score against rate. **3. Take the knee, not the extreme.** The useful point is where probe score has flattened but reconstruction error has not yet fallen off a cliff. Reporting only the highest-beta model because it "looks disentangled" hides the fidelity you paid. **4. Guard the low end of the rate.** Very high beta drives the rate toward zero and the code toward being ignored. Watch per-dimension KL during the sweep and treat all-zero as a failed run rather than an endpoint. ## The alternative to a fixed beta If you need a reproducible operating point, invert the problem: **constrain the rate and let the multiplier adapt**. Set a target KL in nats and adjust the weight during training to hit it — this is a constrained-optimisation view of the same trade-off, where beta becomes the Lagrange multiplier rather than a hyperparameter you guess. A per-dimension floor on the KL is a coarser version of the same idea. Both give you models that are comparable across runs, datasets and resolutions, because the thing you fixed is the thing that means something. ## The honest caveat to state Unsupervised disentanglement is not identifiable in general. There are infinitely many equally good factorisations of a latent space, and without inductive bias or some supervision, nothing in the objective privileges the axes a human would name. Raising beta encourages *axis-aligned, low-rate* codes; whether those axes correspond to interpretable factors depends on the data's structure and on architectural bias, and it is not guaranteed. A candidate who presents high beta as a reliable route to interpretability is overselling it. The defensible position is: use beta to control rate and axis alignment, validate interpretability empirically against factors you actually know, and if interpretability is a hard requirement, buy it with weak supervision rather than hoping the sweep delivers it. ## Making the call State the consumer first. A latent feeding a controller or a downstream classifier usually wants rate — keep beta low and accept an entangled code. A latent a human will inspect or manipulate wants axis alignment and can afford softer reconstructions. A latent used for generation wants an aggregate posterior that actually fills the prior, which is a different check again: decode prior draws and look. One beta cannot serve all three, and pretending otherwise is the failure mode.