Why did image models move from U-Net DDPM to DiT backbones with flow matching?

## Two independent changes, usually shipped together It is worth separating them, because interviewers often conflate them. One change is the *backbone*: what neural architecture does the denoising. The other is the *objective and sampling formulation*: what the network is trained to predict and how you integrate it at generation time. Modern image models changed both, and the reasons differ. ## From U-Net to diffusion transformer The original latent diffusion models used a convolutional U-Net: a downsampling encoder, a bottleneck, an upsampling decoder with skip connections, and cross-attention layers where the text conditioning entered. It works, and convolution's locality bias suits images. A diffusion transformer (DiT) instead patchifies the latent — cuts it into a grid of small squares, embeds each as a token — and runs plain transformer blocks over that sequence, with the noise level supplied through conditioning such as modulated layer norm. Later designs let the text tokens and image tokens attend to one another in shared blocks rather than routing text only through cross-attention. The motivations: - **Scaling.** Transformer families have a well-charted relationship between parameters, data, compute and quality. Scaling a U-Net was more art. Being able to buy quality with compute predictably is a strategic advantage for anyone training frontier image models. - **Global reasoning.** Every patch attends to every other patch from the first block, rather than needing depth to widen a receptive field. That helps long-range consistency: a repeated pattern across a wall, symmetry, coherent perspective. - **Text handling.** Joint attention between the prompt tokens and image patches improved compositional adherence and, notably, rendering legible text inside images — historically the most visible weakness of image generators. - **Reuse.** Transformer training and serving infrastructure, kernels, parallelism strategies and quantisation tooling all transfer from the language side. The costs are the usual transformer costs: attention over patches is quadratic in sequence length, so high resolutions need care, and DiTs are generally data-hungry compared with a convolutional model of similar size. ## From denoising diffusion to flow matching The classic formulation defines a fixed forward process that gradually adds Gaussian noise over many timesteps, and trains a model to reverse one step at a time. Sampling then walks that chain backwards. Reversing a stochastic, curved trajectory accurately takes many small steps; step counts around fifty were routine, and pushing below roughly twenty degraded output noticeably without a specially trained sampler. Flow matching reframes the task as learning a *velocity field*. Define a simple interpolation between a noise sample and a data sample — in the rectified-flow case, a straight line — and train the network to predict the velocity along that path at any point. Generation becomes solving an ordinary differential equation from noise to data with that velocity field. The payoff is directly about curvature. If the learned trajectories are close to straight, a coarse solver with few evaluations tracks them accurately, because a straight path is exactly what a large Euler step assumes. Curved trajectories punish large steps. So the same quality is reachable in far fewer network evaluations. Rectified-flow training explicitly pursues that straightness, and step-distillation techniques push it further, down to a handful of steps or even one for interactive use. Secondary benefits: the training objective is simpler and generally more stable, with fewer schedule hyperparameters to tune, and the same framework transfers cleanly across modalities. ## Why it matters in practice Sampling steps are the unit of cost and latency for image generation. Halving or quartering them changes what products are feasible: live previews that update as a user types, batch catalogue runs that fit an overnight window, on-device generation. When you are asked to justify a model migration, this is the concrete argument — not that the new model is 'better', but that the quality-per-step curve moved, so your cost per accepted image and your time-to-first-image both drop. ## Honest caveats Fewer steps is not free quality. Aggressively distilled few-step variants often trade away diversity and fine detail, and they frequently change or disable the guidance knob, so prompt-steering habits do not transfer. Step counts are model-specific and must be swept, exactly like guidance. And a DiT is not automatically better than a well-trained U-Net at a fixed small scale — the architecture's advantage shows up as you scale data and compute. As of mid-2026 the DiT-plus-flow-matching pairing is the default for new models, but the older stack still runs in plenty of production pipelines and remains perfectly serviceable.