How does a text-to-image diffusion model turn random noise into an image?

## The starting point is noise, not a canvas A diffusion image generator does not paint. It does not draw a subject and then a background, and it does not build the image left to right or top to bottom the way an autoregressive text model emits tokens. It begins with a tensor of random values — visually, television static — and progressively refines *the entire image at once* over a fixed number of passes. ## What the model was trained to do Training is the easy half to understand. Take a real image, pick a random noise level, add that much Gaussian noise to it, and hand the noisy image plus the noise level to a network. The network's job is to predict the noise that was added. Because the clean image and the noise are both known during training, this is ordinary supervised learning with a mean-squared-error loss. Do this over hundreds of millions of image-caption pairs at every noise level, and you get a network that, for any noisy image, can say "here is roughly the junk in this, and here is roughly the picture underneath". Some model families parameterise the target slightly differently — predicting the clean image, or a velocity term, rather than the raw noise — but they are all equivalent views of the same denoising signal. ## Sampling: running it backwards Generation inverts the process. Start from pure noise, which is the highest possible noise level. Ask the network what the noise is. Remove a *fraction* of it, taking the sample from "100% noise" to something like "96% noise", and repeat. Each step lands you at a slightly lower noise level until, at the last step, you have a clean image. The reason it is a loop and not a single forward pass is that the network's estimate from pure noise is extremely vague — it can only guess at broad structure. As some noise is removed, the partially formed image gives the network far more to condition on, and its next estimate is sharper. Composition and layout settle in the early steps; texture, edges and fine detail arrive late. This is why an aborted run at step 5 of 30 looks like a coloured blur with the right general arrangement. ## Where the prompt enters The text prompt is encoded once by a text encoder into a sequence of vectors. Those vectors are supplied to the denoiser at *every* step, usually through cross-attention or joint attention inside the backbone. That is the crucial point for interviews: conditioning is not a one-time seed, it is a continuous pull applied to every nudge. It is also why prompt adherence can be dialled up or down independently of step count — that is the guidance scale, a separate control. ## The scheduler A scheduler (or sampler) decides *how many* steps there are and how much noise comes off at each one. Cutting steps makes each removal larger and coarser; below the model's usable range you get mushy or malformed output. Raising steps far above it buys almost nothing and costs linear time. Modern models are tuned for a specific band, and few-step distilled variants are trained specifically to work at a handful of steps. ## Why two runs of the same prompt differ The starting noise is drawn from a random seed. Fix the seed, the prompt, the model, the step count and the guidance value, and generation is essentially reproducible; change the seed and you get a different sample of the same described distribution. This is the standard mechanism for "give me four options" and for reproducing a specific result later — you keep the seed. ## What this rules out Because the network only ever outputs a noise estimate, there is no database lookup and no collage step. The model's knowledge lives entirely in its weights. And because the whole frame is refined jointly, there is no notion of "finishing" one region before another — which is exactly why localized edits need a separate mechanism such as masking, rather than just asking for a smaller change.