The Diffusion Model

An image generator starts with an array of numbers that look like random noise and keeps changing that array until it lands somewhere near the distribution of images a prompt describes. Diffusion training makes that possible by running the process backward first: take real images and wreck them, adding ordinary bell-curve noise — Gaussian noise — in small steps whose sizes you chose in advance. Because you know exactly how much you added at every step, you can train a network to point back toward the clean version. Flow matching skips the wrecking story and learns a velocity field instead: for any point, which direction to move and how fast. Both are describing transport — moving probability mass from a simple source to structured data.

This drawing keeps one two-dimensional distribution in the same coordinates across every sheet, deliberately small enough that you can look at it. Forward noise, reverse denoising, compression, guidance, numerical sampling, probability flow, and flow matching all act on those same points. A real image model works in thousands or millions of dimensions and learns from image-text data, so what the toy explains is geometry — never output quality.

The lineage runs from denoising diffusion probabilistic models (DDPMs), through classifier-free guidance and latent diffusion, out to Diffusion Transformers (DiT) and flow matching. Architectures and product defaults will keep changing. What survives is the pair of endpoint distributions, a time-dependent field, conditioning, a numerical solver, and the representation the path gets learned in.

Say the clean samples follow a curved two-moon distribution in coordinates (x₁, x₂). Destroying that structure is easy: shrink the existing signal a little and add independent Gaussian noise, over and over. After enough steps the moons blur out into something close to a standard normal cloud. Every forward marginal has a closed form, which means training can jump to any noise level it wants without simulating all the steps before it.

Generation starts at the far end. Draw a fresh Gaussian cloud, ask a learned network which way the points should move at the current time, take a numerical step, repeat. The network learned those local directions from corrupted training examples. It is not fishing a hidden finished image out of the initial noise — the initial array just supplies a random coordinate inside the learned distribution, and the prompt conditioning bends the route from there.

Ho, Jain, and Abbeel's 2020 DDPM paper^[1] connected a working image model to denoising score matching and a variational objective. The operational split it drew still organizes the whole subject: the forward process is fixed, the reverse process is learned. Everything since has changed the representation, the conditioning strength, the network backbone, or the path taken at sampling time — not that split.

Let x₀ ∈ ℝᵈ be one clean datum. A discrete diffusion defines βₜ ∈ (0,1), sets αₜ = 1 − βₜ, and multiplies them along as ᾱₜ = ∏ₛ₌₁ᵗ αₛ. The transition preserves dimension — a d-vector stays a d-vector — and at each step it shrinks the signal that's there and adds a fresh standard-normal vector of the same shape.

Gaussian composition hands you the shortcut below, and it's the reason training is cheap. Sample a clean example, a timestep, and one noise vector ε ∼ 𝒩(0, I), then build xₜ in a single shot. No loop through 1 … t needed.

The noise schedule controls the signal-to-noise ratio over time. A linear schedule raises βₜ evenly; a cosine-style schedule shapes the cumulative signal more gently near the endpoints. "Noise" here means sampled numeric perturbations with the same tensor shape as the representation, nothing more exotic. In pixel diffusion those entries perturb pixel channels; in latent diffusion they perturb VAE latent channels. The array may look like television static once you decode or plot it, and the mathematical object is a Gaussian vector either way.

The destruction is chosen rather than learned, and that's what makes the whole thing work. It gives the learner unlimited paired examples (xₜ, x₀, ε, t) out of finite clean data. It also sets a contract: at the terminal time, the corrupted distribution has to be close enough to the simple prior you'll start generation from. Too little terminal noise leaves data structure sitting in the endpoint, and a badly allocated schedule burns model evaluations in regions where almost nothing changes.

The forward variables describe distributions, not one irreversible edit history. Fix x₀ and draw ε twice and you land at two different xₜ values that obey the same marginal. Training asks the network to do well across all of those draws. At high noise several clean examples can plausibly explain a single corrupted state, so a squared-error predictor averages the compatible directions together. Different prior samples can select different valid modes, and stochastic samplers add more variation on the way back.

Signal-to-noise ratio is a far more portable clock than a raw integer timestep. Two implementations can both report t=500 while running different step counts and different cumulative schedules. Comparing ᾱₜ/(1−ᾱₜ) tells you how much clean signal is left relative to the perturbation, which is usually the easier coordinate for reasoning about training weights, distillation schemes, and solver grids. A timestep without its schedule attached doesn't mean much.

The exact reverse conditional depends on the data distribution, which nobody has. So a neural network takes the noisy tensor xₜ, a representation of time t, and optional conditioning c, and puts out a d-dimensional tensor. A sampler turns that output into a mean, a velocity, a score, or a clean estimate, and advances toward lower noise.

Three prediction parameterizations show up in practice. Epsilon prediction estimates the injected ε. x₀ prediction estimates the clean sample directly. Velocity prediction uses a time-dependent rotation of signal and noise, usually written v = √ᾱₜ ε − √(1−ᾱₜ)x₀. All three targets have the same dimension as xₜ, and conversion formulas connect them once you know the schedule coefficients. Parameterization changes the numerical weighting and the training behavior, not the data type coming out the other end.

Calling every network output "the noise" papers over that choice, which is where a lot of confusion starts. A model configured for velocity prediction doesn't return the sampled ε at all, though a scheduler can derive an epsilon estimate from what it does return. Model and scheduler have to agree on the parameterization. Mix them and you get a tensor with perfectly valid dimensions, the wrong semantics, and a trajectory that goes nowhere.

The reverse mean stays uncertain even when the network is accurate. Close to the data distribution, a noisy point can sit between several plausible textures or edges with nothing to break the tie. A stochastic reverse process samples from a modeled conditional distribution; a deterministic ODE traces a reproducible transport path from wherever it started. Neither one means the model recovered the specific training example behind a local direction. What the field summarizes is evidence from the learned distribution as a whole.

Song and colleagues' score-based SDE framework^[2] makes the field interpretation explicit. The score ∇ₓ log pₜ(x) points toward increasing density at time t. The reverse-time SDE combines the known forward drift and diffusion with an estimated score, and its probability-flow ordinary differential equation has matching time marginals while following deterministic paths. One trained field can therefore feed both stochastic and deterministic numerical procedures, which is why samplers proliferate the way they do.

You can derive DDPM training from a variational bound, and the practical objective that ended up mattering is much simpler than the derivation. Sample x₀ from the data, sample t, sample ε, build xₜ with EQ. 2.2, and minimize squared error between the perturbation you know you added and the prediction. The expectation runs over data, time, and Gaussian noise.

The target is available for free because training created the corruption in the first place. One example turns into many tasks: faint corruption teaches fine detail, heavy corruption teaches broad structure and global semantics. Timestep sampling and loss weighting decide how often and how strongly each regime gets to contribute. The model sees a time embedding because the same noisy coordinate can call for a different interpretation depending on the signal-to-noise ratio it arrived at.

For x₀ or velocity prediction, swap the target and keep a declared weighting. A raw mean-squared error compared across parameterizations doesn't imply equal emphasis, because target scale moves with time. Production systems pile on learned variance, perceptual, adversarial, reconstruction, or distillation terms as well. The simple denoising loss is still the cleanest account of why paired clean and noisy data needs no hand annotation at all.

Minibatch training updates one shared field. There's no lookup table of reverse paths anywhere in it. Generalization comes from visual structure and conditioning relationships that recur across the dataset, which means dataset composition, captions, VAE loss, resolution, augmentation, and filtering all shape what ends up inside the learned distribution. No amount of diffusion math recovers a concept the training evidence never contained.

The loss also explains a fine-tuning failure people hit constantly. Feed a small dataset that keeps pairing one concept with the same narrow backgrounds, poses, or lighting, and the update entangles them. Noise augmentation manufactures many corruption levels; it does not manufacture independent semantic examples. Caption quality and genuine visual variation still have to come from somewhere.

Batch construction usually mixes timesteps so one optimizer update spans several noise regimes at once. The time embedding is what lets shared weights specialize their response without training a separate network per step. At low noise, spatial evidence is strong and the target leans toward texture, color boundaries, and residual artifacts. At high noise, local evidence is weak and conditioning plus learned global regularities carry more of the prediction. The division is gradual, not a fixed early-versus-late job chart.

A loss value on its own can't rank visual models across datasets and parameterizations. It moves with target scaling, timestep distribution, weighting, representation, and preprocessing. Validation should keep the denoising loss under one fixed contract, then add sample-based measurements and human inspection suited to the actual use. Lower training MSE coexists comfortably with a worse decoder, weaker caption alignment, or narrower data coverage.

Text-to-image training encodes a caption into conditioning vectors and hands them to the denoiser, usually through cross-attention or adaptive modulation. The spatial tensor carries where and what to change; the text features supply the semantic direction being asked for. The Transformer covers the query-key-value mechanism that DiT and cross-attention both run on.

Classifier-free guidance^[4] trains one network on conditional examples and on examples where the conditioning was deliberately dropped. At sampling time it evaluates both an unconditional and a conditional prediction, then extrapolates from the first toward the second. In epsilon parameterization:

At w = 1 the expression is just the conditional prediction. Values above one amplify the conditional difference. That knob trades prompt adherence against diversity, and pushed far enough the extrapolation drags samples into oversaturated or distorted territory. Where the sweet spot sits depends on training, parameterization, scheduler, prompt, and implementation, so treat it as a trade rather than a quality dial.

Guidance changes the vector field while leaving the coordinates alone. In the two-moon drawing, unconditional motion restores both modes. Condition on the upper arm and the direction changes near the ambiguous points. Turn guidance up further and more paths concentrate there, at the cost of coverage on the other arm. The same geometric statement scales straight up to a latent image tensor, even though nobody can eyeball those axes.

The interactive reuses exactly one clean cloud and one fixed set of perturbations. Timestep changes the mixture of signal and perturbation. Schedule changes how slider time maps to noise level. Guidance applies a prepared conditional displacement. Step count and solver apply a small deterministic integration error. These transformations illustrate separate controls; they are not learned and produce no evidence about an image model.

Pixel diffusion runs every network evaluation against the full image tensor, which gets expensive fast. Latent diffusion trains an autoencoder first. Its encoder maps an image x ∈ ℝᴴˣᵂˣ³ down to a smaller tensor z ∈ ℝʰˣʷˣᶜ, and its decoder maps a generated latent back to pixels. The diffusion model then learns entirely in z space. For a representative spatial factor f, h = H/f and w = W/f; channel count can grow while the total spatial workload drops sharply.

Compression throws away some pixel detail and reorganizes what's left into learned coordinates. Training and sampling get cheaper because the denoiser has fewer spatial positions to process. It also installs a ceiling: whatever the VAE discarded or biased, no latent denoiser — however perfect — can put back faithfully. The Embedding Space builds intuition for learned coordinates and neighborhoods, with the caveat that image latents are structured tensors rather than one semantic point.

Rombach and colleagues' latent diffusion work^[5] paired pretrained autoencoding with diffusion and cross-attention conditioning. "Stable Diffusion" names the product and checkpoint families built on that recipe; latent diffusion names the architectural separation underneath. The explanation outlives changes in checkpoint, text encoder, resolution, and user interface, which is why it's worth learning at that level.

The denoising backbone was historically a U-Net — a convolutional encoder-decoder with skip connections, mixing local detail against multi-scale context. Peebles and Xie's DiT^[6] swapped that for a transformer over latent patches. Patchify turns z into a token sequence, transformer blocks mix information across it, and an output projection returns a tensor-shaped prediction. Diffusion is still the training and generation process. DiT is just the backbone doing the field estimation.

Changing the architecture shifts the inductive bias and the scaling behavior with it. Convolutions bake locality and translation structure into the operations themselves. A transformer instead exposes latent patches to content-dependent mixing through attention, with positional information preserving the patch grid. Time and class or text conditions can enter through tokens, through cross-attention, or through modulation. The final projection unpatchifies the sequence so the scheduler receives a tensor lined up exactly with the noisy latent — that shape contract is the whole reason a DiT can drop in for a U-Net without touching the forward process.

The memory savings from latent diffusion are real and don't reduce to the spatial factor alone. Attention can scale quadratically with latent token count, convolutional activations scale with feature-map sizes, and the text encoder and VAE bring their own costs. In a basic pipeline the decoder runs after sampling; image-to-image and inpainting also encode inputs and hold onto masks or source latents. Report component, shape, precision, and evaluation count rather than pinning the whole runtime on "latent space."

Training learns a field across time. Sampling picks discrete times and numerically follows it. A sampler — or scheduler — specifies the time grid, the update equation, how much stochasticity to inject, the parameterization conversion, and sometimes correction stages on top. "Thirty steps" means roughly thirty denoiser evaluations only for a one-evaluation method; a second-order method can call the network more than once per reported step.

Euler treats the current field as constant across a small interval. Heun evaluates a provisional endpoint and averages the two slopes, usually cutting local integration error in exchange for another field evaluation. Ancestral methods inject noise during the reverse steps, so the same intermediate state can produce different results. Deterministic DDIM-style paths^[7] strip that per-step randomness out and can reach a result using only a subsequence of the training times. Solver order, network-evaluation count, and stochasticity are three separate attributes that get conflated constantly.

Fewer evaluations cut latency and cost while raising discretization error and asking more of every individual prediction. More evaluations refine the path until the returns saturate, and a mismatched or unstable solver can make things worse no matter how many you spend. Guidance changes the stiffness too: a strongly amplified conditional field can demand different step sizes than the unguided one. Compare configurations by total network evaluations, wall time, memory, matched seeds, prompts, dimensions, and model revision.

The fairest contact sheet holds the initial noise, prompt, negative conditioning, resolution, decoder, and model fixed, then varies exactly one of step count, time grid, solver, or guidance. Even a "same seed" comparison can betray you if the software maps seeds to different initial arrays or makes extra random draws along the way, so archive the starting tensor whenever exact correspondence matters.

Image-to-image generation starts partway along the route. Encode an input through the VAE, corrupt its latent to a chosen noise level, and reverse from there under new conditioning. Start from higher noise and you discard more of the source, allowing bigger structural changes; start lower and more of the encoded input survives. Inpainting adds a spatial constraint that keeps reconciling generated regions against preserved context. Both are boundary conditions on the same state path rather than separate definitions of diffusion.

Video extends the tensor with time and demands coherence across frames. The field network might add temporal attention, three-dimensional operations, factorized space-time blocks, or conditioning on earlier frames. Sampling still integrates a time-indexed generative field. Worth being careful here: "time" in the diffusion schedule and time along the video axis are different coordinates, and conflating them produces wrong shape descriptions and wrong cost estimates.

Continuous-time diffusion defines a forward stochastic differential equation with drift f(x,t) and diffusion scale g(t). Its reverse-time SDE uses the learned score and carries stochastic noise along with it. Song and colleagues also derived a probability-flow ODE whose deterministic trajectories share the same marginal distributions pₜ as the SDE. Individual paths differ; the distribution at each time matches, under the stated field.

That ODE view recasts generation as velocity integration. Lipman and colleagues' flow matching^[8] trains a vector field vθ(x,t) by regressing it against a target conditional probability path. The pedagogical version linearly interpolates paired endpoints, xₜ = (1−t)x₀ + t x₁, whose conditional velocity is just x₁−x₀. Practical constructions choose their probability paths and couplings with much more care — straight lines in a drawing guarantee neither straight unconditional transport nor easy image generation.

Diffusion and flow matching can share a VAE, a DiT backbone, text conditioning, CFG-like guidance, and an ODE solver. What differs is the path construction and the supervised target. Diffusion typically learns a score- or noise-related quantity along a Gaussian corruption path; flow matching regresses a velocity for a path it chose. Both end up approximating a time-dependent field and integrating it from a simple source distribution over to data.

Agreeing on endpoints doesn't make every path equally easy to travel. A path can produce crossing or rapidly changing conditional velocities that demand more network capacity and smaller solver steps. Coupling choices determine which source samples get associated with which data samples during conditional construction. Research on rectified and related flows is chasing straighter or easier transport, and "straight" only means something relative to a declared coordinate system and coupling. A route that curves in pixel space can look much simpler after learned compression, and the reverse happens too.

Probability flow is the conceptual bridge between the two, which is not the same as saying every diffusion sampler and flow-matching model is interchangeable. A score model determines a particular ODE from its forward SDE. A flow-matching model is trained against the velocity of whichever path it selected. At inference the solvers can look identical, because both expose dx/dt, while the fields they're integrating came from different objectives. Check the training contract before you name a generated path.

Which leaves a short checklist that outlasts the product names. Say what the source and target distributions are. Say whether you're in pixel or latent coordinates, and which decoder. Say what the time convention and path are. Say whether the network predicts epsilon, clean data, score, velocity, or some other equivalent target. Say what the conditioning and guidance are. Say what solver, time grid, stochasticity, and evaluation count you used. Those six facts let a reader compare two systems long after the preferred samplers have all been replaced.