The Diffusion Model

An image generator begins with an array whose entries look like random noise and repeatedly changes that array until it lies near the distribution of images described by a prompt. Diffusion training makes this possible by corrupting real examples with a known Gaussian process and teaching a network how to infer the clean direction. Flow matching learns a velocity field between two distributions more directly. Both views describe transport: move probability mass from a simple source to structured data.

This drawing keeps one two-dimensional distribution in the same coordinates through every sheet. The object is deliberately small enough to inspect. Forward noise, reverse denoising, compression, guidance, numerical sampling, probability flow, and flow matching all act on the same points. A real image model works in thousands or millions of dimensions and learns from image-text data; the toy drawing explains geometry, never output quality.

The lineage runs from denoising diffusion probabilistic models (DDPMs), through classifier-free guidance and latent diffusion, to Diffusion Transformers (DiT) and flow matching. Architectures and product defaults will change. The durable parts are the endpoint distributions, a time-dependent field, conditioning, a numerical solver, and a representation in which the path is learned.

Suppose clean samples follow a curved two-moon distribution in coordinates (x₁, x₂). It is easy to destroy that structure: shrink the existing signal and add independent Gaussian noise a little at a time. After enough steps, the moons blur into a nearly standard normal cloud. Every forward marginal is available in closed form, so training can choose any noise level without simulating all earlier steps.

Generation starts at the opposite endpoint. Draw a fresh Gaussian cloud, ask a learned network which way points should move at the current time, take a numerical step, and repeat. The network has learned local directions from corrupted training examples. It does not retrieve a hidden finished image from inside the initial noise. The initial array supplies a random coordinate in the learned distribution; prompt conditioning bends the route.

Ho, Jain, and Abbeel’s 2020 DDPM paper^[1] connected a practical image model to denoising score matching and a variational objective. Its clean operational split still organizes the subject: the forward process is fixed; the reverse process is learned. Later work changed the representation, the conditioning strength, the network backbone, and the path used at sampling time.

Let x₀ ∈ ℝᵈ be one clean datum. A discrete diffusion defines βₜ ∈ (0,1), sets αₜ = 1 − βₜ, and multiplies them as ᾱₜ = ∏ₛ₌₁ᵗ αₛ. The transition preserves dimension: a d-vector remains a d-vector. At each step it shrinks the existing signal and adds a fresh standard-normal vector of the same shape.

Gaussian composition gives the useful shortcut below. Training samples a clean example, a timestep, and one noise vector ε ∼ 𝒩(0, I), then constructs xₜ directly. No loop through 1 … t is required.

The noise schedule controls signal-to-noise ratio across time. A linear schedule increases βₜ evenly; a cosine-style schedule shapes the cumulative signal more gently near the endpoints. “Noise” here means sampled numeric perturbations with the same tensor shape as the representation. In pixel diffusion those entries perturb pixel channels. In latent diffusion they perturb VAE latent channels. The array may look like television static when decoded or plotted, but the mathematical object is a Gaussian vector.

The destruction is chosen, not learned. That gives the learner unlimited paired examples (xₜ, x₀, ε, t) from finite clean data. It also sets a contract: at the terminal time, the corrupted distribution must be close enough to the simple prior used to begin generation. Too little terminal noise leaves data structure in the endpoint; a badly allocated schedule wastes model evaluations where little changes.

The forward variables describe distributions, not one irreversible editing history. For a fixed x₀, two draws of ε land at different xₜ values while obeying the same marginal. Training asks the network to perform well across those draws. At high noise, several clean examples can plausibly explain one corrupted state, so a squared-error predictor averages compatible directions. Different initial prior samples can select different valid modes; stochastic samplers can add further variation during the reverse path.

Signal-to-noise ratio supplies a more portable clock than a raw integer timestep. Two implementations can both say t=500 while using different step counts and cumulative schedules. Comparing ᾱₜ/(1−ᾱₜ) states how much clean signal remains relative to perturbation. Training weights, distillation schemes, and solver grids are often easier to reason about in that coordinate. Always pair a timestep with its schedule.

The exact reverse conditional depends on the unknown data distribution. A neural network receives the noisy tensor xₜ, a representation of time t, and optional conditioning c. It outputs a d-dimensional tensor. A sampler converts that output into a mean, velocity, score, or clean estimate and advances toward lower noise.

Three prediction parameterizations are common. Epsilon prediction estimates the injected ε. x₀ prediction estimates the clean sample directly. Velocity prediction uses a time-dependent rotation of signal and noise, often written v = √ᾱₜ ε − √(1−ᾱₜ)x₀. Each target has the same dimension as xₜ; conversion formulas link them when the schedule coefficients are known. Parameterization changes numerical weighting and training behavior, not the endpoint data type.

Calling every network output “the noise” hides this choice. A model configured for velocity prediction does not literally return the sampled ε, though a scheduler can derive an epsilon estimate from its output. Model and scheduler must agree on parameterization. Mixing them produces a dimensionally valid tensor with the wrong semantics and a failed trajectory.

The reverse mean is uncertain even when the network is accurate. Near the data distribution, a noisy point may sit between several plausible textures or edges. A stochastic reverse process samples from a modeled conditional distribution; a deterministic ODE chooses a reproducible transport path from its initial state. Neither procedure means the model has recovered the exact training example responsible for a local direction. The field summarizes evidence from the learned distribution.

Song and colleagues’ score-based SDE framework^[2] makes the field interpretation explicit. The score ∇ₓ log pₜ(x) points toward increasing density at time t. The reverse-time SDE combines the known forward drift and diffusion with an estimated score. Its probability-flow ordinary differential equation has matching time marginals while following deterministic paths. One trained field can therefore support stochastic and deterministic numerical procedures.

DDPM training can be derived from a variational bound, yet the influential practical objective is compact. Sample x₀ from data, sample t, sample ε, construct xₜ with EQ. 2.2, and minimize squared error between the known perturbation and the prediction. The expectation spans data, time, and Gaussian noise.

The target is available because training created the corruption. One example yields many tasks: faint corruption teaches fine detail; heavy corruption teaches broad structure and global semantics. Timestep sampling and loss weighting decide how often and how strongly those regimes contribute. The model sees a time embedding because the same noisy coordinate can require different interpretations at different signal-to-noise ratios.

For x₀ or velocity prediction, replace the target and retain a declared weighting. A raw mean-squared error across parameterizations does not imply identical emphasis because target scale changes with time. Production systems may add learned variance, perceptual, adversarial, reconstruction, or distillation terms. The simple denoising loss remains the cleanest account of why paired clean/noisy data does not need hand annotation.

Minibatch training updates a shared field, never a lookup table of reverse paths. Generalization comes from recurring visual structure and conditioning relationships across the dataset. Dataset composition, captions, VAE loss, resolution, augmentation, and filtering influence what the learned distribution contains. The diffusion equation cannot recover concepts absent from training evidence.

The loss also explains a common fine-tuning failure. If a small dataset repeatedly pairs one concept with narrow backgrounds, poses, or lighting, the update entangles them. Noise augmentation creates many corruption levels, not independent semantic examples. Caption quality and visual variation still matter.

Batch construction usually mixes timesteps so one optimizer update covers several noise regimes. The time embedding lets shared weights specialize their response without training a separate network for every step. At low noise, spatial evidence is strong and the target may emphasize texture, color boundaries, or residual artifacts. At high noise, local evidence is weak and conditioning plus learned global regularities carry more of the prediction. That division is gradual rather than a fixed early-versus-late job chart.

Loss value alone cannot rank visual models across datasets and parameterizations. It depends on target scaling, timestep distribution, weighting, representation, and preprocessing. Validation should retain the denoising loss under the same contract, then add sample-based measurements and human inspection appropriate to the intended use. A lower training MSE can coexist with a worse decoder, weaker caption alignment, or narrower data coverage.

Text-to-image training encodes a caption into conditioning vectors and gives them to the denoiser, commonly through cross-attention or adaptive modulation. The spatial tensor asks where and what to change; text features supply a requested semantic direction. Read The Transformer for the query-key-value mechanism used by DiT and cross-attention.

Classifier-free guidance^[4] trains one network on conditional and deliberately dropped-conditioning examples. At sampling time it evaluates an unconditional prediction and a conditional prediction, then extrapolates from the former toward the latter. For epsilon parameterization:

At w = 1, the expression equals the conditional prediction. Values above one amplify the conditional difference. The guidance-scale knob trades prompt adherence against diversity and can push samples into oversaturated or distorted regions when extrapolation becomes excessive. Its numeric sweet spot depends on training, parameterization, scheduler, prompt, and implementation; it is not a universal quality dial.

Guidance changes the vector field while preserving coordinates. In the two-moon drawing, unconditional motion restores both modes. A condition selecting the upper arm changes the direction near ambiguous points. Stronger guidance concentrates more paths there and may reduce coverage of the other arm. The same geometric statement scales to a latent image tensor even though its axes are unavailable to human inspection.

The interactive reuses exactly one clean cloud and one fixed set of perturbations. Timestep changes the mixture of signal and perturbation. Schedule changes how slider time maps to noise level. Guidance applies a prepared conditional displacement. Step count and solver apply a small deterministic integration error. These transformations illustrate separate controls; they are not learned and produce no evidence about an image model.

Pixel diffusion applies every network evaluation to the full image tensor. Latent diffusion first trains an autoencoder. Its encoder maps an image x ∈ ℝᴴˣᵂˣ³ to a smaller tensor z ∈ ℝʰˣʷˣᶜ; its decoder maps a generated latent back to pixels. The diffusion model learns in z space. For a representative spatial factor f, h = H/f and w = W/f. Channel count may grow while the total spatial workload falls sharply.

Compression removes some pixel detail and organizes the remaining signal into learned coordinates. That reduces training and sampling cost because the denoiser processes fewer spatial positions. It also creates a ceiling: details discarded or biased by the VAE cannot be recovered faithfully by a perfect latent denoiser. The Embedding Space provides intuition for learned coordinates and neighborhoods; image latents are structured tensors rather than a single semantic point.

Rombach and colleagues’ latent diffusion work^[5] paired pretrained autoencoding with diffusion and cross-attention conditioning. “Stable Diffusion” names product and checkpoint families built from this broad recipe, while latent diffusion names the architectural separation. The explanation survives changes in checkpoint, text encoder, resolution, and user interface.

The denoising backbone historically favored a U-Net: a convolutional encoder-decoder with skip connections that mixes local detail and multi-scale context. Peebles and Xie’s DiT^[6] replaced the common U-Net backbone with a transformer over latent patches. Patchify converts z into a token sequence; transformer blocks mix information; an output projection returns a tensor-shaped prediction. Diffusion remains the training/generation process. DiT is the field estimator’s backbone.

The architecture change shifts inductive bias and scaling behavior. Convolutions build locality and translation structure into their operations. A transformer exposes latent patches to content-dependent mixing through attention, while positional information preserves the patch grid. Time and class or text conditions can enter through tokens, cross-attention, or modulation. The final projection unpatchifies the sequence so the scheduler receives a tensor exactly aligned with the noisy latent. That shape contract is why a DiT can replace a U-Net without redefining the forward process.

Memory savings from latent diffusion are substantial but not described by spatial reduction alone. Attention may scale quadratically with latent token count, convolutional activations scale with feature-map sizes, and the text encoder plus VAE add their own costs. Decoder execution happens after sampling in a basic pipeline, while image-to-image and inpainting also encode inputs and preserve masks or source latents. Report component, shape, precision, and evaluation count rather than attributing the whole runtime to “latent space.”

Training learns a field across time. Sampling chooses discrete times and numerically follows it. A sampler or scheduler specifies the time grid, update equation, stochasticity, parameterization conversion, and sometimes correction stages. “Thirty steps” means roughly thirty denoiser evaluations only for a one-evaluation method; a second-order method may call the network more than once per reported step.

Euler takes the current field as constant over a small interval. Heun evaluates a provisional endpoint and averages slopes, usually reducing local integration error at extra field cost. Ancestral methods inject noise during reverse steps and can produce variation from the same intermediate state. Deterministic DDIM-style paths^[7] remove that per-step randomness and can reach a result with a subsequence of training times. Solver order, network-evaluation count, and stochasticity are separate attributes.

Fewer evaluations reduce latency and cost but increase discretization error and demand more from each prediction. More evaluations can refine a path, though returns saturate and a mismatched or unstable solver can worsen it. Guidance also changes stiffness: a strongly amplified conditional field can require different step sizes. Compare configurations by total network evaluations, wall time, memory, matched seeds, prompts, dimensions, and model revision.

The fairest contact sheet holds the initial noise, prompt, negative conditioning, resolution, decoder, and model fixed. Vary one of step count, time grid, solver, or guidance. A “same seed” comparison can still fail if software maps seeds to different initial arrays or performs extra random draws, so archive the starting tensor when exact correspondence matters.

Image-to-image generation begins partway along the route. Encode an input through the VAE, corrupt its latent to a chosen noise level, and reverse from there under new conditioning. A higher starting noise discards more source information and permits larger structural changes; a lower level preserves more of the encoded input. Inpainting adds a spatial constraint that repeatedly reconciles generated regions with preserved context. These are boundary conditions on the same state path, not separate definitions of diffusion.

Video extends the tensor with time and requires coherence across frames. The field network may add temporal attention, three-dimensional operations, factorized space-time blocks, or conditioning on prior frames. Sampling still integrates a time-indexed generative field; “time” in the diffusion schedule and time in the video axis are distinct coordinates. Confusing them leads to incorrect shape descriptions and cost estimates.

Continuous-time diffusion defines a forward stochastic differential equation with drift f(x,t) and diffusion scale g(t). Its reverse-time SDE uses the learned score and includes stochastic noise. Song and colleagues also derived a probability-flow ODE whose deterministic trajectories have the same marginal distributions pₜ as the SDE. Individual paths differ; the distribution at each time matches under the stated field.

This ODE view turns generation into velocity integration. Lipman and colleagues’ flow matching^[8] trains a vector field vθ(x,t) by regressing against a target conditional probability path. A simple pedagogical path linearly interpolates paired endpoints, xₜ = (1−t)x₀ + t x₁, whose conditional velocity is x₁−x₀. Practical flow-matching constructions choose probability paths and couplings carefully; straight lines in a drawing do not guarantee straight unconditional transport or easy image generation.

Diffusion and flow matching can share a VAE, DiT backbone, text conditioning, CFG-like guidance, and ODE solver. What changes is the path construction and supervised target: diffusion commonly learns a score/noise-related quantity along a Gaussian corruption path; flow matching directly regresses a velocity for a chosen probability path. Both must approximate a time-dependent field and integrate it from a simple source distribution to data.

Endpoint agreement does not make all paths equally easy. A path can create crossing or rapidly changing conditional velocities that demand more network capacity and smaller solver steps. Coupling choices determine which source samples are associated with which data samples during conditional construction. Research on rectified and related flows seeks straighter or easier transport, but “straight” must refer to a declared coordinate system and coupling. A curved route in pixel space may look simpler after learned compression, and the reverse can also occur.

Probability flow supplies the conceptual bridge without claiming that every diffusion sampler and flow-matching model is interchangeable. A score model determines a particular ODE from its forward SDE. A flow-matching model is trained against the velocity of its selected path. Solvers can look identical at inference because both expose dx/dt, while their fields came from different objectives. Check the training contract before naming a generated path.

The durable checklist is therefore short. Declare the source and target distributions. Declare pixel or latent coordinates and the decoder. Declare the time convention and path. Declare whether the network predicts epsilon, clean data, score, velocity, or another equivalent target. Declare conditioning and guidance. Declare solver, time grid, stochasticity, and network evaluations. Those facts let a reader compare systems after product names and preferred samplers change.