Diffusion models: how pure noise becomes an image

The forward process of adding noise, a network that estimates the noise, DDPM and DDIM written from scratch, eight clusters without mode collapse, how many sampling steps are needed, a comparison with VAE and GAN on digits and the road to images from text.

12 minutes Matěj Žalmánek 2026-10-16

Series AI basics Part 46 of 52

Full series syllabus 52 of 83 topics published

What you will take away

How the forward and reverse diffusion processes work.
Why diffusion trains more stably than a GAN and what the number of steps costs.
How diffusion becomes a text-to-image generator.

Useful to know first: Generative adversarial networks (GAN): the forger versus the detective, Variational autoencoders (VAE): new images from a continuous latent space

Stable Diffusion, Midjourney, DALL·E 3 and the models that generate video all rest on the same idea, and it is surprisingly simple. Destroying an image is easy — just keep adding noise to it until only noise is left. If we learned to undo each small step, we could start from pure noise and peel an image out of it step by step. In this part we program it from scratch and compare it with VAE and GAN on the same data.

It builds on the parts Generative adversarial networks and Variational autoencoders.

1. Forward process: an image dissolves in noise

Definition — forward (diffusion) process

Let d ≥ 1 be the dimension of the data, x₀ ∈ ℝ^d an image from the data, T ≥ 1 the number of steps and β₁, …, β_T ∈ (0, 1) the variance schedule. Let ε₁, …, ε_T ~ N(0, I) be mutually independent random vectors, independent of x₀ (I is the d × d identity matrix). The forward (diffusion) process is the sequence of random vectors x₁, …, x_T ∈ ℝ^d given by the recurrence

x_t = √1 − β_t · x_t−1 + √β_t · ε_t, t = 1, …, T.

Denote

ᾱ_t = ∏_s=1^t (1 − β_s) for t = 1, …, T, ᾱ₀ = 1.

Then for every t the closed form holds (theorem below)

x_t = √ᾱ_t · x₀ + √1 − ᾱ_t · ε, ε ~ N(0, I) independent of x₀,

i.e. given x₀, x_t has the distribution N(√ᾱ_t x₀, (1 − ᾱ_t) I). The sequence ᾱ_t is decreasing, and when it is almost zero for large enough t, x_t has almost the distribution N(0, I) — pure noise in which almost no trace of x₀ is left.

In plain words

A drop of ink in a glass of water: for a moment you can see the shape of the drop, then it spreads out, and in the end the water is evenly coloured and the original shape cannot be recognised. The closed formula says you can jump straight to any moment — there is no need to simulate all the steps.

x_t = √ᾱ_t · x₀ + √1 − ᾱ_t · ε

x_t: the noisy image at step t — exactly what the network gets as input during training
√ᾱ_t: how much of the original image is left; ᾱ_t is the product of all (1 − β_s) up to step t, so it falls with every step
x₀: the original image from the data
√1 − ᾱ_t: how much noise there is; the square roots are there because it is the variances that add up: (√ᾱ)² + (√1 − ᾱ)² = 1, so data with variance 1 have variance 1 at every step and at the end pass smoothly into N(0, I)
ε: one sample of noise from N(0, I) — exactly what the network will guess; knowing it turns training into ordinary regression

Theorem — why one jump is enough

Let x₀, x₁, …, x_T be the forward process with schedule β₁, …, β_T ∈ (0, 1) and ᾱ_t = ∏_s=1^t (1 − β_s). Then for every t = 1, …, T there exists a random vector ε ~ N(0, I) independent of x₀ such that

x_t = √ᾱ_t x₀ + √1 − ᾱ_t ε.

Proof. By induction on t. For t = 1 it is directly the definition of the step, since ᾱ₁ = 1 − β₁. Suppose the claim holds for t − 1, i.e. x_t−1 = √ᾱ_t−1 x₀ + √1 − ᾱ_t−1 ε′ with ε′ ~ N(0, I) independent of x₀. Substitute into step t:

x_t = √1 − β_t √ᾱ_t−1 x₀ + √1 − β_t √1 − ᾱ_t−1 ε′ + √β_t ε_t.

The first term is √ᾱ_t x₀, because ᾱ_t = (1 − β_t) ᾱ_t−1. The other two are independent normal vectors with mean 0 (ε′ depends only on x₀ and ε₁, …, ε_t−1); their sum is again normal with mean 0 and the variances add up:

(1 − β_t)(1 − ᾱ_t−1) + β_t = 1 − (1 − β_t) ᾱ_t−1 = 1 − ᾱ_t.

The sum therefore equals √1 − ᾱ_t ε for some ε ~ N(0, I) independent of x₀. ∎

Example — our noise schedule in numbers

β grows linearly: β_t = β₁ + (β_T − β₁) · t − 1/T − 1, that is by β_T − β₁/T − 1 = 0.00101 per step.

`t`	β_t	ᾱ_t	√ᾱ_t (signal)	√1 − ᾱ_t (noise)
1	0.00010	0.9999	1.000	0.010
2	0.00111	0.9988	0.999	0.035
25	0.02432	0.7351	0.857	0.515
37	0.03643	0.5045	0.710	0.704
38	0.03744	0.4856	0.697	0.717
50	0.04955	0.2830	0.532	0.847
100	0.10000	0.0056	0.075	0.997

Let us check the second row by hand: ᾱ₂ = (1 − β₁)(1 − β₂) = 0.9999 · 0.99889 = 0.99879. Signal and noise are equal (ᾱ = 0.5) between steps 37 and 38; at step 100 the original image has a weight of only 0.075 against 0.997 for the noise (colours as in the formula above).

In our case T = 100 and β grows linearly from 0.0001 to 0.1. The share of the original signal √ᾱ_t is 0.86 at step 25, 0.53 at step 50 and only 0.07 at step 100.

A three in the forward process (from the left t = 0, 10, 25, 50, 75, 100)

2. Reverse process: a network that guesses the noise

Definition — DDPM (denoising diffusion probabilistic model)

Let T, β₁, …, β_T and ᾱ_t be as in the forward process, let p_data be the data distribution on ℝ^d, and let ε_θ : ℝ^d × {1, …, T} → ℝ^d be a neural network with parameters θ that estimates the added noise from a noisy image and the step number. DDPM (denoising diffusion probabilistic model; Jonathan Ho, Ajay Jain and Pieter Abbeel, 2020; the idea of Jascha Sohl-Dickstein et al., 2015) is a pair of procedures:

Training: the parameters θ are chosen by minimizing the mean squared error L(θ) = E_{x₀, t, ε} ‖ε − ε_θ(√ᾱ_t x₀ + √1 − ᾱ_t ε, t)‖², where x₀ ~ p_data, t is uniformly distributed on {1, …, T}, ε ~ N(0, I), and all three are independent.
Sampling (input: the trained network ε_θ; output: a sample x₀ ∈ ℝ^d):
1. draw x_T ~ N(0, I);
2. for t = T, T − 1, …, 1 draw z ~ N(0, I) (for t = 1 set z = 0) and compute x_t−1 = 1/√1 − β_t (x_t − β_t/√1 − ᾱ_t · ε_θ(x_t, t)) + √β_t · z;
3. after step t = 1 return x₀.

The network thus learns by ordinary regression to estimate the noise, and during sampling it subtracts it step by step.

In plain words

The network gets a blurry, noisy photo and the question “how much noise is in it and what does it look like?”. If it can estimate the noise, it can also subtract a little of it. We repeat this a hundred times, each time with a bit of fresh randomness — and an image emerges from the noise. Training is ordinary regression: no opponent as in a GAN, no compromises as in a VAE.

L(θ) = E_{x₀, t, ε} ‖ε − ε_θ(√ᾱ_t x₀ + √1 − ᾱ_t ε, t)‖²

ε: the noise we added to the image ourselves — the correct answer, which is why we know it without labelling any data
ε_θ(…, t): the network’s estimate of the noise; it also gets the step number t, because at t = 2 it has little noise to find and at t = 100 x_t is almost pure noise
√ᾱ_t x₀ + √1 − ᾱ_t ε: the noisy image x_t produced in one jump — thanks to the closed formula it is enough to draw t and ε for each example; nothing is simulated step by step

The expectation is over images from the data, a uniformly drawn step t ∈ {1, …, T} and noise ε ~ N(0, I); in practice an average over the batch. Ho et al. obtained this simple loss by simplifying the variational lower bound (the one a VAE also optimises) — they dropped the weights of the individual steps.

x_t−1 = 1/√1 − β_t · (x_t − β_t/√1 − ᾱ_t · ε_θ(x_t, t)) + √β_t · z

x_t: the current noisy state — we start from pure noise x_T ~ N(0, I)
ε_θ(x_t, t): the estimate of the noise that is in x_t
β_t/√1 − ᾱ_t: how big a piece of the estimated noise to subtract — only the share belonging to one step (β_t), rescaled to the total noise level √1 − ᾱ_t; in our case 0.01 at step 1 and 0.10 at step 100
1/√1 − β_t: undoes the shrinking from the forward step (there it was multiplied by √1 − β_t); at step 100 it is 1.054
√β_t · z: fresh noise z ~ N(0, I) (in the last step z = 0) — the reverse process is random like the forward one; without it the samples would be pulled towards average images. The size √β_t is one of the two choices Ho et al. tried

The bracket multiplied by the first factor is an estimate of the mean of an image one step less noisy; then as much randomness is added to it as belongs to the step.

# training (DDPM)
repeat 16,000×:
    for each of the 64 examples in the batch:
        x₀ = random example from the data
        t  = random from 1 … 100
        ε  = vector of random numbers from N(0, 1)
        x_t = √ᾱ_t · x₀ + √(1 − ᾱ_t) · ε
        error = ‖ε − ε_θ(x_t, code(t))‖² / dimension
    Adam step on the mean error

# sampling (DDPM, 100 steps)
x = vector from N(0, 1)
for t = 100, 99, …, 1:
    ε̂ = ε_θ(x, code(t))
    z = vector from N(0, 1), zero at step t = 1
    x = (x − β_t / √(1 − ᾱ_t) · ε̂) / √(1 − β_t) + √β_t · z
return x

Definition — sinusoidal time encoding

Let T ≥ 1 be the number of steps and t ∈ {1, …, T}. The sinusoidal time encoding is the vector code(t) ∈ [−1, 1]⁸,

code(t) = (sin πt/T, cos πt/T, sin 2πt/T, cos 2πt/T, sin 4πt/T, cos 4πt/T, sin 8πt/T, cos 8πt/T),

i.e. pairs (sin fπt/T, cos fπt/T) for the frequencies f = 1, 2, 4, 8. Instead of the number t the network gets these eight numbers. The slow frequencies say “roughly where in the process we are”, the fast ones tell neighbouring steps apart; all the numbers lie between −1 and 1, so no input outweighs the others.

For t = 25 (a quarter of the process, angles π/4, π/2, π and 2π): code(25) = (0.707, 0.707, 1.000, 0.000, 0.000, −1.000, 0.000, 1.000).

Our network is the ordinary multilayer network from the part on GANs: the input is x_t plus eight numbers of the sinusoidal encoding of time t (like the positional encoding in a transformer), the output is the estimate of the noise. On the circle it has two hidden layers of 64 neurons, on the digits of 128. It is trained with Adam for 16,000 steps.

3. Eight clusters without collapse

The same data as in the previous part: eight clusters on a circle. This is what 400 samples look like during the reverse process:

Reverse process: from noise (t = 100) to data (t = 0)

Even at step 25 the points are scattered almost uniformly (quality 3%), at step 10 the circle starts to show, and in the last ten steps the points gather into clusters. At the end all 8 clusters are covered and each got 28 to 50 of the 500 samples. The quality (share within 3σ of a centre) is 63%.

The difference from a GAN is not in quality but in reliability. The GAN covered all eight clusters in only one of four runs; diffusion did it in the single run we tried (one run is no proof, but it agrees with what is known about diffusion). The diffusion loss is not a game but a regression, and the network is penalised for every data sample — it cannot “give up” on any cluster.

4. How many steps are needed

Definition — DDIM

Let ε_θ be the trained DDPM network, ᾱ_t as in the forward process and ᾱ₀ = 1. For times 0 ≤ s < t ≤ T and a state x_t ∈ ℝ^d we define the estimate of the clean image and the DDIM step from time t to time s:

x̂₀ = x_t − √1 − ᾱ_t ε_θ(x_t, t)/√ᾱ_t, x_s = √ᾱ_s x̂₀ + √1 − ᾱ_s ε_θ(x_t, t).

DDIM sampling with n steps (Jiaming Song, Chenlin Meng and Stefano Ermon, 2020):

choose times T = t_n > t_n−1 > … > t₀ = 0 (in our case evenly spaced);
draw x_T ~ N(0, I);
for i = n, n − 1, …, 1 perform the DDIM step from time t_i to time t_i−1;
return x₀ (in the last step ᾱ₀ = 1, so x₀ = x̂₀).

DDIM thus uses the same trained network as DDPM, but after choosing x_T it draws nothing more (it is deterministic) and it can jump in arbitrarily large steps.

In plain words

Instead of a hundred tiny steps we take twenty bigger ones. In each, the network asks “what would the finished image probably look like?” and jumps part of the way accordingly. When the jumps are too big, the network misses.

x_s = √ᾱ_s · x̂₀ + √1 − ᾱ_s · ε_θ(x_t, t), x̂₀ = x_t − √1 − ᾱ_t ε_θ/√ᾱ_t

x_t: the state we are in right now
ε_θ: the noise estimate — the same network as for DDPM, nothing is retrained
x̂₀: the estimate of the finished image: the closed formula of the forward process solved for x₀; it divides by √ᾱ_t, which is small at the start, so this is where the estimate is least accurate
√ᾱ_s: how much of the image there should be at the target time s — the same factor as in the forward process, just for a different time
√1 − ᾱ_s: how much noise should remain; DDIM uses the same estimated noise and draws no new one — that is why it is deterministic

A step is “denoise completely, then add noise again up to the level of time s”. That is why DDIM can jump arbitrarily far.

Example — why two steps are not enough (simplified)

DDIM with n steps goes through evenly spaced times: with 5 steps 100 → 80 → 60 → 40 → 20 → 0, with 2 steps 100 → 50 → 0. The first jump is always from t = 100, where √ᾱ₁₀₀ = 0.075.

If the network’s noise estimate is off by δ, the estimate x̂₀ is off by δ · √1 − ᾱ₁₀₀/√ᾱ₁₀₀ = 13.3 δ — the error is amplified more than tenfold. After substituting into the jump to time s, the new state differs from the correct one by δ · (√1 − ᾱ_s − √ᾱ_s · 13.3) (colours as in the DDIM formula above):

5 steps, jump 100 → 80: factor −1.59,
2 steps, jump 100 → 50: factor −6.23,

With a small jump the bad estimate x̂₀ is mixed in with a small weight √ᾱ₈₀ = 0.19 and the following steps correct it. With a jump to the middle of the process it enters with weight √ᾱ₅₀ = 0.53 and only a single step remains. The argument follows only one error in one step and assumes the network is otherwise accurate; but it shows why the first, noisiest steps are the most sensitive.

Quality and coverage on the circle by number of sampling steps DDPM (100 steps) DDIM best GAN run

DDIM with 50 steps reaches a quality of 79% — more than DDPM with 100 steps (63%), most likely because it adds no random noise that our imprecise network fails to remove in the last steps. With 10 steps it is still 70%, with 5 steps 60%. With 2 steps sampling falls apart (0%). The number of steps is the main cost of diffusion: each step is one pass through the network, whereas a GAN generates an image in a single pass.

5. Digits: comparing three generators

On 8 × 8 digits (pixels rescaled to the range −1 to 1) the network is trained for 16,000 steps.

The first ten samples

model	distance to the nearest digit	least represented digit (of 200)	digits with fewer than 10 samples
VAE (k = 8)	0.054	10	0
GAN, seed 1	0.049	3	3
GAN, seed 2	0.042	0	6
diffusion (100 steps)	0.063	12	0

Here we have to be honest: our diffusion has a distance of 0.063, worse than the VAE (0.054) and the GAN (0.042). Residual noise is visible in the images. But the digits are evenly represented — the least frequent digit has 12 samples, while the GAN with seed 2 misses two digits entirely.

Why is the sharpness worse? A small multilayer network has to denoise images at all hundred noise levels at once, and that is a harder task than the single mapping noise → image of a GAN. In return it improves steadily with compute: the distance of the samples fell during training 0.096 (step 4,000), 0.082 (step 8,000), 0.068 (step 12,000), 0.063 (step 16,000) (a quick estimate from 100 DDIM samples with 20 steps). No collapse and no swings. Real diffusion models use convolutional U-Net networks or transformers with hundreds of millions of parameters and train for weeks — and then they win on sharpness too.

6. From digits to images from text

Conditioning. The network ε_θ(x_t, t, c) additionally gets a description c — a text embedding from a language model, attached through attention (the part on attention). It is trained on billions of image–caption pairs.
Classifier-free guidance (Jonathan Ho and Tim Salimans, 2021). The network is trained both with and without the description, and during sampling the estimate is shifted in the direction “with description minus without description”, amplified by a coefficient. The image then matches the prompt better at the cost of less variety.
Latent diffusion. Diffusion runs in the small latent space of an autoencoder, not in pixels — see the part on VAEs. Thanks to that, Stable Diffusion (2022) runs even on a gaming graphics card.
Fewer steps. Better samplers and distillation (a network learns to imitate many steps in one or a few) have cut generation from hundreds of steps to a handful.

ε̃ = ε_θ(x_t, t, ∅) + w · (ε_θ(x_t, t, c) − ε_θ(x_t, t, ∅))

ε_θ(…, ∅): the noise estimate without a description (empty condition ∅) — “what any image looks like”; the network learns it because during training the description is occasionally dropped at random
ε_θ(…, c) − ε_θ(…, ∅): how much the estimate changes when the network knows the description — the direction in which the description pulls the image
w: guidance strength: w = 0 ignores the description, w = 1 is an ordinary conditional model, w > 1 exaggerates the influence of the description — the image matches the prompt better, but the samples are more alike

During sampling ε̃ is used in place of ε_θ; the cost is two passes through the network per step. Ho and Salimans write the same formula as (1 + w′) ε_θ(…, c) − w′ ε_θ(…, ∅), that is w = 1 + w′. The diffusers library uses a default of w = 7.5 for Stable Diffusion.

7. Summary

The forward process gradually adds noise; you can jump to any step with the formula √ᾱ x₀ + √1 − ᾱ ε.
The network learns to estimate the added noise — ordinary regression. Generation runs backwards from pure noise.
On eight clusters diffusion covered all of them in a single run (DDIM with 50 steps 79%); the GAN only in one of four runs.
Fewer steps (DDIM) speed up sampling; with 10 steps the quality is 70%, with 2 steps it falls apart.
On digits diffusion covers all of them, but with our small network it lags behind in sharpness; it improves steadily with compute, though.

Want to get hands-on? Have an AI agent train a small DDPM on two-dimensional data (say, points in the shape of your initials) and plot how the points assemble into letters during the reverse process.

Do you see it differently? Write to support@coden-app.com.

Diffusion models: how pure noise becomes an image

1. Forward process: an image dissolves in noise

2. Reverse process: a network that guesses the noise

3. Eight clusters without collapse

4. How many steps are needed

5. Digits: comparing three generators

6. From digits to images from text

7. Summary

An Agent on your computer, driven from your phone