How AI Image Generators Work: Diffusion, Explained

Many AI images start as pure static. Lots of image generators work this way, and it's called diffusion. This 500x Explained, drawn entirely in code, shows how it turns that static into a picture: training, generating, guidance and the seed, plus latent space as a bonus here in the post.

Training: add noise, learn to predict it

Training starts with real images. Noise is added to each one in small steps until nothing but noise is left. The model is trained to look at a noisy image and predict the noise that was added. Ho, Jain and Abbeel's 2020 paper, Denoising Diffusion Probabilistic Models (DDPM), describes this as a forward process that "gradually adds Gaussian noise to the data", and the model they train predicts that noise.

Generating: run it in reverse

To make a new image, the model starts from fresh random noise and runs the process backwards. Each step it predicts the noise and removes a little of it. After many steps, a picture appears. The Stable Diffusion 3 paper sums it up: diffusion models "create data from noise by inverting the forward paths of data towards noise."

The step count is a setting. In Hugging Face's diffusers library, the Stable Diffusion pipeline defaults to 50 steps, and its docs say more steps "usually lead to a higher quality image at the expense of slower inference."

Guidance: your prompt steers every step

Your prompt is fed to the model at every step, so each guess about the noise is nudged toward the image you described. Stable Diffusion uses classifier-free guidance (Ho and Salimans, 2022), and the strength of that nudge is the guidance scale: diffusers says a higher value "encourages the model to generate images closely linked to the text prompt at the expense of lower image quality."

Latent space: a smaller canvas

Running all those steps on every pixel is expensive. Latent diffusion (Rombach et al., 2022), the approach behind Stable Diffusion, trains in the smaller latent space of an autoencoder, so generation starts from latent noise and is decoded to pixels at the end. For Stable Diffusion v1, the latent is 8 times smaller than the image on each side. The paper reports this cuts compute significantly compared with pixel-based diffusion models.

Try this: the seed is the starting static

The starting static comes from a number called the seed. When your tool shows the seed and you like an image, reuse that seed and tweak the prompt slightly. The same seed gives every try the same starting static, so the result can stay close to the one you liked. It can also change a lot: Hertz et al. (2022) note that "even a small modification of the text prompt often leads to a completely different outcome." Hugging Face's diffusers docs suggest exactly this workflow: pick an image tied to a seed and "iteratively improve on it by crafting a more detailed prompt." Not every app shows the seed, and some don't let you set it.

Want variety instead? Keep the prompt and change the seed. Keep the seed, prompt and settings the same and you'll usually get the same picture back on the same setup.

Static in, picture out, one step at a time.

Other approaches

The video says "many", and that matters. Some newer models use flow matching, such as the rectified flow in Stable Diffusion 3, which also goes from noise to image but along a straighter path. And OpenAI describes GPT-4o's image generation as an autoregressive model built into ChatGPT, and describes DALL·E as a diffusion model.

The tile boards in the video (16x16 pictures, a cat and a yellow X, and a 10-step noise meter) are an illustration made for this video. Real models work with far more numbers.