Diffusion Models

Overview#

Diffusion models are the dominant generative architecture for continuous-valued data — images, video, audio, 3D, molecular structures — in 2026. They generate samples by reversing a forward Gaussian noising process: corrupt clean data with progressively more noise over T steps until it is indistinguishable from white noise, then train a network to undo that corruption one step at a time. At inference, sample noise, denoise iteratively, and the result is a sample from the learned data distribution.

The framework was proposed by Sohl-Dickstein et al. in 2015 as a way to fit complex distributions with tractable likelihood. It sat at the periphery of generative modelling for five years while GANs (sharp samples, unstable training) and VAEs (stable training, blurry samples) dominated. The 2020-2022 period transformed it: DDPM (Ho et al. 2020) made training stable and quality competitive with GANs; DDIM (Song et al. 2020) cut sampling cost by an order of magnitude; Latent Diffusion (Rombach et al. 2021) moved the process into a VAE latent space and made training affordable on academic budgets; Stable Diffusion (August 2022) released open-weights frontier-quality image generation that fit in 4 GB of VRAM.

The downstream cascade was the Cambrian explosion of open generative AI. Stable Diffusion's open release enabled tens of thousands of community fine-tunes, LoRAs, ControlNets, custom samplers and inference UIs. By 2024 every frontier image model — closed (DALL-E 3, Imagen 3, Midjourney) and open (SDXL, SD3, FLUX.1) — was a latent diffusion model with a Transformer backbone. Video models followed (Sora, Veo, Kling, Runway Gen-3 in 2024-2025), then audio (Stable Audio, MusicGen, AudioLDM), then 3D (DreamFusion, RFdiffusion for proteins), each adapting the same recipe to its modality's data layout.

This entry is the reference for the operator and applied researcher who needs to understand diffusion as a systems primitive: what the forward and reverse processes actually compute, why classifier-free guidance is everywhere, what changed from U-Net to DiT to MM-DiT, the trade-offs between samplers, and where the 2026 stack actually sits (diffusers, ComfyUI, FLUX, LoRA fine-tuning). This entry helps you understand diffusion well enough to pick the right model and sampler for your latency budget, plan the GPU footprint a FLUX.1-class workload actually needs, and steer clear of the licence trap that catches teams shipping FLUX.1 [dev] commercially by mistake. If you are deploying image-generation workloads on Yobibyte, this matters because the catalogue (SDXL, SD3, FLUX.1 [schnell]) and the GPU picker (L40S for batch, H100 / H200 for interactive, B200 for FLUX-scale) both encode the architectural reasoning below.

How it works: forward noising and learned reverse denoising#

The forward process is fixed and parameter-free. Take a clean data sample x_0 (an image, a latent, a spectrogram). Add Gaussian noise according to a schedule {beta_1, ..., beta_T}: at step t, x_t = sqrt(1 − beta_t) · x_{t−1} + sqrt(beta_t) · noise. Equivalently, x_t can be computed in closed form from x_0: x_t = sqrt(alpha_bar_t) · x_0 + sqrt(1 − alpha_bar_t) · epsilon, where alpha_bar_t is the cumulative product of (1 − beta_s) for s ≤ t. After T steps (typically T = 1000), x_T is essentially pure Gaussian noise.

The reverse process is learned. A neural network is trained to undo one step of the noising: given a noisy x_t and the timestep t, predict the noise epsilon that was added (the 'epsilon-prediction' parameterisation) or the original x_0 ('x_0-prediction') or a particular weighted combination ('v-prediction', from Salimans & Ho 2022). All three parameterisations are mathematically equivalent up to algebra; v-prediction tends to be more numerically stable, especially for video and audio, and is the modern default.

The training loss is shockingly simple. Sample a random timestep t and a random Gaussian noise epsilon. Compute x_t from a real x_0. Predict the noise. Use mean-squared-error between the prediction and the true epsilon. That's it. There is no adversarial term, no balancing two networks, no mode collapse failure mode. Quality scales predictably with compute and data — the property that made diffusion the architecture of choice for frontier generative models.

The deep reason this works was clarified by Yang Song and collaborators in 2020-2021. Predicting noise at every noise level is mathematically equivalent to learning the score function — the gradient of the log density of the data — at every scale. Once you have the score at every noise level, you can solve a stochastic or ordinary differential equation backwards from pure noise to a sample from the data distribution. The DDPM training loss is one specific way to learn that score; DDIM sampling is one specific way to integrate the resulting ODE.

At inference, start with x_T sampled from N(0, I). Iteratively run the reverse process: at each step t, ask the network for the noise prediction, use it to compute a less noisy x_{t-1}, and repeat. After T steps you have a sample x_0 from the learned distribution. DDPM samples stochastically with T = 1000 steps. DDIM samples deterministically and lets you skip steps (50-step sampling is standard, 20-step works). Higher-order solvers (DPM-Solver++, UniPC) push to 10-20 steps. Consistency distillation (Song et al. 2023) pushes to 1-4 steps at the cost of training a separate distilled model.

# diffusion_train_loop.py — illustrative DDPM training step.
# pip install torch diffusers
import torch
import torch.nn.functional as F
from diffusers import UNet2DModel, DDPMScheduler

device = "cuda" if torch.cuda.is_available() else "cpu"
unet = UNet2DModel(
    sample_size=64, in_channels=3, out_channels=3,
    layers_per_block=2,
    block_out_channels=(64, 128, 256, 256),
).to(device)
scheduler = DDPMScheduler(num_train_timesteps=1000)
opt = torch.optim.AdamW(unet.parameters(), lr=1e-4)

def train_step(x_0):
    # 1. Sample a random timestep per example.
    batch = x_0.shape[0]
    t = torch.randint(0, scheduler.config.num_train_timesteps, (batch,), device=device)

    # 2. Sample noise; add it to x_0 according to the schedule.
    noise = torch.randn_like(x_0)
    x_t = scheduler.add_noise(x_0, noise, t)

    # 3. Predict the noise; MSE loss against the true noise.
    noise_pred = unet(x_t, t).sample
    loss = F.mse_loss(noise_pred, noise)

    opt.zero_grad(); loss.backward(); opt.step()
    return loss.item()

# Replace with a real DataLoader over an image dataset.
# x_0 = next(iter(loader)).to(device)
# print(train_step(x_0))

# Inference (after training):
# from diffusers import DDIMScheduler, DiffusionPipeline
# scheduler_inf = DDIMScheduler.from_config(scheduler.config)
# # ... run scheduler_inf.set_timesteps(50) and iterative denoising loop ...

Variants: parameterisation, schedule, sampler and backbone#

Diffusion has four axes of choice, and the modern model designer picks one option on each. The combinations matter — SD 1.5 (epsilon-prediction + linear schedule + DDIM sampler + U-Net) and FLUX.1 (rectified flow + linear schedule + Euler sampler + MM-DiT) have very different training and inference characteristics despite both being 'diffusion models'.

• DDPM (Ho et al. 2020) — original noise-prediction, stochastic 1000-step sampling, linear schedule. Stable but slow. Almost never used directly in production today; superseded by DDIM at inference.
• DDIM (Song et al. 2020) — same trained model, deterministic sampler, skip steps to 25-50. The bridge to practical inference. Modern UIs ship it as a fallback option.
• Improved DDPM / iDDPM (Nichol & Dhariwal 2021) — introduced the cosine noise schedule that dominated for several years; remains a strong baseline.
• DPM-Solver and DPM-Solver++ (Lu et al. 2022, 2023) — higher-order ODE integrators; same trained model as DDIM, but 10-20 steps for equivalent quality. The default sampler in diffusers and ComfyUI for SDXL-era models.
• UniPC (Zhao et al. 2023) — predictor-corrector solver, often 5-10 steps for SDXL-quality output.
• Consistency models (Song et al. 2023) and distillation (LCM, SDXL Turbo, SDXL Lightning, Hyper-SD) — train a separate distilled model that generates in 1-4 steps. Quality lower than a full sampler but production-realtime.
• Rectified flow (Liu et al. 2023, arXiv:2209.03003) — re-derives the schedule from an optimal-transport perspective so trajectories are straight lines, needing fewer sampling steps. SD3 and FLUX use it; the new default for frontier training.
• DiT (Peebles & Xie 2022) — replaces U-Net with a pure Transformer over latent patches. Better scaling than U-Net, used by SD3.
• MM-DiT (Multimodal DiT, Esser et al. 2024) — two parallel Transformer streams (one for text, one for image), joined via cross-stream attention. FLUX.1 and SD3 use it; the de-facto frontier architecture.

Where it is used today: image, video, audio, 3D, science#

Diffusion is the dominant generative architecture in 2026 for every continuous-valued modality that tolerates iterative sampling. Text remains the exception — autoregressive Transformers still dominate language generation — but everything else has converged on diffusion.

Image: Stable Diffusion XL (open, U-Net, ~3.5B), Stable Diffusion 3 / 3.5 (open, MM-DiT, 2-8B), FLUX.1 [dev] (open, MM-DiT, 12B), FLUX.1 [pro] (closed API), DALL-E 3 (OpenAI, closed), Imagen 3 (Google, closed), Midjourney v7 (closed). The open / closed split is roughly: closed labs lead on aesthetic polish and prompt-following at the frontier; open releases dominate workflow integration, fine-tuning ecosystems and on-prem deployment.

Video: Sora and Sora 2 (OpenAI), Veo 2 / Veo 3 (Google DeepMind), Kling (Kuaishou), Runway Gen-3 (Runway), Pika, Movie Gen (Meta), Open-Sora and CogVideoX on the open-weights side. All use 3D DiT / 3D MM-DiT backbones with space-time attention over patches.

Audio: Stable Audio 2 (Stability AI), AudioLDM 2 (open), MusicGen (Meta, technically a Transformer LM over audio tokens but with diffusion components), Suno and Udo for music generation. Generally operate on mel-spectrograms or learned audio latents with U-Net or DiT backbones.

3D and scientific: DreamFusion / Magic3D for text-to-3D (diffusion guides NeRF rendering); Zero-1-to-3 for image-to-3D; RFdiffusion for protein backbone design (Baker lab, 2023) — the technique that produced the first diffusion-designed proteins to bind real targets; GenIE for small molecules; AlphaFold 3 uses diffusion components for structure prediction.

Inverse problems: diffusion models trained on natural images serve as plug-and-play priors for inpainting, super-resolution, deblurring and JPEG artefact removal. DPS (Diffusion Posterior Sampling), Pi-GDM and DDRM are the standard techniques.

Yobibyte customers needing on-prem or UK / EU sovereign image generation deploy SDXL, SD3 or FLUX.1 [schnell] through the same workspace and spend-cap surface as their LLM workloads. The diffusion-specific runtime selection (NVIDIA Triton Inference Server and TensorRT-tuned graphs, not vLLM) happens transparently; what the customer chooses is model, region, and per-image latency target.

Trade-offs and known limitations#

Diffusion won the modality wars by being stable to train and high-quality to sample. Its weaknesses are correspondingly well-understood and have well-developed mitigations.

Sampling cost remains the headline. A 50-step diffusion model needs 50 full forward passes per sample, compared to one for a one-shot generator. Distillation techniques (Latent Consistency Models, SDXL Turbo, SDXL Lightning, Hyper-SD, FLUX.1 [schnell]) cut this to 1-4 steps at the cost of training a separate distilled model and losing some quality. For real-time applications (browser-side image generation, VR), distilled diffusion is the only practical option.

Classifier-free guidance (Ho & Salimans 2022) doubles inference cost because each step requires both a conditional and an unconditional pass — the model is run twice per step, then linearly combined. CFG scale tuning matters: higher scale = more prompt adherence and saturation, lower = more diversity and natural images. SDXL uses CFG = 5-9; FLUX uses ~3.5 because its training already encodes strong conditioning; SD3 uses guidance distillation to eliminate the double pass.

Aspect ratio and resolution generalisation. Most diffusion models are trained at a fixed resolution (SD 1.5 at 512², SDXL at 1024², FLUX at 1024² with bucketing). Generating outside this range often produces composition artefacts (duplicate heads, missing limbs) unless multi-aspect-ratio bucketing was used in training. SDXL introduced multi-aspect training; modern frontier models train at many aspect ratios from the start.

Text rendering in images was famously broken until ~2024. SDXL could not reliably spell three-letter words; SD3, DALL-E 3 and FLUX changed this by using larger text encoders (T5-XXL in addition to CLIP) and by training on more text-heavy data. FLUX in particular renders text robustly.

Compositional reasoning. Diffusion models struggle with prompts involving counting, spatial relationships and rare object combinations ('a red cube on top of a blue sphere, to the left of a green pyramid'). Mitigations: better text encoders, structured prompting techniques, ControlNet for explicit layout control, regional prompting (e.g. Attention Couple). The gap is closing but has not closed.

Licence fragmentation: this is operationally important. Stable Diffusion 1.5 was Creative ML OpenRAIL-M (permissive commercial). SDXL was the same. Stable Diffusion 3 introduced a Stability AI Community Licence with revenue thresholds. FLUX.1 [dev] is non-commercial; FLUX.1 [schnell] is Apache 2.0; FLUX.1 [pro] is API-only. Read the actual licence on the checkpoint, not the announcement blog, before deploying commercially.

Bias and safety. Diffusion models inherit biases from their training data and can generate harmful, NSFW or copyrighted content. Standard mitigations: safety checker classifiers (CLIP-based, blocking NSFW), prompt-filter lists, watermarking (SynthID for Google models, C2PA metadata for Adobe and others). Open models typically ship with safety checkers; production deployments add their own filtering layer.

Practical implementation notes#

Libraries that matter in 2026: Hugging Face diffusers is the canonical Python library — every common model (SDXL, SD3, FLUX, AudioLDM, Stable Video Diffusion) ships as a Pipeline class with one-line inference. ComfyUI is the visual node-graph editor used by most power users for image and video workflows. AUTOMATIC1111 / Forge are the older browser-UI options, still widely deployed. Invoke and SwarmUI cover similar ground with cleaner UX. For training, diffusers has built-in trainers; kohya-ss / sd-scripts is the most popular community trainer for LoRA and DreamBooth fine-tuning; OneTrainer is the modern GUI alternative.

Fine-tuning paths: LoRA (Hu et al. 2021, arXiv:2106.09685) is the universal cheap path — train a small low-rank adapter (typically 50-200 MB for SDXL, 100-400 MB for FLUX) on a few dozen to a few thousand images, applied at inference by adding the LoRA weights to the base model. DreamBooth (Ruiz et al. 2022) is the per-subject fine-tune for capturing a specific identity (person, character, product) into a unique token. Full fine-tuning of the base model is rare outside frontier labs because the storage and compute cost is order-of-magnitude higher than LoRA for marginal quality gain.

Distillation for speed: SDXL Turbo (1-step, single image), SDXL Lightning (2/4/8-step variants), Hyper-SD (1-step with adversarial distillation), FLUX.1 [schnell] (4-step Apache-2.0 distilled FLUX). All are separate downloaded checkpoints that work with diffusers' standard pipelines via the corresponding scheduler config.

ControlNet (Zhang et al. 2023) lets you condition diffusion on structural inputs — edge maps, depth maps, pose skeletons, segmentation masks — by training small adapter networks that inject conditioning into the U-Net or DiT. The standard tool for layout-controlled generation. SDXL ControlNets are abundant in the community; FLUX ControlNets exist for canny, depth and pose.

Inference cost ranges: SDXL at 1024² with 30 steps DPM-Solver++ on an H100 is ~2-3 seconds per image at FP16, ~1-1.5 seconds at FP8; on an L40S, ~5-8 seconds at FP16. FLUX.1 [dev] at 1024² with 28 steps Euler on an H100 is ~6-10 seconds at FP16; SD3 medium ~3-5 seconds. SDXL Turbo or FLUX.1 [schnell] (1-4 steps) drop to 0.3-1.0 seconds per image. Video models are orders of magnitude heavier — a 5-second 720p clip on Sora-class models takes minutes on multiple H100s.

Deployment patterns: for low-latency single-image inference, vLLM is not the right tool (it is built for autoregressive LMs); instead use the diffusers pipeline directly, NVIDIA TensorRT for an optimised graph, Triton Inference Server to host it, or specialised stacks like Replicate Cog / fal.ai serverless. For batch generation, diffusers + accelerate for multi-GPU distribution. For workflow automation, ComfyUI's API mode.

Compliance and licensing: training-data provenance is a live legal question in 2026 — Getty vs Stability AI (UK), NYT vs OpenAI (US), and EU AI Act Article 28 transparency obligations are all relevant. For commercial deployment, prefer models with clear training-data disclosure (Adobe Firefly trained on Adobe Stock; FLUX with documented filtered sources) and watermarking pipelines (SynthID, C2PA metadata). Always verify the actual checkpoint licence before commercial use — the licence on the announcement blog may not match the LICENSE file in the repository.

Where diffusion sits in the Yobitel stack#

Yobibyte's model catalogue includes the open frontier image-generation checkpoints (Stable Diffusion XL, SD3, FLUX.1 [schnell]) for customers who need on-prem or sovereign image generation. Inference is routed through industry-standard runtimes (NVIDIA TensorRT and Triton Inference Server for diffusion-specific deployments; vLLM and friends do not apply here). Customers see the same workspace, region pin, spend cap and OIDC binding as for LLM workloads — the diffusion-specific plumbing (scheduler choice, CFG scale, VAE handling) is exposed where it matters and abstracted where it does not.

Omniscient Compute routes diffusion workloads to the appropriate GPU SKU per workload: L40S for cost-efficient batch generation, H100 / H200 for low-latency interactive use, B200 for FLUX-scale and video models. The picker reasons about VRAM (FLUX.1 [dev] needs ~24 GB at FP16, ~12 GB at FP8) and per-step latency, both of which are diffusion-architecture-derived signals.

Yobitel's first-party AI applications do not currently expose image generation as an end-customer feature — the platform is positioned for sovereign LLM and embedding workloads in 2026 — but the catalogue and the routing logic support diffusion workloads for customers who deploy them through Yobibyte directly.