Guide

Production and economics

Serving diffusion models

Latency is steps times step time. Distillation beats micro-optimizing a 50-step loop. Interactive image gen is often batch 1.

133 of 138 · 2 min read

Diffusion and flow-matching image models run a denoising loop: often 20–50 steps (fewer for distilled models), each a full forward through a large UNet or DiT. The work is strongly compute-bound on GPUs. There is little "decode memory bandwidth" analogy to LLM tokens/s.

Latency ≈ steps × step_time. Cutting steps (distillation, consistency models, cached adapters) is the main product lever. Batch size helps throughput for non-interactive queues. Interactive "one image now" is often batch=1 with step distillation for UX. Video multiplies cost by frames × steps. Caching and latent reuse matter a lot.

Concept

A denoising step is one full forward that takes a noisy latent closer to a clean image. Distilled models learn to do the same job in fewer steps. See generative models for the training side.

Walking through step math

If one step is 80 ms on a given GPU and the base model needs 40 steps → 3.2 s before VAE. A distilled 8-step model at 90 ms/step → 0.72 s. That is the difference between a toy and a product — and it is mostly model choice, not kernel micro-optimization.

Serving SDXL at 50 steps on an interactive button with no distillation is how users bounce.

Caching. Reuse text-encoder outputs for repeated prompts. Cache LoRA merges carefully. Some servers cache intermediate latents for img2img workflows. Treat cache keys as part of the API contract (prompt + seed + params).

Export steps executed, end-to-end seconds, queue depth and VRAM high-water. Autoscaling diffusion on CPU util while GPUs are saturated leaves pending pods forever. Scale on GPU util or queue time.

The big idea

A photographer who takes 50 exposures of the same scene is slower than one who learned to get the shot in 8. Each exposure is still a full camera cycle. The picture breaks for video: you now multiply by frames, so even 8 steps can be expensive unless you reuse latents across time.

Common mistakes

  • Micro-optimizing kernels on a 50-step interactive loop instead of shipping a distilled checkpoint.
  • Autoscaling on CPU while the GPU is the saturated resource.
  • Forgetting cache keys include seed and params, so two "same prompt" hits return the wrong image.

Check yourself

  1. 40 steps at 80 ms, versus 8 steps at 90 ms. What are the two e2e times before VAE? Answer: 3.2 s versus 0.72 s.
  2. Why is diffusion unlike LLM decode on the roofline? Answer: each step is a fat compute-bound forward, not a memory-bound single-token weight stream.

Key takeaways

  • Diffusion cost is step count times step time.
  • Distillation is the product lever. Batch helps queues, not the "one image now" button.
  • Video multiplies by frames. Cache text-encoder outputs and latents on a real key.
  • Scale on GPU queue or util, not CPU.

Further learning

Next, classic ML serving is the opposite extreme: CPU, sub-5 ms, no transformer.

Serving embeddings, rerankers, speech and diffusion

Inference beyond chat LLMs. Encoders, speech, diffusion and classic ML each have a different bottleneck and a different batcher.

Generative models beyond text

How images, video and speech are generated. Compares the main model families, builds diffusion from first principles, and explains why diffusion inference costs so differently from chat.

Serving speech models

ASR cost scales with audio duration. Batch similar clip lengths. Streaming TTS is judged on time-to-first-audio, not realtime factor alone.