Diffusers & Multimodal
Generating images, video, and audio with the diffusers library — pipelines, per-component quantization, and managing memory on models that don't fit
Diffusers & Multimodal
TL;DR
diffusers is the library for generative image, video, and audio models — Stable Diffusion, FLUX, and the current wave of unified any-to-any models. The API is one call, DiffusionPipeline.from_pretrained(...), and the interesting engineering happens in memory management: these pipelines are made of several large sub-models, and the single most useful diffusers-specific skill is quantizing them one component at a time, not the whole pipeline uniformly.
| Property | Value |
|---|---|
| Level | Intermediate |
| Reading time | ~15 minutes |
| Prerequisites | Deployment & Inference |
| You will understand | How a diffusion pipeline is put together, and how to fit a large one into limited VRAM |
One Call, A Pipeline of Models
from diffusers import DiffusionPipeline
import torch
pipe = DiffusionPipeline.from_pretrained(
"black-forest-labs/FLUX.1-dev",
dtype=torch.bfloat16,
).to("cuda")
image = pipe("a fox reading documentation, watercolor").images[0]DiffusionPipeline.from_pretrained looks the same as loading a transformers model, but a single pipe object here is actually several separate neural networks working together:
What's inside a typical text-to-image pipeline
Text encoder(s)
Transformer / UNet backbone
VAE
That backbone is almost always the largest and slowest piece by a wide margin, which is exactly why the next section quantizes it more aggressively than everything else.
The task families
| Task | What goes in | What comes out |
|---|---|---|
| Text-to-image | A text prompt | A generated image |
| Image-to-image | An image + a prompt | A modified image, guided by the prompt |
| Image-to-video | A single image (+ optional prompt) | A short generated video animating it |
| Inpainting / editing | An image, a mask, a prompt | The masked region regenerated to match the prompt |
| Unified any-to-any | Increasingly, one model handles several of the above | Depends on the request |
The model landscape moves fast here. The specific model you'd reach for today — a FLUX variant, an image-to-video model, a unified generation model — will very likely not be the best choice in six months. What's stable is the pipeline shape and the loading/quantization API; treat the specific repo IDs in this page as examples, not recommendations, and check the Hub's trending models for what's current when you actually build something.
Quantizing One Component at a Time
A full-size diffusion pipeline routinely needs more VRAM than a single consumer GPU has. Section 7 of the crash course covered quantization for language models; diffusers takes the same idea further, because a pipeline is made of clearly separable components with very different sizes — so you can quantize the huge one hard and leave a smaller one alone.
import torch
from diffusers import DiffusionPipeline
from diffusers.quantizers import PipelineQuantizationConfig
quant_config = PipelineQuantizationConfig(
quant_backend="bitsandbytes_4bit",
quant_kwargs={
"load_in_4bit": True,
"bnb_4bit_quant_type": "nf4",
"bnb_4bit_compute_dtype": torch.bfloat16,
},
components_to_quantize=["transformer", "text_encoder_2"],
)
pipeline = DiffusionPipeline.from_pretrained(
"black-forest-labs/FLUX.1-dev",
dtype=torch.bfloat16,
quantization_config=quant_config,
).to("cuda")components_to_quantize is the part worth internalizing: name the sub-modules that actually dominate memory (usually the transformer/UNet backbone, sometimes a large secondary text encoder), and leave the rest — typically the VAE and a smaller text encoder — at full precision, because they're cheap to keep and quantizing them buys little.
Quantizing everything uniformly is the naive move, and it's usually the wrong one. The VAE decodes the final image and is comparatively tiny; quantizing it saves little memory but can visibly hurt output quality. Profile where the memory actually goes before deciding what to quantize — it is almost never evenly distributed across components.
When Quantization Alone Isn't Enough
For pipelines that still don't fit, CPU offloading moves components to system RAM when they're not actively being used in the current denoising step:
pipe.enable_model_cpu_offload()This trades speed for memory — components are moved to and from the GPU as needed rather than sitting resident the whole time — and it stacks with quantization rather than replacing it. A common working combination is: quantize the backbone in 4-bit, and offload whatever still doesn't fit.
Memory levers, in the order to reach for them
Per-component quantization
RecommendedThe biggest win for the least quality cost, when applied to the right (largest) component. Start here.
CPU offloading
Free of quality cost, costs speed instead. Reach for it when quantization alone still doesn't fit your VRAM.
A smaller/distilled model variant
Many popular pipelines ship faster, smaller "turbo" or distilled variants at a real quality trade-off — worth it when speed matters more than peak fidelity.
Newer techniques — FP8 quantization and dynamic quantization of the attention layers — are appearing as ways to trade a little accuracy for speed rather than memory, and are worth checking the diffusers documentation for as they mature; the API surface (quant_backend, components_to_quantize) is designed to accommodate new backends without changing how you call it.
Concept Checks
Check yourself
Why does per-component quantization exist as its own idea, rather than just quantizing the whole pipeline?
Because a diffusion pipeline's components differ enormously in size and in how much quantizing them costs in quality. The backbone (transformer/UNet) is usually most of the memory and tolerates aggressive quantization well; the VAE is small and quantizing it barely helps memory while risking visible output degradation. Targeting the actual bottleneck gets most of the memory win with the least quality loss.
A pipeline still doesn't fit in VRAM after 4-bit quantizing the backbone. What's the next lever, and what does it cost?
CPU offloading via enable_model_cpu_offload(). It costs speed, not quality — components move between system RAM and the GPU as each is needed rather than staying resident, so generation gets slower but the memory ceiling drops further. It stacks with quantization rather than being an alternative to it.
Why does this page describe the specific example models as examples rather than recommendations?
Because the generative model landscape changes on a timescale of months, and naming a "best" model today would be stale quickly. What's durable is the pipeline shape (text encoder → backbone → decoder) and the loading/quantization API — those transfer to whatever model is current when you're actually building, which the Hub's trending listings will show you at the time.
Key Concepts Recap
| Concept | One-line summary |
|---|---|
DiffusionPipeline | One from_pretrained call loads a pipeline of several component models |
| Pipeline anatomy | Text encoder(s) → transformer/UNet backbone → VAE decoder |
| The backbone dominates memory | Quantize it first and hardest |
PipelineQuantizationConfig | Names which components to quantize, leaving others at full precision |
| CPU offloading | Trades speed for memory when quantization alone isn't enough |
| Distilled/turbo variants | A model-level trade of quality for speed, orthogonal to quantization |
| The model landscape moves fast | Treat specific repos as examples; the API and technique are what's durable |
Next
Now the discipline that keeps any of this reliable once it's live: Production & Operations.
Deployment & Inference
Three shapes for turning trained weights into something callable — Inference Providers, Spaces, and Inference Endpoints — and how to pick between them
Production & Operations
Revision pinning, safe weight formats, licensing, caching, cost control, and monitoring — the operational discipline around a deployed HuggingFace stack