Diffusion Models

Weights: every checkpoint is hosted under the zeromodels organization as zm_config.json + Keras weights + tokenizer.json. Load one with ZMDiffusion.from_weights("zeromodels/<variant>"); no conversion is needed. On-the-fly hf: conversion is not supported for diffusion models.

Text-to-image generation (and image-to-image with the SDXL refiner) in pure Keras 3, on JAX, PyTorch or TensorFlow. Every family loads through two classes from zeromodels.diffusers:

• ZMDiffusion: the model and its generate.
• ZMDiffusionTokenizer: the prompt tokenizer.

The checkpoint's model_type (in its zm_config.json) selects the family, so switching from Stable Diffusion to Qwen-Image only changes the repo id.

Supported models

Family	model_type	Hosted checkpoints (zeromodels/...)
Stable Diffusion 1.x	stable_diffusion	stable-diffusion-v1-1 to stable-diffusion-v1-5
Stable Diffusion 2.x	stable_diffusion_2	stable-diffusion-2, stable-diffusion-2-base, stable-diffusion-2-1, stable-diffusion-2-1-base, sd-turbo
Stable Diffusion XL	stable_diffusion_xl	stable-diffusion-xl-base-1.0, sdxl-turbo
Stable Diffusion XL refiner	stable_diffusion_xl_refiner	stable-diffusion-xl-refiner-1.0 (image-to-image)
Stable Diffusion 3	stable_diffusion_3	stable-diffusion-3-medium
Stable Diffusion 3.5	stable_diffusion_3_5	stable-diffusion-3.5-large, stable-diffusion-3.5-large-turbo, stable-diffusion-3.5-medium
Qwen-Image	qwen_image	qwen-image
Qwen-Image-2.1	qwen_image_21	qwen-image-2.1

Stable Diffusion 3 and 3.5 can also attach the separately hosted T5-XXL text encoder, zeromodels/t5-v1_1-xxl-encoder, with text_encoder_3=.

Example: Stable Diffusion 3

import os

os.environ["KERAS_BACKEND"] = "torch"  # or "jax" / "tensorflow"

from PIL import Image
from zeromodels.diffusers import ZMDiffusion, ZMDiffusionTokenizer

model = ZMDiffusion.from_weights(
    "zeromodels/stable-diffusion-3-medium",
    text_encoder_3="zeromodels/t5-v1_1-xxl-encoder",  # optional: omit to save memory
)
tokenizer = ZMDiffusionTokenizer.from_weights("zeromodels/stable-diffusion-3-medium")

inputs = tokenizer(
    "a fluffy corgi wearing round sunglasses, sitting on a surfboard at a sunny beach, photo"
)
images = model.generate(**inputs, num_inference_steps=28, guidance_scale=7.0, seed=4)

print(images.shape, images.dtype)  # (1, 1024, 1024, 3) uint8
Image.fromarray(images[0]).save("corgi.png")

Example: Qwen-Image

import os

os.environ["KERAS_BACKEND"] = "torch"  # or "jax" / "tensorflow"

from PIL import Image
from zeromodels.diffusers import ZMDiffusion, ZMDiffusionTokenizer

model = ZMDiffusion.from_weights("zeromodels/qwen-image")
tokenizer = ZMDiffusionTokenizer.from_weights("zeromodels/qwen-image")

inputs = tokenizer(
    "a coffee shop entrance with a chalkboard sign reading 'Open', warm afternoon light"
)
images = model.generate(
    **inputs,
    height=1024,
    width=1024,
    num_inference_steps=50,
    guidance_scale=4.0,
    seed=1,
)

print(images.shape, images.dtype)  # (1, 1024, 1024, 3) uint8
Image.fromarray(images[0]).save("coffee.png")

Outputs

Generated with the hosted checkpoints, same code as above with the repo, prompt and settings shown.

stable-diffusion-v1-5, 50 steps, guidance 7.5, seed 1 a red fox sitting in a field of lavender, golden hour, wildlife photography, sharp focus	stable-diffusion-2-1-base, 50 steps, guidance 7.5, seed 2 a steaming bowl of ramen on a wooden table, food photography, shallow depth of field
sd-turbo, 1 step, no guidance a cinematic shot of a baby raccoon wearing an intricate italian priest robe	sdxl-turbo, 1 step, no guidance a cinematic shot of a baby raccoon wearing an intricate italian priest robe
stable-diffusion-xl-base-1.0, 30 steps, guidance 5.0, seed 3 a lighthouse on a rocky coast at dawn, dramatic clouds, cinematic, highly detailed	base + stable-diffusion-xl-refiner-1.0, base to 80% then refiner (denoising_start=0.8) the same lighthouse prompt
stable-diffusion-3-medium + T5, 28 steps, guidance 7.0, seed 4 a fluffy corgi wearing round sunglasses, sitting on a surfboard at a sunny beach, photo	stable-diffusion-3.5-medium + T5, 40 steps, guidance 4.5, seed 5 a whimsical treehouse village at twilight, lanterns glowing, watercolor illustration

Verified against diffusers

The same prompt and initial latent through ZeroModels (left) and 🤗 diffusers (right). Stable Diffusion 1.5 matches to within one pixel level (99.3% of pixels identical, PSNR 74.7 dB).

generate

Pass the tokenizer's output, plus any of:

Argument	Meaning
negative_input_ids	a tokenized negative prompt (tokenizer("...")["input_ids"])
num_inference_steps, guidance_scale	sampler steps and guidance strength; default to the repo's own settings
seed	reproducible noise on one backend
latents	explicit initial noise, for identical images on every backend
image, strength	image-to-image: start from an image instead of noise
denoising_end, denoising_start, output_type="latent"	split a run between the SDXL base and refiner
height, width	output size (Qwen-Image families)

It returns (batch, height, width, 3) uint8 NumPy images. Prompts batch through the tokenizer: tokenizer(["a cat", "a dog"]).

Loading options

from_weights also takes load_dtype="float16" / "bfloat16" / "float32", quantization="int8" / "int4" for lower memory, and size overrides such as unet_sample_size=96, vae_sample_size=768 to build for another resolution. The sampler can be swapped at any time with model.scheduler = ... (zeromodels.base.base_scheduler), and the trained components are attributes (.unet or .transformer, .vae, .text_encoder), each callable on its own.

The same repos also load through the Auto API: AutoZModel.from_weights(repo) returns a ZMDiffusion and AutoZMTokenizer.from_weights(repo) a ZMDiffusionTokenizer.