Skip to content

Gemma 3

The third Gemma generation, ported to pure Keras 3, and the first multimodal one. The 1B variants are text-only; 4B and larger pair the text decoder with a SigLIP vision tower and an average-pool projector, so the same class handles text or image+text.

What changed from Gemma 2:

  • Vision (4B+): a SigLIP tower plus Gemma3MultiModalProjector, which 4x4 average-pools patch features into a fixed 256 soft tokens per image.
  • 5:1 attention pattern: five local (sliding_window) layers to each global one, set by sliding_window_pattern.
  • Dual RoPE bases: global layers use rope_theta (1e6), local layers rope_local_theta (1e4), with optional rope_scaling_factor for long context.

Links:

See also gemma.md, gemma2.md, gemma4.md.

Variants

Load any of these with from_weights("<variant>"). -pt is the pretrained (base) checkpoint, -it is instruction-tuned (use the chat template).

Variant Hub Vision
gemma-3-1b-pt google/gemma-3-1b-pt text only
gemma-3-1b-it google/gemma-3-1b-it text only
gemma-3-4b-pt google/gemma-3-4b-pt yes
gemma-3-4b-it google/gemma-3-4b-it yes
gemma-3-12b-pt google/gemma-3-12b-pt yes
gemma-3-12b-it google/gemma-3-12b-it yes
gemma-3-27b-pt google/gemma-3-27b-pt yes
gemma-3-27b-it google/gemma-3-27b-it yes

API

Gemma3Model

The backbone: text decoder, plus the SigLIP tower and projector when vision_num_layers > 0. Returns {"last_hidden_state": (batch, seq, embed_dim)}.

Arg Default Meaning
vocab_size 262144 token vocabulary size
embed_dim 1152 model width
mlp_dim 6912 GeGLU inner width
num_layers 26 decoder blocks
num_heads 4 query heads
num_kv_heads 1 key/value heads
head_dim 256 per-head width
query_pre_attn_scalar 256.0 query scaling divisor before attention
sliding_window 512 local attention span
sliding_window_pattern 6 one global layer every N (5 local : 1 global)
norm_eps 1e-6 RMSNorm epsilon
rope_theta 1000000.0 rotary base on global layers
rope_local_theta 10000.0 rotary base on local layers
rope_scaling_factor None long-context rope scaling
tie_embeddings True reuse the embedding matrix as the LM head
vision_embed_dim 1152 SigLIP tower width
vision_mlp_dim 4304 SigLIP MLP width
vision_num_layers 0 SigLIP depth; 0 builds a text-only model

Gemma3Generate

Gemma3Model plus a (tied) LM head. Returns {"logits": (batch, seq, vocab_size)} and adds .generate() for text or image+text to text. Same constructor arguments as Gemma3Model.

generate(input_ids, attention_mask=None, max_new_tokens=None,
         eos_token_id=None, sampler=None, seed=None, **prefill_inputs)

Image inputs ride along as **prefill_inputs (pixel_values), which the processor produces for you.

Gemma3VisionModel

The SigLIP vision tower: biased conv patch embed and learned position embeddings into pre-LN encoder blocks. Built automatically by Gemma3Model; you rarely construct it directly.

Gemma3VisionModel(embed_dim, mlp_dim, num_layers, num_heads,
                  image_size=896, patch_size=14, norm_eps=1e-6)

Gemma3MultiModalProjector

Maps vision features to text-embedding space: 4x4 average pool, soft-token RMSNorm, then a matmul with the learned projection.

Gemma3MultiModalProjector(vision_dim, text_dim, patches_per_image=64,
                          tokens_per_side=16, norm_eps=1e-6)

Gemma3ImageProcessor

Bicubic resize to a fixed square, rescale to [0, 1], normalize.

Gemma3ImageProcessor(size=896, image_mean=(0.5, 0.5, 0.5), image_std=(0.5, 0.5, 0.5))

Gemma3Tokenizer

SentencePiece-BPE tokenizer on the tokenizers backend.

Gemma3Tokenizer(hf_id=None, tokenizer_file=None)

Gemma3Processor

Image plus text to model inputs. Renders the chat template, preprocesses images, and expands each image placeholder into mm_tokens_per_image soft tokens.

Gemma3Processor(hf_id=None, mm_tokens_per_image=256, tokenizer=None, image_processor=None)
Arg Default Meaning
hf_id None Hub repo for the tokenizer files
mm_tokens_per_image 256 soft tokens each image expands to
tokenizer None override the default Gemma3Tokenizer
image_processor None override the default Gemma3ImageProcessor

End-to-end example

Single input (text only)

import os
os.environ["KERAS_BACKEND"] = "torch"   # or "jax" / "tensorflow"

from kerasformers.models.gemma3 import Gemma3Generate, Gemma3Tokenizer

model = Gemma3Generate.from_weights("gemma-3-1b-it")
tokenizer = Gemma3Tokenizer.from_weights("gemma-3-1b-it")

inputs = tokenizer([{"role": "user", "content": "Explain rotary embeddings in one sentence."}])
outputs = model.generate(**inputs, max_new_tokens=64)

print(tokenizer.decode(outputs[0]))

Single input (image + text)

Use Gemma3Processor with a 4B or larger variant:

from PIL import Image
from kerasformers.models.gemma3 import Gemma3Generate, Gemma3Processor

model = Gemma3Generate.from_weights("gemma-3-4b-it")
processor = Gemma3Processor.from_weights("gemma-3-4b-it")

image = Image.open("photo.jpg")
inputs = processor(conversation=[{
    "role": "user",
    "content": [
        {"type": "image", "image": image},
        {"type": "text", "text": "Describe this image in one sentence."},
    ],
}])
outputs = model.generate(**inputs, max_new_tokens=64)

print(processor.decode(outputs[0]))

Batch

prompts = [
    "The capital of France is",
    "In one sentence, what is a transformer?",
    "Write a haiku about GPUs.",
]
inputs = tokenizer(prompts)              # {"input_ids": (3, seq), "attention_mask": (3, seq)}
outputs = model.generate(**inputs, max_new_tokens=64)

for text in tokenizer.batch_decode(outputs):
    print(text)

Backbone only

from kerasformers.models.gemma3 import Gemma3Model

backbone = Gemma3Model.from_weights("gemma-3-1b-pt")
hidden = backbone(inputs)["last_hidden_state"]   # (batch, seq, embed_dim)

Loading from the Hub

model = Gemma3Generate.from_weights("hf:google/gemma-3-4b-it")

Lower memory

The 12B and 27B checkpoints benefit from bf16 or weight-only quantization. See quantization.md:

model = Gemma3Generate.from_weights(
    "gemma-3-27b-it", quantization="int8", low_memory=True, load_dtype="bfloat16"
)