T5¶

Weights: pretrained Keras weights live on Hugging Face under kerasformers/<variant> (each repo carries kf_config.json + model.weights.h5). Load with from_weights("kerasformers/<variant>").

Google's T5 (Text-to-Text Transfer Transformer) in pure Keras 3: an encoder-decoder transformer where every task is framed as text-to-text. The blocks use T5-style RMSNorm (scale only, pre-LayerNorm), a learned relative-position bias instead of absolute or rotary embeddings, no 1/sqrt(d) attention scaling, and an LM head tied to the shared embedding. One implementation runs unmodified on TensorFlow / Torch / JAX, bit-exact with Hugging Face on real checkpoints.

This port targets original T5 (a ReLU dense-relu-dense MLP, a shared token embedding, and a tied LM head scaled by embed_dim ** -0.5); v1.1 / Flan-T5, which use gated-GELU and an untied head, are not covered.

Variants¶

Load any of these with from_weights("kerasformers/<variant>").

Variant	Hub	layers / dim
`t5_small`	`kerasformers/t5_small`	6 / 512
`t5_base`	`kerasformers/t5_base`	12 / 768
`t5_large`	`kerasformers/t5_large`	24 / 1024
`t5_3b`	`kerasformers/t5_3b`	24 / 1024
`t5_11b`	`kerasformers/t5_11b`	24 / 1024

API¶

All classes are imperative SubclassedBaseModels; the heads extend the backbone. The one hosted model.weights.h5 (declaring T5Model) is the full encoder-decoder; every class loads its own subset (T5ConditionalGenerate ties the LM head, so it adds no weight; the classification / QA heads are randomly initialized for fine-tuning).

`T5Model`¶

The encoder-decoder backbone. Takes input_ids / attention_mask / decoder_input_ids and returns {"last_hidden_state": (B, T, d), "encoder_last_hidden_state": (B, S, d)}.

Arg	Default	Meaning
`vocab_size`	`32128`	token vocabulary size
`embed_dim`	`768`	model width
`key_value_dim`	`64`	per-head width (`num_heads * key_value_dim` may != `embed_dim`)
`mlp_dim`	`3072`	feed-forward inner width
`num_layers`	`12`	encoder blocks
`num_decoder_layers`	`12`	decoder blocks
`num_heads`	`12`	attention heads
`relative_attention_num_buckets`	`32`	relative-position buckets
`relative_attention_max_distance`	`128`	max bucketed distance
`hidden_act`	`"relu"`	feed-forward activation
`layer_norm_eps`	`1e-6`	RMSNorm epsilon
`tie_word_embeddings`	`True`	reuse the shared embedding as the LM head
`pad_token_id` / `eos_token_id` / `decoder_start_token_id`	`0` / `1` / `0`	special tokens

`T5ConditionalGenerate`¶

T5Model plus the tied LM head. Returns {"logits": (B, T, vocab_size), ...} and adds .generate() (runs the encoder once, then greedily decodes, cross-attending to the frozen encoder output at each step). Takes the same constructor arguments as T5Model.

generate(
    input_ids,
    attention_mask=None,
    max_new_tokens=None,
    eos_token_id=None,
    sampler=None,
    seed=None,
)

Arg	Default	Meaning
`input_ids`	required	`(B, S)` encoder token ids
`attention_mask`	`None`	`(B, S)` 1 = keep, 0 = padding
`max_new_tokens`	`None`	tokens to generate
`eos_token_id`	`None`	stop token (defaults to the tokenizer's)
`sampler`	`None`	sampling strategy; greedy when unset

Other classes¶

Class	HF equivalent	Output
`T5EncoderModel`	`T5EncoderModel`	`{"last_hidden_state": (B, S, d)}`
`T5SequenceClassify`	`T5ForSequenceClassification`	`(B, num_classes)`
`T5TokenClassify`	`T5ForTokenClassification`	`(B, S, num_classes)`
`T5QnA`	`T5ForQuestionAnswering`	`{"start_logits": (B, T), "end_logits": (B, T)}`

`T5Tokenizer`¶

SentencePiece (Unigram) tokenizer on the tokenizers (Rust) backend: the ▁ metaspace, $A </s> post-processing (T5 appends EOS), and the 100 <extra_id_N> sentinels are baked in. Returns input_ids / attention_mask (T5 has no token-type ids).

T5Tokenizer(variant="t5_base", tokenizer_file=None, max_seq_len=512)

End-to-end example¶

Text-to-text generation¶

import os

os.environ["KERAS_BACKEND"] = "torch"  # or "jax" / "tensorflow"

from kerasformers.models.t5 import T5ConditionalGenerate, T5Tokenizer

model = T5ConditionalGenerate.from_weights("kerasformers/t5_base")
tokenizer = T5Tokenizer.from_weights("kerasformers/t5_base")

inputs = tokenizer("translate English to German: The house is wonderful.")
output_ids = model.generate(
    inputs["input_ids"], inputs["attention_mask"], max_new_tokens=40
)
print(tokenizer.decode(output_ids[0]))  # "Das Haus ist wunderbar."

Encoder features¶

from kerasformers.models.t5 import T5EncoderModel, T5Tokenizer

encoder = T5EncoderModel.from_weights("kerasformers/t5_base")
tokenizer = T5Tokenizer.from_weights("kerasformers/t5_base")
feats = encoder(tokenizer("The quick brown fox."))["last_hidden_state"]  # (1, S, 768)

Scoring a target sequence¶

from kerasformers.models.t5 import T5Model

model = T5Model.from_weights("kerasformers/t5_base")
out = model(
    {
        "input_ids": inputs["input_ids"],
        "attention_mask": inputs["attention_mask"],
        "decoder_input_ids": decoder_ids,  # right-shifted targets
    }
)
out["last_hidden_state"]  # decoder states (B, T, d)

Loading from the Hub¶

model = T5ConditionalGenerate.from_weights("hf:google-t5/t5-base")

Architecture notes¶

Relative position bias, not absolute or rotary: a learned Embedding(num_buckets, num_heads) per stack, bucketed by memory - query (bidirectional in the encoder, causal in the decoder) and shared across all layers of that stack. Cross-attention uses no relative bias.
T5LayerNorm is RMSNorm (scale only, no mean subtraction, no bias). Residuals are pre-LayerNorm, and each stack ends with a final RMSNorm.
No 1/sqrt(d) attention scaling and no biases in q/k/v/o or the MLP. The decoder starts from the pad token (decoder_start_token_id = 0); the LM head is the transposed shared embedding scaled by embed_dim ** -0.5.

Parity¶

Bit-exact with Hugging Face transformers (eager, float32): T5ConditionalGenerate logits and the T5EncoderModel last hidden state match to 0.0 max-abs difference; the classification / QA heads match to < 2e-7. See convert_t5_hf_to_keras.py.

T5¶