ELECTRA

Weights: pretrained Keras weights live on Hugging Face under kerasformers/<variant> (each repo carries kf_config.json + model.weights.h5). Load with from_weights("kerasformers/<variant>").

Google's ELECTRA in pure Keras 3: a BERT-style bidirectional text encoder pre-trained as a replaced-token discriminator (with a smaller generator producing the corrupted tokens), with a masked-LM head plus sequence / token classification, question-answering, and multiple-choice heads. Unlike BERT it embeds tokens at a separate embedding_size and projects up to the hidden size when they differ, and it has no pooler. One implementation runs unmodified on TensorFlow / Torch / JAX, bit-exact with Hugging Face on real checkpoints.

• Paper: ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators (arXiv:2003.10555)
• HF docs: transformers/model_doc/electra

See also bert.md, modernbert.md, roberta.md, deberta.md.

Variants

ELECTRA ships two checkpoints per size, hosted as one repo each. The discriminator repo (kf_config declares ElectraModel) serves the encoder + the classify / QA / token / multiple-choice heads; the generator repo (kf_config declares ElectraMaskedLM) serves the masked-LM. Load with from_weights("kerasformers/<variant>").

Size	Discriminator (encoder / downstream)	Generator (masked-LM)
small	kerasformers/electra_small_discriminator	kerasformers/electra_small_generator
base	kerasformers/electra_base_discriminator	kerasformers/electra_base_generator
large	kerasformers/electra_large_discriminator	kerasformers/electra_large_generator

API

ElectraModel

The encoder backbone (no pooler). Takes a dict of input_ids / attention_mask / token_type_ids (all (B, L) int) and returns {"last_hidden_state": (B, L, embed_dim)}. Defaults below are for the small variant; base / large keep embedding_size == embed_dim (no projection).

Arg	Default	Meaning
vocab_size	30522	token vocabulary size
embedding_size	128	token-embedding width (projected up when != embed_dim)
embed_dim	256	hidden width
num_layers	12	transformer blocks
num_heads	4	attention heads
mlp_dim	1024	feed-forward inner width
max_position_embeddings	512	position-table size
type_vocab_size	2	token-type embeddings
hidden_act	"gelu"	feed-forward activation
layer_norm_eps	1e-12	LayerNorm epsilon
pad_token_id	0	padding token id

Task heads

Each composes an ElectraModel backbone; all take the same backbone constructor args, plus the extras below. The discriminator heads take the pretrained encoder and a randomly-initialized task layer (ready for fine-tuning, or a hf: fine-tune); ElectraMaskedLM loads from the generator repo with real head weights.

Class	Repo	Extra args	Output
ElectraMaskedLM	generator		MLM logits (B, L, vocab_size)
ElectraSequenceClassify	discriminator	num_classes	(B, num_classes)
ElectraTokenClassify	discriminator	num_classes	(B, L, num_classes)
ElectraQnA	discriminator		{"start_logits": (B, L), "end_logits": (B, L)}
ElectraMultipleChoice	discriminator	num_choices	(B, num_choices)

ElectraTokenizer

WordPiece tokenizer on the tokenizers (Rust) backend (the discriminator and generator of a size share one vocabulary), with [CLS] A [SEP] B [SEP] post-processing and segment ids.

ElectraTokenizer(
    variant="electra_base_discriminator", tokenizer_file=None, max_seq_len=512
)

End-to-end example

Backbone features (discriminator)

import os

os.environ["KERAS_BACKEND"] = "torch"  # or "jax" / "tensorflow"

from kerasformers.models.electra import ElectraModel, ElectraTokenizer

model = ElectraModel.from_weights("kerasformers/electra_base_discriminator")
tokenizer = ElectraTokenizer.from_weights("kerasformers/electra_base_discriminator")
out = model(tokenizer("Hello, world."))["last_hidden_state"]  # (1, L, 768)

Fill-mask (generator)

from kerasformers.models.electra import ElectraMaskedLM, ElectraTokenizer

mlm = ElectraMaskedLM.from_weights("kerasformers/electra_base_generator")
tokenizer = ElectraTokenizer.from_weights("kerasformers/electra_base_generator")
logits = mlm(tokenizer("The capital of France is [MASK]."))  # (1, L, vocab_size)

Classification (fine-tune the discriminator)

from kerasformers.models.electra import ElectraSequenceClassify, ElectraQnA

clf = ElectraSequenceClassify.from_weights(
    "kerasformers/electra_base_discriminator", num_classes=2
)  # encoder pretrained, classifier random -> fine-tune
qa = ElectraQnA.from_weights("hf:org/electra-base-squad2")  # or a community fine-tune

num_classes is read from a hf: fine-tune's config. ElectraMultipleChoice takes a static num_choices at build; its classifier head is shape-independent of it.

Loading from the Hub

model = ElectraModel.from_weights("hf:google/electra-base-discriminator")
mlm = ElectraMaskedLM.from_weights("hf:google/electra-base-generator")

Architecture notes

• Separate embedding_size: embeds at embedding_size and projects to embed_dim with an embed_project linear only when they differ (small: 128 -> 256; base / large keep them equal).
• BERT-style encoder: post-LayerNorm blocks (multi-head self-attention + GELU feed-forward), no pooler.
• Heads: classification reads the first ([CLS]) token through a dense + GELU + linear head; the masked-LM head is a dense -> GELU -> LayerNorm at embedding_size, then a decoder tied to the word embeddings.

Parity

Bit-exact with Hugging Face transformers (eager, float32): every class matches the reference forward to < 1e-6 max-abs difference (ElectraModel, ElectraMaskedLM, and each task head), including the embed_project projection on the small variant. See convert_electra_hf_to_keras.py.