Skip to content

Fix double scaling when exporting Gemma embeddings - #1

Open
ybochkov wants to merge 1 commit into
Goader:masterfrom
ybochkov:fix/double-scaling
Open

Fix double scaling when exporting Gemma embeddings#1
ybochkov wants to merge 1 commit into
Goader:masterfrom
ybochkov:fix/double-scaling

Conversation

@ybochkov

Copy link
Copy Markdown

PartlyFrozenEmbeddings.to_embeddings() previously rebuilt the embedding table by calling self.forward() for every token. Gemma's specialized wrapper applies embed_scale in forward(), so the exported checkpoint contained already-scaled vectors. Gemma then applied the same scale again at inference time after loading the checkpoint.

Reconstruct the standard embedding table directly from the frozen and trainable parameter partitions instead. Preserve the source padding index and dtype while rebuilding the layer.

Add regression tests that distinguish runtime Gemma scaling from persisted raw weights and verify that ordinary embedding reconstruction retains weights and metadata.

PartlyFrozenEmbeddings.to_embeddings() previously rebuilt the embedding table by calling self.forward() for every token. Gemma's specialized wrapper applies embed_scale in forward(), so the exported checkpoint contained already-scaled vectors. Gemma then applied the same scale again at inference time after loading the checkpoint.

Reconstruct the standard embedding table directly from the frozen and trainable parameter partitions instead. Preserve the source padding index and dtype while rebuilding the layer.

Add regression tests that distinguish runtime Gemma scaling from persisted raw weights and verify that ordinary embedding reconstruction retains weights and metadata.
@ybochkov
ybochkov force-pushed the fix/double-scaling branch from 7d947f3 to 5deee77 Compare August 12, 2026 14:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant