Repository navigation
What is the purpose of SmearGate cross-document 'leak fix'? #1988
Description
Activity
I see CocoHearts mentioned its a ruling here: #1797 (comment).
This is surprising to me. For anyone using these techniques in their own repos on small models, I would not recommend paying the extra compute cost to mask. Looks like in parameter golf this is a custom rule.
yeah it's interesting
agreed - it looks to me honestly like something mis-identified through some sort of auto-audit flagging data leakage, which has been a problem with a ton of PRs here, but obviously seeing a prior token doesn't constitute any form of leakage.
Actually I felt the exact same about the fix
Why are PRs, such as the current record #1855, doing this?
Looking at prior documents does not break causality. Any LLM that doesn't use intra-document masking is already looking at prior documents through attention. There is no 'cheating' involved here, unless the maintainers have created an arbitrary ruling on this. When all of these 1 position techniques like smear gate and bigram hash were created, both the masked and unmasked versions were tested, and the unmasked version was intentionally selected because it ran faster, didn't hurt loss, and obeyed the causal mask.
I am concerned that every record is going to copy paste this, and the final record is going to have this janky inefficiency for no good reason.