Repository navigation
FIX: attribute fuzzer jailbreak scores through the scored piece id - #3300
Merged
Roman Lutz (romanlutz) merged 9 commits intoOct 20, 2026
Merged
Roman Lutz (romanlutz) merged 9 commits into
Roman Lutz (romanlutz) merged 9 commits into
Conversation
Roman Lutz (romanlutz)
approved these changes
Oct 19, 2026
Roman Lutz (romanlutz)
enabled auto-merge
October 19, 2026 18:36
auto-merge was automatically disabled
October 19, 2026 18:52
Head branch was pushed to by a user without write access
Contributor
Author
|
The pre-commit failure was an unused Message import my test no longer needed, ruff removed it as a fixup and the hook treats modified files as a failure. Dropped in c30822b, and 41c94e3 merges current main so the branch is not behind. Ruff check and format are clean on the touched files and the fuzzer suite passes (24 tests). |
Exercise deterministic batched scoring through execute_async and isolated SQLite across one-to-one, multiple, skipped, absent, and successful score batches. Verify stable conversation attribution despite reversed completion, persisted metadata, and response-based query limits. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 4a77f1bc-5948-4351-ad5f-c7f02e414a3b
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 4a77f1bc-5948-4351-ad5f-c7f02e414a3b
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Fixes #3299
_process_scoring_resultspaired the flattenedscore_batch_asyncresult with the responses by position, but that list can hold zero or several scores per response, so position is not the response index. A jailbreak verdict is now attributed through the scored piece id, so it always lands on the conversation the score was actually stored on. The target query counter now counts responses instead of scores, since a response is one target query regardless of how many scores its scorer produced.Tests and Documentation
Added a regression test that feeds two responses with a three score batch (two scores on the first response, the jailbreak on the second). It fails on main with an IndexError and passes with the change, and also covers the skipped response case where only the second response produced a score. The full fuzzer suite passes (24 tests).
No documentation change needed, so I did not run JupyText.