Repository navigation
FIX: match non-ASCII values in SQLite seed filters - #3060
Merged
Roman Lutz (romanlutz) merged 3 commits intoOct 13, 2026
Merged
Conversation
Reuse dialect-aware JSON array expansion for substring filters so Unicode escape text cannot select unrelated seeds or groups. Preserve existing LIKE behavior and exact Unicode matching, and cover retrieval, deletion, SQL dialects and reopened databases. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: b608c312-73dc-400b-8245-de2cbb19461a
Roman Lutz (romanlutz)
approved these changes
Oct 13, 2026
Resolve overlapping imports while retaining the decoded-element seed filtering fix and current main behavior. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: b608c312-73dc-400b-8245-de2cbb19461a
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
On
SQLiteMemory, seed filters miss stored values that contain non-ASCII characters. The shippedairt_harmsdataset listsFrédéric Dubutas an author, yet filtering by that name returns nothing:There are two causes:
get_seeds_asyncandget_seed_groups_async(harm_categories,authors,groups,parameters) runLIKEover the JSON column text. SQLAlchemy writes that text withjson.dumps, which escapes non-ASCII characters, so the column holds["Frédéric Dubut"]and%Frédéric%can never match._add_list_conditionsnow searches for the value as JSON spells it (json.dumps(value)[1:-1]). This also lets values that contain"or\match. It works for rows already in a database, with no migration.remove_seeds_from_memory_async,get_seed_examples_async, and the other callers of_get_condition_json_array_matchand_get_condition_json_property_match) compare SQLite'sLOWER(value), which folds ASCII only, with Python'smatch_value.lower().LOWER('ÉTHIQUE')givesÉthiquewhile Python giveséthique, so a value with a non-ASCII capital never matches.SQLiteMemorynow folds the stored side with the Unicode lowercase function it already registers on every connection for analytics (UnicodeLower). Azure SQL'sLOWERalready folds Unicode, so that backend needs no change.After the fix, the same calls return 4 seeds and remove 1 seed. The substring filters match a value as it is stored; their case folding is unchanged (SQLite
LIKEfolds ASCII letters only).Tests and Documentation
test_get_seeds_with_list_filters_matches_non_ascii_values: full and partial non-ASCII authors, non-ASCII harm categories, and authors containing a quote or a backslash, throughget_seeds_async.test_remove_seeds_by_list_filters_matches_non_ascii_valuesandtest_get_seed_examples_harm_filter_folds_non_ascii_case: exact matches with non-ASCII capitals, through the public delete and browse paths.test_json_match_conditions_fold_non_ascii_case_async: the SQLite property, partial, case-sensitive, and nested-array conditions, with negative controls.main; the 3 negative controls pass on both.tests/unit/memory, Ruff, andtypass.