Skip to content

FIX: match non-ASCII values in SQLite seed filters - #3060

Merged
Roman Lutz (romanlutz) merged 3 commits into
microsoft:mainfrom
bIackr0se:fix/non-ascii-seed-filters
Oct 13, 2026
Merged

Roman Lutz (romanlutz) merged 3 commits into
microsoft:mainfrom
bIackr0se:fix/non-ascii-seed-filters

Conversation

@bIackr0se

Copy link
Copy Markdown
Contributor

Description

On SQLiteMemory, seed filters miss stored values that contain non-ASCII characters. The shipped airt_harms dataset lists Frédéric Dubut as an author, yet filtering by that name returns nothing:

get_seeds_async(dataset_name="airt_harms", authors=["Dubut"])           -> 4 seeds
get_seeds_async(dataset_name="airt_harms", authors=["Frédéric Dubut"])  -> 0 seeds (expected 4)
remove_seeds_from_memory_async(harm_categories=["ÉTHIQUE"])             -> removed 0 (expected 1)

There are two causes:

  • The default list filters of get_seeds_async and get_seed_groups_async (harm_categories, authors, groups, parameters) run LIKE over the JSON column text. SQLAlchemy writes that text with json.dumps, which escapes non-ASCII characters, so the column holds ["Frédéric Dubut"] and %Frédéric% can never match. _add_list_conditions now searches for the value as JSON spells it (json.dumps(value)[1:-1]). This also lets values that contain " or \ match. It works for rows already in a database, with no migration.
  • The exact filters (remove_seeds_from_memory_async, get_seed_examples_async, and the other callers of _get_condition_json_array_match and _get_condition_json_property_match) compare SQLite's LOWER(value), which folds ASCII only, with Python's match_value.lower(). LOWER('ÉTHIQUE') gives Éthique while Python gives éthique, so a value with a non-ASCII capital never matches. SQLiteMemory now folds the stored side with the Unicode lowercase function it already registers on every connection for analytics (UnicodeLower). Azure SQL's LOWER already folds Unicode, so that backend needs no change.

After the fix, the same calls return 4 seeds and remove 1 seed. The substring filters match a value as it is stored; their case folding is unchanged (SQLite LIKE folds ASCII letters only).

Tests and Documentation

  • test_get_seeds_with_list_filters_matches_non_ascii_values: full and partial non-ASCII authors, non-ASCII harm categories, and authors containing a quote or a backslash, through get_seeds_async.
  • test_remove_seeds_by_list_filters_matches_non_ascii_values and test_get_seed_examples_harm_filter_folds_non_ascii_case: exact matches with non-ASCII capitals, through the public delete and browse paths.
  • test_json_match_conditions_fold_non_ascii_case_async: the SQLite property, partial, case-sensitive, and nested-array conditions, with negative controls.
  • 12 of the 15 new cases fail on main; the 3 negative controls pass on both.
  • Unit tests under tests/unit/memory, Ruff, and ty pass.

Reuse dialect-aware JSON array expansion for substring filters so Unicode escape text cannot select unrelated seeds or groups. Preserve existing LIKE behavior and exact Unicode matching, and cover retrieval, deletion, SQL dialects and reopened databases.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: b608c312-73dc-400b-8245-de2cbb19461a
Resolve overlapping imports while retaining the decoded-element seed filtering fix and current main behavior.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: b608c312-73dc-400b-8245-de2cbb19461a
@romanlutz
Roman Lutz (romanlutz) merged commit d2a1fb2 into microsoft:main Oct 13, 2026
55 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants