fix: decide separated action text by shape, and label identifiers by recognizer

CLAUDE.md

@@ -85,7 +85,7 @@ scrubbed = scrub_dict(action, scrubber)
 | EMAIL_ADDRESS | john@example.com | <EMAIL_ADDRESS> |
 | PHONE_NUMBER | 555-123-4567 | <PHONE_NUMBER> |
 | US_SSN | 923-45-6789 | <US_SSN> |
-| CREDIT_CARD | 4532-1234-5678-9012 | <CREDIT_CARD> |
+| CREDIT_CARD | 4111111111111111 | <CREDIT_CARD> |
 | DATE_TIME | 01/15/1985 | <DATE_TIME> |
 | LOCATION | Toronto, ON | <LOCATION> |
 

README.md

@@ -38,15 +38,16 @@ Scrubbing is one control inside a reviewed egress process. It is not a
 guarantee that an artifact is free of protected data, and the evidence behind
 it is synthetic, not clinical.
 
-`tests/test_phi_recall.py` is the regression gate: 24 synthetic identifiers
+`tests/test_phi_recall.py` is the regression gate: 26 synthetic identifiers
 across names, contact details, financial identifiers, dates of birth,
 addresses, network identifiers, medical record numbers, member IDs, and
-provider licenses. It requires 24 out of 24, and it also checks that ordinary
-operational UI text comes back untouched.
+provider licenses. It requires 26 out of 26, it pins the entity type each one
+must produce, and it also checks that ordinary operational UI text comes back
+untouched.
 
 Detection is contextual, so the same value scrubs differently depending on what
-surrounds it. Every output below was produced by running 1.0.3 on 2026-08-28,
-not written by hand. Some of it will surprise you:
+surrounds it. Every output below was produced by running this code on
+2026-08-28, not written by hand. Some of it will surprise you:
 
 ```python
 >>> from openadapt_privacy.providers.presidio import PresidioScrubbingProvider
@@ -59,14 +60,20 @@ not written by hand. Some of it will surprise you:
 '<ORGANIZATION>: <US_SSN>'
 
 >>> s.scrub_text("Card 4111111111111111 on file")
-'<DATE_TIME> on file'
+'Card <CREDIT_CARD> on file'
+
+>>> s.scrub_text("Card: 4532-1234-5678-9012")
+'Card: <DATE_TIME>'
 ```
 
-The last one is the important one. A card number gets redacted, but as
-`DATE_TIME`, not as `CREDIT_CARD`, and the label "Card" goes with it. The
-redaction holds; the entity type you get back is not the one you would predict.
-Do not build a policy that keys off the placeholder name without measuring it
-against your own data first.
+The last two are the interesting pair. Both card numbers get redacted, but only
+the first is labelled `CREDIT_CARD`, because only the first passes a Luhn
+check. An identifier that no recognizer can validate falls to whatever the
+spaCy model makes of it, which for a run of digits is usually `DATE_TIME`.
+
+So: the redaction holds either way, and a validated identifier now carries its
+own type. An unvalidated one does not. Measure the placeholder names against
+your own data before you route on them.
 
 For production egress: scrub a copy, verify every output file, and bind the
 human or policy approval to the verified artifact. A model that ran without
@@ -115,10 +122,23 @@ Only the keys in `PrivacyConfig.SCRUB_KEYS_HTML` are scrubbed, and non-string
 values pass through, which is why the coordinates survive. Pass `scrub_all=True`
 to scrub every string regardless of key.
 
-One caveat in 1.0.3: the `text` key is treated as character-separated action
-text, joined by `ACTION_TEXT_SEP` (`-`). Scrubbing a plain sentence under that
-key returns it hyphenated, one character at a time. Use `value` or `title` for
-ordinary prose until that's fixed.
+The keys in `PrivacyConfig.SCRUB_KEYS_SEPARATED` (`text` and `canonical_text`)
+can also hold recorded keystrokes: one typed character per `ACTION_TEXT_SEP`,
+as in `j-o-h-n-@-e-x-a-m-p-l-e-.-c-o-m`. Those are reassembled before analysis,
+scrubbed, and separated again, because the PII is invisible in the split form.
+Whether that happens is decided by the value, not the key, so prose under
+`text` is scrubbed as prose:
+
+```python
+scrub_dict({"text": "Email: john@example.com"}, scrubber)
+# {'text': '<PERSON>: <EMAIL_ADDRESS>'}
+
+scrub_dict({"text": "-".join("john@example.com")}, scrubber)
+# {'text': '<-E-M-A-I-L-_-A-D-D-R-E-S-S->'}
+```
+
+Pass `separated_keys=[]` to turn keystroke handling off for a call, or
+`separated_keys=["keys"]` to move it to your own field name.
 
 ## Scrubbing screenshots
 
@@ -183,9 +203,9 @@ PrivacyConfig(
 ```
 
 The full field list is `SCRUB_CHAR`, `SCRUB_LANGUAGE`, `SCRUB_FILL_COLOR`,
-`SCRUB_KEYS_HTML`, `ACTION_TEXT_NAME_PREFIX`, `ACTION_TEXT_NAME_SUFFIX`,
-`ACTION_TEXT_SEP`, `SCRUB_CONFIG_TRF`, `SCRUB_PRESIDIO_IGNORE_ENTITIES`, and
-`SPACY_MODEL_NAME`.
+`SCRUB_KEYS_HTML`, `SCRUB_KEYS_SEPARATED`, `ACTION_TEXT_NAME_PREFIX`,
+`ACTION_TEXT_NAME_SUFFIX`, `ACTION_TEXT_SEP`, `SCRUB_CONFIG_TRF`,
+`SCRUB_PRESIDIO_IGNORE_ENTITIES`, and `SPACY_MODEL_NAME`.
 
 The analyzer's supported entity set comes from Presidio: `CREDIT_CARD`,
 `CRYPTO`, `DATE_TIME`, `EMAIL_ADDRESS`, `IBAN_CODE`, `IP_ADDRESS`, `LOCATION`,

openadapt_privacy/base.py

@@ -191,6 +191,7 @@ def scrub_dict(
         list_keys: list[str] | None = None,
         scrub_all: bool = False,
         force_scrub_children: bool = False,
+        separated_keys: list[str] | None = None,
     ) -> dict[str, Any]:
         """Scrub PII/PHI from a nested dictionary.
 
@@ -204,18 +205,29 @@ def scrub_dict(
             scrub_all: If True, scrub all string values regardless of key.
             force_scrub_children: If True, use aggressive scrubbing for
                 child values after PII is detected in parent.
+            separated_keys: Keys that may hold character-separated action text.
+                Defaults to config.SCRUB_KEYS_SEPARATED. Pass an empty list to
+                scrub every value as prose.
 
         Returns:
             Scrubbed dictionary with PII/PHI removed.
         """
+        policy = effective_config()
         if list_keys is None:
-            list_keys = effective_config().SCRUB_KEYS_HTML
+            list_keys = policy.SCRUB_KEYS_HTML
+        if separated_keys is None:
+            separated_keys = policy.SCRUB_KEYS_SEPARATED
 
         scrubbed_dict: dict[str, Any] = {}
         for key, value in input_dict.items():
             if self._should_scrub_text(key, value, list_keys, scrub_all):
-                scrubbed_text = self._scrub_text_item(value, key, force_scrub_children)
-                if key in ("text", "canonical_text") and self._is_scrubbed(value, scrubbed_text):
+                scrubbed_text = self._scrub_text_item(
+                    value,
+                    key,
+                    force_scrub_children,
+                    separated_keys=separated_keys,
+                )
+                if key in separated_keys and self._is_scrubbed(value, scrubbed_text):
                     force_scrub_children = True
                 scrubbed_dict[key] = scrubbed_text
             elif isinstance(value, list):
@@ -228,6 +240,7 @@ def scrub_dict(
                             list_keys,
                             scrub_all=list_scrub_all,
                             force_scrub_children=force_scrub_children,
+                            separated_keys=separated_keys,
                         )
                         if self._should_scrub_list_item(
                             item,
@@ -246,6 +259,7 @@ def scrub_dict(
                     value,
                     list_keys,
                     scrub_all=scrub_all or (isinstance(key, str) and key == "state"),
+                    separated_keys=separated_keys,
                 )
             else:
                 scrubbed_dict[key] = value
@@ -256,17 +270,22 @@ def scrub_list_dicts(
         self,
         input_list: list[dict[str, Any]],
         list_keys: list[str] | None = None,
+        separated_keys: list[str] | None = None,
     ) -> list[dict[str, Any]]:
         """Scrub PII/PHI from a list of dictionaries.
 
         Args:
             input_list: List of dictionaries to be scrubbed.
             list_keys: List of keys whose values should be scrubbed.
+            separated_keys: Keys that may hold character-separated action text.
 
         Returns:
             List of scrubbed dictionaries.
         """
-        return [self.scrub_dict(input_dict, list_keys) for input_dict in input_list]
+        return [
+            self.scrub_dict(input_dict, list_keys, separated_keys=separated_keys)
+            for input_dict in input_list
+        ]
 
     def _should_scrub_text(
         self,
@@ -305,18 +324,24 @@ def _scrub_text_item(
         value: str,
         key: str,
         force_scrub_children: bool = False,
+        separated_keys: list[str] | None = None,
     ) -> str:
         """Scrub a single text value.
 
         Args:
             value: Text value to scrub.
             key: Dictionary key associated with the value.
             force_scrub_children: If True, use aggressive scrubbing.
+            separated_keys: Keys that may hold character-separated action text.
 
         Returns:
             Scrubbed text.
         """
-        if key in ("text", "canonical_text"):
+        if separated_keys is None:
+            separated_keys = effective_config().SCRUB_KEYS_SEPARATED
+        if key in separated_keys:
+            # A permission, not an instruction: the provider applies separated
+            # handling only to a value that is genuinely a key sequence.
             return self.scrub_text(value, is_separated=True)
         if force_scrub_children:
             return self.scrub_text_all(value)
@@ -351,6 +376,7 @@ def _scrub_list_item(
         list_keys: list[str],
         force_scrub_children: bool = False,
         scrub_all: bool = False,
+        separated_keys: list[str] | None = None,
     ) -> Any:
         """Scrub a single list item.
 
@@ -360,6 +386,7 @@ def _scrub_list_item(
             list_keys: List of keys that should be scrubbed.
             force_scrub_children: If True, use aggressive scrubbing.
             scrub_all: If True, scrub every string at every list depth.
+            separated_keys: Keys that may hold character-separated action text.
 
         Returns:
             Scrubbed item.
@@ -370,6 +397,7 @@ def _scrub_list_item(
                 list_keys,
                 scrub_all=scrub_all,
                 force_scrub_children=force_scrub_children,
+                separated_keys=separated_keys,
             )
         if isinstance(item, list):
             return [
@@ -380,6 +408,7 @@ def _scrub_list_item(
                         list_keys,
                         scrub_all=scrub_all,
                         force_scrub_children=force_scrub_children,
+                        separated_keys=separated_keys,
                     )
                     if self._should_scrub_list_item(
                         nested_item,
@@ -391,7 +420,7 @@ def _scrub_list_item(
                 )
                 for nested_item in item
             ]
-        return self._scrub_text_item(item, key)
+        return self._scrub_text_item(item, key, separated_keys=separated_keys)
 
 
 class ScrubbingProviderFactory:

openadapt_privacy/config.py

@@ -67,6 +67,10 @@ class PrivacyConfig:
         SCRUB_LANGUAGE: Language code for NLP analysis (default: "en").
         SCRUB_FILL_COLOR: BGR color value for image redaction (default: blue 0x0000FF).
         SCRUB_KEYS_HTML: List of dict keys that should be scrubbed.
+        SCRUB_KEYS_SEPARATED: Dict keys whose values may hold character-separated
+            action text rather than prose. A value under one of these keys is
+            reassembled before analysis only when it is actually in separated
+            form; prose under the same key is scrubbed as prose.
         ACTION_TEXT_NAME_PREFIX: Prefix for action text names (e.g., "<").
         ACTION_TEXT_NAME_SUFFIX: Suffix for action text names (e.g., ">").
         ACTION_TEXT_SEP: Separator for action text sequences (e.g., "-").
@@ -103,6 +107,16 @@ class PrivacyConfig:
         ]
     )
 
+    # Keys whose values may hold a key sequence joined by ACTION_TEXT_SEP.
+    # Membership here only permits separated handling; the value's own shape
+    # decides whether it is applied.
+    SCRUB_KEYS_SEPARATED: list[str] = field(
+        default_factory=lambda: [
+            "text",
+            "canonical_text",
+        ]
+    )
+
     # Action text formatting (for handling separated text like key sequences)
     ACTION_TEXT_NAME_PREFIX: str = "<"
     ACTION_TEXT_NAME_SUFFIX: str = ">"

openadapt_privacy/pipelines/dicts.py

@@ -50,6 +50,7 @@ def scrub_dict(
     scrubber: ScrubbingProvider,
     list_keys: list[str] | None = None,
     scrub_all: bool = False,
+    separated_keys: list[str] | None = None,
 ) -> dict[str, Any]:
     """Scrub PII/PHI from a nested dictionary.
 
@@ -61,6 +62,10 @@ def scrub_dict(
         list_keys: List of keys whose values should be scrubbed.
             Defaults to config.SCRUB_KEYS_HTML.
         scrub_all: If True, scrub all string values regardless of key.
+        separated_keys: Keys that may hold character-separated action text
+            (a key sequence joined by ACTION_TEXT_SEP). Defaults to
+            config.SCRUB_KEYS_SEPARATED. Pass an empty list to scrub every
+            value as prose.
 
     Returns:
         Scrubbed dictionary with PII/PHI removed.
@@ -75,13 +80,19 @@ def scrub_dict(
         >>> scrubbed = scrub_dict(event, scrubber)
     """
     helper = DictScrubber(scrubber)
-    return helper.scrub_dict(input_dict, list_keys=list_keys, scrub_all=scrub_all)
+    return helper.scrub_dict(
+        input_dict,
+        list_keys=list_keys,
+        scrub_all=scrub_all,
+        separated_keys=separated_keys,
+    )
 
 
 def scrub_list_dicts(
     input_list: list[dict[str, Any]],
     scrubber: ScrubbingProvider,
     list_keys: list[str] | None = None,
+    separated_keys: list[str] | None = None,
 ) -> list[dict[str, Any]]:
     """Scrub PII/PHI from a list of dictionaries.
 
@@ -91,6 +102,7 @@ def scrub_list_dicts(
         input_list: List of dictionaries to be scrubbed.
         scrubber: The ScrubbingProvider to use for text scrubbing.
         list_keys: List of keys whose values should be scrubbed.
+        separated_keys: Keys that may hold character-separated action text.
 
     Returns:
         List of scrubbed dictionaries.
@@ -105,4 +117,8 @@ def scrub_list_dicts(
         >>> scrubbed = scrub_list_dicts(events, scrubber)
     """
     helper = DictScrubber(scrubber)
-    return helper.scrub_list_dicts(input_list, list_keys=list_keys)
+    return helper.scrub_list_dicts(
+        input_list,
+        list_keys=list_keys,
+        separated_keys=separated_keys,
+    )