Repository navigation
FEAT: Add native ASSERT prompt generation - #3271
Richard Lundeen (richlundeen) wants to merge 4 commits into
Conversation
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: ede3dafb-0eaa-472d-9638-2bf9e8661437
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: ede3dafb-0eaa-472d-9638-2bf9e8661437
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: ede3dafb-0eaa-472d-9638-2bf9e8661437
Check group prompt contracts before attack construction. Retry whitespace-only system prompts and use canonical behavioral inputs for seed storage. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: ede3dafb-0eaa-472d-9638-2bf9e8661437
| prompts = seed_group.adversarial_prompts | ||
| if prompts is None: | ||
| return | ||
| if not self._uses_adversarial or "attack_adversarial_config" not in self._compatibility_helper.accepted_params: |
There was a problem hiding this comment.
[P2] Reject named prompt slots that the attack does not use. Checking for attack_adversarial_config is not enough to establish support for every slot in AdversarialPrompts.
For example, a group containing only AdversarialPrompts(first_message=SeedPrompt(value="Ask about the public return policy.")) passes this check for both Crescendo and TAP. create(seed_group=...) then succeeds, but both attacks resolve their configuration with resolve_user_messages=False and discard that first message. A named feedback_template is also ignored. Their effective configurations contain first_message=None and the default feedback template instead. The same inputs are preserved by RedTeamingAttack.
Could the compatibility check reject unsupported slots, or explicitly declare which slots each attack consumes? Otherwise the run reports a distinct group identity for instructions it never actually uses. Please cover first-message-only and feedback-only groups for these attacks.
| raise IncompatibleTechniqueError(f"Factory '{self._name}' does not accept group adversarial prompts.") | ||
| if self._has_custom_adversarial_prompt: | ||
| raise IncompatibleTechniqueError(f"Factory '{self._name}' already has a custom adversarial prompt.") | ||
| if prompts.system_prompt is not None: |
There was a problem hiding this comment.
[P2] Check the response schema against the effective default system prompt. When a group supplies only first_message, this validation skips the system-prompt branch. However, RedTeamingAttack then loads its default system prompt, which already declares response_json_schema.
A valid AdversarialPrompts(first_message=SeedPrompt(..., response_json_schema=get_common_json_schema("adversarial_chat"))) therefore passes preflight, but create() raises Both the adversarial system prompt and first message declare a response_json_schema. I reproduced this through a matrix containing that group plus a compatible ordinary group, with the default skip policy and baseline enabled. The constructor error aborts the entire matrix instead of filtering the incompatible group. Removing the first-message schema is a passing control.
Could preflight account for the effective default prompt's schema too and route this known conflict through IncompatibleTechniqueError before construction?
Description
Assert (https://github.com/responsibleai/ASSERT) allows users to specify policies using english language and test those policies.
This PR adds native ASSERT-style prompt generation using PyRIT targets, normalizers, and seeds.
AssertGeneratorresearches an English behavior requirement, builds a policy taxonomy, resolves coverage dimensions, and returns validated nativeAttackSeedGroupobjects.SeedPromptinputs. Existing technique factories bind those inputs through constructor-levelAttackAdversarialConfig; existing attacks own JSON replies, feedback, and turn handling.AssertProviderfor memory storage. Group associations, literal text, response schemas, identity, and bounded generation references survive serialization and storage.seed_datasets/generated, preserving the publicTargetObjectiveProviderimport. Add lazy exports and retain MIT attribution to ASSERT v0.3.0, revision04b713fe73879208369750b3bd5ad53544f3eecc(2026-09-04).This PR covers generation only. ASSERT-specific scoring and foundry scenario integration are separate work. No ASSERT runtime dependency or dependency-manifest changes are included.