Skip to content

Commit 6363cda

Browse files
authored
Merge branch 'main' into dependabot/github_actions/minor-and-patch-c673fd7efc
2 parents 5ea0f0d + 1f575a5 commit 6363cda

123 files changed

Lines changed: 5781 additions & 1139 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.env_example‎

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -14,9 +14,12 @@
1414
# other OpenAI-compatible endpoint. See doc/setup/populating_secrets.md
1515
# for provider-specific examples.
1616
#
17-
# If you are using Entra authentication for Azure resources,
18-
# keys for those resources are not needed. PyRIT auto-detects: if an API key
19-
# is set, it uses key auth; otherwise it falls back to Entra ID automatically.
17+
# For Azure resources you can authenticate with Microsoft Entra ID instead of a
18+
# key. Make it an explicit choice: run `az login` and pass auth_mode="identity"
19+
# to the target, e.g. OpenAIChatTarget(endpoint=..., auth_mode="identity").
20+
# Leaving the key unset still selects Entra ID automatically, but that fallback
21+
# is deprecated and is removed in 1.4.0.
22+
# See doc/code/setup/1_configuration.ipynb for details.
2023
#
2124
# ============================================================================
2225

‎.github/instructions/scenarios.instructions.md‎

Lines changed: 14 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -92,7 +92,12 @@ scenario.set_params_from_args(args={"objective_target": target, "max_concurrency
9292
await scenario.initialize_async()
9393
```
9494

95-
The base `Scenario` declares the common run inputs once in `_common_scenario_parameters()`: `objective_target` (a `RegistryReference` — resolved by name or supplied as an instance), the `opaque` live objects `scenario_techniques` / `technique_converters` / `dataset_config` / `memory_labels` (passed by identity, never coerced or deep-copied), and the scalars `max_concurrency` / `max_retries` / `include_baseline`.
95+
The base `Scenario` declares the common run inputs once in `_common_scenario_parameters()`: `objective_target` (a `RegistryReference` — resolved by name or supplied as an instance), the `opaque` live objects `scenario_techniques` / `technique_converters` / `dataset_config` / `memory_labels` (passed by identity, never coerced or deep-copied), and the scalars `max_concurrency` / `max_retries` / `include_baseline` / `incompatible_technique_policy`.
96+
97+
`incompatible_technique_policy` accepts `skip` (default) or `raise`; `None` uses
98+
`INCOMPATIBLE_TECHNIQUE_POLICY`. Forward `context.incompatible_technique_policy` to matrix
99+
builders and apply shared requirement helpers in manual builders. Skip only known incompatibility,
100+
with one summary per technique/dataset. Raise if no selected technique can run, even with a baseline.
96101

97102
### Declaring custom parameters — add via `additional_parameters`
98103

@@ -273,6 +278,11 @@ config, memory labels, baseline flag, and seed groups), snapshots them into an i
273278
`_build_atomic_attacks_async` (see the Baseline section below). Scenario authors never read
274279
half-initialized `self._*` state to build attacks — read everything from `context`.
275280

281+
On resume, `context.stored_run_plan` carries the original plan. Restore committed
282+
per-input technique choices from it instead of selecting again. The base restores
283+
the planned source population before builders check compatibility. Saved adaptive
284+
choices must identify the full technique, including its prompts, not only its inner attack.
285+
276286
### Zero-boilerplate matrix scenarios
277287

278288
Scenarios whose construction is the plain technique × dataset cross-product delegate to the
@@ -339,6 +349,9 @@ Key points:
339349
`True` to force opt-in.
340350
- `kwargs` are validated against the attack class constructor signature at
341351
factory-construction time, so typos fail loudly and early.
352+
- Check factory and attack-class requirements through `TargetRequirements.check` to retain custom validation.
353+
- Use shared seed preparation before adding technique seeds. Keep source groups unchanged and record adaptation.
354+
- Reject saved plans whose technique identity or accepted inputs cannot be restored.
342355

343356
The canonical catalog factories live in `pyrit/setup/initializers/techniques/`; see
344357
[setup-techniques.instructions.md](setup-techniques.instructions.md) for how to add one

‎.github/instructions/setup-techniques.instructions.md‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -42,3 +42,13 @@ stop — put the value back inline.
4242
- Keep entries free of long explanatory comments and paper citations. The factory should read as
4343
plain configuration; if a technique's mechanics are subtle, document them on the reusable
4444
converter/model it uses, or here in this file, not as an inline comment block.
45+
46+
## Requirements
47+
48+
Declare `TechniqueRequirements` inline; attack-class target requirements also apply.
49+
Use `TargetRequirements.required` only when configured adaptation preserves the technique;
50+
use `native_required` otherwise. Declare modality combinations explicitly.
51+
52+
`objective_only=True` rejects extra dataset seeds and is the simulated-conversation default.
53+
Set `try_adapt=True` only when removal of extra dataset seeds preserves the technique's behavior.
54+
Scenarios, not factories, choose skip or raise.

‎.pyrit_conf_example‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -140,6 +140,16 @@ allow_custom_initializers: false
140140
# ~/.pyrit/custom_initializers.
141141
# custom_initializers_source: https://account.blob.core.windows.net/container/custom_initializers
142142

143+
# Upload Folder for API-Created Targets
144+
# -------------------------------------
145+
# Folder that targets created through the REST API may upload local files from
146+
# (HTTPXAPITarget). The server passes it to the target; API callers cannot choose it.
147+
# When unset, those targets cannot be created through the API; register them in
148+
# Python or with an initializer instead.
149+
#
150+
# Default: unset
151+
# target_upload_directory: /path/to/upload/files
152+
143153
# Local Backend Server
144154
# --------------------
145155
# Client settings used by pyrit_scan when connecting to or launching a backend.

‎build_scripts/stamp_compatibility.py‎

Lines changed: 7 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -6,9 +6,9 @@
66
import argparse
77
import json
88
import os
9-
import re
109
import runpy
1110
import subprocess
11+
import tomllib
1212
import warnings
1313
from collections.abc import Callable
1414
from pathlib import Path
@@ -31,8 +31,12 @@ def _version(root: Path) -> str:
3131
raise ValueError("Python package version must be a string")
3232
project_file = root / "pyproject.toml"
3333
if project_file.is_file():
34-
project_version = re.search(r'^version\s*=\s*"([^"]+)"', project_file.read_text(encoding="utf-8"), re.MULTILINE)
35-
if project_version is None or project_version[1] != version:
34+
project_data = tomllib.loads(project_file.read_text(encoding="utf-8"))
35+
project = project_data.get("project", {})
36+
if not isinstance(project, dict):
37+
raise ValueError("Python package and project versions differ")
38+
project_version = project.get("version")
39+
if not isinstance(project_version, str) or project_version != version:
3640
raise ValueError("Python package and project versions differ")
3741
return version
3842

‎doc/code/framework.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -168,6 +168,7 @@ If you are contributing to PyRIT, that work will most likely land in one of the
168168

169169
- A scenario takes user input and uses it to package datasets with attack techniques
170170
- A scenario orchestrates resiliency and parallelism from a high level
171+
- A scenario chooses whether to skip or raise for incompatible techniques.
171172
- No result should depend on previous results (that is an attack's job)
172173
- **Does not own**: the per-objective conversation logic. Branching, turn-by-turn adaptation, and scoring-based decisions belong to the attack; a scenario selects and packages existing attack techniques rather than defining new attack algorithms or datasets.
173174

@@ -187,6 +188,7 @@ If you are contributing to PyRIT, that work will most likely land in one of the
187188
- Tags describe a technique (behavioral tags like `single_turn`/`multi_turn`, owner tags like `airt`); they don't decide what a scenario runs. There is deliberately **no global `default` tag** — a default is scenario-relative, declared per scenario via `build_technique_class_from_factories` (the `factories` list is the pool, catalog tags become named aggregate presets, and `default_tags` / `default_names` set what runs when nothing is chosen).
188189
- Generated technique enums follow their factory pool's lifecycle. Registry-derived enums and their lazy exports are discarded on setup registry reset so new scenarios see the replacement catalog. Enums built exclusively from fixed scenario-owned factories, such as `DoctorTechnique` and `JailbreakTechnique`, retain their identity across registry resets, so previously imported members remain valid for new scenario instances.
189190
- Factories opt into additive request-converter composition with `supports_additional_request_converters=True`. This is a semantic capability, not just constructor-signature detection; the factory validates that opted-in attacks accept `attack_converter_config`.
191+
- A technique declares target and dataset requirements and permitted adaptation; targets own capabilities and adapters.
190192
- **Does not own**: the conversation algorithm itself. Branching, turn management, and scoring decisions live in the executor it wraps — a technique only selects and configures existing components, and shouldn't implement new sending, scoring, or branching logic.
191193

192194
**Framework Plans**:

‎doc/code/scenarios/0_attack_techniques.ipynb‎

Lines changed: 45 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -155,6 +155,47 @@
155155
"cell_type": "markdown",
156156
"id": "3",
157157
"metadata": {},
158+
"source": [
159+
"## Target and dataset requirements\n",
160+
"\n",
161+
"Declare `TechniqueRequirements` on a factory. The attack class's `TARGET_REQUIREMENTS` also apply.\n",
162+
"\n",
163+
"```python\n",
164+
"from pyrit.executor.attack import PromptSendingAttack\n",
165+
"from pyrit.models import SeedGroupRequirements\n",
166+
"from pyrit.prompt_target import CapabilityName, TargetRequirements\n",
167+
"from pyrit.scenario import AttackTechniqueFactory, TechniqueRequirements\n",
168+
"\n",
169+
"factory = AttackTechniqueFactory(\n",
170+
" name=\"native_history\",\n",
171+
" attack_class=PromptSendingAttack,\n",
172+
" requirements=TechniqueRequirements(\n",
173+
" objective_target=TargetRequirements(native_required=frozenset({CapabilityName.MULTI_TURN})),\n",
174+
" seed_group=SeedGroupRequirements(objective_only=True),\n",
175+
" ),\n",
176+
")\n",
177+
"```\n",
178+
"\n",
179+
"| Requirement | Behavior |\n",
180+
"|-------------|----------|\n",
181+
"| `required` | Native target support, or an `ADAPT` policy with a configured adapter. |\n",
182+
"| `native_required` | Native target support only. |\n",
183+
"| `objective_only=True` | Only the objective before technique seeds are added. |\n",
184+
"| `objective_only=True, try_adapt=True` | Permit removal of extra dataset seeds from an execution copy. |\n",
185+
"\n",
186+
"Required modality combinations must be supported together. Simulated-conversation factories\n",
187+
"default to objective-only input. Source groups stay unchanged; applied adaptation is recorded\n",
188+
"in result metadata.\n",
189+
"\n",
190+
"Incompatible inputs raise `IncompatibleTechniqueError`. Scenarios can\n",
191+
"[skip them](./1_common_scenario_parameters.ipynb#incompatible-techniques).\n",
192+
"Changes to requirements or permitted adaptation can prevent older runs from resuming."
193+
]
194+
},
195+
{
196+
"cell_type": "markdown",
197+
"id": "4",
198+
"metadata": {},
158199
"source": [
159200
"## Executing a single technique\n",
160201
"\n",
@@ -176,7 +217,7 @@
176217
{
177218
"cell_type": "code",
178219
"execution_count": null,
179-
"id": "4",
220+
"id": "5",
180221
"metadata": {},
181222
"outputs": [
182223
{
@@ -595,7 +636,7 @@
595636
},
596637
{
597638
"cell_type": "markdown",
598-
"id": "5",
639+
"id": "6",
599640
"metadata": {},
600641
"source": [
601642
"## How techniques are selected\n",
@@ -619,7 +660,7 @@
619660
},
620661
{
621662
"cell_type": "markdown",
622-
"id": "6",
663+
"id": "7",
623664
"metadata": {
624665
"class": "col-page-right"
625666
},
@@ -637,7 +678,7 @@
637678
},
638679
{
639680
"cell_type": "markdown",
640-
"id": "7",
681+
"id": "8",
641682
"metadata": {},
642683
"source": [
643684
"## Relationship to single-turn attacks\n",

‎doc/code/scenarios/0_attack_techniques.py‎

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -117,6 +117,42 @@
117117
pd.set_option("display.max_colwidth", None)
118118
print(pd.DataFrame(rows).to_string(index=False))
119119

120+
# %% [markdown]
121+
# ## Target and dataset requirements
122+
#
123+
# Declare `TechniqueRequirements` on a factory. The attack class's `TARGET_REQUIREMENTS` also apply.
124+
#
125+
# ```python
126+
# from pyrit.executor.attack import PromptSendingAttack
127+
# from pyrit.models import SeedGroupRequirements
128+
# from pyrit.prompt_target import CapabilityName, TargetRequirements
129+
# from pyrit.scenario import AttackTechniqueFactory, TechniqueRequirements
130+
#
131+
# factory = AttackTechniqueFactory(
132+
# name="native_history",
133+
# attack_class=PromptSendingAttack,
134+
# requirements=TechniqueRequirements(
135+
# objective_target=TargetRequirements(native_required=frozenset({CapabilityName.MULTI_TURN})),
136+
# seed_group=SeedGroupRequirements(objective_only=True),
137+
# ),
138+
# )
139+
# ```
140+
#
141+
# | Requirement | Behavior |
142+
# |-------------|----------|
143+
# | `required` | Native target support, or an `ADAPT` policy with a configured adapter. |
144+
# | `native_required` | Native target support only. |
145+
# | `objective_only=True` | Only the objective before technique seeds are added. |
146+
# | `objective_only=True, try_adapt=True` | Permit removal of extra dataset seeds from an execution copy. |
147+
#
148+
# Required modality combinations must be supported together. Simulated-conversation factories
149+
# default to objective-only input. Source groups stay unchanged; applied adaptation is recorded
150+
# in result metadata.
151+
#
152+
# Incompatible inputs raise `IncompatibleTechniqueError`. Scenarios can
153+
# [skip them](./1_common_scenario_parameters.ipynb#incompatible-techniques).
154+
# Changes to requirements or permitted adaptation can prevent older runs from resuming.
155+
120156
# %% [markdown]
121157
# ## Executing a single technique
122158
#

‎doc/code/scenarios/1_common_scenario_parameters.ipynb‎

Lines changed: 30 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -664,6 +664,31 @@
664664
"cell_type": "markdown",
665665
"id": "19",
666666
"metadata": {},
667+
"source": [
668+
"## Incompatible techniques\n",
669+
"\n",
670+
"Set `incompatible_technique_policy` to `skip` (default) to warn and omit incompatible\n",
671+
"techniques, or `raise` to stop initialization.\n",
672+
"\n",
673+
"```python\n",
674+
"scenario = RedTeamAgent()\n",
675+
"scenario.set_params_from_args(\n",
676+
" args={\"objective_target\": objective_target, \"incompatible_technique_policy\": \"raise\"}\n",
677+
")\n",
678+
"await scenario.initialize_async()\n",
679+
"```\n",
680+
"\n",
681+
"Use `--incompatible-technique-policy raise` on the CLI or\n",
682+
"`incompatible_technique_policy: raise` in YAML. `None` uses the scenario class default.\n",
683+
"\n",
684+
"This policy does not enable adaptation. Initialization fails if no selected technique can run,\n",
685+
"even with a baseline. Other construction errors are not skipped."
686+
]
687+
},
688+
{
689+
"cell_type": "markdown",
690+
"id": "20",
691+
"metadata": {},
667692
"source": [
668693
"## Custom Scorers\n",
669694
"\n",
@@ -677,7 +702,7 @@
677702
{
678703
"cell_type": "code",
679704
"execution_count": null,
680-
"id": "20",
705+
"id": "21",
681706
"metadata": {},
682707
"outputs": [
683708
{
@@ -790,7 +815,7 @@
790815
},
791816
{
792817
"cell_type": "markdown",
793-
"id": "21",
818+
"id": "22",
794819
"metadata": {},
795820
"source": [
796821
"## Reporting Every Conversation\n",
@@ -805,7 +830,7 @@
805830
{
806831
"cell_type": "code",
807832
"execution_count": null,
808-
"id": "22",
833+
"id": "23",
809834
"metadata": {},
810835
"outputs": [
811836
{
@@ -876,7 +901,7 @@
876901
},
877902
{
878903
"cell_type": "markdown",
879-
"id": "23",
904+
"id": "24",
880905
"metadata": {},
881906
"source": [
882907
"`output_scenario_full_async` adds the run overview, like `--view full`, and writes the report as\n",
@@ -887,7 +912,7 @@
887912
{
888913
"cell_type": "code",
889914
"execution_count": null,
890-
"id": "24",
915+
"id": "25",
891916
"metadata": {},
892917
"outputs": [
893918
{

‎doc/code/scenarios/1_common_scenario_parameters.py‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -161,6 +161,26 @@
161161
# await scenario.initialize_async()
162162
# ```
163163

164+
# %% [markdown]
165+
# ## Incompatible techniques
166+
#
167+
# Set `incompatible_technique_policy` to `skip` (default) to warn and omit incompatible
168+
# techniques, or `raise` to stop initialization.
169+
#
170+
# ```python
171+
# scenario = RedTeamAgent()
172+
# scenario.set_params_from_args(
173+
# args={"objective_target": objective_target, "incompatible_technique_policy": "raise"}
174+
# )
175+
# await scenario.initialize_async()
176+
# ```
177+
#
178+
# Use `--incompatible-technique-policy raise` on the CLI or
179+
# `incompatible_technique_policy: raise` in YAML. `None` uses the scenario class default.
180+
#
181+
# This policy does not enable adaptation. Initialization fails if no selected technique can run,
182+
# even with a baseline. Other construction errors are not skipped.
183+
164184
# %% [markdown]
165185
# ## Custom Scorers
166186
#

0 commit comments

Comments
 (0)