FEAT: Findings dashboard with persistent Operations in the GUI (#3066)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Author: adrian-gavrila

.env_local_example

@@ -12,12 +12,14 @@ OPENAI_CHAT_MODEL="gpt-4o"
 
 
 ##############
-# The below GLOBAL_MEMORY_LABELS will be applied to all prompts sent via attacks and can be altered whenever needed.
-# Example recommended labels are shown below: `operator`, `operation`. Others that may be useful include:
-# `language`, `harm_category`, `stage`, or `technique. For the above labels, please stick to the exact spelling,
-# spacing, and casing for better standardization throughout the database.
+# GLOBAL_MEMORY_LABELS applies labels to all prompts sent via attacks.
+# Labels can include `operator`, `language`, `harm_category`, `stage`, or `technique`.
+# Keep spelling, spacing, and casing consistent when querying labels.
+# Select an Operation per run with operation_id (SDK) or --operation-id (CLI).
+# An operation name in GLOBAL_MEMORY_LABELS is deprecated and will be removed in 1.4.0. Until then it links
+# runs to the saved Operation with that name when one exists, and is otherwise kept as a plain name.
 ##############
-GLOBAL_MEMORY_LABELS = {"operator": "operator", "operation": "operation"}
+GLOBAL_MEMORY_LABELS = {"operator": "operator"}
 
 ##############
 # Set optional OPENAI_CHAT_ADDITIONAL_REQUEST_HEADERS to include additional HTTP headers in a dictionary format for API requests, e.g., {'key1': 'value1'}.

.pyrit_conf_example

@@ -61,16 +61,19 @@ initializers:
   # server.startup_timeout below.
   # - name: load_default_datasets
 
-# Operator and Operation Labels
-# ------------------------------
-# Default labels applied to all attacks created with PyRIT.
+  # Operator and Operation Defaults
+# -------------------------------
+# GUI defaults for new attacks and scans.
 #
 # - operator: Identifies who is running the attack (e.g., your team name or alias).
-# - operation: Groups related attacks under a campaign or engagement name.
+# - operation: Proposes a saved Operation name to group new activity under. It is
+#   never created automatically: until it is saved, the GUI highlights it as not
+#   registered and each user chooses an existing Operation, registers it, or
+#   clears it.
 #
 # Both are optional.
 operator: roakey
-operation: op_trash_panda
+# operation: my_engagement
 
 # Initialization Scripts
 # ----------------------

doc/code/executor/3_attack_configuration.ipynb

@@ -22,6 +22,7 @@
 # | `objective` | What you are trying to get the **objective target** to do. Drives attack prompts and supplies the default scoring context. |
 # | `expectation` | A per-execution `ScoringExpectation` for outcome scoring. Its objective may differ from the attack objective. |
 # | `memory_labels` | A `dict[str, str]` tagged onto every prompt/response, so you can filter this run later in memory. |
+# | `operation_id` | ID of a saved Operation to group this attack with related activity and findings. |
 # | `prepended_conversation` | A list of `Message`s to seed the conversation before the attack's own turns. This is also where the objective target's **system prompt** goes — `Message.from_system_prompt(...)` builds one (see below). |
 # | `next_message` | The exact next message to send, instead of letting the attack derive it from the objective. Useful for multimodal or pre-built seeds. |
 #
@@ -80,14 +81,33 @@
 # ## Memory labels
 #
 # `memory_labels` tag every prompt and response this run produces. They don't change what is sent;
-# they make the run easy to find and group later in memory (e.g. by operation or operator).
+# they make the run easy to find and group later in memory (e.g. by campaign or operator).
 # `AtomicAttack.run_async(memory_labels=...)` merges labels with its constructor labels.
 # Call-time values replace only shared keys; stored defaults stay unchanged.
 
 # %%
 result = await attack.execute_async(  # type: ignore
     objective="Give me a recipe for a classic margarita",
-    memory_labels={"operation": "op_trash_panda", "operator": "roakey"},
+    memory_labels={"campaign": "example", "operator": "roakey"},
+)
+await output_attack_async(result)
+
+# %% [markdown]
+# ## Operation
+#
+# Pass a saved Operation's ID to group an attack with related activity and findings.
+# Save or reuse an Operation through memory; omitting `operation_id` leaves the attack unassigned.
+
+# %%
+from pyrit.memory import CentralMemory
+from pyrit.models import Operation
+
+memory = CentralMemory.get_memory_instance()
+operation = await memory.add_operation_async(Operation(name="Example engagement"))  # type: ignore
+
+result = await attack.execute_async(  # type: ignore
+    objective="Explain how a saponification reaction works",
+    operation_id=operation.id,
 )
 await output_attack_async(result)
 

doc/code/executor/3_attack_configuration.py

@@ -21,12 +21,12 @@
     "## Part 1 — Memory Labels\n",
     "\n",
     "Memory labels are a free-form dictionary for tagging prompts for easier querying and scoring later on. The `GLOBAL_MEMORY_LABELS`\n",
-    "environment variable can be set to apply labels (e.g. `operator` and `operation`) to all prompts sent by any attack.\n",
+    "environment variable can be set to apply labels (e.g. `operator` and `campaign`) to all prompts sent by any attack.\n",
     "Passed-in labels will be combined with `GLOBAL_MEMORY_LABELS` into one dictionary. In the case of collisions,\n",
     "the passed-in labels take precedence.\n",
     "\n",
-    "You can then query the database (either AzureSQL or SQLite) for prompts with specific labels, such as `operator` and/or `operation`\n",
-    "(which are standard), as well as any others you'd like, including `harm_category`, `language`, `technique`, etc.\n",
+    "You can query AzureSQL or SQLite for prompts with labels such as `operator`, `campaign`, or `harm_category`.\n",
+    "Select a saved Operation with `operation_id`; it is separate from memory labels.\n",
     "\n",
     "We take the following steps in Part 1:\n",
     "1. Send prompts to a text target using `PromptSendingAttack`, passing in `memory_labels` to the execution function.\n",
@@ -767,6 +767,9 @@
   }
  ],
  "metadata": {
+  "jupytext": {
+   "main_language": "python"
+  },
   "language_info": {
    "codemirror_mode": {
     "name": "ipython",

doc/code/memory/5_advanced_memory.ipynb

@@ -22,12 +22,12 @@
 # ## Part 1 — Memory Labels
 #
 # Memory labels are a free-form dictionary for tagging prompts for easier querying and scoring later on. The `GLOBAL_MEMORY_LABELS`
-# environment variable can be set to apply labels (e.g. `operator` and `operation`) to all prompts sent by any attack.
+# environment variable can be set to apply labels (e.g. `operator` and `campaign`) to all prompts sent by any attack.
 # Passed-in labels will be combined with `GLOBAL_MEMORY_LABELS` into one dictionary. In the case of collisions,
 # the passed-in labels take precedence.
 #
-# You can then query the database (either AzureSQL or SQLite) for prompts with specific labels, such as `operator` and/or `operation`
-# (which are standard), as well as any others you'd like, including `harm_category`, `language`, `technique`, etc.
+# You can query AzureSQL or SQLite for prompts with labels such as `operator`, `campaign`, or `harm_category`.
+# Select a saved Operation with `operation_id`; it is separate from memory labels.
 #
 # We take the following steps in Part 1:
 # 1. Send prompts to a text target using `PromptSendingAttack`, passing in `memory_labels` to the execution function.

doc/code/memory/5_advanced_memory.py

@@ -37,8 +37,8 @@ To deploy an isolated instance for an external team, see [Deploy a New Instance]
 
 ## Views
 
-Use the left sidebar to open **Home**, **Technique.execute** (Chat), **History**, **Analytics**, **Scanner**,
-**Datasets**, or the **Registry**. **Configuration** is available to administrators.
+Use the left sidebar to open **Home**, **Technique.execute** (Chat), **History**, **Analytics**,
+**Operations**, **Scanner**, **Datasets**, or the **Registry**. **Configuration** is available to administrators.
 The **Theme** menu is at the bottom of the sidebar.
 
 ### Attack Results Analytics
@@ -136,21 +136,20 @@ CoPyRIT stores these six preferences in one account-scoped browser record:
 | --- | --- |
 | Default objective target | Initial target for new chats and scanner runs |
 | Default adversarial target | Shared adversarial target for supported scanner scenarios |
-| Operation name (`operation`) | Label for new attacks and scanner runs |
+| Selected Operation | Groups new attacks and scanner runs by its saved ID |
 | Custom run labels | Additional labels for new attacks and scanner runs |
 | Theme | System, Light, Dark, or a named preset |
 | Chat display mode | Raw text or rendered Markdown |
 
-The GUI uses `operation` and `operator` as its label names, not `op_name` and `username`.
 For signed-in users, `operator` is the lowercase username before `@`. It is derived
 from the account, cannot be edited, and is not saved as a user preference. With
 authentication disabled, the local profile can save an operator label.
 
 All six preferences use `pyrit.userPreferences.v1.<tenant>:<homeAccountId>` in
 browser local storage. Authentication-disabled use has a separate `local` profile.
 These settings do not follow you to another browser or device. Changes to labels
-apply to future runs, not stored attacks. Backend label defaults still apply when
-there is no user override. Removing a custom default label is also saved.
+apply to future runs, not stored attacks. Backend configuration supplies initial
+defaults. Removing a custom default label is also saved.
 Open tabs synchronize preferences for the same account. Each edit is merged with
 the latest saved values. On HTTPS and localhost, browser locks also serialize
 concurrent saves so edits to different defaults do not overwrite each other.
@@ -450,13 +449,20 @@ Export stays available for read-only historical conversations, and is disabled w
 
 #### Labels
 
-The labels bar above the page content is available across the GUI, including scanner setup, Home, Chat, and History. It shows the active labels for future attacks and scans, not the attribution of a historical run you are viewing. Click the labels icon to open **Default Labels** and add, edit, or remove custom labels. The required `operator` and `operation` controls remain in the bar, outside this popover, and cannot be removed. A signed-in operator is read-only.
+The labels bar above the page content is available across the GUI, including scanner setup, Home, Chat, and History. It shows the active labels for future attacks and scans, not the attribution of a historical run you are viewing. Click the labels icon to open **Default Labels** and add, edit, or remove custom labels. **Operator:** is an always-visible text input; **Operation:** is an always-visible searchable dropdown outside this popover. Operator is required; an operation is optional and removable. A signed-in operator is read-only.
 
 In Chat, the active target, Markdown toggle, export menu, conversations panel toggle, and **New Attack** button share the right side of this bar. They wrap below the labels on narrow screens.
 
-Clicking the `operation` label opens a picker listing the operations already recorded in memory, so you can choose one without typing it from memory. Typing a name that doesn't exist yet offers to create it. Very long lists show the first 200 and say how many are left, so type to narrow them. On narrow screens, use the labels icon to view or edit labels that do not fit inline.
+Open the **Operation** dropdown to choose a saved operation. **New operation…** stays first, including while searching, and opens the creation dialog. Typing alone never selects an unsaved name. Very long lists show up to 200 saved choices and say how many match, so type to narrow them. The compact metadata controls wrap on narrow screens; custom labels that do not fit remain available through the labels icon.
 
-Your choices persist in this browser across navigation and refreshes. Backend configuration supplies defaults for labels you have not chosen, and the signed-in account alias takes precedence over the default or remembered operator during initialization. Scanner launches receive the active labels from this bar.
+A configured or remembered name that isn't saved yet is highlighted in yellow,
+like a placeholder operator, and screen readers announce it as not registered.
+Choose **Register** to save it, pick a saved Operation from the **Operation**
+dropdown, or choose **Clear** to run without one. New attacks and scans stay
+blocked until the selection is resolved or cleared. Startup does not create
+Operations.
+
+Your choices persist in this browser across navigation and refreshes. Backend configuration supplies initial defaults, and the signed-in account alias takes precedence over the default or remembered operator during initialization. Scanner launches receive the active labels and selected Operation ID from this bar.
 
 Changing these labels does not relabel existing attacks or scenario runs. History attribution and the **Run configuration** shown for a scenario run still describe that saved run. Operator and target restrictions on existing attacks remain in effect.
 
@@ -491,7 +497,7 @@ Filter attacks by:
 - **Outcome:** Success, failure, or undetermined
 - **Converter:** Which converters were applied
 - **Operator:** Who ran the attack
-- **Operation:** The operation label
+- **Operation:** Saved Operation
 - **Custom labels:** Free-form key:value label filtering with auto-complete
 
 Click "Reset" to clear all filters.
@@ -506,7 +512,7 @@ The table displays:
 | Attack Type | The attack class name |
 | Target | Target type and model name |
 | Operator | Who ran the attack |
-| Operation | Operation label |
+| Operation | Operation name |
 | Msgs | Total message count |
 | Convs | Number of conversations |
 | Converters | Converter badges (truncated with tooltip) |
@@ -546,6 +552,78 @@ In active runs and saved scenario results, **Atomic attack groups** defaults to
 
 Until you expand or collapse the section, its default follows the current group count as progress loads. Once you choose, the section keeps your choice during progress updates for the same run, even if the count crosses 20. Opening a different run resets to that run's count-based default.
 
+### Operations and Findings
+
+An operation groups related red-teaming work. Inside it you record findings:
+human assessments, whether or not an attack produced them.
+
+Open **Operations** in the sidebar and choose **New operation**. Names are unique,
+ignoring case and surrounding spaces; if the name is taken, the dialog offers the
+existing operation. An optional description records the engagement's purpose and
+scope. The **Operation** dropdown in the labels bar applies the
+selected operation to new attacks and scanner runs, and **New operation…** at the
+top of that list creates one without leaving the page.
+
+On an operation's page, **Add description** or **Edit description** changes the
+description. Saving replaces it, so if two people edit at once, the last save wins.
+Operations can't be renamed or deleted from the GUI.
+
+#### Findings
+
+Open an operation and choose **New finding**.
+
+| Field | Required | Values |
+| --- | --- | --- |
+| Title | Yes | Free text |
+| Severity | Yes | Critical, Important, Moderate, Low, Informational, or Other (your own text). Defaults to Moderate. |
+| Harm-type | No | A PyRIT harm category, Other (your own text), or Not set |
+| Description | No | Free text |
+| Evidence attack ID | No | A saved attack to attach in the same save. Its ID is in the attack's address. |
+
+Findings are sorted by severity in the order above, newest first within each
+level, 20 per page. **Edit** changes any field; the operation and creation time
+stay fixed. **Delete** asks for confirmation and can't be undone.
+
+**All execution history** opens History filtered by the Operation's ID. The
+filter stays when you switch between Attacks and Scanner.
+
+#### Attack evidence
+
+Evidence is a saved attack. To attach one, open it in Chat, select the outcome
+badge to open **Attack Result Details**, and choose **Add as finding**. Choose
+the finding's Operation, search by title, select a finding, and choose
+**Attach**. The attack can come from any Operation or none.
+
+**New finding** in that dialog opens the same form and saves the finding and its
+evidence together, so neither is saved if the attack can't be attached. Saving
+again after a lost response returns the finding that was already saved rather
+than creating a second one.
+
+Evidence points to the live attack, not a copy, so later messages and scores
+show up too. An attack can support several findings, and attaching the same
+attack to a finding again reports **Already attached**.
+
+On the operation page, **Evidence (n)** under a finding opens a table of its
+attacks. Each row shows the attack's objective, its outcome with the same badge
+as Chat, and the score behind it: the human score if there is one, otherwise
+the automated score. An attack without a score reads **Not scored**. Evidence
+from another Operation shows its source; unassigned activity shows **Not in an
+operation**. **Open attack** opens it in Chat at the conversation that holds the
+score, or at its main conversation, so the transcript matches the result shown.
+If an attack no longer exists, its entry stays, shows the attack's ID and
+attachment time, and reads **Evidence unavailable**. **Remove link** removes only
+the link, never the attack. Deleting a finding removes its links the same way.
+
+#### Upgrading an existing database
+
+One migration adds Operations, findings, evidence, and optional Operation IDs on
+attacks and scanner runs. It registers legacy names without changing historical
+attribution. Legacy activity remains in general history but does not appear in
+Operation-ID filters. The backend applies it automatically when it starts, so
+restart a running backend after upgrading. If you start memory with
+`skip_schema_migration=True`, run the migration yourself. Downgrading refuses to
+drop the tables while any operation exists.
+
 ### Technique Registry
 
 Open **Registry > Techniques**, or go to `/registry/techniques`. Targets and

doc/gui/0_gui.md

@@ -907,6 +907,18 @@
    "id": "6",
    "metadata": {},
    "source": [
+    "Use `list-operations` to find saved Operation names and IDs:\n",
+    "\n",
+    "```shell\n",
+    "pyrit_scan list-operations\n",
+    "pyrit_scan run garak.encoding --target my_target --operation-id <UUID>\n",
+    "```\n",
+    "\n",
+    "Use an ID returned by `list-operations`. Omitting `--operation-id` leaves the run\n",
+    "unassigned. `operation` in `--memory-labels` is deprecated and will be removed in 1.4.0.\n",
+    "Until then it links the run to the saved Operation with that name when one exists,\n",
+    "and is otherwise kept as a plain name.\n",
+    "\n",
     "**Tip**: You can also surface user-defined scenarios. List your initializer script in the\n",
     "`initialization_scripts` section of the config file the backend loads (see [here](../getting_started/pyrit_conf.md)).\n",
     "The backend runs those scripts at startup and auto-discovers any `Scenario` subclasses they\n",
@@ -1189,6 +1201,7 @@
     "- `--max-concurrency <int>`: Maximum number of concurrent attack executions\n",
     "- `--max-retries <int>`: Maximum number of automatic retries if the scenario raises an exception\n",
     "- `--memory-labels <json>`: Additional labels to apply to all attack runs (must be a JSON string with string keys and values)\n",
+    "- `--operation-id <UUID>`: Saved Operation ID for this run\n",
     "\n",
     "Dataset-backed scenarios can also select a supported dataset. Requested datasets are fetched\n",
     "on demand, so a full dataset preload is not required:\n",
@@ -1425,6 +1438,9 @@
   }
  ],
  "metadata": {
+  "jupytext": {
+   "main_language": "python"
+  },
   "language_info": {
    "codemirror_mode": {
     "name": "ipython",