Skip to content

Commit ca54989

Browse files
romanlutzCopilot
andauthored
FEAT: Attack analytics REST (#3149)
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
1 parent 04e3f84 commit ca54989

30 files changed

Lines changed: 2837 additions & 149 deletions

‎doc/code/analytics/0_attack_results.md‎

Lines changed: 228 additions & 5 deletions
Large diffs are not rendered by default.

‎doc/code/framework.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -434,6 +434,7 @@ The below describes the user-facing surfaces built on top of the framework.
434434
- Surfaces targets, scenarios, and health/version endpoints; served via `uvicorn` with Swagger/ReDoc docs.
435435
- Wherever possible it should reuse other components rather than reimplementing them (e.g. the registry to build components, `pyrit.models` for its model layer), while adding presentation-specific information on top as needed.
436436
- Organized into `routes/`, `services/`, `models/`, `mappers/`, and `middleware/`, and launched through the `pyrit_backend` command (configurable via `PYRIT_API_HOST` / `PYRIT_API_PORT` / `PYRIT_API_RELOAD`).
437+
- Saved-result analytics routes consume the cached SDK directly and serialize shared models. The shared runtime owns admission, busy checks, SDK draining and cache invalidation before memory replacement/disposal, including work still cleaning up after an HTTP timeout. It does not calculate statistics or add an SDK authorization scope.
437438

438439
## [CLI](../scanner/0_scanner)
439440

‎doc/contributing/11_memory_models.md‎

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -103,9 +103,11 @@ The query grain is a distinct stored `AttackResultEntry.id`. Different result ID
103103
sharing a conversation remain different results. For callers that need full result
104104
objects, `MemoryInterface.get_attack_results_async` accepts the opt-in
105105
`result_selection=AttackResultSelection.ALL_RESULTS` mode. The deprecated sync
106-
method also accepts this mode. Both default to
107-
`LATEST_PER_CONVERSATION`, including existing History callers. Turn bounds apply
108-
after selection, and neither mode deletes or rewrites stored duplicates.
106+
method also accepts this mode. Both still default to `LATEST_PER_CONVERSATION`.
107+
The backend's Attack History listing explicitly requests `ALL_RESULTS` and
108+
fingerprints that mode in its cursor; unrelated SDK and scenario callers keep
109+
their existing selection. Turn bounds apply after selection, and neither mode
110+
deletes or rewrites stored duplicates.
109111

110112
Revision `901e6c7bf9d4` follows the published `main` head `34a18645c7e9`. It
111113
bounds `outcome` to 16 characters and adds the computed

‎doc/gui/0_gui.md‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -404,6 +404,12 @@ Human score changes do not require a registered objective target. The original o
404404

405405
The History view lists all past attacks with filtering and pagination.
406406

407+
Each saved result ID is a separate row, even when several results share one
408+
conversation. Failed attempts followed by successful retries are not collapsed.
409+
Existing filters and recency ordering are unchanged. A pagination cursor from
410+
before this selection change restarts at the first page. Scenario run statistics
411+
still count their latest execution units, not these raw result rows.
412+
407413
<img width="1664" alt="Attack history view" src="images/history.png" />
408414

409415
#### Filters
@@ -578,6 +584,10 @@ runtime operation is active. Wait for the work to finish, or cancel it with its
578584
When the runtime is idle, PyRIT closes admission and checks again before it changes runtime state. This second check
579585
prevents newly admitted work from overlapping replacement.
580586

587+
Analytics also remains busy while a query's native database/session cleanup is
588+
running after an HTTP timeout. Live apply waits for the user to retry once that
589+
work has drained; receiving a timeout response does not make memory safe to replace.
590+
581591
Operation status survives a browser disconnect or navigation; other connected clients detect runtime generation
582592
changes and refresh catalogs without discarding chat or configuration drafts.
583593

@@ -588,6 +598,10 @@ a single request or cleanup failure is re-raised unchanged, while multiple failu
588598
Shutdown can therefore wait for an outstanding operation. Runtime readiness reports `ready: false` and
589599
`state: stopping` as soon as shutdown closes admission, including while an accepted live apply finishes.
590600

601+
Analytics shutdown closes its shared SDK lifetime on the owning event loop and
602+
clears the cached dependency before memory disposal. If that drain fails, the
603+
backend retains the owner and memory rather than disposing resources still in use.
604+
591605
If validation fails, PyRIT does not change the live runtime. Repair the saved source and retry. If startup fails, or
592606
if live initialization fails after replacement starts, runtime operations stay unavailable until you restart the
593607
backend. The administration UI stays available for configuration repair. Authentication and authorization retain

‎pyrit/analytics/_execution.py‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -115,6 +115,11 @@ def is_closed(self) -> bool:
115115
"""Whether shutdown has drained all started operations, not just their callers."""
116116
return self._closed
117117

118+
@property
119+
def has_active_work(self) -> bool:
120+
"""Whether admission, operations, or shutdown cleanup still occupy this lifetime."""
121+
return (self._closing and not self._closed) or any(lane.active or lane.queued for lane in self._lanes.values())
122+
118123
async def run_async(self, *, report: bool, task: Callable[[QueryControl], Awaitable[T]]) -> T:
119124
"""
120125
Admit one operation and await its response without blocking the owning loop.

‎pyrit/analytics/attack_result_analytics.py‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -75,6 +75,21 @@ def __init__(self, *, memory: MemoryInterface | None = None) -> None:
7575
self._controller: AnalyticsExecution | None = None
7676
self._closed = False
7777

78+
@property
79+
def has_active_work(self) -> bool:
80+
"""
81+
Whether this facade's shared controller still owns admitted work or cleanup.
82+
83+
This includes work whose caller timed out or was cancelled. Reading the
84+
status never creates a controller or opens a session.
85+
"""
86+
return self._controller is not None and self._controller.has_active_work
87+
88+
@property
89+
def is_closed(self) -> bool:
90+
"""Whether this facade is terminal and its attached controller has finished draining."""
91+
return self._closed or (self._controller is not None and self._controller.is_closed)
92+
7893
async def __aenter__(self) -> Self:
7994
"""
8095
Acquire the shared execution lifetime without opening a database session.

‎pyrit/backend/main.py‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,7 @@
2727
from pyrit.backend.middleware.request_size import RequestSizeLimitMiddleware
2828
from pyrit.backend.middleware.runtime import RuntimeAdmissionMiddleware
2929
from pyrit.backend.routes import (
30+
analytics,
3031
attacks,
3132
auth,
3233
configuration,
@@ -121,6 +122,7 @@ async def lifespan(app: FastAPI) -> AsyncGenerator[None, None]:
121122

122123

123124
# Include API routes
125+
app.include_router(analytics.router, prefix="/api", tags=["analytics"])
124126
app.include_router(attacks.router, prefix="/api", tags=["attacks"])
125127
app.include_router(message_sends.router, prefix="/api", tags=["attacks"])
126128
app.include_router(configuration.router, prefix="/api", tags=["config"])

‎pyrit/backend/middleware/error_handlers.py‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,13 +13,41 @@
1313

1414
from pyrit.backend.middleware.request_size import RequestTooLargeError, request_too_large_response
1515
from pyrit.backend.models.common import FieldError, ProblemDetail
16+
from pyrit.exceptions.analytics_exception import AnalyticsException
1617

1718
logger = logging.getLogger(__name__)
1819

1920

2021
def register_error_handlers(app: FastAPI) -> None:
2122
"""Register all error handlers with the FastAPI app."""
2223

24+
@app.exception_handler(AnalyticsException)
25+
async def analytics_exception_handler_async(request: Request, exc: AnalyticsException) -> JSONResponse:
26+
"""
27+
Surface overload, deadlines, and stored-data failures without empty-report fallbacks.
28+
29+
Returns:
30+
JSONResponse: The shared error contract, with retry guidance for overload.
31+
"""
32+
logger.warning(
33+
"Analytics request failed on %s request_id=%s: %s",
34+
request.url.path,
35+
getattr(request.state, "request_id", None),
36+
exc.message,
37+
)
38+
problem = ProblemDetail(
39+
type="/errors/analytics",
40+
title="Analytics unavailable",
41+
status=exc.status_code,
42+
detail=exc.message,
43+
instance=str(request.url.path),
44+
)
45+
return JSONResponse(
46+
status_code=exc.status_code,
47+
content=problem.model_dump(exclude_none=True),
48+
headers={"Retry-After": "1"} if exc.status_code == status.HTTP_503_SERVICE_UNAVAILABLE else None,
49+
)
50+
2351
@app.exception_handler(RequestValidationError)
2452
async def validation_exception_handler( # pyrit-async-suffix-exempt
2553
request: Request,

‎pyrit/backend/routes/analytics.py‎

Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,60 @@
1+
# Copyright (c) Microsoft Corporation.
2+
# Licensed under the MIT license.
3+
4+
"""Read-only saved-result analytics through the shared SDK."""
5+
6+
from typing import Annotated
7+
8+
from fastapi import APIRouter, Depends
9+
10+
from pyrit.analytics import AttackResultAnalytics
11+
from pyrit.backend.models.common import ProblemDetail
12+
from pyrit.backend.services.analytics_service import get_analytics_service_async
13+
from pyrit.models import (
14+
AttackAnalyticsFacetQuery,
15+
AttackAnalyticsFacets,
16+
AttackAnalyticsQuery,
17+
AttackAnalyticsReport,
18+
AttackAnalyticsResults,
19+
AttackAnalyticsResultsQuery,
20+
)
21+
22+
router = APIRouter(
23+
prefix="/analytics/attacks",
24+
tags=["analytics"],
25+
responses={
26+
400: {"model": ProblemDetail, "description": "Invalid or mismatched result cursor"},
27+
422: {"model": ProblemDetail, "description": "Invalid analytics query"},
28+
500: {"model": ProblemDetail, "description": "Stored data or unexpected query failure"},
29+
503: {"model": ProblemDetail, "description": "Analytics capacity is busy; retry after one second"},
30+
504: {"model": ProblemDetail, "description": "The analytics query exceeded its deadline"},
31+
},
32+
)
33+
34+
35+
@router.post("/query", response_model=AttackAnalyticsReport)
36+
async def query_analytics_async(
37+
*, query: AttackAnalyticsQuery, analytics: Annotated[AttackResultAnalytics, Depends(get_analytics_service_async)]
38+
) -> AttackAnalyticsReport:
39+
"""Return SDK statistics and a coherent first result page."""
40+
return await analytics.query_async(query=query)
41+
42+
43+
@router.post("/results", response_model=AttackAnalyticsResults)
44+
async def query_results_async(
45+
*,
46+
query: AttackAnalyticsResultsQuery,
47+
analytics: Annotated[AttackResultAnalytics, Depends(get_analytics_service_async)],
48+
) -> AttackAnalyticsResults:
49+
"""Return a fresh lightweight page without recalculating reports."""
50+
return await analytics.results_async(query=query)
51+
52+
53+
@router.post("/facets", response_model=AttackAnalyticsFacets)
54+
async def query_facets_async(
55+
*,
56+
query: AttackAnalyticsFacetQuery,
57+
analytics: Annotated[AttackResultAnalytics, Depends(get_analytics_service_async)],
58+
) -> AttackAnalyticsFacets:
59+
"""Return options for one dimension without recalculating reports."""
60+
return await analytics.facets_async(query=query)
Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
# Copyright (c) Microsoft Corporation.
2+
# Licensed under the MIT license.
3+
4+
"""Backend ownership of the direct SDK dependency, not a forwarding service."""
5+
6+
from functools import lru_cache
7+
8+
from pyrit.analytics import AttackResultAnalytics
9+
10+
11+
@lru_cache(maxsize=1)
12+
def get_analytics_service() -> AttackResultAnalytics:
13+
"""
14+
Resolve one long-lived SDK facade after startup has configured CentralMemory.
15+
16+
Returns:
17+
AttackResultAnalytics: The direct SDK dependency for this runtime.
18+
"""
19+
return AttackResultAnalytics()
20+
21+
22+
async def get_analytics_service_async() -> AttackResultAnalytics:
23+
"""
24+
Resolve the cached SDK on the runtime loop, avoiding concurrent threadpool cache misses.
25+
26+
Returns:
27+
AttackResultAnalytics: The same SDK owner for concurrent HTTP dependencies.
28+
"""
29+
return get_analytics_service()
30+
31+
32+
async def close_analytics_service_async() -> None:
33+
"""Drain on the owning loop before releasing the cached memory reference."""
34+
if get_analytics_service.cache_info().currsize:
35+
analytics = get_analytics_service()
36+
try:
37+
await analytics.close_async()
38+
finally:
39+
# Cancellation may propagate after draining; a failed drain must retain its owner.
40+
if analytics.is_closed:
41+
get_analytics_service.cache_clear()

0 commit comments

Comments
 (0)