Skip to content

Feat/market trends agent drift detection - #2056

Open
akomandooru wants to merge 6 commits into
awslabs:mainfrom
akomandooru:feat/market-trends-agent-drift-detection
Open

Feat/market trends agent drift detection#2056
akomandooru wants to merge 6 commits into
awslabs:mainfrom
akomandooru:feat/market-trends-agent-drift-detection

Conversation

@akomandooru

Copy link
Copy Markdown

Amazon Bedrock AgentCore Samples Pull Request

Important

  1. We strictly follow a issue-first approach, please first open an issue relating to this Pull Request.
  2. Once this Pull Request is ready for review please attach review ready label to it. Only PRs with review ready will be reviewed.

Issue number:
#2055

Concise description of the PR

Changes to the Market Trends Agent sample, adding a quality drift detection
feature under drift_detection/, because the sample's evaluators already score
live traffic but nothing watches whether those scores get worse over time.
Also fixes an evaluator IAM permissions policy that had unscoped Resource: "*"
on some log actions, and a trust-policy condition that didn't cover the
runtime's own source ARN.

User experience

Before: no signal exists when the agent's live quality degrades (e.g. after a
model version update with no corresponding deploy event). Degradation is only
found via manual inspection of Evaluations dashboards or user complaints.

After: running drift_detection/scripts/deploy.py stands up a scheduled Lambda,
DynamoDB state table, one CloudWatch alarm per evaluator, and a dashboard. Each
evaluator's score stream is tracked against its own learned baseline, and an
alarm fires naming the specific evaluator that degraded and stayed degraded.
watch.py, shape_report.py, and induce_drift.py give an operator CLI visibility
into detector state and a way to demo drift without waiting for it.

Checklist

If your change doesn't seem to apply, please leave them unchecked.

  • [ x] I have reviewed the contributing guidelines
  • [ x] Add your name to CONTRIBUTORS.md
  • [ x] Have you checked to ensure there aren't other open Pull Requests for the same update/change?
  • Are you uploading a dataset?
  • [ x] Have you documented Introduction, Architecture Diagram, Prerequisites, Usage, Sample Prompts, and Clean Up steps in your example README?
  • I agree to resolve any issues created for this example in the future.
  • [ x] I have performed a self-review of this change
  • [ x] Changes have been tested
  • [ x] Changes are documented

Acknowledgment

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of the project license.

Anand Komandooru added 5 commits September 3, 2026 15:41
- Centralize Bedrock model selection into tools/model_config.py so
  MODEL_ID can be overridden via environment variable without editing
  three separate call sites (market_trends_agent.py, browser_tool.py,
  broker_card_tools.py)
- Fix provider resolution for cross-region inference profile IDs
  (langchain-aws cannot infer provider from a "global." prefix)
- Widen the agent execution role's trust policy to allow runtime/* in
  addition to ab-test/*, fixing "Role validation failed" on startup
- Add aws-opentelemetry-distro dependency; without it the agent
  emitted no spans and AgentCore Evaluations had nothing to score
- Add bedrock:InvokeModel permission for the evaluators' execution
  role so LLM-judge evaluators can actually invoke their judge model
- Scope evaluators/iam/permissions-policy.json resources to this
  account instead of wildcarding the account segment (log reads,
  Lambda invoke, and Bedrock inference-profile invoke)
Adds a scheduled detector that watches this agent's 8 evaluators over
time and alarms when a stream degrades and stays degraded, distinct
from evaluate/optimize which compares variants rather than an agent
against its own past.

- detector/: EWMA and z-score methods with warm-up, persistence, and
  latching; the method used per evaluator is chosen from its measured
  score shape rather than applied uniformly
- scripts/deploy.py: state table, Lambda, EventBridge schedule,
  per-evaluator CloudWatch alarms, and a dashboard
- scripts/shape_report.py: measures each evaluator's live score shape
  and flags any evaluator whose configured method disagrees with it
- scripts/induce_drift.py: three demo triggers (model_swap,
  stale_prices, skip_profile_step) to exercise the detector against
  real, induced drift instead of synthetic data
- scripts/traffic.py, watch.py, attach_evaluators.py, teardown.py:
  supporting operational scripts
- market_trends_agent.py, tools/browser_tool.py: agent-side support
  for the skip_profile_step and stale_prices triggers

See drift_detection/README.md for design rationale, the detection
method comparison, and measured results from three induced causes of
drift on this agent.
Previously defaulted to a hardcoded guess that could silently mismatch
the real service.name computed by evaluators/scripts/deploy.py, leaving
the dashboard's raw evaluator score widget empty. Now derived the same
way (from agentRuntimeArn in .deploy_output.json), with an explicit
env var override and a warning if it falls back to the guess.
…ent-drift-detection

# Conflicts:
#	02-use-cases/01-conversational-agents/market-trends-agent/uv.lock
#	CONTRIBUTORS.md
@github-actions github-actions Bot added the 02-use-cases 02-use-cases label Sep 3, 2026
@github-actions

github-actions Bot commented Sep 4, 2026

Copy link
Copy Markdown

Latest scan for commit: 0042bf7 | Updated: 2026-09-04 01:48:52 UTC

Security Scan Results

Scan Metadata

  • Project: ASH
  • Scan executed: 2026-09-04T01:42:36+00:00
  • ASH version: 3.0.0

Summary

Scanner Results

The table below shows findings by scanner, with status based on severity thresholds and dependencies:

Column Explanations:

Severity Levels (S/C/H/M/L/I):

  • Suppressed (S): Security findings that have been explicitly suppressed/ignored and don't affect the scanner's pass/fail status
  • Critical (C): The most severe security vulnerabilities requiring immediate remediation (e.g., SQL injection, remote code execution)
  • High (H): Serious security vulnerabilities that should be addressed promptly (e.g., authentication bypasses, privilege escalation)
  • Medium (M): Moderate security risks that should be addressed in normal development cycles (e.g., weak encryption, input validation issues)
  • Low (L): Minor security concerns with limited impact (e.g., information disclosure, weak recommendations)
  • Info (I): Informational findings for awareness with minimal security risk (e.g., code quality suggestions, best practice recommendations)

Other Columns:

  • Time: Duration taken by each scanner to complete its analysis
  • Action: Total number of actionable findings at or above the configured severity threshold that require attention

Scanner Results:

  • PASSED: Scanner found no security issues at or above the configured severity threshold - code is clean for this scanner
  • FAILED: Scanner found security vulnerabilities at or above the threshold that require attention and remediation
  • MISSING: Scanner could not run because required dependencies/tools are not installed or available
  • SKIPPED: Scanner was intentionally disabled or excluded from this scan
  • ERROR: Scanner encountered an execution error and could not complete successfully

Severity Thresholds (Thresh Column):

  • CRITICAL: Only Critical severity findings cause scanner to fail
  • HIGH: High and Critical severity findings cause scanner to fail
  • MEDIUM (MED): Medium, High, and Critical severity findings cause scanner to fail
  • LOW: Low, Medium, High, and Critical severity findings cause scanner to fail
  • ALL: Any finding of any severity level causes scanner to fail

Threshold Source: Values in parentheses indicate where the threshold is configured:

  • (g) = global: Set in the global_settings section of ASH configuration
  • (c) = config: Set in the individual scanner configuration section
  • (s) = scanner: Default threshold built into the scanner itself

Statistics calculation:

  • All statistics are calculated from the final aggregated SARIF report
  • Suppressed findings are counted separately and do not contribute to actionable findings
  • Scanner status is determined by comparing actionable findings to the threshold
Scanner S C H M L I Time Action Result Thresh
bandit 0 0 0 0 1 0 1.3s 0 PASSED MED (g)
cdk-nag 0 0 0 0 0 0 6.3s 0 PASSED MED (g)
cfn-nag 0 0 0 0 0 0 10ms 0 PASSED MED (g)
checkov 0 0 0 0 0 0 4.9s 0 PASSED MED (g)
detect-secrets 0 0 0 0 0 0 1.3s 0 PASSED MED (g)
grype 0 0 0 0 0 0 1m 7s 0 PASSED MED (g)
npm-audit 0 0 0 0 0 0 190ms 0 PASSED MED (g)
opengrep 0 0 0 0 0 0 <1ms 0 SKIPPED MED (g)
semgrep 0 0 0 0 0 0 <1ms 0 MISSING MED (g)
syft 0 0 0 0 0 0 2.0s 0 PASSED MED (g)

- Restore the narrow ruff rule selection (E4, E7, E9, F) in the repo's
  root pyproject.toml. Ruff 0.16 expanded its default rule set from
  ~59 to 413 rules; the repo's ignore list was written against the old
  defaults and never set select, so any file touched by a PR now
  fails against hundreds of newly-enabled rules unrelated to this
  repo's actual style conventions. Verified this affects untouched
  pre-existing files too, not just this branch's changes.
- Fix an actual issue caught after restoring the intended ruleset:
  drop redundant f-string prefixes on two string literals with no
  placeholders in teardown.py.
- Apply ruff format to the files this branch touches.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

02-use-cases 02-use-cases

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant