Feat/market trends agent drift detection - #2056
Open
akomandooru wants to merge 6 commits into
Open
Conversation
added 5 commits
September 3, 2026 15:41
- Centralize Bedrock model selection into tools/model_config.py so MODEL_ID can be overridden via environment variable without editing three separate call sites (market_trends_agent.py, browser_tool.py, broker_card_tools.py) - Fix provider resolution for cross-region inference profile IDs (langchain-aws cannot infer provider from a "global." prefix) - Widen the agent execution role's trust policy to allow runtime/* in addition to ab-test/*, fixing "Role validation failed" on startup - Add aws-opentelemetry-distro dependency; without it the agent emitted no spans and AgentCore Evaluations had nothing to score - Add bedrock:InvokeModel permission for the evaluators' execution role so LLM-judge evaluators can actually invoke their judge model - Scope evaluators/iam/permissions-policy.json resources to this account instead of wildcarding the account segment (log reads, Lambda invoke, and Bedrock inference-profile invoke)
Adds a scheduled detector that watches this agent's 8 evaluators over time and alarms when a stream degrades and stays degraded, distinct from evaluate/optimize which compares variants rather than an agent against its own past. - detector/: EWMA and z-score methods with warm-up, persistence, and latching; the method used per evaluator is chosen from its measured score shape rather than applied uniformly - scripts/deploy.py: state table, Lambda, EventBridge schedule, per-evaluator CloudWatch alarms, and a dashboard - scripts/shape_report.py: measures each evaluator's live score shape and flags any evaluator whose configured method disagrees with it - scripts/induce_drift.py: three demo triggers (model_swap, stale_prices, skip_profile_step) to exercise the detector against real, induced drift instead of synthetic data - scripts/traffic.py, watch.py, attach_evaluators.py, teardown.py: supporting operational scripts - market_trends_agent.py, tools/browser_tool.py: agent-side support for the skip_profile_step and stale_prices triggers See drift_detection/README.md for design rationale, the detection method comparison, and measured results from three induced causes of drift on this agent.
Previously defaulted to a hardcoded guess that could silently mismatch the real service.name computed by evaluators/scripts/deploy.py, leaving the dashboard's raw evaluator score widget empty. Now derived the same way (from agentRuntimeArn in .deploy_output.json), with an explicit env var override and a warning if it falls back to the guess.
…ent-drift-detection # Conflicts: # 02-use-cases/01-conversational-agents/market-trends-agent/uv.lock # CONTRIBUTORS.md
|
Latest scan for commit: Security Scan ResultsScan Metadata
SummaryScanner ResultsThe table below shows findings by scanner, with status based on severity thresholds and dependencies: Column Explanations: Severity Levels (S/C/H/M/L/I):
Other Columns:
Scanner Results:
Severity Thresholds (Thresh Column):
Threshold Source: Values in parentheses indicate where the threshold is configured:
Statistics calculation:
|
- Restore the narrow ruff rule selection (E4, E7, E9, F) in the repo's root pyproject.toml. Ruff 0.16 expanded its default rule set from ~59 to 413 rules; the repo's ignore list was written against the old defaults and never set select, so any file touched by a PR now fails against hundreds of newly-enabled rules unrelated to this repo's actual style conventions. Verified this affects untouched pre-existing files too, not just this branch's changes. - Fix an actual issue caught after restoring the intended ruleset: drop redundant f-string prefixes on two string literals with no placeholders in teardown.py. - Apply ruff format to the files this branch touches.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Amazon Bedrock AgentCore Samples Pull Request
Important
review readylabel to it. Only PRs withreview readywill be reviewed.Issue number:
#2055
Concise description of the PR
User experience
Before: no signal exists when the agent's live quality degrades (e.g. after a
model version update with no corresponding deploy event). Degradation is only
found via manual inspection of Evaluations dashboards or user complaints.
After: running drift_detection/scripts/deploy.py stands up a scheduled Lambda,
DynamoDB state table, one CloudWatch alarm per evaluator, and a dashboard. Each
evaluator's score stream is tracked against its own learned baseline, and an
alarm fires naming the specific evaluator that degraded and stayed degraded.
watch.py, shape_report.py, and induce_drift.py give an operator CLI visibility
into detector state and a way to demo drift without waiting for it.
Checklist
If your change doesn't seem to apply, please leave them unchecked.
Acknowledgment
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of the project license.