Skip to content

TEST: Stabilize target startup and analytics deadline tests - #3268

Merged
Roman Lutz (romanlutz) merged 2 commits into
microsoft:mainfrom
hannahwestra25:hannahwestra25-macos-ci-reliability
Oct 17, 2026
Merged

Roman Lutz (romanlutz) merged 2 commits into
microsoft:mainfrom
hannahwestra25:hannahwestra25-macos-ci-reliability

Conversation

@hannahwestra25

@hannahwestra25 hannahwestra25 commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

Description

Merged current main (ba20617d5eaddf8ae302d7f32454296ba50a9cc7) and narrowed this PR after #3278 landed. The diff is now five files instead of 21. It preserves #3278's completion-wait and analytics-budget fixes, #3221's frozen analytics-identity clocks, and #3255's deterministic SQLite contention checks.

Post-#3278 CI still exposed two target-test timing failures and one separate analytics assertion failure:

Evidence Targeted change
macOS Python 3.13 merge-group job: test_cancelled_send_preserves_cancellation_when_retirement_fails_async exceeded its two-second wait for the mocked send to start. Use the existing finite 300-second deadlock-backstop convention for that startup wait only. Keep the separate two-second cancellation-completion assertion unchanged.
macOS Python 3.14 merge-group job: the discovery provider-state test's success variant reported "Awaited 0 times" with a five-second probe deadline that also covers preceding memory access. Give this test a finite 300-second correctness backstop. For its timeout variant, wait until the mocked request is pending, then expire it through the real asyncio.wait_for cancellation path. Assert cancellation occurred and drain the owned task on every exit.
Windows Python 3.12 post-merge job: the event-controlled analytics lifecycle test failed on assert 300.0000000000002 <= 300. Freeze both deadline clocks in that test and assert the exact 300-second budget. Expiration still comes from its explicit event, with the existing HTTP 504, busy-capacity, repeated-cancellation, and cleanup assertions intact.

The shared TEST_WATCHDOG_SECONDS names the existing 300-second backstop; wait_for_completion_async retains its existing default and behavior.

Removed the blanket watchdog increases, changes to unrelated correctness-query budgets, and the Makefile/workflow worker override. CI still uses four workers. The distinct five-second scorer-cancellation cleanup check is preserved.

No production code, dependencies, provider settings, retries, matrix policy, or JUnit/reporting behavior changes. The targeted startup waits remain finite, but a genuine startup deadlock can now take five minutes to fail. These changes do not establish that reducing workers would improve reliability or that every historical failure was a runner stall.

Tests and Documentation

Local validation on Windows with Python 3.14.4, after uv sync --frozen --extra github-copilot --link-mode=copy:

uv run --frozen --no-sync -m pytest -n 4 --dist=loadfile -q -p no:cacheprovider --tb=short tests\unit\backend\test_analytics_lifecycle.py tests\unit\analytics\test_execution.py tests\unit\backend\test_analytics_routes.py tests\unit\prompt_target\target\test_github_copilot_target.py tests\unit\prompt_target\test_discover_target_capabilities.py

Result: 319 passed, with 32 existing deprecation warnings. This includes the complete changed test modules, analytics deadline/cancellation contracts, and related HTTP route tests.

uv run --frozen --no-sync ruff check tests\unit\async_utils.py tests\unit\backend\test_analytics_lifecycle.py tests\unit\prompt_target\target\test_github_copilot_target.py tests\unit\prompt_target\test_discover_target_capabilities.py
uv run --frozen --no-sync ruff format --check tests\unit\async_utils.py tests\unit\backend\test_analytics_lifecycle.py tests\unit\prompt_target\target\test_github_copilot_target.py tests\unit\prompt_target\test_discover_target_capabilities.py
uv run --frozen --no-sync -m ty check --output-format concise tests\unit\async_utils.py tests\unit\backend\test_analytics_lifecycle.py tests\unit\prompt_target\target\test_github_copilot_target.py tests\unit\prompt_target\test_discover_target_capabilities.py
uv run --frozen --no-sync -m build_scripts.validate_docs
git diff --cached --check

Results: all passed; all four Python files were already formatted.

A temporary, uncommitted pytest plugin also checked the actual baseline and revised test bodies under identical bounded conditions: a three-second mocked Copilot startup, six-second mocked discovery startup, and a frozen clock where (268.72848822480245 + 300) - 268.72848822480245 rounds above 300. The five baseline cases intentionally failed with the expected timeout, zero-await, or budget-assertion signatures; all five revised cases passed. No real providers or live workflows were invoked.

doc/contributing/4_running_tests.md describes the targeted watchdogs and frozen deadline clocks. JupyText is not applicable because no notebooks or executable documentation samples changed. The full unit suite, all-extras environment, and macOS execution were not run locally.

The macOS build_and_test leg failed 80 times in two days; 71 were single
tests whose short watchdogs fired while the 3-vCPU runner stalled for
20-68s, not genuine hangs.

- Add TEST_WATCHDOG_SECONDS (300s) in tests/unit/async_utils.py as the
  shared deadlock backstop and use it for positive test watchdogs,
  analytics query budgets, and the pytest-timeout markers that capped
  those waits.
- Make the discover-capabilities timeout variant deterministic: expire
  the probe deadline through the real wait_for cancellation path once
  the request is pending, instead of a 5s wall-clock deadline that a
  stalled runner could exhaust before the request started.
- Reuse the shared analytics control in the eval-identity tests.
- Run 3 xdist workers on macOS via a PYTEST_WORKERS Makefile override.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Comment thread tests/unit/memory/memory_interface/test_interface_observations.py Outdated
Comment thread tests/unit/executor/attack/test_scoring_expectation_transport.py Outdated
Keep the targeted Copilot startup and discovery probe fixes, freeze analytics lifecycle deadline clocks, and drop superseded broad watchdog and CI-worker changes.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 67cf8740-114f-4201-b752-812381bbc33b
@romanlutz Roman Lutz (romanlutz) changed the title TEST: Make macOS unit-test gate resilient to runner stalls TEST: Stabilize target startup and analytics deadline tests Oct 17, 2026
@romanlutz

Copy link
Copy Markdown
Contributor

hannahwestra25 I took the liberty to update this after #3279 merged. The more of these fixes go in and the sooner the better.

@romanlutz
Roman Lutz (romanlutz) added this pull request to the merge queue Oct 17, 2026
Merged via the queue into microsoft:main with commit 8283aac Oct 17, 2026
64 checks passed
Roman Lutz (romanlutz) added a commit to richlundeen/PyRIT that referenced this pull request Oct 17, 2026
Preserve the watchdog, request-entry synchronization and task cleanup merged in microsoft#3268. Keep only the returned JSON capability assertions as the remaining delta.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: e52c81e0-1386-4099-9cfc-b52c7632320f
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants