Repository navigation
TEST: Stabilize target startup and analytics deadline tests - #3268
Merged
Roman Lutz (romanlutz) merged 2 commits intoOct 17, 2026
Merged
Roman Lutz (romanlutz) merged 2 commits into
Roman Lutz (romanlutz) merged 2 commits into
Conversation
The macOS build_and_test leg failed 80 times in two days; 71 were single tests whose short watchdogs fired while the 3-vCPU runner stalled for 20-68s, not genuine hangs. - Add TEST_WATCHDOG_SECONDS (300s) in tests/unit/async_utils.py as the shared deadlock backstop and use it for positive test watchdogs, analytics query budgets, and the pytest-timeout markers that capped those waits. - Make the discover-capabilities timeout variant deterministic: expire the probe deadline through the real wait_for cancellation path once the request is pending, instead of a 5s wall-clock deadline that a stalled runner could exhaust before the request started. - Reuse the shared analytics control in the eval-identity tests. - Run 3 xdist workers on macOS via a PYTEST_WORKERS Makefile override. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
varunj-msft
approved these changes
Oct 16, 2026
This was referenced Oct 17, 2026
Keep the targeted Copilot startup and discovery probe fixes, freeze analytics lifecycle deadline clocks, and drop superseded broad watchdog and CI-worker changes. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 67cf8740-114f-4201-b752-812381bbc33b
Roman Lutz (romanlutz)
approved these changes
Oct 17, 2026
Contributor
|
hannahwestra25 I took the liberty to update this after #3279 merged. The more of these fixes go in and the sooner the better. |
This was referenced Oct 17, 2026
Roman Lutz (romanlutz)
added a commit
to richlundeen/PyRIT
that referenced
this pull request
Oct 17, 2026
Preserve the watchdog, request-entry synchronization and task cleanup merged in microsoft#3268. Keep only the returned JSON capability assertions as the remaining delta. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: e52c81e0-1386-4099-9cfc-b52c7632320f
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Merged current
main(ba20617d5eaddf8ae302d7f32454296ba50a9cc7) and narrowed this PR after #3278 landed. The diff is now five files instead of 21. It preserves #3278's completion-wait and analytics-budget fixes, #3221's frozen analytics-identity clocks, and #3255's deterministic SQLite contention checks.Post-#3278 CI still exposed two target-test timing failures and one separate analytics assertion failure:
test_cancelled_send_preserves_cancellation_when_retirement_fails_asyncexceeded its two-second wait for the mocked send to start.asyncio.wait_forcancellation path. Assert cancellation occurred and drain the owned task on every exit.assert 300.0000000000002 <= 300.The shared
TEST_WATCHDOG_SECONDSnames the existing 300-second backstop;wait_for_completion_asyncretains its existing default and behavior.Removed the blanket watchdog increases, changes to unrelated correctness-query budgets, and the Makefile/workflow worker override. CI still uses four workers. The distinct five-second scorer-cancellation cleanup check is preserved.
No production code, dependencies, provider settings, retries, matrix policy, or JUnit/reporting behavior changes. The targeted startup waits remain finite, but a genuine startup deadlock can now take five minutes to fail. These changes do not establish that reducing workers would improve reliability or that every historical failure was a runner stall.
Tests and Documentation
Local validation on Windows with Python 3.14.4, after
uv sync --frozen --extra github-copilot --link-mode=copy:Result: 319 passed, with 32 existing deprecation warnings. This includes the complete changed test modules, analytics deadline/cancellation contracts, and related HTTP route tests.
Results: all passed; all four Python files were already formatted.
A temporary, uncommitted pytest plugin also checked the actual baseline and revised test bodies under identical bounded conditions: a three-second mocked Copilot startup, six-second mocked discovery startup, and a frozen clock where
(268.72848822480245 + 300) - 268.72848822480245rounds above 300. The five baseline cases intentionally failed with the expected timeout, zero-await, or budget-assertion signatures; all five revised cases passed. No real providers or live workflows were invoked.doc/contributing/4_running_tests.mddescribes the targeted watchdogs and frozen deadline clocks. JupyText is not applicable because no notebooks or executable documentation samples changed. The full unit suite, all-extras environment, and macOS execution were not run locally.