Skip to content

TEST: Bound scenario estimate concurrency waits - #3273

Merged
Roman Lutz (romanlutz) merged 2 commits into
microsoft:mainfrom
romanlutz:romanlutz-pr-3239-ci-failure-investigation
Oct 16, 2026
Merged

Roman Lutz (romanlutz) merged 2 commits into
microsoft:mainfrom
romanlutz:romanlutz-pr-3239-ci-failure-investigation

Conversation

@romanlutz

Copy link
Copy Markdown
Contributor

Description

Two scenario concurrency tests inherited the production 3-second estimate deadline even though they test task sharing and cancellation, not latency. In #3239's macOS merge-group job, the sharing test took 10.5 seconds and received an unavailable estimate instead of its mocked result. If construction times out before estimation starts, its unbounded started.wait() can also wait indefinitely.

  • Give these two tests a test-only 300-second estimate budget and bounded waits. This is a deadlock backstop, not an added delay; normal execution remains fast.
  • Release blocked estimates and drain caller and service-owned tasks in finally blocks, including after assertions fail.
  • Add a deterministic constructor-expiry regression: block construction, explicitly expire the wait, then release construction and verify estimation never starts and all workers drain.

Production code and deadlines are unchanged. Explicit timeout tests remain separate. Related #3268 hardens analytics budgets and other test watchdogs but does not change these scenario tests. The earlier Windows six-hour stall had no test identity or stack, so this PR does not claim to establish or fix its actual cause.

Tests and Documentation

Local validation on Windows, Python 3.14.4:

  • uv run --no-sync python -m pytest -q -p no:cacheprovider tests\unit\backend\test_scenario_service.py: 78 passed, 9 deprecation warnings, 5.37 seconds.
  • uv run --no-sync ruff check tests\unit\backend\test_scenario_service.py: passed.
  • uv run --no-sync ruff format --check tests\unit\backend\test_scenario_service.py: passed.
  • uv run --no-sync ty check --output-format concise tests\unit\backend\test_scenario_service.py: passed.
  • uv run --no-sync pre-commit run --files tests\unit\backend\test_scenario_service.py: passed.

Temporary, uncommitted pytest probes also exercised both hardened tests with 3.25-second constructor delays and 3.25-second event-loop stalls: 2/2 passed in each mode. Injected assertion failures remained visible, and probe teardown verified no outstanding estimates, tracked workers, or cached estimates. The temporary probes were removed.

Not run locally: the full unit suite or the macOS CI matrix. Documentation/JupyText: N/A, test-only change.

Use test-only estimate budgets, bounded waits, and unconditional task cleanup. Cover expiry during constructor execution without a scheduling-dependent deadline.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 6121f36e-74c0-4fff-9c0c-091c3b6abcef
Raise only the analytics fixture budgets from 30 to 300 seconds and document the finite watchdog. Production and explicit deadline tests are unchanged.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 6121f36e-74c0-4fff-9c0c-091c3b6abcef
@romanlutz
Roman Lutz (romanlutz) added this pull request to the merge queue Oct 16, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Oct 16, 2026
@romanlutz
Roman Lutz (romanlutz) added this pull request to the merge queue Oct 16, 2026
Merged via the queue into microsoft:main with commit f7cda2f Oct 16, 2026
64 checks passed
@romanlutz
Roman Lutz (romanlutz) deleted the romanlutz-pr-3239-ci-failure-investigation branch October 16, 2026 21:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants