Skip to content

TEST: Remove merge-queue correctness-test timing races - #3278

Merged
Roman Lutz (romanlutz) merged 2 commits into
microsoft:mainfrom
richlundeen:richlundeen-queue-test-reliability
Oct 17, 2026
Merged

Roman Lutz (romanlutz) merged 2 commits into
microsoft:mainfrom
richlundeen:richlundeen-queue-test-reliability

Conversation

@richlundeen

Copy link
Copy Markdown
Contributor

Description

Fix the unrelated test timing failures seen while #3221 was in the merge queue. This PR starts from current main. It does not change #3221 or production code.

Failed run Evidence Change
38952712158 macOS Python 3.12: the progress-cache handoff test expired its five-second wait_for(gather(...)); the traceback shows the gather finished with valid snapshots. Keep event-controlled ordering and every cache, cursor, lock, and cleanup assertion. Await owned tasks directly, with one finite 300-second test watchdog instead of per-step latency limits. Only the test's explicit owner cancellation can cancel the owner.
38953955155 The direct-link modal test found the destination route, but the background was still aria-hidden. Its immediate button query found no accessible roles. Use fake timers and await the accessible destination button. Keep the dialog-close, route, visibility, and expanded-state assertions. Restore real timers in finally.
38970854575 macOS Python 3.11: fresh launch exceeded the ten-second idle polling wait; the SDK-equivalence query exceeded the test fixture's 30-second report budget. Other test setup took 52.83 seconds on this runner. Fresh launch uses the task-completion helper already added by #3207. Extend its event-controlled terminal-handoff regression to both fresh and resumed launches. Give the backend analytics correctness fixture the same finite 300-second budgets used by analytics correctness tests.

#3138 and #3207 are already merged. The fresh-launch metadata test still used the old polling helper. Reuse the completion helper rather than add another polling deadline.

The analytics fixture retains real SDK execution, SQL, serialization, admission, cancellation, and cleanup. Production request deadlines do not change. The explicit timeout test still forces expiry through its controlled event and checks the 504 response and retained cleanup ownership.

Open #3268 overlaps the analytics fixture and its coupled test assertion. It also changes the old resume polling helper. This PR does not depend on it or include its wider CI changes; the fresh-launch test no longer uses that polling helper.

Tests and Documentation

Local validation on Windows, using the worktree environment from uv sync --frozen --extra all:

Command Result
uv run --no-sync pytest -q -n 4 --dist=loadfile tests\unit\backend 2726 passed, 7 skipped
uv run --no-sync pytest -q tests\unit\backend\test_scenario_resume.py -k 'run_completion_waits_for_terminal_handoff or fresh_launch_saves_only_nonsecret' 3 passed, including both handoff modes
uv run --no-sync pytest -q tests\unit\analytics\test_execution.py 78 passed, including deadline and cancellation contracts
From frontend: npm test -- --runInBand --runTestsByPath src/components/Scenarios/ScenarioRunPage.test.tsx --silent 63 passed
From frontend: npm run type-check Passed
From frontend: npx eslint src/components/Scenarios/ScenarioRunPage.test.tsx --max-warnings 0 Passed
uv run --no-sync ty check --output-format concise tests\unit\backend\conftest.py tests\unit\backend\test_analytics_lifecycle.py tests\unit\backend\test_scenario_resume.py tests\unit\backend\test_scenario_progress_read_model.py Passed
With UV_NO_SYNC=1: uv run --no-sync pre-commit run --all-files All hooks passed

A temporary, uncommitted progress-test probe expired the old gather wait without a real sleep. All three baseline exit modes raised TimeoutError; all three changed modes passed the full assertions.

The frontend failure was diagnosed from the CI DOM and trace. A local immediate-query probe did not reproduce it. The changed test passed with fake timers and the normal accessible-role query.

doc/contributing/4_running_tests.md now describes task-completion waits, finite correctness budgets, and deferred modal accessibility. JupyText is not applicable; no notebooks or executable documentation samples changed. The macOS CI suite has not been run locally.

Await scenario task completion and modal accessibility. Use finite correctness watchdogs without changing production deadlines.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 0c4473a3-e3cd-44e2-9b2c-f11dd78a19de
@azure-pipelines

Copy link
Copy Markdown
There was an error handling pipeline event 2f349945-6948-4214-9c59-4ea8e5444760.

@romanlutz
Roman Lutz (romanlutz) added this pull request to the merge queue Oct 17, 2026
Merged via the queue into microsoft:main with commit 7c21a4b Oct 17, 2026
64 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants