Trim the split-build-dir harness test, or decide it is not worth the fidelity

[leynos]

harness_compiles_under_a_split_build_dir is the last heavy test on the Windows
gate. It spawns a Cargo build of test_support, which depends on
netsuke-build, so it compiles the whole crate plus roughly 350 dependencies
into private directories to prove two things: that the collected
-L dependency= set spans a split build.build-dir, and that a control
fixture type-checks against the resulting rlib.

What it costs now

The figure has moved, and the change is the point of filing this rather than
building it.

Before #687, the test ran a median of 274.7s and its exclusive tail, the period
after every other test had reported, implied that trimming both isolated-Cargo
tests was worth 156s. #687 removed the other one from Windows, and the two had
been throttling each other on a four-vCPU runner, so this test got faster
without being touched.

Measured on the three runs after #687:

run	test duration	exclusive tail
34075197897	125.3s	62.3s
34079222917	170.6s	87.2s
34080385050	170.7s	85.5s

So trimming it to near zero is now worth about 85s, not 156s. Nobody should
start this expecting the larger number.

For scale, the whole Windows lane fell from a 1468s median to about 848s across
#687, #690 and #691. 85s is roughly 10 percent of what remains.

The design, if it is built

Build a minimal fixture crate under the split layout instead of test_support.
It needs at least one dependency so that dependency rlibs land in the split
build directory while the fixture's own uplifted rlib lands in the target
directory, which is the arrangement the regression exists to catch: a single
derived directory that missed the dependencies entirely.

Two things it must carry, and they are the hard part:

A fidelity argument in a doc comment, stating exactly which regression it
guards and why a fixture crate suffices where the real test_support build was
used before. The current test is a regression test for a defect that was found
once; replacing its subject with a stand-in weakens it unless the argument is
explicit about what is and is not still covered.

The Windows response-file pressure. TestSupportRlib::compile passes its
arguments through a rustc response file because Cargo 1.99 gives every crate
its own artefact directory, and this test adds long temporary roots on top, so
passing them directly exceeds the Windows CreateProcess limit and fails with
Os { code: 206 }. A fixture crate with one dependency produces far fewer
directories and would stop exercising that. Either the fixture must generate
enough search paths to keep the pressure, or the response-file contract needs
its own test and the doc comment must say so.

What was already ruled out

• cargo check instead of cargo build. Timed cold at -j 4 on a
  32-core host: 114s against 102s, twelve percent. It also writes nothing into
  the target directory, so the uplift the regression exists to catch stops
  happening and the test passes vacuously.
• Reusing the lane's compiler cache. It already reaches the spawned build;
  ci-windows.yml sets RUSTC_WRAPPER at job scope and the test adds to the
  child environment rather than clearing it. Warming it is worth about 3
  percent.
• Sharing a target directory. The test needs private roots to avoid racing
  the #[once] fixture with E0460.

When to revisit

Wait until ten runs of the split lane exist, so the harness test's share of the
build-test-windows job is known under the new shape rather than estimated from
three runs. If its tail has settled below the 85s measured here, or if
Test has stopped being the lane's critical path, this is not worth the
fidelity risk.

Related

• Measure and reduce the Windows cost of the isolated-Cargo-build tests #673, which measured the lane
• Measure the Windows cost of the isolated-Cargo-build tests #687, which relocated the packaging verification build and changed this
  test's cost
• Fold the Windows native-recipe smoke job into the gate job #690, which folded the native-recipe smoke job into the gate job
• Split the Windows lints and tests into parallel jobs #691, which splits the Windows lints from the tests

[leynos]

@coderabbitai are all of the tests that require a full rebuild of netsuke-build now isolated in a separate nextest group allowing them to monopolize all cores on the runner?

[coderabbitai]

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo '=== Issue 693 ==='
gh issue view 693 --repo leynos/netsuke --comments

echo '=== Nextest configuration candidates ==='
fd -HI 'nextest.*|.*nextest.*|config\.toml' . \
  -E .git -E target -E node_modules | sort

echo '=== netsuke-build references in tracked test/config files ==='
rg -n -i -C 4 \
  -g '*.rs' -g '*.toml' -g '*.yml' -g '*.yaml' -g '*.md' \
  -g '!target/**' \
  'netsuke-build|nextest.*group|test-group' .

Length of output: 53774

---

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo '=== .config/nextest.toml ==='
cat -n .config/nextest.toml

echo '=== Tests and support code that create isolated Cargo build roots or invoke Cargo build/test ==='
rg -n -i -C 8 \
  -g '*.rs' -g '!target/**' \
  '(CARGO_TARGET_DIR|CARGO_HOME|target[-_ ]dir|split[_ -]build|private[_ -]root|isolated.*(?:cargo|build)|Command::new\(.*cargo|\.arg\("build"\)|\.arg\("test"\))' \
  tests test_support

echo '=== Developer-guide section on the private build ==='
sed -n '4040,4100p' docs/developers-guide.md

Length of output: 97002

---

@leynos, no.

.config/nextest.toml defines no [test-groups] entry and no test-group override. Therefore, no test can reserve the runner cores through a nextest group.

The two isolated Cargo-build tests are:

• harness_compiles_under_a_split_build_dir
• packaged_manifest_retains_build_script_sources

They only receive immediate successful output. The first also receives a Windows-specific slow-timeout override.

Both tests run with normal nextest concurrency. They do not monopolize all runner cores.

You are interacting with an AI system.

[leynos]

@coderabbitai would doing so be likely to improve performance?

[coderabbitai]

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo '=== Issue 693 ==='
gh issue view 693 --repo leynos/netsuke --json number,title,body,comments

echo '=== Nextest configuration ==='
cat -n .config/nextest.toml

echo '=== CI references to nextest and test threads ==='
rg -n -i -C 5 \
  -g '*.yml' -g '*.yaml' -g '*.toml' -g '*.sh' -g '*.md' \
  -g '!target/**' \
  '(cargo[[:space:]]+nextest|nextest[[:space:]]+run|test-threads|jobs|threads|NEXTEST)' .

echo '=== Candidate test definitions ==='
rg -n -C 18 \
  -g '*.rs' -g '!target/**' \
  'fn[[:space:]]+(harness_compiles_under_a_split_build_dir|packaged_manifest_retains_build_script_sources)\b' .

Length of output: 200346

---

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo '=== .config/nextest.toml ==='
cat -n .config/nextest.toml

echo '=== Cargo rebuild test scheduling and measurements ==='
sed -n '2654,2771p' docs/developers-guide.md

echo '=== Full bodies of the two tests ==='
sed -n '233,310p' tests/locale_stub_ui_tests.rs
sed -n '170,260p' tests/packaging_smoke_tests.rs

Length of output: 17277

---

@leynos, do not add a nextest group for this purpose.

A nextest test group limits concurrency within that group. It does not reserve runner cores or prevent tests outside the group from running.

The major contention source no longer exists on Windows. packaged_manifest_retains_build_script_sources now skips Cargo’s verification build there. It takes 4.2–7.6 seconds. The remaining expensive test, harness_compiles_under_a_split_build_dir, then fell from a 274.7-second median to 125.3 and 170.6 seconds. The full nextest phase fell from 365 seconds to 185.5 and 258.0 seconds.

A group that serializes the remaining build could reduce variance if concurrent short tests still contend materially for CPU, disk, or the compiler cache. It could also increase total wall time by leaving cores idle while that test waits for or blocks other tests.

The Linux package-verification build is outside nextest configuration because the coverage runner executes it. A nextest group cannot improve that lane.

Measure an A/B run before changing configuration. Compare at least ten Windows runs with the current configuration against a configuration that runs the split-build test without concurrent nextest tests. Retain the change only if it reduces the Test step’s median and tail duration.

You are interacting with an AI system.