ci: evidence runs under 2 minutes on every platform

Evidence runs should finish in under 2 minutes on every platform when the remote cache is warm, and in under 1 minute on Linux.

Measure each step first (runner start, checkout, setup, Bazel start, fetch, analysis, build, test, upload) on all five platforms with //cli/process/.... Then work through these, keeping only what helps:

Fetch only what the scope needs. Make every heavy toolchain repo lazy (LLVM, JDK, Android, OCaml, emsdk, Go, Python, Node), so an unrelated scope never downloads it.
Download through BuildBuddy's remote downloader (--experimental_remote_downloader), so repo archives come from the cache instead of upstream.
Use --remote_download_minimal in CI so cached outputs stay remote.
Skip work that a cache hit makes pointless: no upload of unchanged results, no extra Bazel commands per job (validate, info, query).
Windows: output root on D:, no server restarts, fewer startup option changes between commands.
Split wide scopes across parallel jobs when one job would pass 2 minutes.

Done when, with a warm cache, a //cli/process/... evidence run takes under 2 minutes of job time on linux_x86_64, linux_arm64, macos_arm64, windows_x86_64 and windows_arm64, with Linux under 1 minute. Put the before and after table in the PR.

PR #8917 (stacked on #8908): converters only for rst/asciidoc sources, BuildBuddy remote downloader, --remote_download_minimal in evidence, one Bazel server per job.

Evidence, scope //cli/process/..., task test. Cold 42315723528 and warm 42316556203 both green on all five platforms.

platform	cold job / Bazel elapsed	warm job / Bazel elapsed
linux_x86_64	146s / 133s	90s / 79s
linux_arm64	130s / 121s	103s / 92s
macos_arm64	184s / 157s	120s / 96s
windows_x86_64	183s / 157s	184s / 155s
windows_arm64	437s / 380s	364s / 317s

Gap to target (under 2 min, linux under 1 min): linux_x86_64 is 30s over 1 min, windows_x86_64 64s over 2 min, windows_arm64 244s over 2 min. Critical path is under 9s everywhere, so analysis dominates: about 86k configured targets on Linux and 96k on Windows for 7 top-level targets. Next levers: #8914 (Android SDK hub), #8915 (rust_host_tools on every cold start), and reducing what the scope analyses / splitting wide scopes. Leaving this open.

ci: pass the remote repo contents cache as a startup option (part of #…

PR #8935 (remote repo contents cache + --jobs=64 in evidence). Seed run 42318234269, warm run 42319132533, baseline 42316556203; scope //cli/process/....

Warm (job s / Bazel s), before -> after:

platform	before	after
linux_x86_64	90 / 79	107 / 94
linux_arm64	103 / 92	99 / 88
macos_arm64	120 / 96	110 / 85
windows_x86_64	184 / 155	100 / 79
windows_arm64	364 / 317	196 / 139

windows_x86_64 meets the target. Linux, macOS and windows_arm64 do not: about 1150 repos are still fetched per job (mostly npm through @dx_npm_link, plus dx_androidsdk and llvm-project).

Remaining levers: #8931 (keep npm out of Rust scopes), #8914 (dx_androidsdk), #8915 (crate extension), #8932 (reproducible MSVC layout), and splitting wide scopes.

quality: real_source_target depends on converters only for rst and as…

ci: download repository archives through the BuildBuddy remote downlo…

ci: evidence build and test leave cached outputs remote

ci: evidence jobs start Bazel once

ci: lazy converters, remote downloader, minimal downloads and one Baz…

ci: evidence reuses fetched repos from the remote repo contents cache

ci: pass the remote repo contents cache as a startup option (part of #…

ci: evidence reuses fetched repos from the remote repo contents cache…

ci: make reusable workflow credential steps match setup-bazel (part of …

PR #9066 (parallel git checkout on Windows evidence jobs), round 2 evidence on //cli/process/...: main 42346380877, branch 42346382734, all green.

windows_arm64: job 124s -> 122s, checkout 26s -> 25s (incl. the 1s new step)
windows_x86_64: job 97s -> 92s, checkout 12s -> 11s

The gain is small. Still open: windows_arm64 is about 2s over 2 minutes, Linux is not under 1 minute yet. The next lever on windows_arm64 is splitting the scope (item 6).

🤖 Generated with Claude Code

https://claude.ai/code/session_01Xut2hzpb4qWNZKpDCrwUz7

Evidence for PR #9207 (notrack flags in evidence.yml only) passed:

42369262003: task=test //cli/process/..., all 5 platforms succeeded.
42369269569: task=build relocate=true, linux_x86_64 succeeded (determinism path).

PR #9207 is now ready for review. Still open for this issue: hot spots #9201, #9202, #9203, and scope splitting (item 6) as the fallback for windows_arm64.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Xut2hzpb4qWNZKpDCrwUz7

#9135 closed: Merkle 49.9 to 11.2 core-s, windows_arm64 job 92s to 86.5s (PR #9138); the rest is under 2s, see #9135.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Xut2hzpb4qWNZKpDCrwUz7

ci: reusable-consumer and msix-install check out with plain git on Wi…

evidence: analyse with --notrack_incremental_state --nokeep_state_aft…

modules: families.json drops removed unused aspects and lists japicmp…

cuda: regenerate third_party/rules_dx (part of #8912)

quality runner: rustfmt the real module (part of #8912)

tests: fix quality and modules tests red on main (#9192)

Evidence for #9262 (rules_dotnet toolchain_type moved to //dotnet/toolchain), scope //cli/process/..., task=build, two warm pairs. Times are Bazel Elapsed time.

Platform	Pair	Branch	Main	Delta
linux_x86_64	1 (42371503269 / 42371513098)	34.30s	29.72s	+4.58s
linux_x86_64	2 (42371517954 / 42371522891)	29.62s	34.10s	-4.48s
linux_x86_64	mean	31.96s	31.91s	+0.05s
windows_x86_64	1	48.06s	43.94s	+4.12s
windows_x86_64	2	37.38s	35.73s	+1.65s
windows_x86_64	mean	42.72s	39.84s	+2.88s

All four runs passed. Both branches load the same number of packages (426 on Linux, 398 on Windows), since //dotnet/toolchain takes the place of //dotnet. Linux comes out even and Windows is slower, so the change fails the >=2s gate. I am closing #9262 without merging.

#9201 stays open until upstream bazel-contrib/rules_dotnet#598 (issue #597) lands. The #9203 upstream issues are rules_go#4779 and rules_py#1675.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Xut2hzpb4qWNZKpDCrwUz7

Round 8 result (#9105, PR #9270): I fixed the hidden-lockfile cache from #9119 and seeded it, then measured it. It does not pay, so PR #9270 removes it.

Scope //cli/process/..., task=test, all five platforms. The four runs ran at the same time. Branch head ab7e42f had the order fix plus a temporary seed step, and every branch job logged Cache restored from key: dx-ext-lock-....

Pair 1: main 42372790531, branch 42372804674
Pair 2: main 42372826347, branch 42372839830

setup-bazel + "Run evidence task" step (s)

platform	main p1, p2	branch p1, p2	main median	branch median	delta
windows_arm64	57, 56	54, 54	56.5	54.0	-2.5
windows_x86_64	61, 35	41, 40	48.0	40.5	-7.5 (main p1 outlier)
macos_arm64	36, 33	38, 37	34.5	37.5	+3.0
linux_arm64	36, 32	33, 35	34.0	34.0	0.0
linux_x86_64	40, 33	38, 38	36.5	38.0	+1.5

Bazel elapsed for the evidence task (s)

platform	main p1, p2	branch p1, p2	main median	branch median	delta
windows_arm64	46.3, 44.8	42.1, 41.7	45.5	41.9	-3.6
windows_x86_64	53.1, 28.4	33.7, 32.1	40.7	32.9	-7.8 (main p1 outlier)
macos_arm64	29.0, 27.3	29.4, 30.2	28.1	29.8	+1.7
linux_arm64	31.1, 26.6	26.1, 29.7	28.8	27.9	-0.9
linux_x86_64	33.8, 25.6	31.7, 32.3	29.7	32.0	+2.3

windows_arm64 clears the 2s bar, but macos_arm64 (+3.0s step time) and linux_x86_64 (+2.3s Bazel elapsed) are 2s or more slower. This matches the parse-cost risk from #9097 for a 12 MB hidden lockfile. The cache does not pay, so it is removed.

Next candidate: on Windows, @local_config_apple_cc_toolchains takes 2.3-4.8s of fetch time.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Xut2hzpb4qWNZKpDCrwUz7

Step breakdown of the latest evidence.yml runs on main: 42372790531 and 42372826347, scope //cli/process/..., task=test. Times are in seconds, run1/run2, taken from gh run view --json jobs.

platform	job	checkout	setup-bazel	evidence task	upload
windows_arm64	81/75	15/11	11/10	46/46	2/2
windows_x86_64	71/44	5/4	7/6	54/29	1/1
linux_x86_64	49/39	3/2	6/7	34/26	2/1
linux_arm64	42/38	2/2	5/5	31/27	1/1
macos_arm64	49/43	3/3	7/6	29/27	3/1

windows_x86_64 differs by 27s between the two runs (71 vs 44), almost all of it in the evidence task (54 vs 29). That is runner noise.
Checkout split (fetch / git checkout): windows_arm64 2.4 / 9.1, windows_x86_64 3.2 / 1.0, linux 1.7 / 0.3. The windows_arm64 checkout is the biggest step outside Bazel. Defender real-time scanning causes it, and Tamper Protection on that image blocks turning it off ([Possible Bug] Windows Defender may not be properly disabled actions/runner-images#14326).
setup-bazel on windows_arm64: action download 1.0, setup-bazel 1.4, registry restore 2.8, bazel info server start 3.0, lockfile restore 0.9. PR ci: drop the hidden extension lockfile cache, it does not pay (closes #9105) #9270 removes the last two. The server start is moved, not cut. No other part can save 2s.

Already measured and rejected:

Defender Add-MpPreference exclusions: slower, 20-22s (PR ci: evidence checks out with plain git, skips jq on Windows and excludes Defender on arm64 (part of #8912) #9096).
Set-MpPreference -DisableRealtimeMonitoring: reports success but does nothing on windows-11-arm (runner-images#14326).
Dev Drive: +24-27s per job (ci: windows_arm64 checkout takes 14-15s because Defender scans each file #9101).
Sparse checkout: at most 26% of files can be skipped (ci: evidence sparse checkout of only the directories the scope loads #9102).
actions/checkout: at least 2s slower (evidence.yml: use actions/checkout with parallel checkout like ci.yml #9115).
Bazel prefetch or installing Bazelisk directly: 1.4-1.8s, under the 2s bar (ci: windows_arm64 Bazel download and extraction takes 6-7s in setup-bazel #9099, ci: setup-bazel installs Bazelisk directly instead of through bazel-contrib/setup-bazel #9104).
Dropping the registry cache: about 0.7s (ci: windows_arm64 Bazel download and extraction takes 6-7s in setup-bazel #9099).
upload-artifact tuning: the step takes 1-3s, so it cannot save 2s.

Every job is under 120s and every Linux job is under 60s. Whether this issue is done is the owner's call. The only lever left is the Bazel evidence task (#9201, #9202, #9203, all upstream). Revisit the windows_arm64 checkout only if runner-images ships the image with Tamper Protection off. Measure setup-bazel again after #9270 merges before proposing any cut there.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Xut2hzpb4qWNZKpDCrwUz7

Done. Every warm evidence job is under 2 minutes, and every Linux job is under 1 minute. Latest jobs on main, //cli/process/... (seconds, two runs):

platform	job
linux_x86_64	49 / 39
linux_arm64	42 / 38
macos_arm64	49 / 43
windows_x86_64	71 / 44
windows_arm64	81 / 75

Every option still open is listed in the breakdown above and saves under 2s. The cuts that are left depend on upstream fixes, and we patch them here: #9201, #9202, #9203.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Xut2hzpb4qWNZKpDCrwUz7

#9293 keeps the local llvm patch for hermetic-llvm#925 because it pays on Linux.

Evidence on //cli/process/... (task=test). The baseline is fix-5152-usrepo (#9282). Pair 1 is branch 42374171056 vs base 42374180449; pair 2 is branch 42374183251 vs base 42374185584. All four runs were dispatched together, so they ran at the same time.

Platform	Targets configured (base -> branch)	Pair 1 elapsed (base -> branch)	Pair 2 elapsed, warm (base -> branch)
linux_x86_64	83980 -> 71214	31.7s -> 25.9s (-5.8s)	23.4s -> 19.5s (-3.9s)
windows_x86_64	91991 -> 91991	44.9s -> 47.7s (+2.8s)	46.5s -> 36.9s (-9.5s)
windows_arm64	92767 -> 92767	47.7s -> 44.4s (-3.3s)	43.0s -> 42.1s (-0.9s)

The gain is real on Linux: about 12.8k fewer configured targets and about 4s or more faster in both pairs. On Windows the number of configured targets does not change, because clang-cl does not go through bootstrap_transition, so the Windows timing differences are noise. No platform is slower in the warm pair, so the gate passes (>=2s on one platform, no slower elsewhere).

🤖 Generated with Claude Code

https://claude.ai/code/session_01Xut2hzpb4qWNZKpDCrwUz7

Round 9 result (#9203, PR #9303): the rules_go root_sdks_only patch and the rules_py platform_flags_package patch. They do what they should, but Windows is much slower, so this is dropped. PR #9303 is closed.

Scope //cli/process/..., task=test, linux_x86_64 and windows_x86_64. All four runs started at the same time and all passed. Branch head 62ea925.

Pair 1: main 42374830074, branch 42374831990
Pair 2: main 42374833810, branch 42374835892

setup-bazel + "Run evidence task" step (s)

platform	main p1, p2	branch p1, p2	main median	branch median	delta
windows_x86_64	40, 36	55, 56	38.0	55.5	+17.5
linux_x86_64	35, 37	34, 30	36.0	32.0	-4.0

Bazel elapsed for the evidence task (s)

platform	main p1, p2	branch p1, p2	main median	branch median	delta
windows_x86_64	34.8, 31.8	48.5, 51.2	33.3	49.8	+16.5
linux_x86_64	30.3, 32.5	30.1, 25.0	31.4	27.5	-3.9

What the patches do:

Targets configured drop by about 40%: linux goes from 83,980 to 50,716 and windows from 91,991 to 55,703. Packages loaded stay the same (426 and 398).
In the windows profiles, package creation for @rules_py+//uv/private/constraints/platform (0.79s) and @go_toolchains// (0.37s) is gone.

Why windows is slower, from the p1 profiles (main vs branch):

The whole branch timeline is slower, including module resolution before any patch could matter: compute main repo mapping takes 5.1s vs 2.2s.
Both fetch the same repos and evaluate the same module extensions, yet repo fetching totals 224.5s vs 166.7s.
Neither profile shows a cache miss or an extra extension evaluation that would explain it. But both branch pairs were slower by the same amount, so I can't call it runner noise.

linux clears the 2s bar, but windows_x86_64 is more than 2s slower on both measures, so the change does not meet the rule. #9203 stays blocked until rules_go#4779 and rules_py#1675 land upstream. The patches are still on branch issue-9203-toolchain-patches if someone wants to measure again.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Xut2hzpb4qWNZKpDCrwUz7

Navigation Menu

Repository navigation

ci: evidence runs under 2 minutes on every platform #8912

44 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

ci: evidence runs under 2 minutes on every platform #8912

Description

Activity

TrapsterDK commented on Nov 7, 2026

TrapsterDK commented on Nov 7, 2026

TrapsterDK commented on Nov 7, 2026

44 remaining items

TrapsterDK commented on Nov 7, 2026

TrapsterDK commented on Nov 7, 2026

TrapsterDK commented on Nov 7, 2026

TrapsterDK commented on Nov 7, 2026

TrapsterDK commented on Nov 7, 2026

TrapsterDK commented on Nov 7, 2026

TrapsterDK commented on Nov 7, 2026

TrapsterDK commented on Nov 7, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions