Manual A/B full-sync benchmark tool (N refs, identical blocks, PR-ready table)

[ElFantasma]

Motivation

Comparing the full-sync throughput of two branches on real mainnet blocks is currently a multi-day manual exercise. We did it for #7008 and #7023 and it required, each time: freezing a DB snapshot, re-checkpointing the CL, hand-writing an orchestrator, restoring per leg, grepping throughput out of logs, and hand-assembling a markdown table for the PR comment. That table got hand-built four separate times.

The methodology is sound and reproducible — the same baseline measured 0.219 / 0.220 / 0.227 / 0.224 across four runs on different days (±1.8%) — it just isn't a tool. This is about turning the prototype into one.

Sibling of #7111 (continuous regression watch). Both consume the same leg primitive, specified in #7111 under Shared architecture; the two must not fork, or an A/B result stops being readable against the historical series.

What it does

make fullsync-bench-ab REFS=main,perf/my-branch [ROUNDS=2] [BLOCKS=21504]

For N refs, from one shared anchor:

1. build each ref, verifying the binaries differ by md5 — the Dockerfile's cargo target cache mount silently served a stale cross-branch binary once and produced a completely bogus comparison; this check is not optional
2. restore the anchor before every leg so all legs execute identical blocks
3. run each leg, collect the same LegResult JSON as Continuous full-sync throughput regression watch (rolling-gap mainnet node) #7111
4. print a PR-ready markdown table: per-batch throughput per ref, mean, delta %, peak memory, wall clock, plus a caveats line (anchor, block range, profile, mem cap, commit shas)

Two improvements over the manual version

Interleaving (--rounds). We ran strictly A-then-B, which cannot distinguish a small delta from within-run drift (thermal, background load). Our unified-vs-baseline result came out at −1.5% and I called it a wash — defensible, but sequential ordering can't actually separate −1.5% from drift. Running A,B,A,B and comparing paired means settles that class of question. Costs 2× runtime, so make it a flag rather than the default.

Anchor reuse. If the #7111 watch is running on the box, the A/B tool should be able to borrow its current base as the anchor. Then the legs are directly comparable to the historical series — same blocks, same box, same schema — turning "is my branch slower than the last 30 days of main?" into a real query. If the watch is not deployed, fall back to a standalone frozen snapshot.

Constraints inherited from #7111

• Same operational rules: graceful docker stop -t 300, mem_limit, cold page cache, eth_blockNumber (not eth_syncing.currentBlock) as the stop condition, never compare absolutes across machines.
• Mutual-exclusion lock with the watch — the two must never run concurrently on the same box or both results are junk.

Prior art

/root/bench/run_ab_main.sh on ethrex-mainnet-test-1 is a working 2-leg prototype (restore, build both, md5 guard, run legs, collect). Productising is mostly: generalise to N refs, move it in-repo so the methodology is reviewable and versioned, emit the shared LegResult schema, and render the table.

Suggested sequencing

#7111 first — it needs the box and the Slack plumbing, and building it establishes the shared primitive. This tool then becomes a thin orchestrator over the same engine. Building it first is also defensible, since it forces the primitive into shape with less surrounding machinery.