[wip] commitment: reuse pooled trie workers across Process calls

[AskAlexSharov]

Problem

On a mainnet node heap profile (alloc_space), ParallelPatriciaHashed.workerPool refills cost 943MB — 27% of all sync.Pool refill bytes process-wide (~9.8K refills at ~93KB each, one full NewHexPatriciaHashed per refill).

The drain was self-inflicted, not GC pressure: resetPool() replaced the entire sync.Pool object, instantly discarding every cached worker trie. It ran from Reset() — once per commitment round via SharedDomains.ClearRam — and from Release() on every SharedDomains teardown, in both ParallelPatriciaHashed and StreamingCommitter.

Fix

Workers were already scrubbed with resetForReuse() before every Put (the same-Process recycle path has always relied on that), so pool contents are safe to reuse across Process calls. The raw sync.Pool is replaced by a small trieWorkerPool that:

• keeps cached workers across Reset() — accountKeyLen/cfg are set once at construction and never mutated afterwards (no assignment site exists), so there is nothing to invalidate and no pool-replacement path remains;
• re-applies accountKeyLen + applyConfig(cfg) on every pooled get(), making a pooled hit state-identical to fresh construction (previously an intra-Process recycled worker kept zeroed cfg/memoizationOff while a fresh one carried the configured values);
• drains cached workers back to the global hphPool on Release() (via HexPatriciaHashed.Release()), so a successor instance — e.g. the fresh SharedDomains built after a commit cycle — refills from cached objects instead of allocating;
• centralizes the resetForReuse-before-Put scrub in put() instead of repeating it at every call site.

Why per-use scrubbing is sufficient

resetForReuse() clears everything a use can observe: root cell + root flags, activeRows/currentKeyLen, per-row depths/branchBefore/touchMap/afterMap, ctx (nil), hadToLoadL, mount state, trace/collapse/witness tracers, leaveDeferredForCaller/memoizationOff, auxBuffer, branchEncoder deferred set + buffer, cfg, depthsToTxNum. Grid cells are intentionally not zeroed — activeRows == 0 means no cell is live and mountTo/unfold re-initialize cells before any read — and cells hold no heap references (cell/Update are pure value types), so no DB/tx data can leak across transactions. Every get() site then establishes context, trace writer, and defer flags before use. This is the exact contract same-Process worker recycling has always run under; this PR extends it across Process calls and instances without changing it.

When the pool is still emptied

Only on Release(), where cached workers are handed to the shared constructor pool (hphPool) rather than dropped for the GC. There is no config-change replacement path because the config provably cannot change after construction.

Tests

• TestParallelWorkerPoolSurvivesReset / TestStreamingWorkerPoolSurvivesReset — written red-first against the old code (a tagged pooled worker must survive Reset(); previously a fresh worker came back).
• TestParallelReuseAcrossResetParity — two batches on one instance with the production per-block Reset() + SetState(blob) lifecycle between them, for both VariantParallelHexPatricia and VariantStreamingHexPatricia, asserting root parity with the sequential engine so batch 2 runs on workers cached by batch 1.
• Full ./execution/commitment/... suite green in both short and non-short mode; ./db/state/... short green; parallel commitment benches smoke-run.

https://claude.ai/code/session_01UYCsHj9HYTJnqUCy8a7W91

[Copilot]

Pull request overview

This PR optimizes the execution commitment pipeline by avoiding sync.Pool churn: pooled HexPatriciaHashed workers are preserved across Reset()/Process() lifecycles and only drained back to the shared constructor pool on Release(), reducing repeated trie construction and refill allocations during commitment rounds.

Changes:

• Introduces trieWorkerPool with get()/put()/drain() to reuse scrubbed trie workers across Process() calls and instances.
• Updates Parallel and Streaming commitment implementations to use the new pool and to stop replacing the pool on Reset().
• Adds tests to assert worker reuse across Reset() and parity across the “Reset + SetState(blob)” lifecycle.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
execution/commitment/streaming_deep_fold.go	Switches deferred storage folding to use trieWorkerPool get/put instead of raw sync.Pool.
execution/commitment/streaming_commitment.go	Replaces sync.Pool reset/replacement with trieWorkerPool init + drain-on-Release, and updates worker checkout/return sites.
execution/commitment/streaming_commitment_test.go	Adds a test asserting worker caching survives Reset().
execution/commitment/parallel_patricia_hashed.go	Adds trieWorkerPool and wires it into ParallelPatriciaHashed Reset/Release and worker acquisition.
execution/commitment/parallel_patricia_hashed_test.go	Adds tests for worker pool survival across Reset() and for parity when reusing an instance across Reset+SetState.
execution/commitment/parallel_mount.go	Updates mount processing to use trieWorkerPool.get/put (centralizing scrub-on-put).

Comments suppressed due to low confidence (2)

execution/commitment/parallel_patricia_hashed.go:66

• trieWorkerPool.drain() has the same typed-nil risk as get(): if a typed-nil *HexPatriciaHashed is present, w.Release() will panic. Add a w==nil guard (and consider wording the comment as best-effort since sync.Pool is GC-managed).

// drain hands cached workers back to the shared constructor pool instead of the GC.
func (wp *trieWorkerPool) drain() {
	for {
		w, ok := wp.pool.Get().(*HexPatriciaHashed)
		if !ok {
			return
		}
		w.Release()
	}
}

execution/commitment/parallel_patricia_hashed.go:55

• trieWorkerPool.put() should defensively ignore nil to avoid inserting a typed-nil pointer into the underlying sync.Pool, which would later trigger a nil dereference in get()/drain().

func (wp *trieWorkerPool) put(w *HexPatriciaHashed) {
	w.resetForReuse()
	wp.pool.Put(w)
}

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[AskAlexSharov]

Follow-up commit: trieWorkerPool is now a thin config handle over the package-level hphPool instead of owning a second sync.Pool. Rationale: get() re-applies accountKeyLen/cfg on every checkout, so workers are fungible across instances — and with that, get() was equivalent to NewHexPatriciaHashed (which already checks hphPool) and put() to HexPatriciaHashed.Release(). The instance-local tier and its drain() only bridged a pool that bought nothing: both tiers had identical GC-drain behavior, scrub, and re-init. Net −32 lines, and workers become globally shareable at put() time rather than at instance teardown. Full commitment suite green in short and non-short modes.