perf(mem_wal)!: serve the shard manifest from the store that wrote it

[hamersaw]

Problem

ShardManifestStore::read_latest scanned the version space on every call: a GET for version_hint.json, a HEAD to confirm the hinted version, then batches of parallel HEADs until a whole batch 404s. The loop only exits on a full batch of misses, so the floor is 2 + manifest_scan_batch_size object-store requests even when the hint is exact — and nothing cached the result, so a reader paid it per request.

Profiling a WAL node, this was essentially all of its GET+HEAD traffic. Per fresh-tier read with N L0 generations: 5 + N requests, of which the 5 were manifest probing and only N were data.

Change

Serve the manifest from the store that wrote it, and make the invariant that permits this an enforced one rather than an assumed one.

Serving

A store keeps the manifest it durably wrote as its position and serves that from latest(). Soundness does not rest on holding the claim. Manifest versions are CAS-allocated and gap-free — every writer commits latest.version + 1 under PUT-IF-NOT-EXISTS — so a successful write at version N proves N was the tip: a peer cannot hold N+1 without N existing first.

The two reads now differ by whether they take a position, which is what the names say:

	goes to storage	adopts a position
latest()	only when it has no position	no
refresh_latest()	always	yes

A plain scan deliberately does not adopt, so a reader that polls keeps observing the writer rather than pinning the first manifest it saw. refresh_latest adopts because a claim reads uncached precisely to see another process, and the tip it finds is what its own write then builds on. check_fenced and claim_epoch use it for exactly that reason.

Enforcing

Nothing previously enforced the gap-free invariant the cache rests on: every caller hand-wrote current.version + 1 and nothing checked the result. Given a gap, a cached commit could be acknowledged behind the durable tip.

The invariant is also load-bearing well beyond the cache. find_latest_version stops at the first absent batch, so a gap wider than manifest_scan_batch_size combined with a lost best-effort hint makes even an uncached read misreport the tip.

The check belongs with whoever holds the predecessor, not with the store's position — that position is shared, and a peer's failed CAS clears it, so a commit can find itself judged against an empty position midway through. commit_update therefore validates the closure's output against the manifest the closure received, which is immutable and local. A version the caller did not intend is rejected, not silently corrected; ShardManifest::next_version() is what callers build with.

write keeps only what its own state can judge: a version at or below this store's position is reported as the collision it is, so callers retry.

The tailer

A store can only check contiguity against a position it holds, and WalTailer was the one writer that could never hold one: it claims no epoch, and its store must serve fresh reads for its own position hints — so no single piece of state could be both stable enough to validate against and fresh enough to hint from.

It turns out it never needed to write at all. Its per-entry manifest write maintained wal_entry_position_last_seen purely as a cursor hint for next_position(), whose only callers are tests; replay, the tailer's sole production user, derives the tip from its own read loop. The tailer now tracks the highest position it has read in memory, and publishing that cursor moves to the replay driver, which already holds the epoch, as an ordinary commit_update.

That removes one manifest write per replayed WAL entry and leaves every manifest writer an epoch holder.

Measured

Against a WAL node driving this code:

	before	after
read, N L0 generations	5 + N requests	N
flush (seal into L0)	132	24
compaction pass	35	10
replay, per unflushed WAL entry	8.81	4.18

The replay figure predates the tailer change, which removes a further manifest write per entry.

Writes are unchanged (one conditional PUT per entry).

Breaking changes

• ShardManifestStore::read_latest → latest
• ShardManifestStore::read_latest_uncached → refresh_latest
• ShardManifestStore::write is now crate-private. Callers reach it through commit_update, claim_epoch, or initialize_shard — the three entrances that derive a version from a manifest they just read. It was public from the commit that introduced MemWAL and never acquired a caller.

Downstream commit_update closures need no change: setting version: current.version + 1 is exactly what the check expects.

Tests

cargo test -p lance --lib dataset::mem_wal:: — 619 passed.

Covering the properties this is allowed to break:

• a peer's claim is visible to check_fenced through a held position
• a reader that polls latest() keeps observing the writer, and never adopts a position
• only refresh_latest adopts — reversing that would either pin pollers or reject valid claims
• commit_update recovers from a stale position instead of spinning on the version it lost
• a non-successor write is refused, and the refusal leaves storage untouched
• a taken version reads as a collision, and drops the position so the retry re-reads
• eight concurrent commits on one handle all land, and none is lost
• a tailer read writes no manifest

Note on the commit stack

The first two commits are pre-existing work from the flush-observer branch that has not landed upstream yet; read this PR as its final commit. Rebasing once those merge will reduce it to the single change.

🤖 Generated with Claude Code

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[lance-gatekeeper]

✅ Gate recommendation: approve.

The manifest mutation boundary now validates each update against the exact predecessor supplied to its closure, while collision-driven cache invalidation remains only a retry concern. The raw-gap and stale-predecessor races are both closed without restoring storage reads to the hot path.