fix(rollout): stop WAL merge from blocking concurrent appends

[beinan]

Fixes the ~17s stop-the-world stall on the rollout write path.

Root cause: claim_epoch, not the lock

A merge held the store's write lock for its whole duration, so every concurrent add blocked. But this is an LSM — a merge folds sealed generations into the base table while add writes the active memtable at the WAL tail. Disjoint data. They should never have serialized.

The lock was covering for something else. To drain merged generations from the shard manifest, the merge called claim_epoch, which bumps writer_epoch and fences every live writer of that shard — including our own. So the merge had to close() the resident writer first, and callers had to hold an exclusive lock to hide that window.

The claim is unnecessary:

• commit_update(local_epoch, ..) only rejects a writer whose epoch is older than the stored one (stored > local ⇒ fenced) — it does not require that you just claimed.
• Lance's own flush path reuses one epoch for every manifest commit a writer ever makes. The epoch is an ownership token, not a per-commit token.
• Lance explicitly sanctions concurrent draining by an external compactor: recovery keys off replay_after_wal_entry_position and deliberately does not consult flushed_generations, "since an external compactor may legitimately drain that vector back to empty" — with a dedicated regression test upstream.

So the merge reuses the shard's current epoch and never closes the writer.

Plus a lock-scope split

phase	receiver	work
prepare	&self	seal memtable, read every flushed generation from object storage — the dominant cost (1.5–3.4s per generation on abfss)
commit	&mut self	append to base table, drain manifest, delete merged dirs

Callers run prepare under a read lock so appends keep flowing, and take the write lock only for the short commit. Both sweepers and the HTTP merge-wal handler updated.

The issue's suggested fix would not have worked

Option 1 in the report was "drop the count trigger from the flush sweeper and let the global sweeper drive merges." But the global sweeper takes the same write lock and calls cleanup_own_shard with threshold 1 — more aggressive, not less. The stall would simply have moved to a different interval.

A subtle ordering that had to be preserved

cleanup_own_shard seals before reading the manifest. That ordering is load-bearing: with ROLLOUT_FLUSH_INTERVAL_SECS=0 nothing else seals the memtable, so reading the manifest first leaves flushed_generations empty forever and rows durable but permanently invisible until a restart replays the WAL. Preserved as prepare_cleanup_merge. (My first attempt got this backwards.)

Tests

New suite, 5 tests. The existing one is explicitly "one store, one shard, fully serial — no concurrency, no fence", so none of this had coverage:

• appends succeed and no row is lost while a merge runs
• a generation sealed during a merge is not dropped by the drain
• concurrent merges do not duplicate rows
• an interrupted merge loses nothing; the next merge converges
• an append is not blocked for the merge's duration — the regression itself

Each test drives merges through a helper reproducing the sweeper's exact lock discipline (read-lock prepare → write-lock commit). Merging under a single exclusive lock in the test would pass while the real stall persisted.

Verified non-vacuous: restoring claim_epoch makes 2 of the 5 fail. They also do not compile against the old &mut self signature — "merge concurrent with append" was inexpressible, which is the design flaw in one line.

Caught during development

The first attempt made merge fully &self (clone-based append). The tests immediately caught data loss: all 8 merged rows vanished, because a &self merge cannot advance its own handle past the append, so the rows were neither in its base snapshot nor in the manifest. That is why the commit phase stays &mut self.

Follow-ups (filed, not folded in)

• RolloutStore::dataset should be interior-mutable so merge/compact can refresh the handle without an exclusive lock #198 — make dataset interior-mutable so merge/compact need no exclusive lock at all (~52 call sites; deserves its own review and soak).
• datagen_store.rs has the identical claim_epoch-then-close pattern.

Verification

166 core + 5 new concurrency + 51 server + 14 master tests pass; cargo fmt and clippy --workspace --all-targets -D warnings clean.

🤖 Generated with Claude Code