fix(core): bound flushed generations folded per merge pass

[beinan]

Problem

prepare_merge materialized an entire shard's flushed generations into memory at once. In production this OOMKilled 8 of 20 workers; worker-2 was at 23.3 GiB RSS / 23.2 GiB anonymous with only 39 resident stores.

Two things make the buffer unavoidable today, and both are why the streaming fix (option 1 in the issue) is off the table:

• Dataset::append takes a synchronous RecordBatchReader (lance-7.0.0/src/dataset.rs:919), and there is no async→sync bridge in this repo.
• A lazy reader would push object-store IO into the write-lock window, breaking the prepare/commit split that wal_merge_concurrency.rs pins (~17s stop-the-world appends observed on abfss).

Option 3 (projection) is also infeasible — merge must write back all columns.

The payload is large because blob-v2 offload reads back as None through the MemWAL LSM scanner, so binary_payload is stored inline as LargeBinary.

Fix

Cap how many flushed generations one merge pass folds in: merge_max_generations, default 8, env ROLLOUT_MERGE_MAX_GENERATIONS. Leftovers stay pending for the next pass, so a backlog drains incrementally at bounded peak memory. 0 opts out.

Why a subset merge is safe: commit_merge's drain is already surgical — it filters out exactly the generations that were merged rather than clearing the list:

flushed_generations: current.flushed_generations.iter()
    .filter(|fg| !merged_generations.contains(&fg.generation))
    .cloned().collect(),

So a partial merge is just a smaller version of a full one, with the same crash-safety argument (immutable rows, read-time dedup by key). Generations are the right granularity because each is a self-contained Lance dataset that the manifest tracks individually.

Why the default actually binds: the time-triggered path (cleanup_own_shard) calls merge_own_shard_if_ready(1) — a hardcoded threshold — so it never consults ROLLOUT_MERGE_AFTER_GENERATIONS (50 in the deployment that OOMed). Lowering that env var alone would not have helped; the cap here applies to both trigger paths.

Tests

• merge_pass_is_bounded_and_leftovers_survive — 10 generations, cap 3: asserts exactly 3 reclaimed per pass, leftovers stay pending, ≥4 passes to fully drain, all 10 rows survive, and 0 leaked generation directories.
• zero_max_generations_merges_everything_in_one_pass — the opt-out escape hatch.

Negative verification: removing .take(budget) fails the first test with left: 10, right: 3; restoring it passes.

Full workspace suite green (222 core, up from 217). wal_merge_concurrency 5/5 — the prepare/commit invariants are intact.

🤖 Generated with Claude Code