[cbeck88]

Draft. An alternative to the migration in #3348, working on raw bytes rather than through the typed Map API. Opened for discussion alongside the findings write-up.

Why raw bytes

A MappingUpdateMap key is (ProgramID, Identifier, Plaintext, HeightBytes) behind a 4-byte [network_id, map_id] context. HeightBytes is [u8; 4], which bincode writes bare with no length prefix, and it is the last tuple field — so the height is the final four bytes of the raw key, and the migration is a suffix byte-reversal that leaves the value bytes untouched.

Going through the typed API instead deserializes a Plaintext and a Value<N> per entry and re-serializes both to identical bytes. Measured, that round-trip costs ~4.1 KB of peak RSS per entry, linear:

entries	#3348 peak RSS	this branch	#3348 rate	this branch
111,750	0.42 GiB	0.032 GiB	71k/s	454k/s
894,000	3.45 GiB	0.055 GiB	108k/s	339k/s
1,788,000	—	0.076 GiB	—	253k/s

Flat rather than linear — 30–90 MB regardless of depth. Extrapolated to a ~7.7-month archive node (2.9B entries): ~3.2 h and ~0.1 GiB, against ~10.9 TiB for the typed approach. That matters because credits.aleo's bonded/delegated/committee are rewritten in full by replace_mapping on every block, so ~507 keys accrue one history entry each per block — and 507 < MIGRATION_BATCH_SIZE, so batching never splits them.

Other differences: a forward cursor (so a pass skips one tombstone rather than every tombstone left by previous passes), bounded chunks, and collision handling that preserves the property 38b02d3df established without needing a key's whole history in memory.

Review history

An adversarial review of the first commit found two defects, both since fixed in ab9e5b8df. Recording them because the second one shaped the design:

1. Resume transposed byte-reversal pairs (critical). The colliding write batch and the heights-row deletion were separate writes, so an interruption between them left every entry moved with the row still listing it; the resumed run read each height back through its partner's destination and transposed the pair, silently, for ~109,100 entries on one staking key at today's tip.
2. The colliding subset was unchunked. It is not "a few" entries — for a key written every block it is #{h <= H : byte_reverse(h) <= H}, growing as H^2/2^24: 109,100 at tip 21.6M, 1,048,578 by tip 67M, previously all held in memory at once.

Both are fixed by the same change: each colliding batch shrinks the heights row to the heights it has not yet moved, in the same atomic write, and batches pack whole partner groups. The only states a crash can leave are "batch not applied, row unchanged" and "batch applied, row already shortened", so a resumed run is never offered a height it has already migrated.

One caveat worth a reviewer's attention. That fix removes the window, not the ambiguity: a migrated colliding entry is byte-identical to its partner's un-migrated source, so a database placed in the old intermediate state by other means would still be transposed. I argue it is unreachable — WriteBatch is atomic, and the only code that could produce that state is the first commit on this branch, which was never released. The repro was updated to exercise the reachable state instead. If that reasoning is too thin, the alternative is a defensive check, and I could not find one that works from the data alone.

A related finding this PR does not fix

test_migration_exposes_unlisted_entries_as_wrong_heights is committed and #[ignore]d, because it documents a constraint on any heights-row-driven migration rather than a bug in this branch.

snarkOS v4.7.5 and v4.8.0 (snarkVM a108e8159..58a1359ea) wrote little-endian history entries while writing no heights rows. Those entries are invisible to a migration driven by the heights row, and they survive it — where an unlisted entry at height h is byte-identical to a migrated entry at byte_reverse(h), a floor seek finds it. A read that was correct before the migration returns a value from an unrelated height after it.

A fully correct migration has to enumerate what is actually under each key prefix and classify by encoding, rather than trusting the heights row as an inventory. That is tractable — heights are ≤ ~21.6M < 2^25, so an LE height read as BE almost always exceeds the chain tip — but it is a design change, not a patch.

The same gap causes #3323 to misread that data on v4.8.1+ today, independently of any migration. That is covered in the findings write-up rather than here.

Verification

cargo test -p snarkvm-ledger-store --features history,rocks — 127 passed, 0 failed, 6 ignored.

adv_bincode_height_suffix_holds_for_every_plaintext is deliberately active: it checks the assumption the whole approach rests on, over 396 combinations covering every literal type, nested structs and arrays, and both encodings.

This branch also deletes test_get_historical_mapping_value_legacy, since it asserts the legacy read path that this change removes.

🤖 Generated with Claude Code

https://claude.ai/code/session_01GhrVwfiKhGgcRQXUTZfC5p

[cbeck88]

closing in favor of #3408