[cbeck88]

Draft, for 4.10.1. Supersedes #3404, and is independent of it — this branch is cut from testnet and contains none of that one's commits, so it merges on its own. It shares ideas with #3404 (byte-reversal collision handling, whole-group chunking, read-all-before-write-any batches) but the driver and the resume mechanism differ, and #3404's measurements, adversarial review and defect repros are why this one is shaped the way it is.

Why this exists

A partner running an archive node with --features history upgraded to v4.9.0 and hit a write-performance regression — roughly 500× write amplification (3.6–3.9 GB/s against ~7 MB/s healthy) and a compaction sawtooth that never drained. They diagnosed it themselves, reverted #3323, and have been running that patched build since.

We do not know how long they ran the unpatched build, nor what state their database is in now. They reverted quickly — the report says the signature reproduced within ~60 seconds on a fully-synced node — but "quickly" is not a number, and every entry written during that window is in a different on-disk encoding from everything around it. Nor do we know whether they ran --history before v4.7.5, which determines whether their keys carry a heights row and are therefore immune to the mixing described below.

That uncertainty is the reason for this work. The point is not only to repair a known-bad database, but to be able to tell whether a given database is repairable at all, and to say so before touching it.

The two changes that caused it

#3312 — "Fix history indexing performance" (a108e815) removed all three sites that wrote MappingUpdateHeightsMap, replacing the index with a prefix scan. Self-consistent at the time: every entry was little-endian and the scan decoded little-endian. Writes stopped rewriting an ever-growing vector, so this was a genuine improvement. Shipped in snarkOS v4.7.5 and v4.8.0.

#3323 — "Speed up history reads" (58a1359e) introduced big-endian height keys, using the presence of a heights row as the sentinel for "this key is legacy":

• write path, finalize.rs#L371 and #L453
• read path, finalize.rs#L875

Shipped in snarkOS v4.8.1, v4.9.0 and v4.9.1.

That sentinel was already false for three weeks of production data. Keys written by v4.7.5/v4.8.0 have no heights row and are little-endian, so #3323's else branch treats them as big-endian. Two consequences:

1. Reads return values from unrelated heights — silently, with a 200.
2. The write amplification returns. For a key that still carries a row from ≤v4.7.4, a108e815 had frozen that row and made writes O(1). Speed up history reads #3323 sees the frozen row, concludes "legacy", and resumes the read-modify-write. That is what the partner measured.

Seeing it fail

#3404 carries a committed test that demonstrates the corruption directly — a read that is correct before a migration and wrong after it, because an unlisted little-endian entry is byte-identical to a migrated big-endian one:

test_migration_exposes_unlisted_entries_as_wrong_heights

seeded: heights 100 and 1000 listed; an unlisted LE entry at height 256
query at height 65,536 after migration -> Some(256u64)
correct answer                          -> Some(1000u64)

LE(256) and BE(65_536) are the same four bytes. That test is the shortest statement of why a migration driven by the heights row is not enough, and it is worth reading before the rest of this description.

Why the heights map cannot drive this

Three incompatible layouts shipped in three weeks, and nothing on disk records which one wrote a given entry:

snarkOS	height encoding	MappingUpdateHeightsMap
≤ v4.7.4	little-endian	written, lists every entry
v4.7.5, v4.8.0	little-endian	not written
v4.8.1+	big-endian for keys with no row; little-endian for keys with one	frozen, or appended again

The middle row is the problem. Those entries have no heights row, so a heights-driven migration cannot see them — and on a node that reverted the v4.8.1 change they are most of its history. This migration reads MappingUpdateMap itself; the heights map is only ever deleted, never trusted. test_heights_map_is_ignored_and_dropped proves that against a row which omits a real height, invents two that never existed, and points at a key with no entries.

What is here

Errors are returned, never panicked: snarkOS wraps ledger loading in spawn_blocking!, which turns a panic into anyhow!("[tokio::spawn_blocking] {error}") — so a panicking migration reports "task N panicked" where the error chain should carry the remedy. We observed exactly that on a real ledger.

A storage schema version. A metadata map (last DataID variant, per that enum's append-only rule) records the version a database was last written under, plus migration resume state. RocksDB::open refuses a database from a newer build instead of misreading it, and otherwise applies migrations in order, advancing the version only once each completes. It runs whatever cargo features are set, since a migration works on raw keys under a prefix — a build that never opens the typed map can still repair it. It only protects forward from this release; older builds do not consult it.

Migration v0 → v1. A suffix byte-reversal on raw keys: HeightBytes is a [u8; 4] written bare by bincode as the final field, so the value bytes never move. The typed API would deserialize a Plaintext and a Value per entry only to re-serialize both identically, at ~4.1 KB peak per entry — unaffordable where a few hundred staking keys accrue one entry each per block forever.

Both legacy paths removed. The write path is the half that matters: it read the heights row, appended, and rewrote it on every update. No rows survive the migration, so removing it is what actually ends the amplification.

Measurements — and a correction

An xhigh review found that the sweep below measures the wrong axis, and I have confirmed it. Peak memory is O(entries in the largest single key), not flat: scan_heights's vector, its bincode copy, the lookup HashSet and the colliding/isolated partition each hold one element per entry of the key being migrated. The sweep scaled key count while holding depth at <= 4,000, so it never varied the dimension that drives memory.

At a constant ~900k entries, varying only the shape:

shape	entries	peak growth
447 keys x 2,000	894,000	0.076 GiB
20 keys x 45,000	900,000	0.110 GiB
5 keys x 180,000	900,000	0.118 GiB

So a real 6.5M-height staking key is on the order of low single-digit GB, not the ~0.1 GiB originally claimed here. Still far below the ~10.9 TiB the typed approach would need, and still workable, but the earlier figure was wrong and the benchmark needs a depth axis before any number here should be trusted.

Throughput, which the sweep does measure correctly: 300-550k entries/s.

Two things a reviewer should attack

Resume semantics. An xhigh review found six defects here, all sharing one cause: progress was recorded in a write separate from the data it described. write_chunk now puts the cursor in the same WriteBatch as its deletes and inserts, which required the work order to be deterministic in the recorded heights alone so a resumed run rebuilds the identical batch list and skips committed batches whole. Three metadata orderings were inverted for the same reason. Every substantive defect found in this work so far has been in the interruption path, and most were found by someone other than me. Specifically here: a key's original heights are recorded to metadata before anything moves, because once partly migrated a key's entries no longer say what they were — a moved entry sits at BE(h) == LE(byte_reverse(h)) and reads back as a plausible height of its own. And migrate writes a completion sentinel before returning, since the runner advances the version afterwards and a crash in between would otherwise re-run a finished migration over big-endian data.

Classifying a mixed database. An earlier revision refused any database containing big-endian entries. Running it against a real three-format ledger showed that was wrong: it panicked on an entry that byte-reversed to height 53,442, and the reason is structural. A key first written during the v4.7.5/v4.8.0 window has no heights row — that is what defines the window — so when a v4.8.1+ build took over it took the big-endian branch for exactly those keys. Single keys hold both encodings, and that is the expected shape of any node that ran v4.7.5/v4.8.0 and then upgraded, so refusing the mixture would refuse the databases this exists to fix.

Entries are classified per key against the chain tip, read from the committee store — CurrentRound then RoundToHeight, both in this same database and reachable by the raw prefix access the migration already uses. A reading above the tip cannot be a height, which settles almost every entry.

Where both readings are plausible:

• a palindromic suffix reads as the same height either way, so there is nothing to decide;
• a key with no big-endian entry was never touched by a v4.8.1+ build, so everything under it is little-endian;
• a key with no little-endian entry was only ever written big-endian;
• a key showing neither — a mapping written exactly once, at an ambiguous height — is judged against the ledger as a whole, which usually excludes one encoding;
• only a ledger genuinely holding both, on a key that shows neither, leaves the entry undecidable, and it is reported rather than guessed at.

That fourth case was found by review, and it mattered: without it such an entry was assumed little-endian, so a correct big-endian entry was deleted and rewritten at a height it never had — on ledgers that had nothing to migrate, while --check reported them as safe.

There is deliberately no boundary heuristic. An earlier revision placed such entries by which side of the encoding changeover they fell on; it produced two classification defects and earns very little once the bound is a real tip rather than a constant, so it was removed. (Thanks to @snarkworld for catching that this description still described it.)

What the residual case costs

Measured on real ledgers built from stock binaries:

ledger	tip	undecidable
genesis-synced, three formats	76,517	6
20.8M rung, v4.8.0 → v4.9.1	20,847,646	2,887

The count is approximately mixed keys × little-endian window ÷ 128, so it grows linearly with how long an operator ran v4.8.0 before upgrading. The second figure came from a five-hour window; a month-long one would be far larger.

Deciding these needs evidence the storage layer does not have — whether a given block actually updated a given mapping key, which is recorded in that block's finalize operations. A phase to do that was written and removed as unjustified complexity; these measurements are the argument for building it, and the decision is open.

Operator impact

Worth carrying into release notes; there is no CHANGELOG in this repo.

A node will not start on a ledger whose history has not been migrated. It reports the schema version it found, the one it needs, and the command to run. This is the intended behaviour: the alternative is the node reading its own history incorrectly, silently, which is the fault being fixed.

This applies regardless of whether the build has the history feature. The gate is on the data, not on the build. A build that never reads history could stamp the schema version without migrating and would never notice — but that stamp is what a later history-enabled build consults, and it would then skip the migration and misread every entry. So a node carrying unmigrated history must migrate it even if it has no interest in it.

The operator most likely to be caught out is one who ran --history at some point, stopped, and now runs a build without it. If that population turns out to be real rather than hypothetical, a --drop-history mode that discards the maps and stamps the version would be the humane answer; it is not built, because it destroys data.

Ledgers with no history are unaffected — one seek finds nothing, the version is stamped in passing, and the node starts normally. A node that never enabled the feature will not notice this change.

Migration is not automatic. It can rewrite billions of entries and take hours, so it is a deliberate operator action:

snarkvm-migrate-db --check <ledger-dir>   # read-only; can this be migrated, and how long?
snarkvm-migrate-db <ledger-dir>           # do it, with the node stopped

--check is safe against a copy of a production ledger. The migration is resumable: interrupt it and run it again.

Verification

cargo test -p snarkvm-ledger-store --features history,rocks — 128 passed, 0 failed, 6 ignored.

Not yet run against a database produced by real binaries. A three-format ledger built from stock v4.7.4 → v4.8.0 → v4.9.1 exists for exactly that and is the next step.

🤖 Generated with Claude Code

https://claude.ai/code/session_01GhrVwfiKhGgcRQXUTZfC5p

[cbeck88]

The bar this is aiming at

Stating the goal explicitly, since it has shifted twice during this work and the code now reflects a deliberate choice rather than a default.

After this change, a history node either answers every query correctly and performantly, or the database is reported as unrepairable and the operator resyncs. There is no third state.

In particular there is no "mostly repaired, a few entries still misread". That is the state we are trying to end — it is what a node running v4.8.1+ is in today, silently, and quietly leaving a residue of it behind would be the same bug with a smaller blast radius. A wrong answer that looks like a right answer is worse than a refusal.

Within that bar, avoiding a resync is strongly preferred. An archive node resyncing mainnet from genesis is a multi-week operation, so "resync" is a real cost to be spent only when the alternative is silent wrongness.

So the shape is: repair what can be repaired from the database alone, and stop with a precise report when something cannot be.

What that means concretely

Almost everything is repairable. An entry's encoding is usually decidable from its bytes: a height is small, and the byte-reversal of a small number is usually large, so only one reading is a plausible height. Where both are plausible, the key itself usually settles it — a key with no big-endian entry was never touched by a v4.8.1+ build, and a key holding both has its two encodings in disjoint height ranges, because every little-endian entry predates every big-endian one.

What is left over is narrow: an entry at one of the byte patterns that is plausible either way, on a key holding both encodings, with its two readings straddling the changeover. For those, the bytes on disk are genuinely consistent with two different heights and no amount of storage-layer reasoning separates them.

The migration records each one — key body, height bytes, both candidate heights — and then fails, naming the count and the first entry's two readings. The operator learns exactly what is wrong instead of being told to resync a multi-terabyte ledger on faith.

Why it does not go further

These are decidable with evidence the storage layer does not have: whether a given block actually updated a given mapping key is recorded in that block's finalize operations. The ledger can answer that; this crate cannot.

A phase to do it was written and then removed. It needed a trait implemented by the ledger, a second entry point running after Ledger::load, and schema-version tracking spanning both phases. That is real complexity, and the case it serves has not been observed in any real database yet — it requires a key first written in the v4.7.5/v4.8.0 window, updated again under v4.8.1+, holding an entry at an ambiguous byte pattern, with both readings straddling the boundary.

The count a failing migration reports is exactly the evidence that would justify building it. The metadata slot the list is written to is kept and documented to that effect, so a future reader understands why a write-only record exists.

Status

A run against a real three-format ledger (stock v4.7.4 → v4.8.0 → v4.9.1, synced from genesis) is in flight to produce that count. It is a measurement rather than a pass/fail: whether the residue is one entry or thousands, and whether it is one byte pattern or many, determines whether the resolver phase is worth building or the approach needs rethinking.

Worth recording that the same ledger has already falsified two earlier designs on their first run — including one that refused any database containing big-endian entries, which turned out to be the expected shape rather than an edge case.

[snarkworld]

Reviewed at 2e7edf8. Two defects reproduced locally against this branch, plus clippy is failing; details inline.

Blocking

1. The storage version is never stamped after a migration (ljedrz's comment above). Confirmed with a test: seed one entry, migrate(), read StorageVersion back → 0, has_history → true. The node refuses forever and rerunning the tool prints success while doing nothing (it returns early on the CURSOR_COMPLETE sentinel).
2. observed_tip is a lower bound on the chain tip, not an upper bound on real heights, so the newest entries can be misclassified as already migrated. Confirmed with a test (inline on observed_tip).
3. cargo clippy -p snarkvm-ledger-store --features rocks --all-targets -- -D warnings fails: dead MetadataKey::StorageMigrationHandoff, dead delete_metadata, dead Encoding, and enum_variant_names on MetadataKey.

Should be addressed or acknowledged
4. The PR description says a mixed key is resolved "by the boundary between" its LE and BE entries. classify_entries has no boundary logic; every ambiguous entry on a mixed key is reported as undecidable (inline).
5. check_storage_version is not gated on the history feature, so a build that never reads the map is still forced to migrate before it can start. Fine if intentional, but it needs a release note.
6. The tool infers the network id from the directory suffix; dev ledgers are .ledger-{network}-{id} so it picks up the dev id (inline).
7. The map is scanned three times per run: plan() in the tool, plan() again inside migrate(), then classify_entries per key (inline).

Process
8. New binary plus a new tracing-subscriber dependency; per AGENTS.md both need explicit sign-off.

No consensus exposure found: storage only, and the appended DataID variant respects the append-only rule.

[snarkworld]

The PR description says a key holding both encodings is "placed by the boundary between them, since every little-endian entry predates every big-endian one" and only entries fitting both sides are reported. There is no boundary check here: any ambiguous entry on a key with saw_little && saw_big is undecidable. That is safer than described, but it refuses more ledgers than the description claims, and reviewers evaluating the residual-case argument are evaluating logic that is not in the code. Either implement the boundary test (LE reading valid iff as_little < min(certain BE), BE reading valid iff as_big > max(certain LE), decide when exactly one holds) or update the description.

[cbeck88]

You are right, and it is the description that is wrong rather than the code. The boundary heuristic existed in an earlier revision, produced two classification defects, and was removed once the plausibility bound became real — at which point it earned nothing. The description was not updated with it.

I am correcting the PR body rather than implementing the boundary test. Your reading of the tradeoff matches mine: refusing more ledgers is the safer error, and with the tip now read rather than inferred the ambiguous population is small enough that the heuristic would be recovering very little at the cost of the logic that went wrong twice.

Thank you for catching that reviewers were being asked to evaluate an argument that was not in the code.

[snarkworld]

Nit: crate convention is std first, external crates second, crate-local third. This file has super first, then serde, then anyhow, then std.

[cbeck88]

Fixed in 1055c3e — imports reordered to std, external, crate-local.

[snarkworld]

This check is not gated on the history feature, so a build that never opens the typed map is still refused until the operator migrates data it does not read. The PR body presents that as a feature ("a build that never opens the typed map can still repair it"), which is fine, but it is a startup-blocking behavior change for non-history nodes with old history data on disk and should be called out in the release notes.

[cbeck88]

Intentional, and on reflection it is a correctness requirement rather than a convenience — so I have recorded the reasoning in the code (0858472) rather than leaving it to be inferred.

The gate is on the data, not on which features the build was compiled with. A build that never reads history could stamp the version without migrating and would never notice the difference. But that stamp is exactly what a later history-enabled build consults, and it would then skip the migration and read little-endian entries as big-endian — silent wrongness, which is the failure this whole change exists to end. Blocking a non-history node that carries unmigrated history is the price of the version meaning what it says.

You are right that it needs calling out. There is no CHANGELOG in this repo, so I have added an Operator impact section to the PR description; tell me if release notes are kept somewhere I have not found and I will move it.

Worth naming the sharp edge for whoever writes the release notes: the operator most likely to be surprised is one who ran --history at some point, stopped, and now runs a build without it. They have no interest in the history data, yet must migrate it before the node will start. If that turns out to be a real population rather than a hypothetical one, the humane answer is probably a --drop-history mode on the tool that deletes the maps and stamps the version — it is a few lines, but it discards data, so I would not add it speculatively.

[snarkworld]

New dependency (and a new [[bin]] above). Per AGENTS.md both need explicit approval; flagging so it is a conscious decision rather than an implicit one.

[cbeck88]

true, and we should discuss whether we want snarkvm-db-migrate to be a standalone binary (so that the operator explicitly runs migration) or something that runs when the node starts up unconditionally.

usually in webapps you want to have a human watching when migration happens, because if migration goes wrong you can get corrupted data base or data loss. for a blockchain it might not be important to do that because all this data is highly replicated anyways by design, and backed up on CDNs etc.

[cbeck88]

Signed off on both the [[bin]] and the tracing-subscriber dependency — thank you for flagging them rather than letting them pass implicitly.

The larger question they point at is genuinely open, and worth settling before this merges: should the migration be a standalone binary at all, or a snarkos ledger migrate subcommand?

The case for standalone, which is what is here now:

• No node machinery — no config, keys, or network. It is a data tool operating on a directory.
• The migration logic lives in snarkvm-ledger-store and works on raw prefixes, so a binary beside it has no dependencies a node would drag in.
• Easy to run against a copy, which is how anyone sensible will first use it.

The case for a subcommand, which I now think is stronger than I judged when I built this:

• Distribution. Operators receive snarkOS releases. They do not build snarkvm-ledger-store with --features rocks, and a binary that lives in this repo has no route to them without one being invented.
• Discovery. snarkos --help is where someone looks after the node tells them to migrate. A tool named in an error message but absent from the binary they have is a poor handoff.
• I had thought a compliance constraint favoured standalone, on the grounds that a partner told us they cannot run non-stable builds. Re-reading it, that constraint is about patched builds, not about snarkOS as such — a stable release carrying a ledger migrate subcommand is precisely what they can run. So that argument does not hold, and I should not have leaned on it.

What does not change either way: the work belongs in snarkvm-ledger-store where the storage layout is, and the node refuses to start on a schema it does not understand rather than migrating implicitly. The question is only which binary the entry point hangs off.

I have no strong preference and would rather defer to whoever owns the release and packaging story. If the answer is a subcommand, the change is small — the two entry points are thin wrappers over plan() and migrate().

[vicsn]

In general, I think it makes sense for migrations not to be run automatically/unconditionally:

• As Chris said, we may want to have a human in the loop in case of corruption
• We don't want a quorum of validators to ever directly activate this and to stall consensus for the duration of the migration

As an exception, the design in #3329 is run automatically/unconditionally but it's a special case: because it only prunes data there is no interference with consensus. And anyhow we can decide later on the final architecture for that PR.

That leaves the question of snarkVM binary v.s. snarkOS migrate. One more advantage of the snarkOS migrate approach is that this could potentially also cover migrations of BFTPersistentStorage. If BFTPersistentStorage would be moved into snarkVM this benefit would be void, but that would be yet another huge work package.

So I'm in favour of a snarkOS migrate command - for which most logic can live in snarkVM, "close" to the db logic.

[veria-ai]

Low: Ambiguous heights are silently assigned the wrong encoding

The absence of an unambiguous big-endian entry does not prove these ambiguous entries are little-endian: a key can contain an older unambiguous LE entry and a newer ambiguous BE entry. For example, with a tip above 19,005,440, a BE update at height 19,005,440 also reads as the plausible LE height 8,705; an attacker who submits that mapping update can cause migration to relocate it to height 8,705, after which historical views expose the value before it existed. Treat entries whose encoding cannot be established individually as undecidable, or require authoritative metadata before choosing an encoding.

[veria-ai]

PR overview

This pull request records a storage schema version and adds migration logic to repair historical data encoding in the RocksDB ledger store.

One security issue remains open in the history migration logic. Under specific chain-height conditions, an attacker-controlled mapping update with an ambiguous encoding could be migrated to an incorrect earlier height, causing historical views to expose the value before it existed. No issues have yet been addressed.

Open issues (1)

• Low: Ambiguous heights are silently assigned the wrong encoding — ledger/store/src/helpers/rocksdb/internal/history_migration.rs:326

Fixed/addressed: 0 · PR risk: 5/10

[cbeck88]

Updated through 29021f266. A second xhigh review found a defect worse than anything previously identified here, so recording it plainly.

The one that mattered

classify_entries assumed little-endian for an entry with no unambiguous evidence in either direction — which is the case for a mapping written exactly once, at a height whose two readings are both plausible.

Reproduced before fixing: a key written once by a v4.9.x build, big-endian at height 256.

before   height 256 -> "correct"
--check  little=1  big=0  undecidable=0     <- ledger reported entirely safe
after    height 256 -> None,  height 65,536 -> "correct"

A correct entry, deleted and rewritten at a height it never had, with the check reporting no problem. The exposure is on ledgers that ran only v4.8.1+ and had nothing to migrate at all, and at mainnet tip roughly one suffix in 256 qualifies, so any single-update mapping key was at risk.

Fixed by falling back to ledger-wide evidence: a key's own entries settle it where they can, and where a key shows nothing, the ledger usually does — if no entry anywhere is unambiguously big-endian then no v4.8.1+ build ever wrote there. Two regression tests pin both directions.

Also fixed

• --check on a half-migrated ledger classified a partly-moved map and reported "must be resynced" about a ledger that would have finished correctly.
• The tool compared the schema version against zero rather than the version the build requires, so it would have refused to run every future migration.
• The "nothing to migrate" shortcut asked a question specific to v0 → v1; it now asks each outstanding migration whether it has work.
• The resume cursor's batching parameters are recorded and checked, rather than a changed CHUNK_SIZE silently replaying committed work and wedging the ledger.
• The destination check is one batched read rather than a point lookup per entry.
• byte_reverse was u32::swap_bytes longhand.

Deferred, deliberately

tracing-subscriber in the library's feature graph. The objection is right. The clean fix is a bin-scoped feature, but the build command for snarkvm-migrate-db is already in a partner's hands for them to run --check against their own ledger, and changing it now would strand them. Dropping the dependency instead would leave a multi-hour migration printing nothing. Restored with the reason recorded in Cargo.toml, to be revisited once they have reported back.

Packaging. Review notes that the node's refusal message names a binary that ships in no snarkOS artifact — an operator whose node will not start is told to run something they cannot obtain. That is a strong argument for a snarkos ledger migrate subcommand, but it is a decision for reviewers rather than something to change under a partner mid-run.

133 tests, clippy clean.

[vicsn]

I feel like this is a bit too much, see: https://github.com/ProvableHQ/snarkOS/pull/4433/changes#diff-a54ff182c7e8acf56acfd6e4b9c3ff41e2c41a31c9b211b2deb9df75d9a478f9R24

[vicsn]

Mild preference for the following setup:

• call it fn requires_migration
• currently there is no compile-time check whether we're including all possible versions and we have to manually remember to add entries here. We could:
  • replace const STORAGE_VERSION: u32 = 1 by an enum StorageVersions similar to what we do for enum ConsensusVersions.
  • match here on the enum, for the latest one we should always return Ok(false)

[cbeck88]

yeah that seems reasonable

[vicsn]

What?

[cbeck88]

Yeah some of the comments are wierd and it's partly the code getting manipulated by tools that don't respect the comments

[cbeck88]

I'm mostly focused on correctness so far but let's get the comments right too. There was still a correctness bug yesterday I think

[cbeck88]

Note that, I tested this extensively against actual mainnet chain data, running 4.7.5, then 4.8.0 then 4.9, then this, and it seems to sync properly and not cause damage even when there are incorrect endian entries. But that doesn't mean it doesn't have bugs.

Ultimately resyncing is going to ensure that you don't have corrupted data. I'm unsure how important it is to avoid that. But it seems like if it's easy to help our partner avoid that, it's a good idea. How we draw the line, like, at what point we give up and say they have to resync, is up for debate of course.

[cbeck88]

these are the tests that got ran over the weekend:

  3. Real ledgers — where everything was actually learned

  ┌─────────────────────────────────┬─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
  │               run               │                                                                       outcome                                                                       │
  ├─────────────────────────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ v4.8.1 node                     │ wrong version: writes BE, nothing to migrate                                                                                                        │
  ├─────────────────────────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ v4.7.4 on a rung                │ RocksDB format_version wall — can't open a modern ledger                                                                                            │
  ├─────────────────────────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ genesis v4.7.4 → v4.8.0 →       │ built the three-format ledger; measured the frozen bonded window — finding A proven on real data                                                    │
  │ v4.9.1                          │                                                                                                                                                     │
  ├─────────────────────────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ migration #1 (ab9e5b8df)        │ completed, after-table byte-identical → proved the heights-map design strands the v4.8.0 window                                                     │
  ├─────────────────────────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ migration #2 (66ba0d64b)        │ panicked → falsified "refuse any mixed database"; mixed is the expected shape                                                                       │
  ├─────────────────────────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ migration #3 (cdc3cf9a9)        │ failed on a genuinely undecidable entry                                                                                                             │
  ├─────────────────────────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ --check sweeps                  │ 3,409 → 12 → 6 undecidable as the constant became a derived then a real tip                                                                         │
  ├─────────────────────────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ realistic-tip ledger (20.8M)    │ 2,887 undecidable; derived mixed keys × LE window ÷ 128; found the FD/memory bug that killed the tool on a 1.7 TiB ledger                           │
  ├─────────────────────────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ final run (29021f266)           │ 234M entries / 52,539 keys in 90 min, 1.12 GB peak, exit 0. SIGKILL at 900s → resume → byte-identical bonded table. Node starts and serves          │
  │                                 │ correctly.                                                                                                                                          │
  └─────────────────────────────────┴─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘

[vicsn]

Superseded by #3449