Repository navigation
Conversation
eranrund
force-pushed
the
eran/authority-prune
branch
from
July 29, 2026 21:59
db3821c to
ea30a9a
Compare
update Cargo.lock Bump retention block count to 100000
eranrund
force-pushed
the
eran/authority-prune
branch
from
July 29, 2026 23:00
ea30a9a to
a814b5c
Compare
vicsn
requested changes
Aug 7, 2026
vicsn
reviewed
Aug 7, 2026
vicsn
reviewed
Aug 7, 2026
vicsn
reviewed
Aug 7, 2026
Staging dropped the compile-time history feature (fb10184) that the pruning code keyed on. Pruning and the windowed startup spot-check now require prune-authority-data; default builds keep all authority data. Startup pruning skips read-only secondary RocksDB instances. Retention is 500,000 blocks, about two weeks.
The block store writes a marker under the storage metadata prefix in the same atomic batch as the first deletion. A build without prune-authority-data refuses to open a store that carries it, since the pruned data cannot be restored. Secondaries read the marker but never write it; no StorageVersion bump is needed, the key is additive.
Startup pruning logs at info when it starts and finishes, and at debug after each batch; insert-time pruning logs at debug every 10,000 blocks. An entry whose block indexes disagree is logged and kept, and a startup pruning failure is a warning rather than a failed open, as the remaining data is pruned on the next startup.
Every primary this build opens is stamped V1, so builds that predate the marker refuse the store instead of opening one that may be pruned. The upgrade from V0 is the stamp alone: an unpruned store lacks the optional key and nothing is rewritten.
The pruned marker is a raw key in the schema module beside the storage version, read and written through two required BlockStorage methods, so the typed metadata map, its duplicated key and its seven atomic-batch registrations go away. The marker is written before the deletes it describes. One batch helper serves both maps, the ledger spot-check has one implementation, the secondary check is inline, and the tests share the block-chain builder with the cache test and the quorum authority with the authority crate.
… data RocksDB writes the pruned marker through the atomic batch in progress, so a rolled-back insertion leaves no marker. BlockStore::get_block reports a block below the retained window as pruned rather than as a missing authority. The startup spot check samples the whole chain and checks the permanent data of blocks below the window.
The VM inserts with atomic writes paused and no enclosing scope, so a marker written before the scope went straight to disk and survived a discarded block. get_block now reports a pruned block only when the authority is the missing piece, and the spot check below the window covers every permanent component.
…ole gap A shared check verifies every component of a block except its authority. get_block reports a block below the window as pruned only when that check passes, and the startup spot check uses it below the window.
The marker is a one-entry map with its own DataID, started, checkpointed, rewound, aborted and finished with the other block maps, so a discarded or rewound batch discards it on both backends. This replaces the raw key beside the storage version.
The marker write is skipped when it is already set, including earlier in the same batch, and a debug assertion checks it is written inside one. Two comments now describe the marker map as it is.
The one-entry marker map becomes a block metadata map keyed by BlockMetadataKey with byte values, in the same DataID slot and the same atomic batches. Each key's getter and setter own its encoding; the authority-pruned marker is the key's presence.
The block metadata key AuthorityPruned now holds W, the height below which every authority is pruned. Inserts prune the one newly expired height when caught up. Startup walks W up to the retained window in chunks of 256 heights, one batch each, so restarts after the first catch-up do no work and an interrupted catch-up resumes from W. get_block reports pruned for heights below W.
A speculative read scans the pending batch, which grows with every removal, so a 256-height chunk was quadratic in its certificate count. The keys the routine reads are never pending. Also drop a redundant genesis filter and dead test setup, and share the test fixtures.
Only certificate indexes are read from confirmed state; block hashes and authorities are read speculatively again, so a caller that batches many inserts cannot skip a height. get_block never reports genesis as pruned. Tests now cover certificate removal across chunks, resume on RocksDB, and an insert behind a lagging watermark.
Decoding an authority checks every curve point and costs tens of milliseconds per block, which made the first-enable catch-up take days and slowed every insert. Pruning now removes the authority by hash without reading it. The certificate indexes of pruned blocks are removed by one pass over the small index rows at startup, and read as absent until then.
With one deletion per height, a 10,000-height chunk matches the certificate batch size; 256 was sized for about 40 deletions per height. Split the authority walk into its own function, inline the certificate flush, and fold the lingering-row checks into the startup test.
get_batch_certificate re-checks the watermark when the authority is missing, so a block pruned between the index read and the authority read returns None rather than a corruption error. Unit tests use small chunk and batch sizes so the certificate pass flushes mid-iteration, and a test pins the corruption error for blocks at or above the watermark.
A pruned store opened without prune-authority-data, or a newer storage schema, is an IncompatibleStore error. Cleaning would destroy a store another build can open.
Collaborator
|
Could you please re-open this PR from a branch in the aleo-labs-protocol/snarkVM repo? CI skips the lint and self-hosted test jobs for PRs from forks, so |
Collaborator
Author
Yes, as said in the comment above, thats my plan once I feel like the code is in good place. |
Collaborator
Author
|
Closing in favor of #3630 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Related to #3230 — Enable pruning of historic state.
The main benefits of the smaller state, is that initial participation and recovery will be much faster, making the network more robust.
BlockStoragecurrently keeps authority-related data forever. As the chain grows, this causes the RocksDB database to grow indefinitely even though consensus only needs recent authority information.In simple terms, a block contains two different kinds of information:
This PR keeps the permanent information, but removes old authority information once it falls outside the recent retention window.
What this PR does
When the
historyfeature is not enabled:Retain authority data for the latest
BatchHeader::MAX_GC_ROUNDSblocks. This is currently 100 blocks.Never prune the genesis authority (this appears to be optional and is currently done as an abundance of caution, mimicking he special treatment the genesis block has in the Ledger code).
When a new block is inserted, atomically remove:
AuthorityMap.CertificateMap.When an existing database is opened, remove authority and certificate entries that are already outside the retention window.
Perform startup cleanup in batches of at most 10,000 map deletions so opening an old production database does not create one enormous write batch.
Restrict the startup random-block integrity check to blocks whose authorities are guaranteed to remain available.
When the
historyfeature is enabled:get_block.The top-level
historyfeature is propagated tosnarkvm-ledger, and CI now testssnarkvm-ledger-storewith bothrocksandrocks,history.What remains stored
This PR does not prune:
get_state_path_for_commitmentpreviously reconstructed the complete block throughget_block. It now reads only the block height, previous hash, header, and transactions that it actually needs. This keeps historical state paths working even after the block’s authority has been pruned.Expected effect on
get_blockget_blockis intentionally unchanged: it still returns a completeBlock, and a complete block requires its authority.Consequently, in a non-history build,
get_blockwill fail for blocks whose authority has been pruned. Returning a partially populatedBlockwould change the meaning and invariants of that API.Estimated production savings
A temporary one-off estimator was written to measure this using real mainnet data. The estimator was intentionally removed after collecting the results and is not part of this PR but I can reproduce it if needed.
The estimator used the following procedure:
history, prune the expired authorities and certificate indexes, verify the retention boundary, and flush and fully compact the database again.The observed results were:
Removing the 10,000 sampled authorities and 449,289 certificate indexes took approximately 412 milliseconds. The forced full compaction used for the subsequent disk measurement took another 310 milliseconds.
For the production storage estimate, the observed on-disk savings were divided by the 10,000 sampled prunable blocks and multiplied by the 20,219,339 blocks that were prunable at the sampled mainnet height:
This estimates approximately:
of reclaimable production storage.
Applying the same linear extrapolation to the measured pruning time gives:
That is approximately 13.9 minutes for the initial production pruning pass.
This timing estimate covers the deletion pass performed while opening the database. This PR does not force a full RocksDB compaction in production. Deletions create tombstones, and the physical disk space is reclaimed later through RocksDB’s normal compaction process.
The estimator forced a full compaction only so that its before-and-after disk measurements would be comparable. The sample compaction took approximately 310 milliseconds, but it would not be reliable to extrapolate that number directly to a full production database: compaction can touch data outside the pruned maps and depends heavily on the database layout, existing compaction state and storage hardware.
These storage and timing numbers are estimates, not exact promises. Authority sizes vary between blocks, and the sample contains 10,000 evenly distributed blocks rather than the entire chain. The production database may therefore prune faster or slower than the straight-line estimate.
The one-off estimator also retained one additional experimental safety block. This PR retains exactly
MAX_GC_ROUNDSblocks, so the storage estimate is slightly conservative relative to the final implementation.Important remaining work: P2P synchronization
Important
The largest remaining question is how non-history nodes should participate in P2P block synchronization after their old authorities have been pruned.
In snarkOS, a node handling a
BlockRequestcurrently callsledger.get_blocks.Ledger::get_blocksreconstructs every requested block throughget_block.After this PR:
ledger.get_blocksfails, and the node cannot return the requestedBlockResponse.This is especially important for a new node attempting to synchronize from old heights. This PR does not yet decide:
The storage pruning itself is implemented and tested here, but the P2P synchronization policy and corresponding snarkOS changes remain a significant TBD before relying on pruning across all production peers.
Test Plan
The new tests cover the following behavior:
historypreserves old authorities and complete historical blocks.BlockMemoryand RocksDB.historypreserves all authority information.Local validation performed:
Noting that
test_bond_and_unbond_validatorfailed but this appears unrelated as it also happens without the changes in this PR.CircleCI continues to run
snarkvm-ledger-storewith--features=rocksand now has an additional--features=rocks,historyjob. Together, these configurations should include all new history and non-history tests.Documentation
The new retention helper, pruning behavior, and tests have code-level documentation.
No external operator documentation is updated in this PR. Once the P2P synchronization policy is decided, operator-facing documentation should explain:
historyafter a database has already been pruned cannot recreate the deleted authority data; the database must be restored or rebuilt.Backwards compatibility
This is an operationally breaking change in two important ways.
P2P block synchronization
A node running this version without the
historyfeature can no longer reconstruct complete blocks outside the authority-retention window.get_blockrequires the block’s authority, and that authority will have been pruned.An unupdated peer may request an old block range from a pruning node without knowing that the node no longer retains complete historical blocks. The pruning node will fail while building the
BlockResponse, and the requesting peer will not receive the blocks it expected.This PR does not yet introduce a way for peers to advertise their history-retention capability or avoid requesting historical blocks from pruning nodes. The P2P synchronization behavior must be addressed before pruning can be safely enabled across nodes expected to serve historical sync requests.
Irreversible database pruning
For a build without the
historyfeature, opening an existing database automatically removes expired entries fromAuthorityMapandCertificateMap. Subsequent block insertions continue pruning entries as they leave the retention window.This deletion is irreversible. Enabling
historylater does not recreate the deleted authority data. An operator who needs the complete historical database would have to rebuild the database from a source that still retains full history.This does not change consensus rules, block serialization, or network message formats, so I believe it should not require a
ConsensusVersionguard. However, the P2P behavior and automatic irreversible database deletion will make the release operationally backwards-incompatible and require explicit upgrade coordination.