Repository navigation
[Feature] Enable pruning of historic state #3230
Description
🚀 Feature
Add a new public prune_rocksdb_state<S, T> method, which takes in a rocksdb store and key type, as well as a range.
When called, it will queue a new type of SequentialOperation, which prunes the respective data.
The main benefits of the smaller state, is that initial participation and recovery will be much faster, making the network more robust.
Reacted by Copilot
Activity
vicsn commentedon Apr 24, 2026
@ljedrz what do you think of the proposed architecture? The only issue I see is that from a performance/blocking perspective we'd have to be mindful that we don't prune too much at once.
ljedrz commentedon Apr 24, 2026
Ok, so this would primarily apply to the
AuthorityMapunderBlockStorage, right? Too bad it's keyed by the block hash, making it unordered, but if this is used consistently (i.e. only the last N entries are ever stored), it would still be quite fast. That being said, if it's used consistently, then onlyMap::removewould be needed to achieve this, perhaps with the addition of a single removal pass of past entries in case some old entry is found, for which we could just introduce a single atomic block that removes past ranges in a loop (which could still just useMap::removeunderneath), until only the desired number is left.vicsn commentedon Apr 24, 2026
One more idea: we can mark any relevant getters to said data behind a compile-time
historycompilation flag, and make sure we only allow deleting certain data types whennot(history).ljedrz commentedon Apr 24, 2026
Also, I just realized that I don't see why this would have to be a sequential operation; if we're only removing old data (that's presumably unused), I don't think it could negatively impact anything else.
eranrund commentedon Jul 14, 2026
I've spent some time trying to wrap my head around this, and I'm wondering what your thoughts are regarding how
get_blockis currently implemented and used.Right now,
get_blockreconstructs a completeBlock, including its authority data. I don't think we would want it to construct a partial block.Consequently, if we prune old entries from
authority_map,get_blockwill fail for blocks outside the retention window. This appears to have several implications:get_state_path_for_commitmentcurrently relies onget_block, but it should be possible to read the data it actually needs - block height/hash, previous hash, header, and transactions - directly from the underlying maps.finalize_state_for_blockrelies onget_block, but it is behind thehistoryfeature. Assuming authority pruning is disabled whenhistoryis enabled, this should be unaffected.Old
get_authority,get_batch_certificate, and full-block queries would naturally be unavailable after pruning. I haven't yet checked what implications this would have for snarkOS APIs or serving historical blocks to syncing peers.Subject to those constraints, my understanding is that we would periodically prune
AuthorityMapand the correspondingCertificateMapentries, using theMAX_GC_ROUNDSround-based horizon, while retaining everything else. Is that what you had in mind for this ticket?vicsn commentedon Jul 15, 2026
That sounds great.
Indeed we could not prune the data if
historyis enabled.I'm not sure what functionality you refer to with rollback - in contrast to other chains our consensus does not allow re-orgs, and the only rollback functionality we support at the moment is plainly replacing the entire db folder.
Along with the PR it would also be nice to have an estimate for how much data we'll be saving in production.
eranrund commentedon Jul 15, 2026
I was referring to
BlockStore::remove_last_n, however looks like this is only ever called code withn=1- so it won't be affected.eranrund commentedon Jul 16, 2026
Draft PR is available at #3329.
The PR description is a bit long, apologies for that - I tried to capture what this does, how it does it, what it gets us, what remains to be done and what are the operational gotchas this will bring on if we choose to proceed with it.
vicsn commentedon Jul 16, 2026
I thought it was a great write-up, there's just a lot of surface area to discuss.
The P2P sync topic needs discussion indeed. Given that non-history nodes drop old Authority data anyway, maybe the Authority field can simply be made optional/empty and effectively ignored by syncing nodes?
For certain infrastructure like our CDN we can, for simplicity, stick to using a history node.
Note snarkOS has:
Validators only sync from other validators, clients from either clients or validator.
As an extra safety measure perhaps we can roll this out to clients first before rolling it out to validators. On the other hand, splitting the feature in two risks abandonement halfway.
eranrund commentedon Jul 21, 2026
Thank you @vicsn!
Here are some of my thoughts on the possible implications/course of action for making
Authorityoptional.Data shape
We can either go with
Option<Authority>or a newAuthorityvariant. I thinkOption<Authority>will serve us better: a new variant flows into everymatchdeep inside the call stack, whileOptionforces explicit handling at the call sites that actually ask for authority data.Wire representation
Blocks that have an authority keep today's existing (v1)
Blockencoding (and JSON shape). Authority-optional blocks get a v2 encoding (bumped version byte). This implies we need to gate pruning behind a ConsensusVersion update, which I think will have other benefits detailed below.ConsensusVersion
We'll gate the onset of pruning - not just the encoding - on the new ConsensusVersion. Pruning isn't technically a consensus change, but it breaks the existing contract that any synced peer can serve any historical block. Once we enter the world of potentially returning blocks with no authority ("v2 blocks"), we only want to return them to nodes that know how to parse them.
In other words: before height H every node retains and serves full history; at H, nodes without the history feature begin pruning and may hand out v2 blocks with no Authority.
Implementation-wise, both pruning sites in the draft PR (the startup pass and insert-time pruning) become conditional on the activation height. We also need a strategy for the backlog when a running node crosses H - a background sweep, amortized deletion on insert, or simply deferring to the next restart.
Note that v2 blocks only apply outside the GC window - blocks inside the GC window always contain the Authority data. Receivers must enforce this too: an authority-less block at a height inside the GC window is invalid and rejected.
Validation guarantees
Today, if I didn't miss anything, P2P blocks are fully verified at every height (checked deserialization + committee-aware
check_next_block); CDN sync verifies nothing (unchecked deserialize, nocheck_next_block).In the pruning world, the validation done on P2P syncing is potentially downgraded for pre-GC blocks.
A few potential paths beyond simply accepting the downgrade as a worthy tradeoff for the disk savings:
BlockLocatorscould advertise whether Authority data is available, letting nodes prioritize fetching from full-history peers.The machinery half-exists -
REDUNDANCY_FACTOR, currently set to 1 in production ("we do not use this in production yet").Caveat: the existing cross-check compares responses by struct equality, and a pruned and a full form of the same
block are no longer equal, so we'll need to take that into account.
This means any node offline for less than a day keeps today's fully-verified sync. Only fresh bootstraps and long-dead nodes ever touch authority-less blocks or the CDN, which also keeps routine catch-up load off the history-node/CDN tier. This also helps deal with the 100 round GC limit edgecase.
Alternative: Maybe we do keep
Authoritymandatory...Building on the deep-retention idea: maybe we don't make
Authorityoptional.BlockLocatorsadvertising each peer's earliest-available height, requesters route deep history to history peers.This eliminates the v2 encoding (every block on the wire has an authority), the
Eqconcern (an authority-lessBlocknever exists as a value), and the entire validation downgrade - every block any node accepts stays fully verified, at every height, exactly as today.vicsn commentedon Jul 23, 2026
Thx for the very readable write-up again!
What do you mean with the 100 round edge case? Also just a reminder nit: 100 consensus rounds translates to at most 50 blocks.
I'm hesitant to make history nodes a core property of consensus, and would like to keep things simple to minimize security analysis. So I think the Optional Authority approach is fine. Client sync logic has a lot of robustness issues anyway which need to be patched separately, and due to the security guaranteed by hash-chaining only fully verifying the most recent tip should be fine.
I now remember we had #2841 (comment) which likely touches similar parts of the code so I'll just share a few notes. Instead of dropping Authority, it introduces a more compact representation. But the problem with that design was that while it saved some bandwith, sender and recipient spent some compute by converting to/from the compact representation. Your new approach will save us bandwith and storage at no compute cost. 🥳
eranrund commentedon Jul 24, 2026
My pleasure, and thanks for taking the time to read it :)
Re the edge case - I was referring to this part of your comment:
My point was that a retention window much larger than the GC window sidesteps those cases: certificate-based Sync only ever requests data within the GC limit, so peers retaining, say, a day of authorities can never have pruned something a validator legitimately asks for. (And thanks for the rounds-vs-blocks correction.)
On the broader direction - I hear you on keeping the security analysis simple, and I want to make sure we're comparing the two options on the same axis before committing. Making
Authorityoptional is simple from a consensus-analysis standpoint, but implementation-wise it means a v2 block encoding,Optionhandling at every authority call site, the struct-equality issue in multi-peer response comparison, and the validation-downgrade question for pre-GC blocks. The deep-retention alternative (keepAuthoritymandatory, retain ~24h–1week of full blocks, don't serve older blocks) avoids all of that, and - since the block format doesn't change - SDK clients, explorers, and anything deserializing JSONBlocks should keep working untouched.Notably, the CDN path for "deep" history already exists: snarkOS CDN sync resumes from
latest_height + 1at startup, so a node offline longer than the window would catch up from the CDN on restart exactly as it does today. The one gap is a node that stays running but falls behind beyond every peer's window without restarting; that would need a small runtime-CDN-fallback change (or an operator restart, which is likely a reasonable approach since if the retention window is large, something has gone wrong for it to fall this far behind...).vicsn commentedon Jul 27, 2026
Good point, indeed since we rely on CDN already, that is an even smaller conceptual change to make. I'm on team "don't make Authority optional" now.
I would suggest that we don't let peers advertise themselves as history or non-history. If a peer falls behind the designated threshold they can just rely on the trusted CDN (happens automatically) or a snapshot (can be done manually).
If we have our ideal 2.5 second blocktimes we process
60*60*24/2.5 = 34560blocks in a day. I would set the threshold to 100.000 blocks from tip for now.eranrund commentedon Jul 30, 2026
@vicsn I just submitted aleo-labs-protocol/snarkOS#4360, which is an attempt at going with the 100k threshold approach + CDN recovery for a node that falls very far behind.
Would appreciate any thoughts or feedback on it!
Thank you :)