Remove the LMDB chunk store and restore the close-group penalty - #218
Open
grumbach wants to merge 76 commits into
Open
Remove the LMDB chunk store and restore the close-group penalty#218grumbach wants to merge 76 commits into
grumbach wants to merge 76 commits into
Conversation
…up chunk The chunk store can only grow. LMDB returns a deleted page to its own free list and never to the filesystem, so a node that deletes chunks frees no disk. Moving the fleet onto a store that does return space means a node short of disk will have to give up some chunks while it moves the rest across. It cannot avoid being seen doing that, and it cannot stop the consequence, because the penalty is the auditor's decision, not the audited node's. So the auditors stop one release ahead of the migration, and this is that release. What is withheld is deliberately narrow: only the accusation "you did not have a chunk you were supposed to be holding". That covers the responsible-chunk audit, the fresh-replication possession check, the prune audit, a sole-source replica hint whose sender then denies possession, and the fetch paths where a peer that answered Present could not serve the bytes. A node giving up chunks produces every one of those, so withholding some and not others would stop only some of its accusers. The commitment-bound subtree audit is untouched and still penalises. That is not a compromise, it is what makes the rest work: a migrating node reduces its signed commitment precisely so its peers hold it to the smaller claim, and suspending that enforcement would make the reduction meaningless. A sole-source hint the close group rejects outright is also still punished, because that is a claim about a key that does not exist rather than about the sender's own storage. Audits of both kinds keep running and keep recording. Only the trust event is withheld, and the record they leave is how we will know when it is safe to switch the penalty back on, which is a later release rather than a compiled-in expiry so the date can move on evidence. The switch is a build constant, not a configuration field: a node writes its effective configuration back to disk, so shipping it as a setting would bake this release's value into every operator's file and the next release would change nothing. It is initialised from that constant rather than defaulting to "penalise", so a construction path that never applies the policy behaves like this release instead of the previous one. Known cost, accepted: between this release and the one that restores the penalty, a peer that publishes no commitment at all can answer Present, fail to serve, and pay nothing for it. It is bounded by the restore and visible in the audit record. See ADR-0012.
…ging feature `AuditType::as_str` was gated on the `logging` feature because every caller was inside a log macro, which compiles to nothing when that feature is off. The penalty helper takes the label as an ordinary argument, and arguments are evaluated whether or not the macro that consumes them survives, so a `--no-default-features` build stopped compiling. Ungated rather than worked around at the call sites: it is a `const fn` over a three-variant enum returning a string literal, so it costs nothing in a build that never logs, and passing hand-written literals instead would let the structured-log labels drift from the enum they are meant to name.
…iled Review found that `FetchResponse::Error` was routed through the suspended lane, and it should not be. Its only producer is the responder's storage read returning an error: an I/O fault, an exhausted descriptor table, or a chunk whose bytes no longer hash to their address. A peer that simply does not hold the chunk answers `NotFound`, which is a separate variant and stays suspended. Nothing about a node giving chunks up produces an error answer, so withholding the penalty for one hid real faults for no benefit. The response mapping and the charging decision are now two small functions used by the real paths, so the meaning a responder puts on the wire and the charge a fetcher applies cannot drift apart. Tests pin both: a key the node does not hold reads as a plain miss and is answered `NotFound`, a failed read is answered `Error`, and the two answers are classified as different faults. This brings the count back to the six call sites the ADR describes, and the ADR now says explicitly that a failed responder read is not one of them.
ADR-0012 is already taken. `origin/main` carries `ADR-0012-unresolved-verification-retry-backoff.md` from a PR that merged after this branch was cut, so this branch does not contain it and nothing here noticed: the governance check looks for duplicate numbers among the files in the branch, sees one file per number, and passes. The duplicate only exists once the two are merged together, at which point main carries two different decisions wearing one number. 0013 is taken as well, by the settlement-version ADR on another open branch, so the next free number is 0014. Numbers are claimed on merge order and nothing reserves them, so this was checked against main and against every open pull request rather than against main alone.
…and migrate onto it
LMDB returns a deleted page to its own free list and never to the filesystem, so
a node that deletes chunks frees no disk. Last week the fleet deleted 2.29
million chunks and got back zero bytes. Operators read that as a bug and are
tempted to wipe node directories to reclaim space, which costs the network real
replicas. There is no partial way out: compaction needs free space equal to the
live data, which is exactly the condition a full node does not meet, and it does
not get us off LMDB anyway. Disk comes back exactly once, when chunks.mdb is
removed whole.
THE STORE
One immutable file per chunk, at chunks/<last two lowercase hex characters of
the address>/<full 64-character lowercase hex address>. 256 shard directories,
one level, recorded in a layout marker at creation.
Suffix, never prefix. A node holds keys it is among the closest to, so its
holdings share roughly log2(N / close_group_size) leading bits with its own node
ID, and that shared prefix grows as the network grows. At today's fleet size a
two-hex prefix already resolves to about two directories, and past a million
nodes even four hex resolves to one. The address is a BLAKE3 output and
close-group membership constrains only its leading bits, so the trailing byte is
uniform by construction at every network size.
Lowercase hex because NTFS and default APFS fold case: under an encoding with
both cases two distinct keys can share one case-folded filename, which is a
silent overwrite. No hex string can spell a reserved Windows device name.
The filesystem is the only authority. The key set is a BTreeSet rebuilt at every
open from directory entries, names only, no stat and no content read. There is
no sidecar index, because a persistent one cannot remove reconciliation: commit
the index first and a crash leaves a phantom key, rename the file first and a
crash leaves an unindexed file, and repairing either means reading the
filesystem anyway. Every in-memory mutation mirrors a filesystem operation that
has already completed, never one that is about to.
Writes are a temp in the destination directory, flushed, renamed, and only then
admitted to the index, so a name can never appear on partial content: the name
is the hash. Reads are bounded, refuse anything that is not a regular file, and
repair a corrupt or missing chunk from the network by dropping it from the key
set. Deletes unlink and return the blocks immediately, which is the entire point.
THE MIGRATION
A node that fits its payload copies everything and then removes chunks.mdb. It
is never unable to serve, so it needs no coordination.
A node that does not fit copies closest-first, then commits to what it can hold
while continuing to serve everything it ever committed to, and only then gives
the rest up. Serving reads the union of both stores; the commitment reads the
file-backed set. A node is at worst over-honest.
Nothing is given up without three things being true, in this order:
- the node is not near the front of the group for that chunk (measured against
the admission width the pruner already refuses to delete inside)
- its close group has demonstrably RECEIVED its reduced commitment, proven by
those peers answering a neighbour sync that carried it; until they have the
smaller key set they audit it against the one it used to hold
- all but one of the chunk's current close group has answered a cryptographic
possession challenge over a nonce it has never seen, and is itself currently
publishing a commitment
That last point is the pruner's own evidence, reused deliberately. The cheap
VerificationRequest was not enough: it carries a self-reported present flag, and
a node that has silently lost a chunk still answers yes.
Close groups migrate in waves derived from a hash of each node's own ID, so about
two of seven give chunks up at a time. If every holder went at once none could
prove to the others that a copy survived and the group would deadlock waiting on
each other. A host-wide advisory lock separately serialises nodes sharing a
volume, which is a different question: one machine's disk rather than one chunk's
replicas.
Before chunks.mdb is removed, every chunk both stores hold is re-hashed and
rewritten from the legacy copy if it disagrees. A filename is not proof the bytes
behind it are good, and the startup scan reads names only. Removal itself renames
the environment aside and flushes the parent before recording the migration as
finished, because remove_dir_all is not atomic and a partial failure would
otherwise leave a node claiming completion over a half-deleted store.
The destructive step is off in this release. It ships enabled in the next one,
once the fleet has been seen bridging without incident.
WHAT THIS COSTS
There is no rollback once a node has removed its legacy store; the staged rollout
is the only control. A node whose close group is also short of disk will not get
possession proofs, will not free its disk, and will tell its operator to add
storage, which is the correct answer under "no data loss" but means the migration
does not complete unattended everywhere.
The paid list is still LMDB. It is a fixed 256 MiB map that contributes nothing
to the disk problem, but it is why heed cannot be dropped yet.
See ADR-0012.
…testing Four end-to-end tests assert that a peer loses trust. With the close-group storage penalty now defaulting to whatever the release ships, those assertions silently became a statement about the release rather than about the mechanism they were written for, and they passed or failed on test ordering: one test happened to leave the switch on for the next. Each now sets the switch explicitly, so it tests possession, pruning or repeated failures rather than the release it was compiled into, and the result no longer depends on which test ran first.
`FileStore::all_keys` is async and awaits nothing on purpose: the key set is already in memory, and its callers across the replication engine cannot all be made synchronous in one change. It carried an allow for `clippy::unused_async`. That lint was renamed, so a newer toolchain than the one this was written on fires `clippy::unused_async_trait_impl` instead and the build fails. Allowing both, under `unknown_lints` so whichever name the compiler in use has never heard of stays quiet, rather than pinning a toolchain or restructuring an interface to satisfy a lint.
Retirement is refused on Windows because NTFS documents no ordering between the rename that publishes a copied chunk and the deletion of the store it came from, and there is no way to flush a directory through the standard library. That guard was reading an environment variable directly, which meant the three tests that exercise retirement could not clear it without mutating process-global state, and they failed on Windows CI while passing everywhere else. It is now `storage.migration.allow_windows_retire`, still defaulting to off and still honouring the environment variable for its default. Unlike the release switches this one does persist in an operator's configuration, deliberately: someone who has tested power loss on their own hardware should not have to re-assert that on every start, and the decision is theirs rather than the release's. The three retirement tests now clear it the way such an operator would, and a Windows-only test covers the refusal itself, so the guard has coverage on the platform it exists for rather than only being asserted away.
…found Five findings from an adversarial review of this branch. Four were real. THE MIGRATION WAS NEVER STARTED `migration::run` had no caller. Porting this work onto a newer base involved resetting a bad copy of `node.rs`, and the hand-written wiring that went with it was never redone: the store opened in its bridging phase, dual-wrote, served the union of both stores, and then sat there. No chunk was ever copied out, no commitment was ever narrowed, no legacy environment was ever removed. The one thing this work exists to do did not happen on any node. Nothing caught it. Every test constructed the store directly, and a node with no legacy environment starts no migration, so the end-to-end suite could not see the difference either. There is now a test that the driver is reachable from its entry point and returns on its own when there is nothing to do. A DELIVERY COULD BE CREDITED TO A ROOT ITS PEER NEVER SAW A neighbour sync snapshots the commitment root it will carry, sends it, and on the reply records that peer as having received it. If a rotation happened in between, the recorded root was the new one. The doc comment claimed a rotation empties the recipient set; nothing did that, and the set was instead cleared lazily by the first late reply, which then credited itself to a root it had never been sent. Rotation now clears the set, and the caller names the root it actually put on the wire so a reply that arrives after a rotation is dropped rather than miscounted. This gates whether a node may give chunks up, so a wrong count here means shedding while the close group still audits against the larger key set. A SUCCESSFUL REPLACEMENT SYNC DID NOT COUNT When the primary peer does not answer, the round retries with a replacement, carrying the same commitment. The reply proves delivery exactly as the primary's does, but only the primary path recorded it. A node whose close group is slow enough to fall through to replacements could never accumulate enough recipients before the next rotation reset the count, and would wait forever. That is the node most likely to be short of disk in the first place. A FAILED FREE-SPACE QUERY READ AS A FULL DISK The capacity verdict collapsed both error cases into "full". The verification cycle treats full as a standing condition worth minutes of backoff, and documents that a failed query must not be read that way, because it says nothing about available space and may succeed on the next pass. A transient fault on a network mount would have stalled probes and promotes across every pending key on a node that was not full at all. The three-way answer is restored. ONE RELEASE SWITCH, NOT TWO Two constants of the same name existed, with two environment overrides, on either side of the same decision: whether peers withhold the penalty for not holding a close-group chunk. Nothing coupled them. Setting one without the other gave a node willing to give chunks up while every peer applied the full penalty, which is the outcome the release ordering exists to prevent. The migration now reads the switch the auditors read. The fifth finding, that two same-named constants could drift, is the one above.
The wiring that starts the migration went missing during a rebase and every test still passed. That was possible because each test built the store directly and drove its pieces, and a node with no legacy store starts no migration, so nothing in the suite could tell a working migration from an absent one. Two tests close that, both driven through `run`, the same entry point node startup calls: A node with room to hold its chunks starts with 24 chunks in a real LMDB store, and finishes with `chunks.mdb` gone from the filesystem, all 24 readable out of files, and each one under the suffix shard its address names. That is the whole purpose of this work, asserted rather than assumed. A node that cannot fit its chunks and cannot prove anyone else holds them keeps both stores, stays in the bridging phase, deletes nothing, and serves every chunk throughout. That is the case that must fail safe: refusing costs the node disk, proceeding would cost the network data.
The wiring that starts the migration went missing once and nothing noticed: the store opened, dual-wrote, served the union of both backings, and never freed a byte. A node without a legacy store starts no migration, so the absence looked exactly like the normal case. Two guards, because one was clearly not enough. At runtime, a node that still has a legacy chunk store and no task migrating it now logs an error naming the condition, rather than running indefinitely in a state where its disk can never be reclaimed. It is a wiring fault, so it says so. In tests, `should_migrate` is the single predicate both the spawn site and its test use, so "does this node need migrating" cannot be answered one way by the wiring and another way by whatever checks the wiring. The test drives both cases: a fresh node must not get a task, and a node with an LMDB store must.
An adversarial review of this branch found three. All are the same family: a gate that looked sufficient but was measured against the wrong thing, or at the wrong moment. THE POSSESSION BAR WAS SET BY WHOEVER HAPPENED TO ANSWER The number of proofs required was computed from the peers that qualified, not from the close group. With one neighbour publishing a commitment, one proof was enough. Two last holders of a chunk could each see only the other as qualifying, each demand a single proof, each receive it from the other, and both delete. The bar now comes from the whole group and only qualifying peers count toward it, and a routing view too thin to see a full group is not evidence about that group at all. THE GATES WERE CHECKED HOURS BEFORE THE DELETION Rank, commitment delivery and possession were established, and then verification ran, which re-reads the entire store and can take hours on a large node. Nothing was rechecked afterwards. In that window peers leave, replicas are pruned elsewhere, and this node can become the last holder while its own reduced commitment no longer names the chunk. Verification now runs first and every network gate is re-asked immediately before the removal. A CHUNK COULD ARRIVE AFTER THE GATES AND BE DELETED BY THEM A write that reaches the legacy store and then fails to write its file adds a legacy-only key. Such a key is in no commitment, so the answerability check could not see it, and it would have been destroyed having passed nothing. The removal now takes the exact set the gates cleared and refuses if anything else has joined it, which the critical section proving sole ownership makes authoritative. Also: a retirement whose rename failed left the store closed, so the node could no longer serve chunks that lived only there, and the next tick saw no handle and reported success. It now reopens the legacy store and says whether that worked.
…to run there
Retirement was refused on Windows, on the grounds that there is no way to flush a
directory through the standard library and Microsoft does not document
`MoveFileEx` as durable at return, so the copied chunk could not be shown to have
reached the disk before the old store was deleted.
That was not a solution. It left Windows operators with exactly the problem this
work exists to remove: a store that only grows. Refusing to solve a problem for a
platform is not the same as solving it.
There is a documented way, and it is to stop using a rename there. Windows now
creates the chunk under its final name with `create_new` and flushes it.
Microsoft documents that creation metadata is cached and that `FlushFileBuffers`,
which `sync_all` calls on Windows, is how it is flushed. A successful create,
write and flush is therefore a durable publication under a documented contract,
with no rename and no directory flush involved. The rename path stays everywhere
else, where the directory flush does the same job.
The cost of publishing in place is that a crash mid-write leaves a partial file
wearing a real chunk name, so three paths now refuse to trust a name:
- a write that finds the name taken re-reads and verifies it, and replaces it
when it is wrong, instead of reporting a duplicate and discarding the good
copy that had just arrived to repair it
- the read path already verified and repaired
- the pre-retirement pass already re-hashed everything both stores hold
Separately, the migration waves did not work at all as configured. They opened at
0, 24, 48 and 72 hours from first start while nothing could shed until hour 72,
so every wave was open the moment the first one could act and a close group would
have migrated together, which is the pile-up the waves exist to prevent. They now
open from the end of that hold, and a test asserts the stagger under the shipped
defaults rather than under either setting alone.
Four fixes from the production review, each one a path where a node could delete its only copy of a chunk or never reclaim its disk at all. Commitment recipients are now intersected with the close group as routing sees it at the moment of the check. Counting a peer that received the reduced commitment and has since left the group is no evidence about the peers that will actually audit this node, and it let a node shed chunks while its real neighbours still held it to the larger key set. A directory flush that fails on the publish path is now reported instead of swallowed. That flush is what makes the rename durable, and a copy reported as successful is what authorises deleting the legacy store, so discarding the failure let a power loss take the directory entry after the only other copy was already gone. The migration is no longer skipped when the replication engine fails to start. A node with a legacy store depends on the engine for the commitment state, the routing view and the possession challenges, so it now refuses to start rather than running on forever serving from both stores. The engine build and the migration spawn moved into one function so the two cannot come apart again. Shutdown stops protocol routing before waiting on the migration, and the wait is bounded at 30s. Inbound traffic kept starting new legacy reads, which could stop the drain from ever completing and hang the process. Tests: a departed peer no longer opens the commitment gate; an unflushed publication is not reported as stored; and a fully built node holding a legacy store is asserted to be migrating it. That last one goes through build() rather than the spawn helper, because the failure that already happened here was the call site going missing, and it was verified by deleting the spawn and watching it turn red.
…ws build The Windows build failed on a dead non-Unix helper, which was the visible half of a real gap underneath it. Splitting the publish path on `windows` rather than `unix` was the wrong boundary. The rename-plus-directory-flush route is the Unix route, and every other platform should take the create-in-place route, which needs no directory flush at all. Gating on `unix` removes the dead definition and stops the two halves drifting apart. The repair path had the same gap the publish path had. `write_and_replace` rewrites a chunk whose bytes do not match its address, from the legacy store, during the pass that decides whether the legacy store can be deleted. It finished with a best-effort directory flush, so a repair could be reported as done while a power loss could still undo it, leaving that chunk with the wrong bytes and no other copy. On Unix the flush failure now propagates. Off Unix the replacement overwrites the existing file and flushes it, changing no directory entry, which is durable under a documented contract. That overwrite is not atomic, which is safe only because a crash means no report was produced and nothing was deleted, so the next start repairs it again from a store that is still there. Small-file writes now go through the rename retry as well. The layout marker and the migration state are rewritten while the node runs, and off Unix a scanner holding a handle for a few milliseconds turned an ordinary rewrite into a hard failure. Both platform families were compiled and linted with warnings denied.
Retirement now ships on. Deleting the legacy environment is the only step that returns disk, and a build with it off migrates every node and reclaims nothing, which is the condition this work exists to end. Every gate in front of it is unchanged, and a single node can still be told to keep both stores. A test asserts the shipped configuration retires with nothing set by hand. Durability. Publishing a chunk whose name was already on disk returned success without flushing the directory, so a previous attempt whose rename landed and whose flush failed could be laundered into a copy that authorises deleting the last other one. The flush now covers every successful return. Creating a shard directory flushed its parent best-effort and marked the shard usable regardless, so the first chunk written into a directory that was never made durable counted as stored. That flush is load-bearing too now. Reads. A verifying read that finds rotted bytes throws the file away, and until the key is put back in the union view it appears to live in neither store. Retirement decides it may delete the environment by proving it is the only holder of the handle, so a read that took its handle afterwards could find its fallback already gone. Both read paths now take the handle before they touch the file. Duplicate writes. A name on disk is not proof of the bytes under it: off Unix a chunk is created under its final name before it is written. The protocol handler acknowledged `AlreadyExists` from names alone and the file store had an index fast path in front of the verification, so a good copy offered to repair a damaged chunk was thanked and discarded. Both now compare the length first, one metadata call, and a mismatch falls through to the real write. Availability. The rollback copy into the legacy environment could veto a PUT the file store had ample room for: the capacity verdict is optimistic and LMDB can still refuse a write. It is best-effort now, as its own comment already said it should be. The file write is what decides the PUT. The volume lock is keyed by the filesystem rather than by the path beside the root, so two nodes on one disk no longer take two different locks and copy at once, and only genuine contention counts as contention: a filesystem without locking used to leave a node waiting forever for a holder that did not exist. A node whose file-backed set is empty commits to nothing, so waiting for its close group to receive a commitment that does not exist stranded its disk permanently. That gate is skipped when there is nothing to commit to; the possession check that protects the data still runs. Shutdown stops the protocol children as well as the loop that spawns them, and a migration that overruns its grace is aborted rather than left detached over the teardown it depends on. Repair takes a real reservation instead of an unreserved check. A directory entry the scan cannot identify now fails the scan rather than being counted as "not a file" and dropped from the index.
Holding a legacy handle for the whole of every read closed the window where a read could lose its fallback, but it replaced one problem with another: the check that authorises retirement is sole ownership of that handle, and on a node serving any traffic there would always be another holder, so retirement would never run and the disk would never come back. Neither ownership alone nor holding a handle states the actual requirement, which is that no read is in progress. A read that has decided the file store cannot answer, and has not yet taken a handle, holds nothing and is invisible to an ownership check while being exactly the reader that must not lose its fallback. So reads now take a shared guard for their whole duration and retirement takes it exclusively before it takes the environment. Because the lock is fair, a waiting retirement stops new readers rather than starving behind them. Test: a read in flight blocks retirement, and retirement completes as soon as that read finishes. Verified by removing the guard and watching it fail.
Retirement moves the legacy environment aside and then deletes it under the new name. The flush in between was best-effort, so if the rename had not reached the disk when the delete landed, a power loss would bring the environment back under its old name with its contents already removed, and the next start would find an environment it cannot open. The flush now reports, and a failure stops before the delete. The migration is still recorded as finished, because the node is serving from files and needs nothing from the environment; the tombstone is simply left for the next start to sweep. Off Unix there is still no way to flush a directory through the standard library, and the helper says so rather than implying otherwise.
The test that copies a whole store and watches the legacy environment go was setting the retirement switch itself, so it proved the machinery worked without proving the release turns it on. It now sets nothing: if the shipped default ever goes back to off, this fails along with the two tests that check the default directly, rather than passing on a value no node would have.
Nine fixes, all on the path that deletes the legacy environment. The set of keys the removal is allowed to destroy was captured after the gates rather than before, so a key that joined between the last gate and the capture counted as approved having passed nothing. It is snapshotted first now, every gate is asked about exactly that set, and a set that moved while the gates ran stops the tick. Retirement is now recorded durably before anything moves. The rename that puts the environment aside cannot be shown to be durable off Unix, so a power loss could bring it back with its contents already deleted and the node would fail to start on it. A marker created with the same create-and-flush that publishes a chunk says the file store was proven to hold everything; a start that finds it finishes the removal instead of opening the remains. It is cleared when the removal completes and when a recoverable failure sends the node back to bridging, so it only survives a crash. The startup sweep of a leftover tombstone now flushes before deleting, for the same reason. The pre-retirement proof establishes that names are durable, not only that bytes are. A publish whose rename landed and whose directory flush failed leaves a name nothing goes back to flush, and re-reading the right bytes from it does not make it survive a power loss. The pass flushes the chunks directory and every populated shard first, and a failure is a proof it did not produce. Writes and deletes take the retirement guard as reads do. Retirement waits for the legacy environment to go idle, and work that could keep starting in it made that wait unbounded. A client offering a chunk this node already holds is now answered from the bytes rather than the name, and a damaged copy is repaired from the offer. Comparing lengths caught an interrupted create but not rot, and either way acknowledging the offer discarded the copy that would have fixed it. The close group was derived one member too wide: the self-excluding routing call returns close_group_size remote peers while the threshold is computed from a group that includes this node, so four real neighbours plus one peer outside the group cleared a bar meant to need five real ones. Both the commitment-delivery check and the possession check now use the self-inclusive call, as the pruner does. A node waiting on its close group no longer holds the volume against every other node on the machine: that wait is a network condition that may never resolve. A six-hour cap backstops any branch that turns out not to give the lock back on its own. Queued request handlers give up when shutdown starts rather than acquiring a permit and beginning fresh storage work under a store being torn down. Off Unix the volume lock is keyed by the volume root rather than by each node's own parent directory. A repair invalidates the capacity measurement, because replacing a short file with a full one adds real bytes the cache does not know about. Tests: a damaged chunk, short or rotted, is repaired from the copy being offered; an interrupted retirement is finished by the next start rather than reopened. Both verified by breaking the fix and watching them fail.
… deleted Holding the exclusive retirement guard through the whole removal meant every chunk request on the node waited behind `remove_dir_all` on a store that can be hundreds of gigabytes. That turns the one moment the migration pays off into an outage. The guard is released once the handle is out and the directory has been renamed aside, which is the point after which nothing can reach the environment: no handle exists and no code looks for the new name. The deletion that follows is slow but reaches nothing anyone is waiting on. It is also released on the path where nothing was taken, rather than being held to the end of the function for a tick that is deferring anyway.
… round Seven fixes. Most are consequences of the previous round's fixes rather than of the original design, which is what a fourth pass is for. The retirement marker was trusted on sight, and it can be stale: a rename that fails recoverably clears it, and that clearing could itself be lost. The environment may have taken a key since that has been through none of the gates. Opening it is now the test. One that opens cleanly is intact, so it is kept and the migration starts again from the copying stage with every gate re-run; only one that cannot be opened is treated as the remains of an interrupted removal, which is the case the marker exists for and the only case where deleting is both safe and the only way the node starts. Clearing the marker is durable now too. The duplicate write path compared lengths, which catches an interrupted create but not rot, and answered "already have it" without reading. It reads now, and repairs from the offered copy on a mismatch. A chunk held only in the legacy environment was assumed good for the same question; those bytes can be wrong too, and when the copier finds out it drops the key from the union view, so refusing the good copy would have left the node holding nothing. It is read and compared, and the offer is taken if it does not match. The six-hour cap on holding the volume lock was armed only on one of the two paths that take it, so a node that took it while copying and then sat waiting could still hold it forever. Acquisition is one function now, which stamps every time, and giving the lock up at the cap starts a cooldown so another node actually gets it rather than losing the race to the one that just had it for six hours. The possession threshold was derived from however many peers routing happened to return. A view that has lost a peer lowered the bar exactly when it should not be trusted; it comes from the configured group size now, and a group that is short, or that still contains this node, is not evidence. Off Unix the volume lock is resolved to an absolute path before the volume is read from it, so two nodes started from different working directories on one drive do not each take their own lock. The deletion of the retired directory runs on its own thread, so shutdown can walk away from a recursive delete of hundreds of gigabytes rather than sitting through it; the marker means the next start finishes it. Tests: an intact environment is never deleted on a stale marker, an unopenable one left by an interrupted removal is finished off, and a legacy-only chunk whose bytes are wrong is replaced by the copy being offered. Each verified by breaking the fix and watching it fail.
Deciding whether a leftover retirement marker is stale means opening the environment, and opening it is a full key scan to derive which keys the file store does not have. The environment was then dropped and opened again by the ordinary path, so a node that came up after an interrupted removal paid for that scan twice. The handle it proved was worth keeping is now the one it keeps.
The fifth review round found that using "failed to open" as evidence of a half-deleted environment was wrong, and it was the load-bearing step of the previous round's fix. Opening an environment queries free space, maps the file, takes a write transaction and scans every key, so a full disk, a permission change, a mapping limit or a transient fault all look exactly like corruption. Deleting on any of those destroys a perfectly good store. The mark now goes inside the directory rather than beside it, and is written only after the rename has already succeeded. A directory that reverts to its old name reverts carrying its own evidence, so what it is no longer has to be inferred from anything. There is nothing to cancel, so nothing can go stale: the previous design needed the mark cleared when a retirement was abandoned, and a clearing that failed or was lost would authorise deleting an environment that had since taken a chunk. A missing handle is no longer read as a finished migration. A rename that failed and could not be reopened leaves the directory on disk with no way to read it, and the driver would have logged the migration complete over a store still holding chunks nothing else could serve. Reading a chunk to check it now has four answers rather than two. "Could not read it this time" was being treated as "wrong", and off Unix replacing a chunk truncates it in place, so a transient fault could turn a healthy sole copy into an empty one. The duplicate check holds the key's critical section for the whole of it, so the pruner cannot delete both backings between the read and the answer and leave the offered copy refused for a chunk the node no longer has. Deleting a retired directory runs on a detached thread that nothing waits for. On the blocking pool a normal runtime shutdown waits for it anyway, and in the migration task an abort is not observed until the call returns, so the advertised shutdown bound did not apply to a recursive delete of hundreds of gigabytes. The startup sweep is detached for the same reason: a node should serve immediately rather than wait out a leftover deletion. The volume-lock cap now distinguishes using the lock from sitting on it. Copying and verifying are the exclusive disk work the lock exists for, and a store large enough that verification runs past the cap would have had the cap interrupt and restart it, which is the cap causing the problem it prevents. Tests: an environment carrying no mark is kept however badly it reads, one carrying its own mark is removed whatever it is named, the mark survives the rename it exists to outlive, and a lost handle beside a live environment blocks retirement.
A start that found a reverted environment cleared any existing tombstone before renaming, which put a synchronous recursive delete back on the path that had just been cleared of one. The node would have waited it out before opening its store. Retired directories are now named so they cannot collide: the environment is moved under whichever retired name is free, and the sweep detaches a deletion for each one it finds rather than assuming there is at most one. Nothing on the startup path deletes anything itself.
…e it durable The sixth review round found that the previous round's mark was neither required nor durable, which meant its guarantee did not hold. The sweep deleted anything wearing the retired name without looking inside it. That name comes from a rename, and the rename happens after every gate, with the mark written straight afterwards; a crash in between leaves a whole environment wearing a name that says otherwise. Such a directory is now restored to its own name and the migration runs again, and when both names are taken neither is touched and the operator is told which the node is using. The mark itself is flushed along with the directory that now contains it: flushing the file makes its contents durable, and the entry naming it lives in the directory. Three places still folded the four-valued read back into two. A metadata call that failed counted as a length mismatch and triggered a destructive replace; that pre-check is gone, since the read that follows answers the question properly. An unreadable indexed chunk returned success from a write, which a client reads as an acknowledgement and acts on by dropping its own copy; it returns an error now. And quarantine treated a failed re-read as an empty file and deleted the chunk, which could throw away a copy a concurrent repair had just published. A node that lost its handle to an environment still on disk now tries to reopen it every tick. Saying so once and waiting for a restart left an otherwise healthy node unable to serve part of what it holds, for a reason that is usually transient. A verification pass that failed no longer counts as work, so a node whose store cannot be read cannot hold the volume against every other node on the machine for good. The completion line now says the space is being returned, and a separate line says when it actually is. Deleting a large environment takes minutes, and an operator could not otherwise tell a slow deletion from a failed one. The background reaper retries with backoff rather than giving up on the first sharing violation. Tests: an unmarked retired directory is restored rather than deleted, and one beside a live environment is left alone. Verified by reverting to name-based deletion and watching both fail.
Three fixes from the seventh review round. It found no blockers. The four-valued read was still collapsed on the path that runs after a chunk is published. Only "wrong" was handled; "not there" and "could not read it" both fell through to success, and a caller that hears success acts on it. A client drops its own copy, replication marks the key held, and the copier takes it out of the legacy-only set. All four answers are handled now, and three of them are failures. Deleting a retired directory takes its mark away last. A recursive delete walks in whatever order the filesystem gives, so it could unlink the mark and then fail on the data file, which is exactly what a sharing violation produces. What was left was a genuinely retired, partly deleted directory carrying no evidence of it, and the next start would have read that as an intact environment and restored it. The reaper also keeps trying for about a day with capped backoff rather than giving up after five attempts and stranding the disk until the next restart. A directory under the live name that says it has been retired is never opened, even when it cannot be moved aside. It may be partly deleted, and opening it would put keys back into a commitment they have already left. The node serves from files alone, which is what the mark records as safe, and says so. Recovering a lost handle no longer scans the whole environment under the exclusive guard, and no longer does it every tick: the open happens outside the guard, which is then taken only to install the result, and a failure backs off. A node in that state also gives the volume lock back, since no amount of exclusive disk access will fix a store it cannot read. The migration tests wait longer for a phase change. The deadline is measured on the wall clock while the driver it waits on runs on the runtime, so on a saturated machine both stretch and a deadline sized for the work rather than for the contention turns a slow build into a failing test. Seen once here while a full lint and a review agent were running alongside it. Tests: a deletion that fails leaves the mark in place, and a marked directory under the live name is not served from. The second forces the rename to fail, because with it succeeding the test passed either way.
…eanup The eighth review round found that the manual deletion walker added last round follows a symlink at the top level. An operator who points the chunk environment at another volume leaves a link there; retirement renames the link, writes the retirement mark through it into the target, and the walker then deletes the target's contents, which are not this node's to delete. A linked environment is now copied out of but never retired, the operator is told to remove it by hand once the migration has settled, a link is never treated as retired whatever is written through it, and the walker refuses to descend one. Recovering a lost handle worked out which keys only the environment holds before taking the guard, then installed that answer after. Reading a large environment takes minutes, and a verifying read in that window can find a file rotted and throw it away; with no handle installed there was nothing to put the key back into, so it would have been missing from every gate and from verification, and retirement would have destroyed the intact copy. The environment is still read outside the guard, but the comparison against the file store happens under it, where nothing can move. A commitment that could not be recorded is no longer reported as progress. It was resetting the volume hold cap every tick, which let one node whose filesystem had gone read-only keep every other node on the machine from migrating for as long as it lasted. A chunk that cannot be read stops being advertised. The error alone was not enough: the index entry stayed, so the copier dropped the key from the legacy-only set on the strength of the name and replication answered "already held" and never repaired it. The file is left alone and a later successful read puts it back. Two removal paths that gave up for the rest of the process now keep trying: the driver retries a cleanup that could not finish, whatever phase it is in, and only exits when there is nothing left on disk. A deletion that removes the contents and the mark and then cannot remove the directory puts the mark back, since an unmarked directory that still exists is the one state the scheme says cannot happen. Tests: a linked environment blocks retirement, is never treated as retired, and deleting through it is refused with nothing touched behind it; a directory that outlives its own deletion still says what it is.
The ninth review round found that last round's fix for an unreadable chunk created a way to lose one. Dropping the index entry stopped the copier and replication treating the name as possession, but a key already copied is not in the legacy-only set either, so it ended up in neither view. Verification skipped it because the file store did not claim it, nothing else looks at anything but those two views, and retirement then deleted the environment holding its only copy. The index entry stays now. Removing one is the quarantine path's job, which removes the file with it after a read that succeeded and proved the bytes wrong, so the index and the disk stay in step. The real gap it exposed is closed at the same time: verification used to skip any key the file store did not have. That is correct for a key this node is giving up, which is in the legacy-only set and has gates of its own, and wrong for anything else. A key in neither view has been through nothing and is protected by nothing, so it now goes back into the legacy-only set where the gates can see it, and refuses the proof for that pass. The driver no longer exits while a directory is still being deleted. Both the finished-retirement path and the file-only phase returned immediately, so if the background deletion ran out of attempts nothing was left to try again until a restart. They keep the loop alive and it exits at the top, once nothing is pending. Only one reaper thread runs per directory, since asking for cleanup on every tick was starting a new one each time. A root directory that cannot be listed reads as "cannot tell" rather than "nothing there", which is what it was doing while deciding cleanup was complete. A linked environment gives the volume lock back rather than holding it for six hours waiting for a retirement that is never going to happen, and says so at warning level once an hour instead of at debug. Test: a key the environment holds that is in neither view refuses the proof and is put back where the gates can see it. Verified by removing the distinction and watching it fail.
Two tests provoke a deletion failure with directory permissions, which is not how the same thing happens on Windows, and their bodies were gated while the variables they set up were not. That left unused variables on Windows and broke the build there. Both are gated whole now, and both assert what they were only conditionally checking before: the deletion is required to fail and the mark is required to survive it, rather than the assertions being skipped if the setup did not produce the failure.
…s one Three failures, each real rather than a runner being slow. **A fixed delay decided whether the migration crash test tested anything.** On the runner the child had copied nothing in its 150 ms; on this machine it had copied some. It uses the same failpoint as the other children now: ten chunks copied, the eleventh interrupted, the rest untouched, the same every time. The progress handshake it used instead is gone with it. **btrfs does not report freed space immediately.** The test slept 200 ms and took one reading, which on ext4 was enough and on btrfs was not. It polls for the space to come back, with a deadline, and returns the last real reading so a genuine failure still fails on the number rather than on the wait. **Two things only CI's toolchain sees.** Clippy 1.98 rejects an unbounded range in a for loop where 1.95 did not, and making the file store public under the test feature put a doc link to a private item in front of rustdoc for the first time. Local checks now include `cargo doc --features test-utils`, which is what would have caught the second one here.
…old one btrfs charges very differently for four hundred small files than for one large one, so comparing what the disk still costs against what the environment used to occupy was a statement about filesystem overhead rather than about the migration. It compares against what the file store actually occupies now. The load-bearing assertion, that retiring hands back most of what the environment held, was passing on btrfs already. The numbers are printed as well as asserted: on this machine the environment held 14.1 MB, the file store holds 6.6 MB, and 14.1 MB came back.
…f Unix where they hold The four new harnesses were sequenced after the e2e testnet suite. That suite flakes on hosted runners for transport reasons unrelated to storage, and a failing step aborts the job, so on the last Windows run the harnesses did not execute on any platform at all. They are fast and deterministic, so they now run first and always report. The sweep test is Unix-only, because the leftover it sweeps only exists on Unix. Off Unix the store creates the chunk under its final name and flushes it, deliberately, since a rename there is not documented to reach the disk. There is no temporary file to find, and the equivalent hazard there is a real chunk name over short or wrong bytes, which the store's own tests cover on every platform. The scale harness runs on Linux only. It plants a hundred thousand files to measure what a restart costs, and that is a fleet question, where every node is Linux. Re-measuring it on the Windows runner would cost minutes of every run for an answer no node needs.
…y platform Off Unix the failpoint sat after the chunk was created under its final name and written, and before the flush. That is not the moment between the two halves of a dual write: the file is already there and already readable, so the crash test asserting that an interrupted chunk stays on the copier's list failed on Windows for a correct reason. It could not have proved anything about the missing flush either. Killing a process does not empty the page cache, so the bytes survive; only losing power loses them, which no test that kills a process can stage. Moved to before the file is created, which is the same point in the sequence as the Unix temporary-file-written-not-yet-renamed halt. Both scale ceilings were far looser than the measurements justify. The scan-time ceiling was a flat thirty seconds against a hundred milliseconds measured, so a ten-second stall passed; it is now fifty microseconds per key, which scales with a larger run. The bytes-read ceiling was a hundredth of the payload, which grows with chunk size and so permitted a 655-byte header read of every file in a test named for not reading contents; it is now a fixed 64 KiB against the 125 bytes measured, so any read that is per-chunk at all fails, and fails harder the larger the store. Also gate the failpoint out of shipped binaries. It is compiled only under test-utils, which is not a default feature and is not passed by the release workflow. CI now proves that instead of trusting it, by looking for the environment variable name in a default-feature build: the literal survives into the binary whenever the code that reads it is compiled, and it is present with the feature on.
The driver is documented as holding the volume from the first copy through retirement and not handing it back in between. Only the copying half had a test. Retirement is the heavier half: re-reading every chunk in the store to verify it, then deleting an environment. A driver that took the lock only for copying would run that pass while its neighbours on the same disk ran theirs, which is the pile-up the lock exists to prevent. The new test is shaped like the copying one, so the answer does not depend on catching a short window: an outsider takes the volume first, the node is put in the phase where retiring is the only work it has left, and it is watched for not doing it. Then the lock is released and it must retire, which is what keeps the test from passing against a node that never retires at all. Removing the lock from the retirement branch of the driver fails it. Copying and committing are done by hand rather than by waiting for the driver, because the driver reaches that phase by waiting out the shed hold, which is days. Both nodes are now checked for holding their own chunks and only their own. Counting just the one that went first would pass for a node that had picked up its neighbour's chunks as well. The bytes-read ceiling constant moves to the top of its file: clippy 1.98 rejects an item after a statement, and CI runs a newer clippy than this machine.
… if a harness runs nothing The new retirement test drives the store through a hook that exists only under test-utils, so the target now declares that feature and every job that runs it passes it. Without this the harness does not compile, which is how it failed on the loopback filesystem jobs. Each harness step now checks that it actually ran something. A test binary that reports no tests, or a target skipped because a feature was not passed, exits zero and reads as a pass, which is a harness quietly going dormant. Those steps run under bash explicitly, since the Windows runner would otherwise use PowerShell. The reclamation harness prints its measurements everywhere it runs, not only in the main test job. What that job is for is the number each filesystem gives back, and capturing the output meant ext4, XFS and btrfs each reported nothing but a pass.
…hat is left The validation section predated the four harnesses. It now says exactly what runs on every commit and what each mutation check confirmed, so the gates that remain are the ones a workstation genuinely cannot close. Two of them are narrowed rather than removed. The loopback filesystem jobs are not offered as closing the power-loss gate: killing a process keeps the kernel page cache, so removing every flush from the publish path would leave them green. What they do cover is the rest of what a filesystem decides. And the scale gate is now 1M and 10M keys rather than 100k, since 100k is answered in CI and printed, though where the curve stops being linear is still a question about a machine holding ten million files.
A crash during retirement is now staged, which is the most destructive moment in the migration: the environment renamed aside and marked, nothing yet deleted, and the process that wrote the mark killed there. The next start has to finish that deletion and never reopen the directory, because the node has already told the network it serves those chunks from the file store. Its recovery had unit tests that planted the mark by hand; what those cannot show is that the mark is really on disk at that moment. Refusing to believe the mark fails the new test. The startup scan's "names only, no stat per entry" claim had no protection. A flat ceiling cannot give it any: one stat per entry costs about three times a bare walk and stays well inside any ceiling loose enough not to flake on a shared runner, which is why the mutation passed. It is now measured against the machine instead of against a number. The same directory is walked twice in the same process, once reading names and once calling metadata on each entry, and the scan must land on the names-only side of the two. Runner speed cancels because it moves all three together. Adding the stat takes the scan from 88 ms to 296 ms against a 123 ms midpoint. The reclamation test read a signal that runner noise could swallow. Chunks are now 128 KiB rather than 16 KiB, which puts the environment at about 59 MB and the recovery threshold an order of magnitude clear of the drift, and the drift itself is measured and printed so a failure says whether the space did not come back or the machine was busy. Two tests promised more than they did. One that never started a migration driver now runs a real one for the whole of its wait, so that mapping lock contention to "available" no longer leaves it green. The other is renamed to what it checks, since it copies and never retires. The index memory test now says plainly what it can catch. Process-wide RSS and allocator reuse mean it finds an index costing several times what it should, not a small regression, and it should not be read as a byte-accurate account of one data structure.
…refusal The record said a node should refuse to delete the legacy environment on Windows until an operator explicitly overrode it after power-loss testing. That predates the two changes that removed the reason for it, and the code has shipped without a platform condition, so the record and the build disagreed. Publishing a chunk off Unix no longer renames at all: it creates the file under its final name and flushes it, which Microsoft documents as flushing the creation metadata with it. Retirement still renames the environment aside, and that rename is not durable there, but the mark now goes inside the directory rather than beside it. A power loss that reverts the rename brings the directory back under its live name still carrying its mark, and a marked directory under the live name is never opened or served from. A loss before the mark leaves it unmarked under either name, and an unmarked directory is always restored and reopened. All four states have tests. Replacing a policy with a mechanism is the better answer here: a switch nobody turns on is a migration that never finishes, and the fleet already deleted 2.29M chunks and got back nothing. The per-node override remains for an operator who wants retirement held off one machine, and forced power-loss testing is still an open gate on every platform. What that run is now checking is directory creation, which has no portable flush. Also corrects the claim that all four harnesses run on three platforms. The three that touch durability do; the fourth measures what one file per chunk costs at scale, which is a fleet question on a fleet that is Linux.
…nything The mark said yes or no and folded every other answer into no. Reading it can fail for reasons that are neither: a permission change, a descriptor limit, a filesystem that has gone away underneath the node. Folding those into "no mark" fails in the worst direction. A retired environment that reads as unmarked is put back under the live name and reopened, and its keys re-enter a commitment they have already left. It is now three states, and the two questions callers actually ask are asked separately. Deleting a directory requires a mark that was read; opening one requires a mark known to be absent. Neither treats "cannot tell" as a yes, and the six call sites each ask the one they mean. A path that is not there is still definitively unmarked, which is the ordinary case and asked on every tick. This is the same defect as the others this branch has been fixing, in a smaller place: a belief that fails open, acted on later as though it had been established.
…orrect stale docs The scan comparison took one sample of each of its three measurements, so a scheduling pause that landed on the scan and not on the two walks decided the result. Runner speed only cancels when it moves all three together, and a preemption does not. Three interleaved rounds and the median of each: headroom on this machine goes from about 25 ms to 52 ms, and the stat mutation still lands at 257 ms against a 123 ms midpoint. The index memory gate allowed 256 bytes a key against 52 measured, which is loose enough to let another 128 through unnoticed. Now 128. Five places said something the code does not do. The record described publishing as always a rename two lines above the table explaining that one platform does not rename at all, and put documented the same thing unconditionally. The Windows section said two uses of rename were gone when one of them is still there and is safe for a different reason. The retirement crash test claimed a killed process settles that the mark is on disk, when the page cache means it settles when the mark is written, not that it survives power loss. The shared-volume file carried a duplicated heading, and the failpoint check named a count that changes whenever a failpoint is added.
…t be classified An unreadable mark and an unopenable store are different problems with different answers, and one message for both sends an operator to the wrong place. A store this node cannot open needs a restart; a store nothing can classify usually needs a permission or a mount looked at, and the node will neither open nor remove it until that is fixed. The predicate itself now logs at debug rather than warn. It is asked on every tick, so a warn there would be a wall of the same line, and the retirement blocker is the message an operator is meant to read. Also converts the comparisons a mechanical edit left as `assert!(a == b)`, which the clippy CI runs rejects. This machine was two minor versions behind CI, which is how three of these reached it; it is now on the same toolchain and the whole lint, doc and test pass is clean there.
…l fell through Making the mark tri-state was only half of it. Two callers asked whether it permitted removal and let every other answer fall through to the opposite action, so "cannot tell" still reached the opening path. A start finding an environment it could not classify opened it, and the tombstone sweep renamed one back under the live name. A mark check that fails for a moment and succeeds the next is enough for that to resurrect a store that really had been retired. Both now match all three answers: read the mark and remove, know there is none and open, or do neither. Doing neither costs disk until somebody looks, which is the right price for not knowing. A node that cannot classify its environment is also work no amount of exclusive disk will finish. It now stands down from the shared volume instead of holding it to the six-hour cap while its neighbours wait, and it says so through the throttled operator warning rather than only at debug. The tests for this were staged by taking every permission off the directory, which staged too much: at mode 000 the operating system refuses the rename as well, so the tombstone test passed with its own protection removed. It also would have failed on any CI running as root, since root can read a mode-000 directory. The mark is now a symbolic link pointing at itself, so looking for it returns a loop while everything else about the directory keeps working, for every user. Disabling either branch turns both tests red.
… branches that check it The recovery asked whether the environment was there before asking what its mark said, and folded an undetermined answer into "nothing here". That skipped both of the branches added to stop exactly this: an environment whose presence could not be determined went straight to the opening path. The mark already tells the three apart. A path that is not there carries no mark and says so; a path that cannot be reached says it cannot be reached. So there is nothing for the extra question to add, and one less place for an answer to be lost on the way.
… whether or not there is a handle Two ways an environment nobody could classify could still be deleted. The mark is written with `create_new`, and a failure saying something is already at that name was taken as "the mark is there" and the environment deleted on the strength of it. What is at that name might be anything. Every other part of this file insists the name is not the evidence; this was the one place taking it. A mark already present is now accepted only when it reads back as one. And the classification was asked only when the node had lost its handle, so on the ordinary path it was never asked at all: a node holding its store open went through every gate, renamed the directory aside and deleted it, whatever the mark said or failed to say. It is now the first question the retirement blocker asks, before anything else and whether or not there is a handle. The start-up recovery also probed the mark twice. The answer can change between two probes, and a second answer of "cannot tell" after a first of "retired" dropped through to opening the very directory the first answer said not to open. One probe, matched exhaustively. An unanswerable "is the environment there" is no longer read as "it is not". The retirement blocker already read that failure as "there is one"; the classifier that decides whether the work needs a person read it the other way, so the node kept the shared volume for the six-hour cap and said nothing an operator would see. Two folds in the LMDB store, on the same theme. A delete whose growth could not be measured was charged nothing, so a copy-on-write delete could spend disk the budget never saw and the reserve stopped meaning anything; it is now charged the whole slack, which costs at worst one assisted delete. And sizing the map read every metadata failure as an empty database, which on a node with a large one produces a map too small to open it; only a missing file means empty now.
…was measuring nothing CI reported the index costing zero bytes per chunk, and the test passed. Resident memory is process-wide and the allocator hands back what earlier work freed, so opening a store in a process that has already opened and dropped one of the same size grows the resident set by nothing at all. Adding the three reopens the scan comparison needs is what tipped it over: from that point the test measured the allocator rather than the index. It now runs in a child that has done nothing else and so has no freed heap to reuse, and it refuses a reading of zero. A gate that cannot tell the difference between an index that costs nothing and a measurement that happened not to be taken is not a gate.
…never established Seven defects of one shape, found by an independent test pass over this branch. Each is a fact recorded before it was true, or a fact left unrecorded because recording it was attached to a caller that had gone away. A cancelled write could claim a chunk nobody had read. The index insert lived in the blocking half of a put, which deliberately outlives the future that starts it, while the check that decides whether the bytes under an existing name are the right bytes runs after the await. A caller that went away skipped that check and left the node claiming, advertising and committing to a name whose contents nothing had looked at. The sharpest case is a name the startup scan refuses on purpose, because what wears it is a fifo, a socket or a directory. The insert now happens only for a chunk this call published; a name that was already taken is admitted by the arm that reads it and proves it. The same cancellation left a chunk hidden. A successful write cleared the marks that suppress a key only after the await, so a cancelled caller left the key indexed and suppressed at once, and a chunk the node really held stayed unanswerable until some later read settled it. Cleared inside the work now. A migration marker that could not be used was never replaced. The shed hold counts from that marker, and the reason it is written at startup rather than at the first phase change is that a later write would restart the clock on every reboot. A marker that was present but unreadable defeated that: it was replaced in memory and left on disk, so every start read the same bad file and stamped a fresh clock, and a node restarting more often than the hold would never become eligible to shed. It is now written whenever the disk does not hold what the process is using. A marker from a newer build is moved aside rather than written over, and its schema is read from the raw JSON, because a marker from a build with a new phase or a changed field is exactly the one that will not parse and exactly the one worth keeping. Quarantining a chunk unlinked it without flushing the directory, so a power loss could bring back a file the node had proven wrong, with the mark that would hold it back living only in memory. Deleting and quarantining ran without the store-lock lease that every other blocking operation carries, so a cancelled caller dropping the last handle could unlink inside a directory another process had already been given. A publish that failed after the bytes reached the disk handed back the reservation for them, admitting the next write against space that was already spent. A publish now reports whether it left anything behind, rather than the caller inferring ownership from a name being occupied, which is the reasoning the rest of this file exists to reject. The journal of writes in flight was a set, so two writes for one key shared one entry and the first to return cleared it for both. A delete arriving in that window saw no announcement, skipped draining the environment, and let the surviving write land afterwards and undo the prune. Counted now, as the file store's equivalent already was. Also: writes made without a rollback copy are counted and reported rather than logged one line per chunk, so the fleet can answer how many nodes are really keeping one before the second release ships; and the volume lock takes a configured directory, because whether the nodes on a host can see each other's lock is a deployment fact no node can check for itself.
Both node generators set PrivateTmp=true, which gives every unit a tmpfs of its own. The per-volume migration lock defaults to the host's temporary directory, so under that setting each node creates the same filename in a different filesystem, every one of them takes it, logs that it did, and starts copying. That is the case the lock exists to prevent: twelve nodes copying a full store at once need twelve times the free space and all twelve stall. Each generator now creates /var/lib/ant/migration, owns it as the node user, and points every unit at it. It holds nothing but the lock, so the nodes can serialise their copies without being able to reach each other's data, which is what the per-node ReadWritePaths exists to prevent and what granting write access to the shared node directory would have undone. The node also logs the path it took the lock at, so whether the lock is doing anything can be answered from a log rather than inferred from a unit file.
…ase branch The duplicate-number check only looked at the files in the branch it was running on. A branch cut before another decision record merged does not contain that record, so the check sees one file per number and passes, and the duplicate comes into existence only when the two are merged together. Two branches in this repository have been green the whole time while claiming a number main already used. New records are now checked against the base branch as well. This cannot catch two open branches claiming the same free number, since nothing reserves numbers and merge order decides who gets one; that still needs a look at the open pull requests before picking.
The record said every gate is rechecked inside the destructive step itself. What is rechecked there is the proof's health generation, the answerability veto, the announced writes, and that every key held only by the old store is in the approved set. The network gates, rank and commitment delivery and possession, are rechecked immediately before that call and outside the guard. The window on those is the seconds it takes to take the guard rather than the hours a verification pass can run for, so the argument holds, but the two are not the same claim. It also said the per-volume lock is held until the space comes back. It is released when the old store is unlinked and its directory renamed aside; the deletion itself runs detached so the node can serve while it happens, and on a large store that takes minutes. The next node in the queue can begin copying while the previous one's directory is still on the disk. That is deliberate and worth stating rather than claiming a tighter guarantee than there is. Adds what the lock's location depends on, which is a deployment fact no node can check for itself, and how to set it.
…p chunk Suspended for two releases so the fleet could move off a chunk store that never returned disk. The penalty is the auditor's decision, so a node that has to give up chunks cannot stop its peers punishing it for that; the peers had to stop first, one release ahead, and the nodes moved in the next one. That is done, so the accusation means what it always meant and is enforced again. The switch and its environment override stay. Restoring the penalty is the moment most likely to need undoing in a hurry, and this is the cheapest way to do it. It suspends only the penalties a node hands out, so an emergency suspension has to reach the fleet rather than the node being penalised. A test now pins the value this release ships. The existing tests set the switch both ways on purpose and so never noticed which way it was compiled, which is how a suspension outlives the thing it was suspended for.
…t replaced it Two releases ago every chunk lived in an LMDB environment that never returned a freed page to the filesystem: the fleet deleted 2.29M chunks and recovered nothing. The release before this one copied every chunk into a file of its own and deleted that environment. This one removes the code that did it. What goes: the LMDB chunk store, the migration driver, and the facade that presented both stores as one while the copying was in flight. About 5,600 lines of bridge and driver, plus the harnesses that existed to prove the bridge worked. What is left is one store, one file per chunk, and it is called `ChunkStore` because that is what every caller already called it. `heed` stays in the dependency list: the paid-key list has its own LMDB environment, which this does not touch. A node that starts with an unretired environment still on disk refuses to start, and says which directory and what to do about it. Starting anyway was the tempting answer and it is wrong: those chunks are unreachable to this build, but the commitment this node published before the upgrade claimed them, and a commitment is good to its neighbours for two hours. The accusation the first release suspended was "you did not have a chunk you were supposed to hold"; the commitment-bound audit was never suspended in any release. So a node that starts half-migrated spends hours failing audits at full weight on the one lane that always counted, for keys it cannot read. Refusing everything would be wrong too. A migration that finished and then failed to delete the directory leaves one behind that is safe to ignore, and a node whose only fault is a failed `remove_dir_all` should not be held offline for it. So the question is not whether an environment is there but whether it was retired, and the evidence is the mark the retirement wrote inside it. Three states, not two: a mark that cannot be read is neither permission to start nor a reason to stay down forever, and it says which case it is. Tombstones are checked as well as the live name, because a crash between the rename and the mark leaves an intact environment wearing a retired-looking name. Nothing is deleted; this build has no migration code and no business deciding that a directory it cannot read is safe to remove. The deployment settings for the per-volume migration lock go with the migration. They configured an environment variable that no longer exists. The loopback filesystem CI job now runs the storage tests rather than the deleted harnesses, so ext4, XFS and btrfs keep covering what the store does on them: publish through a temporary and a rename, flush, delete, and rebuild an index from the names. BREAKING CHANGE: `LmdbStorage`, `LmdbStorageConfig`, `MigrationConfig`, `MigrationPhase` and `MigrationState` are removed from the public API. `storage.migration` and `storage.db_size_gb` are removed from the node configuration; the second capped a memory map that no longer exists. A config file written by the previous release still loads with both keys present, because nothing declares `deny_unknown_fields`, and there is now a test holding that true.
Two of these are the same failure as the one already caught here: a removal that took something load-bearing with it. The workflow lost two job headers. Removing the deleted harnesses' steps by matching on step boundaries also swallowed the `filesystems:` and `doc:` declarations, so their steps were absorbed into the per-OS test job. That job then referenced a matrix key it does not declare and ran `apt-get`, `mkfs` and `mount` on macOS and Windows, the filesystem coverage never ran as its own job, and documentation sat behind three failing jobs. Every matrix reference now belongs to the job that declares it. Shutdown stopped aborting the protocol routing task. That went with the migration ordering it was written next to. The routing loop has no cancellation branch of its own: it waits on `events.recv()` while holding an `Arc` on the P2P node that keeps the sender alive, so nothing left would ever wake it. It would sit there holding the chunk store and its single-process lock open after the node returned. A node with an unmigrated store could start by turning storage off. The refusal lived only in the store's constructor, and a node configured with `storage.enabled = false` never builds a store. Turning storage off is not consent to run beside chunks whose commitment is still live, so the question is now asked before anything is built. The test that covers it goes through `build()` both ways, because the failure worth catching is a route into the node walking past the check, which is exactly what happened. Finding the leftovers folded unreadable into absent. A `try_exists` that failed, a node root that could not be listed, and an unreadable entry inside it were all read as "nothing here", so a root that permits traversal but not listing would hide an unmigrated store and the node would start. That is the same fail-open the classifier itself is three-state to avoid, one step earlier in the same file. Each is now a refusal that says which question could not be answered. Also: the shipped production config still advertised the database cap and the whole migration section, describing a copier that no longer exists; the two tests that mutate the process-wide penalty switch now both serialise rather than one of them; `page_size` was left as a direct dependency for LMDB map alignment that no longer happens; a failpoint for an operation that no longer exists is gone; and the startup narration still told operators the penalty was suspended.
Making the leftover check refuse on an unreadable root made it refuse on a missing one too, because reading a directory that is not there fails like any other read. That is every node starting for the first time. The whole suite passed with it, because every caller in the tree happens to create the root before opening a store. Nothing depended on that being true, and nothing said so. A root that is not there holds nothing, which is an answer rather than a failure to get one. Every other read failure still refuses.
…ng nodes that are fine The facade held a per-key critical section across whole logical transitions, and deleting it took two requirements with it that had nothing to do with the migration. A delete no longer waited for a write already in flight for that key. A write's blocking half outlives the future that started it, so a cancelled put can still be queued when a delete arrives, and the write then lands afterwards and puts back a chunk the node had decided to prune. The key ends up in a store that no longer claims it. Both regression tests for this ordering were deleted with the facade even though the requirement was not. One is back, and it reproduces the resurrection when the wait is removed. The check that decides whether an offered copy is already held no longer excluded deletion. A prune could remove the file between that read and the answer, so the caller was told the chunk was already held while the good copy it was offering was discarded. Both now take the key's lock for the whole operation. Two ways this refused nodes that are fine. An empty leftover directory is what the previous release's cleanup leaves when it is interrupted between removing the mark and removing the directory: fully migrated, nothing in it, and that release recognised the state and tidied it up. Holding a node offline for a directory with nothing in it is an outage for bookkeeping. And a root that does not exist yet is every node starting for the first time, which the unreadable-root refusal had swept up with it. The startup check also ran after the transport was built, so a bind failure could mask it and a node that did see it had already been charged for a transport it was about to throw away. It runs as soon as the root is known. The test proves the ordering by staging a port an ordinary user cannot bind: moving the check back after the transport returns the bind error instead.
…t was not the bridge's The key's lane was taken by the delete and by the check that answers whether a chunk is already held, but not by the put. That is not enough. A put does a lot before it registers itself as in flight: it checks the address, reads to see whether the name is taken, and reserves capacity. A delete arriving in that window finds nothing registered, waits for nothing, and goes ahead, and the put then registers and publishes afterwards. The node keeps a chunk its pruner had already given up. The put now holds the lane for its whole transition. Repair is split into a public entry that takes the lane and a body for the three callers that already hold it, because the lane is not reentrant. Two harnesses were deleted as migration machinery and were not entirely that. A process killed mid-publish leaving no chunk the store cannot serve, and an interrupted write's leftovers being swept, are about the store's own publish path, which is now the only one there is. They are back as `tests/chunk_store_crash_safety.rs` and run in CI. A third property, that engine shutdown waits for a store write whose awaiter was dropped, went with a harness written against the old store; it needs a live P2P node to stage, so the record names it as missing rather than this change pretending otherwise. Three explanations claimed more than the code does. The non-atomic rewrite off Unix was justified by the old store still being there to repair from; the argument now is that every caller reaches it only after a read proved those bytes wrong, so a crash leaves wrong bytes where wrong bytes were. The already-held check linearises a question about the store, and does not follow its answer out to the wire, where a prune can still land before the peer hears it. Aborting the protocol task asked it to stop without establishing that it had; the handle is awaited now, which is what actually releases the store. The record said commitments stay answerable for two hours. The constant says three.
The put-against-delete test did not test what it said. It started a put and a delete together, accepted either ordering, and asserted only that the index and the disk agreed about the outcome. The bug it was written for satisfies that: a delete that finishes and a put that publishes afterwards leaves both agreeing that the chunk is there. Removing the lane it was supposed to be guarding left it green. Staging it properly needed a hook that did not exist. The existing gate parks a put inside its blocking closure, which is after the write has registered itself, so a delete blocks on waiting for that registration and the test cannot tell which mechanism stopped it. The new gate parks a put after it has taken the key's lane and before it registers, which is exactly the window the lane exists for. It is an async lock: a synchronous one there blocks the runtime the put is on, and the first attempt deadlocked rather than observing anything. The restored crash test could pass having checked nothing. The child discards its put results and the failpoint counts arrivals rather than successes, so every publish before the kill could have failed, leaving an empty store and a serviceability loop with nothing to iterate. It now requires the store to hold something first. `repair` claimed atomic replacement and an untouched old file on every error. Off Unix neither is true: there is no durable rename there, so it truncates and rewrites in place. What makes that acceptable is a precondition nothing enforces, so the contract now states it, and the note about which callers establish it counts them correctly. The rest is the rename's prose finishing: comments describing chunk operations as LMDB, and explanations of the health counter and the directory flush that were written for a verification pass that ran before a store that no longer exists.
`fsync_path` and `write_file_durably` existed for the retirement: marking a directory before deleting it, and writing that mark durably. Both lost their only callers when the migration went and neither is reachable now. Nothing local caught them. `clippy --all-features` cannot: with every feature on, the module is public, so a `pub fn` nobody calls is still reachable and not dead. The build CI runs is `--no-default-features`, where the module is crate-private and the same function is dead, and it runs with warnings denied. That combination is what turns this into an error, and it is now part of what gets run here before a push rather than after one.
…ts, and stop overclaiming Both put gates were compiled under `test-utils`, which the e2e suite enables, so every write in that suite went through an await production will not have. Nothing outside the unit tests uses either of them, so both are `cfg(test)` now and e2e exercises the same path a node does. The gate handshake was a hundred-millisecond sleep, which makes the staging a guess and the word "deterministic" in the comment above it untrue. It waits on a counter now. Four claims were larger than what is actually checked. The rollback note said nothing on disk is deleted or rewritten, which is only true of the old store: the node still writes, repairs and prunes its own chunks, and opening the store still creates the store's own files and sweeps orphaned temporaries. The record said a node with a retired leftover "says so" and an unclassifiable one "says which", when only the refusals' messages are asserted. The filesystem CI job said it watched space come back; it watches an unlink. And the config compatibility test read one table out of a file rather than loading a whole previous config through the loader a node uses, which is now what it does.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Linear issue
V2-1033
Risk tier
Compatibility
chunks.mdbcan no longer be read at all. A node that still has oneand never finished migrating refuses to start and names the directory. A node whose
migration finished starts normally, including one whose cleanup was interrupted and left
an empty or already-marked directory behind. Nothing is deleted, so rolling back to the
previous release leaves that node exactly as it was.
LmdbStorage,LmdbStorageConfig,MigrationConfig,MigrationPhaseand
MigrationStateare gone.ChunkStoreandChunkStoreConfigkeep their names andtheir methods; they now name the file store directly rather than a facade over two stores.
storage.migrationandstorage.db_size_gbare gone from the config file. An operator'sexisting config still loads with both keys present, because nothing declares
deny_unknown_fields, and there is a test holding that true.Semver impact
Test evidence
One e2e run in six reported 95/96 with one failure whose name was not captured; the re-run on
the same commit passed at 96. Five of six runs on this branch passed at 96. That is consistent
with the transport flake this suite has on hosted runners, but it is not proven, and it is
recorded here rather than rounded up.
What the tests prove rather than exercise. Every fix below was checked by reverting it and
watching the test go red:
parks the put after it takes the key's lane and before it registers itself, which is the
only window where the lane is the thing doing the work. Without that gate the delete blocks
on the write-drain instead and the test passes either way, which an earlier version of it
did.
a delete resurrects the chunk.
write's leftovers are swept. Real child process, real failpoint, and the test refuses to
pass over an empty store.
storage.enabledeither way, andrefuses before the transport is built. Moving that check back after
P2PNode::newreturns a bind error instead, which is the point.
Adversarial review. Four rounds of codex at xhigh over the complete diff, from 5 blockers
to none at every severity. Seven times a deletion took something load-bearing with it, and I
found two of those myself; the rest came from the dead-code lint and from review:
run_event_loop().awaitwent with a migration guard above it, so the node would have startedeverything and fallen through to shutdown without handling SIGTERM.
and holds an
Arcon the P2P node keeping alive the sender it waits on, so it would haveheld the store and its single-process lock open after the node returned.
it does not declare and ran
mkfson macOS and Windows.prune against the check that accepts an offered copy.
write_and_replace's non-atomicity was justified by a store that no longer exists.New dependency
none.
page_sizeis removed: it aligned an LMDB map that no longer exists.heedstays,for the paid-key list, which has its own environment and is untouched.
ADR
https://github.com/grumbach/ant-node/blob/storage/lmdb-removal-release-3/docs/adr/ADR-0015-remove-the-lmdb-chunk-store.md
Mitigation / rollback
The penalty is a switch:
ANT_SUSPEND_UNHELD_CHUNK_PENALTY=1suspends it again on a nodewithout a new build, which is why the machinery was kept rather than deleted with the rest.
It only suspends the penalties that node hands out, so an emergency suspension has to reach
the fleet.
The removal has no switch and backs out by rolling back the binary. This release does not
migrate, delete, rename or rewrite
chunks.mdb, does not rewrite the migration marker, anddoes not change the file-store layout, so a node that rolls back finds the same layout and
whatever the previous release left behind. It does of course still write, repair and prune
its own chunks while it runs, and opening the store still creates the store's own files and
sweeps orphaned temporaries, exactly as #216 does.
The two changes are separate commits so they can ship in separate releases if the train
prefers that;
90f9b54is the penalty and stands alone.