Skip to content

feat(cli): bdrive verify proves this folder matches the hub (BEA-161) - #192

Open
ssowonny wants to merge 1 commit into
mainfrom
bea-161-ph-idea-prove-this-folder-matches-the-hub
Open

feat(cli): bdrive verify proves this folder matches the hub (BEA-161)#192
ssowonny wants to merge 1 commit into
mainfrom
bea-161-ph-idea-prove-this-folder-matches-the-hub

Conversation

@ssowonny

Copy link
Copy Markdown
Contributor

TL;DR

  • You could never check the product's central claim. bdrive verify hashes every synced file and tells you what drifted, what never pushed, and what's missing.
  • It catches the case bdrive status structurally cannot: bytes changed, size and mtime put back.
  • --remote asks the hub whether it still holds the content — one check per blob, both blobs/<sha> and manifests/<sha>, or every file over 4 MiB reads as missing.
  • Pure read: no lock, no ops, no journal writes, no repair. Exit 0 clean / 1 otherwise, so it works as a pre-flight check.
  • Known gap: it replays this device's local journals, so it proves "matches what I last pulled", not "matches the hub right now". It prints that caveat itself.

The hole this fills

Three read-only inspectors already existed. The fourth — the one that answers the question people actually ask — was missing:

compares hashes content? asks the hub?
Drift()internal/syncer/drift.go folder → state cache no (size+mtime) no
Explain()internal/syncer/explain.go what would sync next no no
bdrive status counts pending / local no never
Verify() — new folder → journal state → hub yes with --remote

What it looks like

Clean:

  project:  verify-e2e (m-a5867365)
  checked:  2 files, 513 B hashed in 1ms
  OK - the folder matches the journal
  (this compares against what this device last pulled — run `bdrive sync` first for the hub's latest)

A file edited behind the daemon's back (exit status 1):

  project:  verify-e2e (m-a5867365)
  checked:  2 files, 531 B hashed in 1ms

  drifted (1) - on disk, but not the content the journal records
    index.md

  1 problem. Run `bdrive sync` to reconcile, or `bdrive log <path>` for history.

Hub gone, --remote. The remote leg degrades, the local verdict still decides:

  project:  verify-e2e (m-a5867365)
  checked:  2 files, 513 B hashed in 1ms
  warning:  hub check incomplete, local result stands (Get "http://127.0.0.1:53151/api/p/…/store/exists?key=blobs%2Fad24…": dial tcp: connect: connection refused)
  OK - the folder matches the journal

The two things that would have made it silently wrong

--remote must probe both key shapes. Files over chunkThreshold (4 MiB) are pushed as content-defined chunks plus a manifest keyed by the file's own sha256 — manifests/<sha>, never blobs/<sha>. A check asking only blobs/ calls every large file missing from the hub. And the size can only order the probe, never filter it, because a large file legitimately lives under blobs/ three ways: the browser upload path always writes blobs/<sha> at any size, pushChunked falls back to a whole-blob Put when the manifest key is refused, and anything pushed before delta sync existed is a whole blob regardless. So: probe both, ordered by size, short-circuit on the first hit. TestVerifyRemoteChunkedFile asserts the big file is reachable only under manifests/, so it can't pass trivially.

Checking the manifest alone is enough for a chunked file — "a manifest exists ⟹ its chunks exist" is enforced hub-side at ingest, not an honest-client convention — so there's no chunk-by-chunk walk.

missing-locally applies the local filter. .bdriveignore rules are symmetric in scan and materialize, so a path the local filter excludes is legitimately absent from disk. Without filter.Skip(rel) || neverSync(rel) — the same guard materialize uses — every project narrowed by bdrive scope --only would scream its entire out-of-scope set as missing. TestVerifyIgnoredPathNotMissing carries a negative control: remove the rule and the very same folder does report it.

Deviations from the reviewed plan

None of substance. The plan's optional HasBlob hint was taken (it costs one field and one line): when missing-locally is really just "not downloaded yet", the output says so and points at bdrive sync instead of reading as damage.

Shape

internal/syncer/verify.go holds the comparison, next to Drift and Explain; cmd/bdrive/verify.go is the thin cobra shell. That split is not cosmetic — neverSync, chunkThreshold and loadFilter are unexported in syncer, and the multi-device test the spec requires cannot drive a func main package.

The pure-read contract is copied from drift.go verbatim and holds: no Session, no store.Lock(), no SaveSync/SaveCache, no AppendOps, no materialize, and no network at all without --remote. The shell uses config.LoadProject rather than mustProject and remote.Open rather than openSession, because both of those reach ResolveMount, which enrolls the device — a read must not do that. store.Open is stat-guarded so a read never creates a volume for a project that has never synced.

Exit status uses the same mechanism bdrive grep uses for its no-match exit (errVerifyProblems + SilenceErrors/SilenceUsage), so a findings exit is status 1 with the findings and no cobra usage block. Every path printed from missing-locally and missing-on-hub comes out of a peer's journal, so it goes through safeField like every other journal string we print.

Architecture changes

architecture/cli-sync.md: two new classes in internal/syncerVerify and VerifyReport — joining Drift/Explain/SyncedFiles as read-only inspectors, with new edges to SyncedFiles (the walk), Filter (its own fresh instance, for missing-locally), Store (AllOps/DeviceOps/LoadSync/HasBlob) and Backend (Exists per blob, --remote only). verify joins the Commands roster. Nothing was removed.

✅ added · ❌ removed (strikethrough) · unmarked = unchanged

flowchart TB
    Verify["<div style='text-align:left'><b>Verify</b><br/>+Verify(ctx, folder, include, st, device, be) VerifyReport<br/>-existsEither(ctx, be, blob, size) bool</div>"]
    VerifyReport["<div style='text-align:left'><b>VerifyReport</b><br/>+Files int / +Bytes int64 / +Elapsed Duration<br/>+Drifted / NeverPushed / MissingLocally paths<br/>+NotYetScanned / MissingOnHub paths<br/>+NotFetched int / +RemoteErr error<br/>+Problems() int</div>"]
    Drift["<div style='text-align:left'><b>Drift</b><br/>+Drift(folder, include, accepted, cache) added, modified, removed</div>"]
    Explain["<div style='text-align:left'><b>Explain</b><br/>+Explain(folder, include, accepted) two lists<br/>+NotSyncedFiles(entries) int</div>"]
    SyncedFiles["<div style='text-align:left'><b>SyncedFiles</b><br/>+SyncedFiles(folder, include, accepted) paths</div>"]
    Filter["Filter"]
    Store["<div style='text-align:left'><b>Store</b><br/>+AppendOps / DeviceOps / AllOps<br/>+LoadSync / SaveSync<br/>+PutBlob / OpenBlob / HasBlob</div>"]
    Backend["Backend"]
    Commands["<div style='text-align:left'><b>Commands</b><br/>init login logout<br/>sync stop scope grep stale <span style='background:#22c55e55;padding:0 4px;border-radius:3px'>✅ verify</span> forget status log<br/>restore url share export import<br/>web daemon hooks read-log<br/>resume autostart</div>"]
    Note["verify.go — the third read-only sibling,<br/>and the only one that HASHES.<br/>Same pure-read contract as Drift/Explain:<br/>no Session, no flock, no ops, no journal write,<br/>no network unless a Backend is passed.<br/>--remote probes BOTH blobs/&lt;sha&gt; and manifests/&lt;sha&gt;;<br/>size only orders the probe.<br/>RemoteErr stops the remote leg, never the local verdict."]

    Verify -- "<span style='background:#22c55e55;padding:0 5px;border-radius:3px'>✅ hashes what syncs</span>" --> SyncedFiles
    Verify -- "<span style='background:#22c55e55;padding:0 5px;border-radius:3px'>✅ own fresh instance, for missing-locally</span>" --> Filter
    Verify -- "<span style='background:#22c55e55;padding:0 5px;border-radius:3px'>✅ AllOps / DeviceOps / LoadSync / HasBlob</span>" --> Store
    Verify -- "<span style='background:#22c55e55;padding:0 5px;border-radius:3px'>✅ Exists per blob, --remote only</span>" --> Backend
    Verify -. "<span style='background:#22c55e55;padding:0 5px;border-radius:3px'>✅ findings and exit status</span>" .-> VerifyReport
    Commands -- "<span style='background:#22c55e55;padding:0 5px;border-radius:3px'>✅ bdrive verify</span>" --> Verify
    Drift --> Filter
    Explain --> Filter
    SyncedFiles --> Filter
    Verify -.- Note

    classDef added fill:#22c55e22,stroke:#22c55e,stroke-width:2px
    classDef removed fill:#ef444422,stroke:#ef4444,stroke-width:2px,stroke-dasharray:4 3
    classDef noteBox fill:#88888822,stroke:#888888,stroke-dasharray:2 2
    class Verify added
    class VerifyReport added
    class Note noteBox
    linkStyle 0 stroke:#22c55e,stroke-width:2px
    linkStyle 1 stroke:#22c55e,stroke-width:2px
    linkStyle 2 stroke:#22c55e,stroke-width:2px
    linkStyle 3 stroke:#22c55e,stroke-width:2px
    linkStyle 4 stroke:#22c55e,stroke-width:2px
    linkStyle 5 stroke:#22c55e,stroke-width:2px
Loading

Acceptance

Check Result
go build ./... pass
go vet ./... pass
go test ./... pass, every package
internal/syncer/verify_test.go 8 multi-device tests, pass
internal/webapp CLI e2e TestCLIVerifyE2E, pass
npm run e2e not run — no frontend change in this branch

The multi-device tests drive the real converge/diverge paths through newDevice/sharedRemote/cycle: clean after converge; a restored-mtime edit reported as drifted while Drift() on the same folder reports zero; a deleted file as missing-locally; an ignored path not reported; an offline cycle's ops as never-pushed; a fresh file as not-yet-scanned; a >4 MiB file present on the hub under manifests/; a blob removed from the remote as missing-on-hub.

Docs

README.md command table and web/docs/src/content/docs/reference/cli.md (table row plus a ### bdrive verify section covering the five states, the both-key-shapes rule and the staleness caveat). No INSTALL_FOR_AGENTS.md change — this touches neither init, login, nor hooks.

Deliberately not here

No hub UI (no device panel, no sync-state page), no --pull that refreshes journals before comparing, no JSON output, no repair, no --fast, no chunk-level walk. The CLI is the whole win in v1.

The open one: --remote proves the hub still holds everything this device knows about — it cannot prove the hub holds nothing extra. Whether that second half is worth a --pull flag depends on whether anyone actually hits it.

Closes BEA-161.

Build session

cd $(git worktree list | grep bea-161-ph-idea-prove-this-folder-matches-the-hub | awk '{print $1}') && claude --resume adf563f0-e627-4127-8d91-cd0495d61c57

(only works on this machine)

"Your folder is the same everywhere" was a belief with no receipt. `bdrive
status` counts pending ops and unscanned changes but never reads a byte of
content, so a file whose bytes changed while its size and mtime stayed put was
invisible to every check we shipped.

`bdrive verify` hashes every synced file and compares it against
journal.Replay(AllOps), reporting drifted / never-pushed / missing-locally /
not-yet-scanned, and with --remote also missing-on-hub. Exit 0 when every
category is empty, 1 otherwise, so it composes as a pre-flight check.

The logic lives in internal/syncer/verify.go next to Drift and Explain — the
two read-only siblings it completes — because neverSync, chunkThreshold and
loadFilter are unexported there, and because a multi-device test cannot drive
a func main package. cmd/bdrive/verify.go is the thin cobra shell.

Two things that would have made it silently wrong:

- --remote must probe BOTH blobs/<sha> and manifests/<sha>. Files over 4 MiB
  are pushed as chunks plus a manifest keyed by the file's own sha, so a check
  asking only blobs/ would call every large file missing from the hub. Size
  only orders the probe — it can never be a filter, because browser uploads
  always write blobs/<sha> at any size, pushChunked falls back to a whole blob
  when the manifest key is refused, and pre-delta-sync history is whole blobs
  regardless.
- missing-locally applies filter.Skip + neverSync, the same guard materialize
  uses. The rules are symmetric in scan and materialize, so a path the local
  filter excludes is legitimately absent — without this, every project narrowed
  by `bdrive scope --only` would report its whole out-of-scope set as missing.

Pure read throughout: no Session, no volume flock, no ops, no journal writes,
no materialize, and no network without --remote. LoadProject rather than
ResolveMount and remote.Open rather than openSession, so a read never enrolls
the device. An unreachable hub degrades to a printed warning and the local
verdict still decides.

The command says its own caveat out loud: the journals it replays are this
device's local copies, so it proves "this folder matches what I last pulled".

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant