Scope Logbook chains by country and dataset line: archives, live store, and tooling

[juaristi22]

Part of workstream A of #665. The Logbook was one chain at the repo root and one global chain in the live store, so the UK migration's build attempts had nowhere to land that did not interleave them with the US pool lineage — and any UK row recorded against a stale global tail while the US ladder appends concurrently would be permanently orphaned (its row_digest bakes in the predecessor; no reconciliation can repair it). This PR completes the split on both sides at the ratified granularity: chain scope = <country>/<dataset-line>, grandfathered so nothing existing moves.

The scope model

A scope is the unit of lineage: one hash chain per scope, in the git archives and in the database. Scope is derived from pipeline (already inside the hashed payload), never stored — every existing row_digest stays valid, and a row cannot be moved between chains without breaking its own digest.

• The three legacy US pipelines (us-2024-release, us-pool-inc2, us-stacked-pool) are hardcoded to the grandfathered scope us. The live data requires this: the 28 archived rows span all three with chain links crossing them, so pipeline itself cannot key the chain. That mixed chain keeps extending under those names forever; nothing is re-scoped retroactively.
• Every other pipeline must declare <country>-<dataset-line>-… and derives scope <country>/<second-token>. A new pipeline name opens a new scope and demands genesis — fail-closed scope opening (a typo'd us-pool-inc3 gets a loud genesis demand, never a silent continuation of us).
• Scope tokens name the line's base data (review round ruling): frs is the FRS-derived survey-microdata line — persons, benunits, and households, with spine, staging, and imputation stages sharing one chain — and locals is the local-areas product line. Not entities (the interim households over-named one), not mechanisms (spine is the microcosm base-frame concept), not epics. Scopes are also where builds serialize: the predecessor is chosen at build time, so concurrent lines need separate chains.
• The vocabulary is closed-world (review round): logbook.scope_declared() carries exactly us and uk/frs — deliberately minimal: a scope is ratified when its line starts archiving. The uk-locals-* drivers keep the convention names and their rows spool locally regardless (recording never consults the list); uk/locals is a one-line ratification when that line first archives. An unratified scope is refused by name even at genesis, so a typo'd pipeline cannot mint a stray scope on an append-only store. Opening a scope = a reviewed three-place diff (migration + DECLARED_SCOPES CLI mirror + README).

Layout (committed archives)

logbook/us.jsonl                     # grandfathered mixed chain — can never be split
logbook/<country>/<dataset>.jsonl    # every new chain
logbook-spool/<country>/<dataset>/   # spool mirrors, same shape

logbook.jsonl → logbook/us.jsonl and its 28 spool rows moved as pure renames (0-line diffs; same sha 44ee49ea…, same tail 6eda41f8…). validate/render walk the directory chain by chain; export names exactly one chain and refuses a directory.

The database half (supabase/migrations/20260818000000_logbook_chain_scopes.sql)

logbook.chain_scope(pipeline) (IMMUTABLE — grandfather list first, then the two-token derivation), a CHECK that every pipeline declares a scope, builds_single_genesis → builds_single_genesis_per_scope, and enforce_build_chain() scoped throughout: per-scope advisory lock (hashtext(scope) — scopes append concurrently, appends within a scope serialize), per-scope count/tail/genesis, and an explicit chains-never-cross-scopes guard. builds_unique_predecessor deliberately stays global: digests are unique table-wide, so two rows claiming one predecessor is a fork wherever it happens — scoping it would weaken it.

Applied by the project owner to prod (same flow as the R2 setup); the writer key is unaffected.

Verified against a real Postgres (embedded server, the four existing migrations applied with pgcrypto's digest shimmed over built-in sha256, then this one, loaded with the actual 28 production rows):

• the 28 rows load under the old trigger; the migration applies to the populated table
• legacy US pipelines still extend the one us chain from its real tail
• a first uk-frs-* row with a US-tail predecessor is refused; genesis accepted
• cross-scope independence: uk/frs opens at genesis while us holds rows; unratified uk/locals and uk/firms are refused even at genesis
• a uk-frs-* row against a stale scope tail is refused, naming the scope
• us-pool-inc3 is refused outright (unratified scope) — and does not disturb us
• an unscopeable pipeline is refused by name
• every archived row_digest is unchanged after all of the above

That battery ships as tests/test_logbook_chain_scopes_pg.py — pytest.importorskip("pgserver")/("psycopg"), so CI and the default venv skip it (no dependency changes) while any reviewer can run it locally; the docstring carries the exact invocation and the pgcrypto-shim caveat.

UK pipelines named before any UK row exists

uk-national-staging → uk-frs-staging (scope uk/frs, joining the E2-named uk-frs-spine which was right all along); uk-rowwise-geography/uk-rowwise-candidate → uk-locals-rowwise/uk-locals-candidate (scope uk/locals, separate because rowwise candidates build concurrently with national runs). Only local spool receipts carry the old names and they never enter an archive or the DB — the one existing UK row is structurally un-insertable either way (its predecessor is the US tail), and stays a local receipt; the uk/frs chain opens at the campaign's next full-scale run, genesis in both the archive and the live store.

Remote export is scope-filtered

export --remote previously fetched every row and required one connected chain — correct on a single-scope table, broken the moment a second scope exists. Both export branches now derive the scope from the archive path (logbook/us.jsonl → us; logbook/<cc>/<type>.jsonl → <cc>/<type>; anything else — including an unratified scope — refused by name). Local exports get the same discipline (adversarial-review finding: the chain verifier authenticates payloads, never filenames, so a wrong spool would have chained validly into the wrong archive); the remote branch additionally filters server-side (pipeline=in.(…) for the legacy set, pipeline=like.<cc>-<type>-* for derived scopes), and re-verifies every fetched row's scope client-side before chain-ordering — the server filter is untrusted input to a chain verifier. Known edge, deliberately loud: a two-token pipeline with no third segment (e.g. exactly uk-households) derives a scope but escapes the like filter; the export then fails on the chain gap rather than producing a wrong chain.

Verification

• Postgres battery above (local harness; committed as the skipped-in-CI test).
• tools/logbook.py validate: 28 rows, tail 6eda41f8…, unchanged.
• New unit tests: migration text assertions (house pattern), the Python _chain_scope twin (documented mirror of the SQL), remote-filter URL construction for both scope shapes, wrong-scope row refusal, non-scope archive refusal.
• Full suite green, ruff clean.

Refs #628, #666, #665.

🤖 Generated with Claude Code

[vahid-ahmadi]

The scope model is right and the hard parts are done carefully. Verified: the Python _chain_scope and SQL logbook.chain_scope twins agree exactly (same grandfather list, same pattern, same split — I checked the regexes against each other, not just the tests); scope being derived from the hashed pipeline rather than stored is the property that keeps every archived row_digest valid and makes rows unmovable between chains; the grandfathered mixed us chain honestly declared unsplittable is better than a retroactive re-scope that would have had to rewrite history; keeping builds_unique_predecessor global is the correct call for exactly the reason the comment gives; and treating the server-side export filter as untrusted input to the chain verifier — re-checking every fetched row's scope client-side — is the right posture for a lineage tool. The archive move as pure renames (validate reports the same 28 rows and tail on this branch, which I ran), the loud two-token edge, and the committed-but-skipped Postgres battery with its invocation in the docstring are all good discipline. Tests pass for me and validate is clean.

One decision this PR makes silently and shouldn't: uk-frs-spine derives scope uk/frs in both twins, and the PR is silent about it.

The body renames every other UK pipeline into the ratified vocabulary (uk-households-staging, uk-locals-rowwise, uk-locals-candidate) — but tools/build_uk_frs_spine.py keeps _PIPELINE = "uk-frs-spine", and the E2–E4 acceptance runs have already landed spool rows under that name. At the next archived spine run, uk/frs opens as a de-facto scope. That's a problem under the README's own definition either way you read it:

• "A scope is a line of data work … households, locals, firms, and similar base-data families. Different base data need different scopes." The spine and the national staging are the same base data — the spine is the households line's early stage, and the staging build consumes its output. By the stated rule they share a line, which argues for uk-households-spine → uk/households: the spine→staging provenance then lives in one chain, and the serialization you get (spine and staging runs in one scope append one at a time) mirrors how they actually run.
• If instead the spine is deliberately its own line — defensible if spine artifacts are meant to version independently of staging runs — then frs is a source name, not a base-data family, and the decision belongs in the README next to households/locals/firms, not in whatever pipeline name E2 happened to pick before this vocabulary existed.

Either answer is fine; what shouldn't survive is the third state, where the scope opens implicitly at the next full-scale run and the vocabulary is ratified by accident.

Related hardening, worth considering here or as a follow-up: genesis currently accepts any well-formed name, so scope opening is fail-closed against continuation but open-world against vocabulary. The genesis-demand mechanism correctly stops a typo'd us-pool-inc3 from silently extending us — but run the typo'd pipeline without a predecessor and it happily opens scope us/pool at genesis; same for uk-huseholds-staging → uk/huseholds. A committed scope allowlist (a declared-scopes resource the CHECK and the CLI both consult, so opening a scope is a reviewed diff rather than a side effect of any matching name) would close that, and it's the same closed-world move this repo already makes for operation kinds and gate entries. The migration being append-only on prod makes this cheaper to add now than after a stray scope exists.

Neither point blocks the mechanism — but the uk-frs-spine question needs an answer before the next full-scale spine run mints the scope, so it's worth settling in this PR while the chain is still empty.

[juaristi22]

Both points taken, plus one finding from an adversarial pass run in parallel — all three landed in this review round as one commit.

The uk-frs-spine question — settled by ratifying uk/frs as the line's scope, which makes the E2 name the correct one. María's ruling: the dataset line is named by its base data, and this line is the FRS-derived survey microdata — persons, benunits, and households together, which is exactly why the interim households token was wrong (it over-named one entity), and why spine can't be a line (it's a microcosm mechanism — the base frame that receives imputations — not a product). So uk-frs-spine stands untouched, uk-households-staging re-renames to uk-frs-staging to join it, and scope uk/frs carries the spine→staging provenance in one chain. Your first reading, formalized: same base data, one line. The serialization between spine runs and staging runs is accepted and documented — both lanes are one operator today. The README now states the naming rule (base data, not epics, not mechanisms) with frs/locals as the ratified examples.

The allowlist — added, exactly as the closed-world move you named. logbook.scope_declared() in the same migration carries the ratified set — deliberately minimal at us and uk/frs (María's call: a scope is ratified when its line starts archiving, so uk/locals waits for the local-areas line's first archival; its drivers keep the convention names and their rows spool regardless); the CHECK becomes scope_declared(chain_scope(pipeline)) and the trigger refuses an unratified scope by name before genesis — so your uk-huseholds typo now dies loudly instead of minting a stray scope, and us-pool-inc3 is refused outright rather than opening us/pool as a side effect. Opening a scope is now a three-place reviewed diff (migration + the DECLARED_SCOPES mirror in tools/logbook.py + the README list). The Postgres battery re-ran against the updated migration with the new checks (unratified-scope geneses refused for both uk/locals and uk/firms; cross-scope independence proven by uk/frs opening while us holds rows): all pass, archived digests still byte-stable.

The adversarial pass found the mirror-image of your export observation: the local --source branch never derived a scope from the archive path, so a uk-locals-* spool exported into logbook/uk/frs.jsonl would have chained validly and mis-scoped lineage permanently — the chain verifier authenticates payloads, never filenames. Fixed by unifying _archive_scope across both branches: local candidates outside the archive's scope are refused by name before anything touches the archive, with a regression test mirroring the remote one.

🤖 Generated with Claude Code

[vahid-ahmadi]

Verified cc673cc in code: DECLARED_SCOPES = {"us", "uk/frs"} mirrored exactly by logbook.scope_declared(), the CHECK now composing scope_declared(chain_scope(pipeline)) so an unratified scope dies by name before genesis, the uk-frs-staging rename in the national driver, and the unified _archive_scope guarding the local --source branch. 238 tests pass for me on the branch and validate still reports the untouched 28-row us chain at the same tail.

The uk/frs ruling is better than the reading I offered, and worth restating because it's now load-bearing vocabulary: naming the line by its base data rather than by an entity (households over-named one of three) or a mechanism (spine is what receives imputations, not what the data is) gives the rule a principled test the next country can apply without adjudication — NZ's line names itself the same way. Ratify-on-first-archival for uk/locals is the right tempo for the allowlist, and making scope-opening a three-place reviewed diff is exactly the closed-world posture I was asking for.

The adversarial finding is the one that would have hurt: a mis-scoped local export chains validly — the verifier authenticates payloads, never filenames — so the lineage corruption would have been permanent and invisible to every downstream check. Catching the mirror-image of the remote fix before any UK chain exists, with a regression test on each branch, closes the class rather than the instance. Nothing further from me — ready from my side.