Create the SPI income channel on the microcosm spine with the FRS-only second stage

[juaristi22]

Closes #683 (WS-E child of #145 under #665).

What

Three new manifest stages inserted contiguously before the (byte-untouched) certified pair, re-homing the existing SPI restoration machinery onto the raw E2–E4 spine:

• frs_hmrc_spine_leaves — the 5 adjudicated retained HMRC leaves materialized from adult.tab/benefits.tab and aligned by raw identity (person_id = sernum×1000+person, a direct reindex — never positional), plus the incumbent's employer_pension_contributions = 3 × employee derive.
• spi_support_channel — stacks exactly 10,000 zero-weight SPI-synthetic households at the pre-clone grain (uniform seeded donor draw, the incumbent's class, deliberately not weight-proportional), gates the stratum count mid-stage (e7_spi_synthetic_preclone, max 10,000 — the pre-clone re-derivation of the June 100k declarations), then allocates the 0.5 prior-mass share per (region,) stratum with the post-SPI prior allocation: spread stratum mass over the sampled quota, not donor weights #710 semantics: flat within stratum, donor weights never propagate, exact-total conservation, weights advance to IMPORTANCE.
• hmrc_spi_income_spine — SPI stage-1 QRF (age/gender/region; FRS-channel gift_aid/charitable_investment_gifts zero-init), FRS-only stage-2 (29 outputs, trains on the base channel's raw dividends), then the base-channel dividend_income redraw from the stage-1 fit — deliberately ordered after stage-2 training — band materialization, the 208-fact classification with the 8 canonical fences byte-equal, and the effective-mass gate (1 ppm floor, strict raise at f100).

Four new generic operation kinds: stack_zero_weight_donors (the first row-appending manifest primitive), gate_zero_weight_strata, allocate_zero_weight_prior_mass, redraw_columns_from_fitted_qrf. The spine driver (tools/build_uk_frs_spine.py) runs all 14 stages, takes required --spi-tab/--hmrc-ods pinned inputs, and emits the .hmrc_replay.json sidecar.

Untouched, verified: the certified national-driver path, uk/gates.json, the data-shard contract.py (zero fingerprint churn), and the frozen hmrc_income_source_stages.json (sha c0341af7… re-verified). New spine stages declare only new columns as outputs; already-produced columns ride the rewrites key, so the 14-stage StagePlan keeps one canonical producer per column.

Stage-count reconciliation

The issue's "29 FRS-only variables" vs the incumbent HEAD's 31-name list: 29 declared stage-2 outputs + 2 reviewed-absent (incapacity_benefit_reported — present-but-all-default under the active nonzero-signal fence, promotion to a 30th output documented for the licensed run; maternity_allowance_reported — never produced by the incumbent's own FRS loader, which silently skips it) = 31. Fail-closed reviewed absences replace the incumbent's silent narrowing. Resolved by the licensed run: the nonzero-signal fence did not trip, so incapacity_benefit_reported stays reviewed-absent and there is no 29→30 promotion in this PR. Nothing rides to E10 — if a future vintage carries real incapacity signal (e.g. the E11 FRS 2024-25 refresh), the fence raises at build time and forces the reviewed promotion then.

Manifest-order coordination (with #714 / E5–E6)

E7's block sits immediately before the certified pair, which must stay at manifest positions [-2:] (frozen lockstep test). E5/E6 stages insert after frs_brma and before frs_hmrc_spine_leaves, so at E10 assembly the SPI donors inherit wealth/consumption columns (the incumbent's position-7 ordering). E7 edits neither uk/gates.json nor contract.py, so the only collision surface with #714 is source_stages.json + the roster lists (UK_SOURCE_STAGE_NAMES, _STAGE_NAMES, uk_stage_implementations): whoever lands second unions the rosters with E5's stages ahead of E7's block and re-runs the targeted manifest tests. No fingerprint recompute is needed on E7's account.

UK-specificity note

Generic (shared vocabulary/handlers): the four new op kinds and the outputs/rewrites producer-split pattern. UK-specific, with reasons: the three build/uk stage declarations and uk_runtime/spi_spine.py (SPI/FRS source semantics), the retained-leaves constituents and crosswalk doctrine, the HMRC ODS fact surface and its fences, and build_uk_spi_support_channel's channel labels/lineage columns.

Signed differences (vs the incumbent at ebf733c, and one internal)

1. RNG/method class (standing, U14): seeded microcosm QRF distributional draws replace microimpute's per-call-reseeded quantile draws; the uniform donor draw is preserved in class but not in realized donor set. Weighted-distributional parity only; twin-build determinism replaces row-level comparison.
2. Build-side prior-mass allocation (SPI prior allocation: spread stratum mass over the sampled quota, not donor weights #710 port): the incumbent ships zero weights and applies the 0.5 share at calibration; the spine allocates it per (region,) stratum at build with exact-total IMPORTANCE weights. Same semantics, moved seam, conservation receipted.
3. Charity-column policy: the spine zero-fills gift_aid/charitable_investment_gifts on the FRS channel (incumbent behavior, parity-bearing); the certified path's NaN-on-FRS record stays for that path.
4. Real-frame dividend re-imputation exists in spine mode only — the certified path inherits the incumbent's already-re-imputed dividends.
5. Fail-closed reviewed absences replace the incumbent's silent stage-2 skip (defect-class improvement).
6. Disability _reported intermediates are not stripped (first-class spine columns per the E2/E3 architecture; category/flag refresh preserved; the incumbent's export strip is deferred to E10's export-surface adjudication).
7. Spine replay report kind is uk_hmrc_spi_income_spine_208_fact_replay (not the certified uk_hmrc_income_208_fact_replay): a deliberate identity separation so no verifier can accept a spine replay where the certified replay is required (the Close the schema-3 terminal-report path for non-grandfathered exact-k UK releases #712 defense-in-depth lesson). Fact surface, fence set, and summary semantics are byte-compatible; only the kind string differs. Flagged here because the implementation run pinned it in tests without logging it as a deviation.
8. Port-as-is quirks documented, not diverged: the employer-pension ×3 estimate; the uniform (not weight-proportional) donor draw; donor age-jitter/region-map/bootstrap semantics inside the reused imputer; ID remap via id_multiplier (synthetic IDs carry no independent meaning).

Verification

• Per-deliverable targeted suites green during implementation (test-iterate protocol; one implementation deviation logged and resolved as an environment artifact — see below).
• Independent re-run with the full all-extras environment: entire repo suite green (the 7 test_us_multispine_pool_tool.py failures in the sandbox run were caused by a uk-only venv missing policyengine-us, not by this diff) and ruff check . clean.
• Invariants re-verified post-hoc: frozen manifest sha unchanged; certified pair at [-2:]; gates.json/contract.py untouched; git diff --check clean.

Licensed acceptance (pending — PR stays draft until it lands)

The e7 evidence run (twin channel-extended spine builds → data/ukds/acceptance/e7/) with eight instruments: twin-build payload identity, the ±0.02 nonzero-share screen vs the 145-column reference, weighted totals via measure_uk_weighted_integrity_baselines.py, the 208-fact replay sidecar with byte-equal fences, the effective-mass floor receipt, the strata-count receipt, the permutation-identity receipt (QRF columns covered by twin-build determinism, the hmrc_spi_income precedent), and the incapacity-fence adjudication record.

🤖 Generated with Claude Code

[juaristi22]

Licensed acceptance run (e7) — results

Twin channel-extended spine builds (a/b) at commit 4da1b09 on the pinned licensed inputs; evidence in data/ukds/acceptance/e7/ (local only). Both builds succeeded on the first attempt.

Determinism — compare_uk_h5_payload.py payload-identical across the twin builds; nonzero-shares files byte-identical; replay sidecars byte-identical. Permutation-identity receipt for the identity-stable layer (frs_hmrc_spine_leaves): stored columns match a by-identity recomputation from the raw tabs, stable under id permutation, and employer_pension_contributions = 3 × employee holds row-for-row. QRF draws and the stack's donor draw are seeded streams, not identity-keyed — attested by the twin-build identity (the hmrc_spi_income precedent).

Strata receipt — exactly 10,000 SPI-synthetic households at the pre-clone grain; SPI channel mass share 0.500000 of total; exact-total conservation delta 0.0 against the sidecar total; per-region quotas and allocated masses recorded in the receipt.

208-fact replay — report_kind: uk_hmrc_spi_income_spine_208_fact_replay; all 208 facts excluded_with_fence under full_frs_tei_band_unavailable; 8 canonical fences byte-equal; summary status reviewed_exclusions_only.

Effective-mass floor — gift_aid ≈ 1.4e-2 and charitable_investment_gifts ≈ 2.9e-4 of all-person effective mass, vs the 1e-6 floor (strict-raise posture; the mid-stage gate did not fire).

Incapacity fence — did not trip: the raw spine carries no nonzero incapacity signal, so incapacity_benefit_reported stays reviewed-absent (no 29→30 promotion).

Nonzero-share screen (±0.02 vs the committed 145-column reference) — 32 columns compared, max |Δ| 0.0436, 6 beyond the band, adjudication pending (María):

column	spine share	reference	Δ
dividend_income	0.0871	0.1307	−0.0436
employee_pension_contributions	0.2758	0.2351	+0.0407
tax_free_savings_income	0.1198	0.1577	−0.0379
savings_interest_income	0.3717	0.3436	+0.0281
gift_aid	0.0106	0.0375	−0.0269
pension_contributions_via_salary_sacrifice	(beyond band; detail in receipt)

All six are QRF-imputed surfaces on the SPI channel — the standing U14 class (seeded microcosm QRF distributional draws vs microimpute's per-call-reseeded beta-quantile draws, which realize different zero-incidence) at ~38% synthetic composition. Proposed disposition: receipt as U14-class signed differences on the unweighted-share instrument; the weighted-distributional comparison is the binding one at E10, per the #713 doctrine (composition-sensitive deltas are receipted, never tuned away). Reference-absent diagnostics (hmrc_spi_* leaves, other_investment_income, and the disability _reported intermediates the incumbent strips before save) are recorded as expected no-reference columns.

Weighted-totals diagnostics — 62 overlap columns vs the licensed register, median |relative Δ| 0.235, receipted with the weight-regime caveat (register minted on the calibrated certified surface; the spine carries uncalibrated design weights + the 0.5-share SPI importance mass — the E4 adjudication precedent). Values stay local per the #609 EUL posture.

PR stays draft until the share-screen adjudication and green CI.

🤖 Generated with Claude Code

[juaristi22]

Adversarial-review disposition (Codex, branch diff vs main — verdict: needs-attention, 2 high / 2 medium)

All four findings adjudicated; three fixed in d53af949, one deferred with an issue. The licensed twin builds and all eight acceptance instruments were re-run at the fix commit: H5 payloads are identical to the first run's (the fixes touch sidecar bindings and guards, never the data path), twin payload/byte identity holds again, and every instrument result is unchanged.

1. [high] SPI/ODS source pins absent from the build receipt — FIXED. _input_artifact_pins now harvests non-table pinned artifacts by role; the SPI donor and HMRC ODS shas land in the build sidecar (input_artifact_pins) and in the Logbook input-pins digest, so two runs with different high-impact inputs can no longer share build-side provenance. Verified on the licensed rebuild: both roles pinned, digest moved.
2. [high] Stale replay sidecar could survive beside a fresh H5 — FIXED. The driver now clears every output before running, and the build sidecar — written last, so its presence marks a complete bundle — binds the replay sidecar's sha256 and report_kind (hmrc_replay block). A regression test pre-writes stale sidecars and asserts both the refresh and the byte-exact binding.
3. [medium] Rewrites invisible to the canonical-producer graph — DEFERRED to Carry manifest rewrites into the shared Stage/StageRecord producer graph #721. A shared-layer change (Stage/StageRecord/plan assembly) that predates E7 — E3's frs_education_grant_split rewrite has the same property, and the driver's hardcoded FRS_EDUCATION_GRANT_REWRITES carve-out is prior evidence of the gap. E7's rewrites are declaration-reviewed and lockstep-tested against runtime written-column constants; the structural fix belongs upstream of any one country stage.
4. [medium] Drift guards partial — FIXED. _support_stage_parameters / _assert_income_stage_parameters now assert every load-bearing manifest parameter against reviewed constants: stage-1/stage-2 predictors and full output lists, the FRS-channel initialization map, base-redraw columns, reviewed-absent sets, the pre-clone gate declaration (name/selector/maximum), allocation weight_kind_out/conservation, support flag/channels, and the effective-mass columns/floor/fail-flag/channel. Six new regression tests mutate the committed manifest one parameter at a time and assert the fail-closed refusal.

Full microcosm-build suite and ruff green at d53af949; the licensed evidence receipt records the rebuild (rebuilt_after_adversarial_review, base 4da1b099).

🤖 Generated with Claude Code

[vahid-ahmadi]

Early read while this is still in draft — the licensed acceptance and adversarial disposition looked complete enough that waiting for the un-draft would just delay the same review. I read spi_spine.py in full, the manifest declarations, and the driver wiring; 138 tests pass for me with the uk extra.

The design center of this PR — the drift asserts pinning every load-bearing manifest parameter to a reviewed code constant (_support_stage_parameters, _assert_income_stage_parameters) — is the right response to the adversarial finding and worth adopting as the house pattern for every stage whose manifest carries semantics: a manifest-only edit to a seed, share, stratum set, or output list now fails closed instead of silently steering the build. Also right: the outputs/rewrites producer split keeping one canonical producer per column across all 14 stages (with the producer-graph gap honestly deferred as #721 rather than half-fixed here); the leaves stage's alignment asymmetry (unknown raw identities fail loud, spine-only persons fill 0.0); the stage-count reconciliation replacing the incumbent's silent narrowing with fail-closed reviewed absences — maternity_allowance_reported never actually produced by the incumbent's own loader is exactly the kind of fact that would otherwise get re-discovered at E10; and the identity/conservation evidence riding checkpoint metadata so resume keeps the receipts.

Three small things, none blocking:

1. UKSPISupportChannelStageTransform.__call__ silently defaults the weight kind: weight_kind=result.household_weight_kind or WeightKind.IMPORTANCE. The manifest assert already requires weight_kind_out: importance, so if the support builder ever returns None here that's a builder bug being papered over, not a legitimate default — an explicit raise would match the fail-closed posture of everything around it.
2. The mutable last_result on frozen transforms (object.__setattr__) makes checkpoint_metadata() correct only for the run-then-harvest sequence the driver performs today. If a future resume path ever reconstructs transforms without executing them, the RuntimeError will fire at harvest time — which is the right failure, but a one-line comment on last_result saying "populated only by a live run; resume must re-run or skip harvesting" would keep the invariant from being discovered by stack trace.
3. The stage-2 seed + 1 convention is asserted but not explained at the assert site — the reader finds "drifted from the reviewed value" without learning that stage-2 deliberately derives a distinct stream from the same reviewed seed. The assert message or a comment naming the convention would make the manifest self-explaining.

One question for the un-draft, not a finding: the incapacity_benefit_reported promotion (present-but-all-default under the nonzero-signal fence, documented for the licensed run) — the acceptance comment says the twin builds succeeded, so did the licensed run settle whether it promotes to a 30th stage-2 output, or does that decision ride to E10? The PR body and the acceptance comment each carry half of that story; whichever way it resolved, one sentence tying them together in the body would close it.

Process note: plan → conformance review → licensed twin acceptance → adversarial pass with re-run instruments, all before leaving draft, is the most complete pre-review gauntlet any PR in this migration has run. From my side there's nothing structural queued for the un-draft — with the three nits and the one question addressed, this is ready when you are.