Port the LCFS consumption, ETB VAT, and ETB services/NHS layers as declarative source stages

[juaristi22]

Workstream E6 of the UK migration (#145, master epic #665). Closes #682. Stacked on #714 (uk-was-wealth-681); base flips to main when #714 merges.

What

Ports the incumbent LCFS consumption, ETB VAT, and ETB services/NHS layers as three declarative source stages inserted between regional_property_uprating and the SPI spine trio (the incumbent's execution order: consumption before SPI row-stacking, so channel clones inherit donor consumption):

• lcfs_consumption (household) — LCFS 2023-24 household+person donor (UKDS SN 9468, DOI 10.5255/UKDA-SN-9468-3, sha+size-pinned, caller-supplied local inputs), the incumbent's donor cleaning verbatim (lowercase headers, the three rename maps, person-income join, WEEKS_IN_YEAR = 365.25/7, weighta×1000, three-code bus fare, the lossy tenure/accommodation collapses, the four-case electricity/gas split with the 0.52 fallback), the training-side single-pass NEED income calibration, the WAS-bridged has_fuel_consumption flag (WAS round-8 donor reused from E5's pin; recipient side an identity-keyed conditional draw on num_vehicles > 0 × the cited NTS 0.90 ICE share), a seeded weighted 18-output fit_weighted_qrf_chain in the incumbent's conditioning order, donor-support clipping (raked energy columns exempt), the four-margin ×50 weighted NEED raking as the new generic iterative_proportional_fit operation (income → tenure → accommodation → region, zeros preserved, unmapped categories and Scotland/NI untouched per NEED's England & Wales scope), the domestic-energy fold recompute, and zero_when_false petrol/diesel zeroing for non-fuel households.
• etb_vat (household) — ETB 1977-24 donor (UKDS SN 8856, DOI 10.5255/UKDA-SN-8856-4), year == 2023 filter, the incumbent's target construction with VAT rates from a cited anchors resource (0.20 / 0.025 @2023, parameter_path-declared, engine-tree lockstep-tested) — the incumbent's silent except → (0.2, 0.03) fallback dies; missing rates fail loud. Weighted single-output QRF, donor-support clip.
• etb_services (household + person) — same ETB donor at year == max (which resolves to 2023 on this tab: the file labels FYE-2024 as year 2023, so the incumbent's apparent VAT/services/fare-index year mismatch is vacuous on this vintage), education counts derived from person current_education (the incumbent's construction), native-entity predictor aggregation, weighted 3-output chain, donor-support clip, rail_usage = rail_subsidy_spending / 1.110 (compute_ratio, fare index anchored with parameter_path), and the deterministic NHS age-gender person allocation from the committed 252-row public table with the signed 85+ fold-in / full-table £202bn denominator / half-open band fixes.

Supporting: three new generic operation kinds (iterative_proportional_fit backed by the country-neutral microcosm/build/raking.py, bridge_donor_column_via_qrf, allocate_per_capita_from_cell_table) plus a condition parameter on assign_binary_from_rate; five cited build/uk/ resources (NEED 2023 kWh margins + Ofgem Q2 2026 rates, LCFS/VAT/services policy anchors with parameter_path fields and chronicle_candidate flags, the NHS age-gender table); sha-bound SDC-rounded support bounds for all three donors (generated from the pinned licensed tabs, placeholder-refusing CI tests, licensed-only round-trip via POPULACE_UK_LCFS_HH_TAB / POPULACE_UK_LCFS_PERSON_TAB / POPULACE_UK_ETB_TAB); the uk_aggregate_admin gate (first UK binding of the shared gate; NEED/Ofgem energy anchors + the NHS budget as cited TargetSpecs, release-blocking, passed at build time); gate-battery digests re-cut; coverage-manifest family entries for the three stages; driver wiring with _input_artifact_pins; 22 FitWeightRecords across the four fits (all explicit); the --check e6 identity receipt; and the hygiene uv.lock regen (the microcosm-frame uk extra from #717 was never locked; CI is unaffected — it syncs without --locked).

UK-specificity note (required by #682)

Generic contributions: iterative_proportional_fit + raking.py, the bridge-QRF and cell-table allocation kinds, the condition parameter, the anchors-with-parameter_path lockstep pattern, the parameterized encode_qrf_predictor_pair. UK-specific: LCFS/ETB column semantics and cleaning, the NEED margin tables and enum maps, the five build/uk/ resources, the three uk_runtime transforms, and the gate declarations. The incumbent package name appears nowhere in the tree.

Signed differences

1. RNG/method class (standing, and decisive here): seeded (0) weighted RegimeGatedQRF chains replace microimpute QRF (implicit seed 42, unweighted, and a fresh default_rng(42) per output — every household draws the identical quantile index for all 18 outputs). For zero-inflated targets this is not cosmetic: the incumbent's comonotone median-biased draws collapse them (its dfe_education_spending lands at weighted mean ~£10 with nonzero share 0.0004 against an ETB donor whose annualized mean is ~£3,348 at share 0.2546; its petrol mean roughly doubles the donor's). Ours reproduces the donor distributions (education ~£3,210 at share 0.227; petrol ~£550 vs donor ~£631). Parity is weighted-distributional; the eFRS register restates the incumbent's collapse for these columns, so register deltas on them are method-class — flagged for reviewer adjudication rather than silently excluded.
2. Weighted fits (adjudicated 2026-08-19): LCFS weighta, ETB hhold_adj_weight, WAS R8xshhwgt as explicit QRF weights; the ETB selections gain the weight column the incumbent discards.
3. NHS defect fixes (adjudicated, defect class): the incumbent's 80+ roll-in is a chained-assignment no-op (85+ rows silently dropped, deflating the £202bn denominator) and its apply step overlaps band boundaries. Fixed: proper fold, full-table denominator, half-open bands. Measured effect: spending ×0.9102, visits ×1.0180.
4. VAT rates from a cited resource, no silent fallback (defect class): the incumbent's bare except masks the live 0.025 reduced-rate share with 0.03.
5. has_fuel draws identity-keyed, seed 0 (standing E4 class): the incumbent draws positional default_rng(42) on both the WAS donor and the FRS recipient.
6. Per-capita ETB round-trip eliminated (mechanical, algebraic identity, tested).
7. Port-as-is quirks documented, not diverged: WEEKS_IN_YEAR 365.25/7 (LCFS) vs 52 (ETB); the lossy tenure/accommodation collapses; CONVERTED_HOUSE/OTHER/UNKNOWN and Scotland/NI margin exemptions; the 0.52 mean_elec_share fallback; donor dropna row loss (logged); NEED targets at FY26/27 prices in a 2023 build; domestic = elec + gas holding only in the build year; recipient predictors at the frame period vs the incumbent's implicit default-period reads; state_pension dead rename dropped.
8. Exact pinned lowercase column names, hard failure on absence (standing).
9. Donor uprating deferred (deviation from the reviewed plan, deliberate): the LCFS donor year equals the 2023 build year, so the incumbent's CPI/litre-proxy uprating is the identity here. Rather than ship a declared-but-dead manifest operation, the machinery lands with the FRS 2024-25 refresh (Legacy 2023-vintage build of the microcosm UK spine (deferred) #687), where a donor/build year gap first exists. The DESNZ pump-price anchors stay committed as the cited litre-proxy denominators for that refresh.

Licensed acceptance — DONE (evidence: data/ukds/acceptance/e6/)

• Twin builds byte-identical (payload + nonzero shares); hash-chained Logbook rows; the full release-blocking terminal battery passed at build time, including the new uk_aggregate_admin NEED anchors.
• e6 identity receipt PASS: the domestic-energy fold, rail_usage ratio, petrol/diesel zeroing, and the NHS allocation recomputed under row permutation by entity id (grossing scale restored for the NHS recompute — the SPI support channel halves survey weights after E6 runs).
• NEED margin fit: energy weighted means match the incumbent head-to-head to −0.0%/−0.1%; categorical margins fit to ≤1.1% worst-case (tenure), 0.13% (accommodation), 0.0% (region). Income-band receipt residuals reflect post-SPI re-bucketing (the raking runs before the certified pair rewrites incomes — in both pipelines), with under_15k a small-denominator band.
• Incumbent head-to-head at matched composition (run_incumbent_consumption_e6.py at pin ebf733c, same FRS/WAS/LCFS/ETB tabs, LCFS uprating target pinned to 2023): weighted quantiles per output; the NHS level shift quantified above; the zero-inflation divergences per signed difference 1.
• The licensed funnel caught six defects PR CI cannot see (fixed in this PR, each with a regression test that executes the previously-untested path): the has-fuel bridge's LCFS→WAS renames + predict API, etb_vat person-entity aggregation, the education-count derivation, the NHS age-label parser, NHS weight threading, and the four-margin raking wiring (the manifest declared four margins; the transform ran one).
• Support bounds regenerate byte-identically from the pinned tabs; tail concentration recorded per output; artifact note: the staged spine carries by-design NaNs on FRS rows for channel-only SPI auxiliaries (E7's surface, rejected by policyengine-uk dataset validation — an E10/loader integration fact).

Reviewer (María) adjudication asks: (1) the donor-faithful vs incumbent-collapse divergence class on zero-inflated outputs (signed difference 1) — proposed disposition: signed method-class receipts, weighted-distributional comparison binds at E10; (2) the income-band re-bucketing residuals as a documented symmetric class.

Carried from the #717 review

Per the recorded disposition ("deferred to E6 to avoid a CI-only re-spin of an already-reviewed PR"): the SPI support transform now fails closed on a non-importance builder weight kind (regression-tested), last_result carries resume-semantics comments, and the stage-2 seed + 1 RNG-stream convention is named at both sites.

Out of scope

The incumbent's post-calibration rail/bus/fuel/bus-fare aggregate scalers run after weights exist and are adjudicated on #623 — DfT-total comparisons do not apply to E6's pre-scaling outputs. DfT anchors + calibration targets: E9 (#685) / WS-C, salvaging #153. mortgage_debt/consumer_debt: #685. FRS 2024-25 refresh incl. the donor uprating machinery: #687. Upstream defect reports (NHS roll-in, VAT fallback) to policyengine-uk-data follow the EDUCQUAL precedent after this lands.

🤖 Generated with Claude Code

[juaristi22]

Addressed the adversarial-review findings in commit fea19ef2, and rebased the PR onto current main after #714 merged.

Resolved:

• E6 VAT, services/NHS, LCFS, and NEED transforms now load their committed anchor resources at runtime; manifest values are checked against the resource-backed configuration.
• Anchor resources now include authoritative source URLs.
• IPF records mapped positive targets that cannot be reached from zero current means, with an explicit strict mode for fail-closed callers.
• ETB VAT donor cleaning now rejects zero denominators and non-finite derived rates.
• Updated the authoritative UK YAML stage projection, source schema, coverage manifest, and identity-receipt tool for the retargeted branch.

Validation:

• microcosm-build: 210 passed, 7 skipped.
• microcosm-data contract suite: 181 passed.
• Ruff and git diff --check: passed.

[vahid-ahmadi]

Reviewed with the usual method — the three transforms and the raking kernel against the incumbent's consumption.py/vat.py/services.py, the anchors and bounds resources, and both review rounds. The E6 test files pass for me (55 + the licensed skip) on the rebased head.

Verified faithful at the quirk level, which is where this layer's risk lives: the NTS 0.90 ICE share, the 0.52 electricity-share fallback, the weighta rename, and — the one I checked most carefully — the NEED raking. Your kernel's semantics match the incumbent's loop exactly (weighted cell means, current <= 0 cells skipped, ×50 iterations, income → tenure → accommodation → region order), and the additions are strictly evidence: the raking_zero_current_cells attr and the strict mode from the adversarial round record what the incumbent silently skips. The MarginSpec means-based design is honest about why it works in means, and microcosm/build/raking.py is genuinely country-neutral.

The VAT fallback death deserves its own sentence. The incumbent's _get_vat_parameters docstring promises the layered fallback exists "so callers never silently get wrong numbers" — while its terminal fallback is the silent wrong number at any vintage where 0.03 isn't the reduced-rate share (2023's is 0.025). Replacing that with a cited anchors resource, parameter_path-declared and lockstep-tested against the engine tree, so a missing rate fails loud, is the correct posture — and extending the E7 drift-assert pattern from manifest parameters to engine-parameter anchors makes it a house pattern now. The queued upstream report (with the NHS roll-in) following the EDUCQUAL precedent is the right disposition; #463's list grows to five.

The licensed funnel's sixth defect is worth a design note, not just a fix. "The manifest declared four margins; the transform ran one" is precisely the class the drift asserts were invented for — but it escaped them, because the asserts verify declaration equality, not execution. The regression tests now execute the previously-untested path, which covers this instance; the general lesson (each declared operation parameter needs either a drift assert or an executed-effect receipt, and the raking margins had neither until the licensed run) belongs wherever the E7 pattern is documented, so E8+ inherits it deliberately.

Also confirmed: the three #717 carryovers are all swept here as recorded (fail-closed weight kind with a regression test, the last_result resume comments, the seed + 1 convention named at both sites) — that closes my #717 review in full except the incapacity_benefit_reported promotion question, which I'll assume rides E10 with the rest of the composition story unless the licensed run said otherwise. The ETB year-quirk note ("year == max resolves to 2023 on this tab, so the incumbent's apparent mismatch is vacuous at this vintage") is the right kind of finding: documented as vacuous now with the reason, so the 2024-25 refresh (#723) knows to re-check it rather than inherit it.

María's two adjudication asks look correctly scoped to her, and both proposed dispositions read right to me — zero-inflation divergence as a signed method class with the weighted-distributional comparison binding at E10, and the income-band re-bucketing residuals as a documented symmetric class (both pipelines rake before the certified pair rewrites incomes, so the residual is definitional, not directional).

Nothing blocking from my side.