Port the WAS wealth imputation and regional property uprating as declarative source stages

[juaristi22]

Workstream E5 of the UK migration (#145, master epic #665). Closes #681. Closes #180. Originally stacked on #709 (E4); rebased onto main after #709 and #706 merged, now based on main. Supersedes stranded draft #181 (its WAS column mappings and citations are salvaged here; its pre-Frame code is not).

What

Ports the incumbent WAS round-8 wealth layer as two declarative source stages appended after frs_brma:

• was_wealth (household) — WAS round-8 household-tab donor (sha+size-pinned, UKDS SN 7215, DOI 10.5255/UKDA-SN-7215-20, caller-supplied local input), strict lower-case exact column matching (no fuzzy r↔w fallback), the incumbent's 11 imputed outputs plus the UK: surface cash ISA and stocks & shares ISA holdings in the wealth imputation #180 ISA split (cash_isa ← DVCISAVR8_aggr, stocks_and_shares_isa ← DVIISAVR8_aggr), a seeded weighted RegimeGatedQRF checkpointed chain mirroring the incumbent's sequential conditioning (output i conditions on outputs 1..i−1; corporate_wealth drawn as corporate_wealth_excl_isa + stocks_and_shares_isa and folded before later outputs condition on it; cash_isa appended last so the 11-chain conditioning is untouched), donor-support clipping, integer num_vehicles, recipient-side NORTHERN_IRELAND→WALES region remap (prediction only), and the id-keyed allocate_within_group_waterfall for student_loan_balance → person.
• regional_property_uprating (household, deterministic, rewrites only) — per-region factor avg_house_price / unweighted_mean(main_residence_value | owner) applied to main_residence_value and property_wealth from the committed regional_land_values.json resource (MHCLG dwellings + ONS UK HPI Dec 2025; regenerator tools/build_uk_regional_land_values.py from the public CSV fixture). NI has no reference row and is never scaled (port-as-is).

Supporting: three new generic operation kinds (fit_weighted_qrf_chain, allocate_within_group_waterfall, uprate_to_regional_reference) plus first semantics for the reserved fold_into; the uk_support gate (release-blocking) against a committed disclosure-safe bounds resource; uk_export_surface allowances for the two ISA columns (no policyengine-uk model variables yet — policyengine-uk#1791 tracks); gate-battery fingerprints re-cut; coverage-manifest families for both stages (the 11 wealth columns were already required); default-off metadata-keyed QRF model cache in microcosm-fit; acceptance instruments (e5_identity_receipt, household-grain weighted-integrity extension, support-bounds builder).

UK-specificity note (required by #681)

Generic contributions: the three operation kinds, fold_into semantics, the QRF chain usage pattern, the model cache, and the support gate binding pattern live in shared modules/vocabularies. UK-specific: WAS column semantics and cleaning, the two build/uk/ resources, the two uk_runtime transforms, and the gate declarations. The incumbent package name appears nowhere in the tree.

Signed differences

1. RNG/method class (standing): seeded (0) weighted (R8xshhwgt) RegimeGatedQRF chain replaces the incumbent's microimpute QRF (implicit seed 42, unweighted fit, per-output re-seeded RNG giving perfectly correlated quantile ranks across the 11 outputs). Parity is weighted-distributional only. The weighted fit is adjudicated in WAS wealth imputation with the ISA split, plus regional property uprating #681; incumbent seeds recorded as documentation.
2. student_loan_balance allocation off-by-one fixed (defect class): the incumbent looks up 1-based sernums in a 0-based RangeIndex (wealth.py:238–241), assigning each household its neighbor's balance and dropping household 1's. Ours allocates by entity id; a regression test pins non-positional alignment.
3. corporate_wealth as component-sum: drawn as excl-ISA + S&S-ISA and folded, per the UK: surface cash ISA and stocks & shares ISA holdings in the wealth imputation #180/WAS wealth imputation with the ISA split, plus regional property uprating #681 split adjudication; later chain outputs condition on the folded total, preserving incumbent conditioning semantics.
4. num_vehicles integer draw (rint + clip ≥0) replaces microimpute's incidental numeric-categorical RandomForestClassifier conditional-mode path.
5. Exact pinned column names replace the fuzzy r↔w rename fallback; missing columns fail loud. DVCISAVR8_aggr resolved against the real tab header (the bare DVCISAVR8 named in UK: surface cash ISA and stocks & shares ISA holdings in the wealth imputation #180/UK: household-wealth imputation stage with cash / S&S ISA split #181 does not exist).
6. Port-as-is quirks documented, not diverged: blanket fillna(0); GOR code 3 absent from the region map; unweighted uprating means; NI unscaled by the uprating; the WAS↔FRS predictor definitional mismatch (FRS-side predictors materialize through the rules-engine adapter with person/benunit values summed to household, the incumbent's map_to="household" semantics).
7. UKDS negative sentinels recoded to zero for the two nonnegative-domain donor columns the licensed audit found carrying them (vcarnr8: 2 rows of −8; HBedRmR8: 95.8% −8 — the bedrooms question is effectively unasked in the WAS household file). The incumbent trains its QRF on the raw sentinel values. DVPriRntR8's −9 is structural not-applicable, so the is_renting mapping is unchanged; genuinely negative domains (net financial wealth, losses) are never recoded. Post-recode deltas measured within realization noise (adjudicated in-PR, María 2026-08-19).

Licensed acceptance — COMPLETE and adjudicated

Full run 2026-08-19 (evidence data/ukds/acceptance/e5/, receipt acceptance_receipt.json; summary in the PR comments): twin builds + a clean-tree third build payload-identical; permutation-identity receipt PASS; out-of-build seed-0 refit reproduces all 13 stored columns bitwise; method-class parity-or-improvement vs the incumbent evidenced by the donor benchmark (the incumbent's correlated-rank draws inflate skewed columns 3.3–4.8× over WAS donor truth; ours 1.3–3.3×). Register gaps adjudicated as composition/calibration class (binds at E10).

Adjudications (María, 2026-08-19): (1) support-bounds SDC treatment accepted — one-significant-figure outward rounding, rationale incl. the top-code caveat recorded in the resource; (2) WAS sentinel audit → recode-to-zero for the two nonnegative-domain columns carrying sentinels (vcarnr8 2 rows; HBedRmR8 95.8% — bedrooms effectively unasked; predictor-quality revisit on #145), measured in-PR per the size rule (deltas within realization noise; signed difference 7); (3) owned_land reviewed input-mass-parity exclusion granted with a one-month expiry (2026-09-19) — the gate fails the expired entry until renewed — with the end-of-workstream revisit registered on #145; (4) the tail-attenuated columns (savings/other_residential/owned_land) signed as method-class differences: E5 is not required to reproduce the incumbent's inflated totals.

Merge coordination with #706 — RESOLVED

#706 merged first, so this PR applied the agreed second-lander recipe on rebase: the gate entries are unioned (#706's re-baselined weight-ratio/input-mass/tail-concentration parameters kept intact; this PR adds uk_support and the two ISA export allowances) and the three _UK_GATE_BATTERY_* digests were re-cut from the producer payload (test_gate_battery_contract_pins driving), with both contract mirrors updated in lockstep.

Implementation notes

Codex implemented under /codex-implement-plan; one recorded deviation (the placeholder support bounds, covered above). Claude review added two fixes pre-commit, both in the class PR CI cannot see: recipient predictors are now aggregated to household grain by native entity (num_adults/num_children are benunit, the four income predictors person — the direct engine arrays would have crashed the licensed build; the fake-engine tests now serve native-grain arrays with entity metadata), and region is one-hot encoded jointly across donor and recipient (matching the incumbent's dummy encoding and the hmrc_spi_income precedent) instead of an ordinal code.

🤖 Generated with Claude Code

[juaristi22]

Adversarial review (Codex, 2026-08-19) dispositioned at 3b79074 — both findings fixed:

1. [critical] Placeholder support bounds → was_wealth_support_bounds.json regenerated from the pinned licensed WAS household tab via tools/build_uk_was_wealth_support_bounds.py. The resource now records source.tab_sha256 (= the stage's donor pin 18b3eb98…), a CI-safe test asserts that binding and fails on any placeholder wording, and a licensed-only round-trip test (POPULACE_UK_WAS_TAB, skips where the tab is absent — PR CI is secrets-free by design) enforces byte-identity between the committed resource and regeneration. SDC treatment (one-significant-figure outward rounding) and the num_vehicles −8 donor sentinel are flagged in the PR body for reviewer adjudication.

2. [high] WAS QRF fit outside the fit-weight audit → impute_was_wealth now returns UKWASWealthImputationResult carrying one FitWeightRecord per chain fit (13, named uk_was_2018_20_wealth:<target>, all resolving explicit); the transform exposes them via the HMRC-stage fit_weight_records precedent; _stage_fit_weight_records aggregates evidence from every fitting stage — detected by declared stage name (hmrc_spi_income, was_wealth) or exposed records, so a hollow/swapped transform forces the failing empty artifact instead of a named absence, and one stage's records cannot absorb another's missing evidence. Tests cover the aggregation, the audit-bypass case, and the WAS names appearing in uk_weights_audit.details.resolved_weight_kinds.

Full microcosm-build/microcosm-fit/microcosm-data suites + ruff green at 3b79074.

🤖 Generated with Claude Code

[juaristi22]

E5 licensed acceptance run COMPLETE (2026-08-19, evidence in data/ukds/acceptance/e5/, receipt acceptance_receipt.json; all data stayed local, aggregates only below).

Determinism & provenance — twin builds payload-identical, shares byte-identical; a third clean-tree build from the committed 14ce3198 reproduced spine-a byte-for-byte; permutation-identity receipt PASS (uprating rewrite bitwise, waterfall re-allocation at fp tolerance, stored-column cross-check); an out-of-build seed-0 chain refit reproduces all 13 stored columns bitwise. All build attempts hash-chained in the Logbook spool (chained onto E7's head). One licensed-only defect found and fixed (14ce3198): the regional uprating factor divided by a row-order-dependent float mean — now order-independent (sorted summation).

Parity vs the incumbent — parity or improvement, with donor-truth evidence:

• Register totals sit below the certified register on all 11 columns, but the incumbent at pin, matched composition is itself at 0.08–0.90 of the register — the gap is composition/calibration class (enhanced-artifact synthetic rows + calibrated weights), binding at E10, the E7 precedent.
• Method class (ours/incumbent at matched composition): num_vehicles 0.996, property_wealth 1.013, main_residence 0.905, financial wealth 0.89–0.91, corporate 0.86. The three gap columns (savings 0.53, other_residential 0.27, owned_land 0.44) are adjudicated by the donor weighted-mean benchmark as incumbent inflation: the incumbent's correlated-rank draws sit 3.3–4.8× above WAS donor truth on those columns; ours 1.3–3.3×, closer on every column.
• Nonzero shares track the WAS donor ground truth on all 13 columns (max ~0.02); num_vehicles at 0.772 vs NTS-truth ~0.78 is an improvement over the certified reference's 0.666.
• student_loan_balance: ours ~£59bn, within 5.4% of the certified register; the incumbent at pin yields ~£13bn (668 recipient persons vs our 1,519) — anomaly is incumbent-side (correlated-rank incidence + the off-by-one we fixed-and-signed).
• ISA split: shares match the donor (cash 0.332 vs 0.333; S&S 0.145 vs 0.133); totals recorded documented-uncalibrated vs HMRC market totals.
• owned_land seed-stability receipt: national total swings 37.7% between adjacent seeds (London 2.41×) — the uk-data#448 class reproduced in-method. Proposed: reviewed exclusion for owned_land register parity at this grain.

🤖 Generated with Claude Code

[juaristi22]

All four adjudications implemented at 274ae2b (María, 2026-08-19):

1. Support-bounds SDC — accepted, rationale recorded. The resource's sdc_treatment now carries the full reasoning: one-significant-figure outward rounding so no committed number is an exact unit-record value, with the top-code caveat considered and accepted (top-codes are published methodology), adjudication date and forum recorded.

2. Sentinel audit + recode — in-PR, measured. Auditing all 33 used WAS columns found sentinels in exactly two nonnegative-domain columns: vcarnr8 (2 rows of −8) and HBedRmR8 (95.8% −8 — the bedrooms question is effectively unasked in the round-8 household tab; no alternative column exists). Both now recode to zero before fitting (signed difference 7 — the incumbent trains on raw sentinels). DVPriRntR8's −9 is structural not-applicable (mapping already correct); legitimate negative domains untouched. Measurement (pre/post rebuild): most columns move <2% — well inside the seed-realization noise band (owned_land −3.9% vs 35% seed noise); the largest is student_loan_balance +12%, moving toward the register (delta now +6.0%); nonzero shares shift ≥0.001 on only two columns. Committed num_vehicles bound is now a clean [0, 10]. The deeper num_bedrooms predictor-quality question is registered on Port UK compact source stages and 2024-25 refresh from uk-data #145 for end-of-workstream.

3. owned_land exclusion — granted with deadline. Entered in input_mass_reviewed_exclusions.json with expires_on: 2026-09-19; the gate fails the expired entry until the adjudication is renewed, so the deadline is self-enforcing. End-of-workstream revisit (renew-or-remedy: winsorised donor / separate land imputation per uk-data#448) registered on Port UK compact source stages and 2024-25 refresh from uk-data #145 (comment 5341180811).

4. Tail-attenuated columns — signed. Recorded in the ledger with the donor-benchmark evidence: E5 is not required to reproduce the incumbent's inflated totals.

Post-adjudication state: twins rebuilt with the recode, all receipts green (seed-0 refit matches stored bitwise; identity receipt PASS; licensed bounds round-trip byte-identical), full suites + ruff green, evidence receipts updated. CI running on 274ae2b.

🤖 Generated with Claude Code

[juaristi22]

Rebased over merged #717 (tip 71851297). Resolution per the pre-agreed collision surface:

• Stage order: frs_brma → was_wealth → regional_property_uprating → the three SPI-channel stages → the certified pair at the tail — honoring the donor-inheritance rule that the SPI stacking must see the wealth-imputed FRS rows.
• Role collision: Create the SPI income channel on the microcosm spine with the FRS-only second stage #717's input-pin collector keys artifacts by role and its SPI donor claims qrf_donor; the WAS donor artifact is now was_qrf_donor, and E7's pin test verifies both.
• Licensed rerun on the merged 16-stage roster: twins payload-identical, shares byte-identical, and the E5 stage-level draws are byte-stable across the rebase (stage shares identical to the adjudicated run). Two composition mechanics documented in the receipts: the final frame now carries 10k stacked SPI-synthetic households (E5 instruments scope to household_is_spi_synthetic == False), and the survey channel's exported weights carry the declared prior-mass share 0.5 (instruments restore the grossing scale from the manifest). With that scoping the adjudicated acceptance numbers reproduce exactly (owned_land −0.785, student_loan_balance +0.060, …); identity receipt PASS at the scoped population; seed-0 refit matches the stored spine bitwise.

Full suites + ruff green locally; CI restarting on the rebased tip.

🤖 Generated with Claude Code

[vahid-ahmadi]

Reviewed the same way as E2–E4: every derivation against the incumbent's wealth.py/regional_property_uprating.py, plus the resources and both adjudication rounds. 258 tests pass for me with the uk extra.

Signed difference 2 is confirmed in the incumbent's code, and it should go upstream. At wealth.py:346 the incumbent passes output_df[column].clip(lower=0) to the allocator — output_df comes straight from model.predict and carries a 0-based RangeIndex — while _allocate_student_loan_balance_to_people iterates .items() treating that index as person_household_id and looks it up in a groupby keyed by real 1-based sernums. Every household receives the balance predicted for its positional neighbor, and the first household's prediction is dropped. This is live in production (and the incumbent's own test_student_loan_balance.py evidently doesn't catch it, since the allocation tiers behave plausibly on whichever balance arrives). Like the EDUCQUAL inversion, this deserves a policyengine-uk-data issue with your line citations — that's now two production defects this migration has found by porting carefully, and the upstream repo shouldn't have to rediscover them at its next release. The HBedRmR8 finding belongs in the same issue: 95.8% of the round-8 donor rows carry −8 in a QRF predictor the incumbent trains on raw, which means the incumbent's bedrooms predictor is effectively a corrupted constant.

Verified faithful against the incumbent: the corporate_wealth composition (your excl-ISA components + stocks_and_shares_isa fold reproduces the incumbent's five-component sum exactly, with the fold ordered before segment 3 so later outputs condition on the total, preserving the incumbent's chain semantics); the segmented chain state accumulating drawn targets within each segment (sequential conditioning intact); the recipient-side NORTHERN_IRELAND → WALES remap; the allocation tier cascade with repayment-proportional split at tier 0; and the uprating design (per-region HPI ratio on owner-occupier means, property_wealth moved proportionally, the unweighted means honestly signed as a port-as-is quirk). The exact-match cleaning replacing the r↔w fuzzy fallback — with DVCISAVR8_aggr resolved against the real header where #180's bare name didn't exist — is the coded-domain checklist rule doing its work at ingest.

The adjudication rounds are the strongest part. María's sentinel ruling is well-evidenced in both directions: the audit measured all 33 used columns rather than spot-checking, the recode's effect is bounded against the seed-realization noise band (owned_land −3.9% vs 35% noise), and the one material move (student_loan_balance +12%) goes toward the register. The post-rebase receipt — E5 stage draws byte-stable across the #717 rebase, adjudicated acceptance numbers reproducing exactly once the instruments scope to household_is_spi_synthetic == False and restore the declared grossing scale — is precisely the composition-aware re-verification the 16-stage roster needed. The was_qrf_donor role rename resolving the #717 pin collision, with E7's pin test verifying both, closes the coordination surface cleanly.

Two small things, neither blocking:

• support_clip_to_donor silently continues when an output column is missing from the donor frame. The bounds resource and its round-trip test make that unreachable today, but everything around it fails loud on missing columns — one raise would keep the posture uniform.
• In run_segment, recipient_base = pd.concat([recipient_encoded.loc[:, list(base_predictors)]], axis=1) is a one-element concat — just a copy wearing a disguise; recipient_encoded.loc[:, list(base_predictors)].copy() says what it does.

Carried over from #717 (merged before my early review's items were addressed — all non-blocking, so no complaint about the merge, but they shouldn't evaporate): the or WeightKind.IMPORTANCE silent default in the support-channel transform, the last_result/resume invariant comment, the unexplained stage-2 seed + 1 convention, and the open question of whether the licensed run settled the incapacity_benefit_reported promotion or deferred it to E10. A one-commit sweep here or a note on #145 would close all four.

Ready from my side once the upstream issue is filed (or explicitly declined with a reason on the record).

[vahid-ahmadi]

The upstream condition from my review is satisfied — I filed it myself rather than leave it on your plate: policyengine-uk-data#463 covers both the student_loan_balance positional mis-allocation (verified at the predict site: output_df's RangeIndex read as household ids) and the sentinel-training finding, crediting your E5 adjudication receipts for the measurements. With that, nothing blocking remains from my review — the two nits and the four #717 carryovers are cleanup-sized and can ride any later commit.