Skip to content

NZ source packages, wave 1: the populace-nz v1 16-margin surface #176

Description

@juaristi22

Lane: ledger-source-ingestor (.github/chronicle-agents.yml) — judges ledger-source-fidelity + ledger-boundary
Depends on: #175 — the R2 country layout blocks every credentialed publish; the tax-year note blocks the IRD batch; the geography rulings block the MSD and MBIE batches. The stats_nz batches (PRs 1–2) can be authored first.

Context

New Zealand has no incumbent target registry to port (unlike the UK waves) — the surface being retrieved is the populace-nz v1 calibration set: 16 margins, all public, declared in docs/nz-take-up-engine-plan.md @ de7451b, with per-source URLs, licences, and fetch-status in the companion docs/nz-calibration-targets.md (every entry live-verified 2026-07-06). All values are published official statistics (Stats NZ / IRD / MSD / MBIE) — no licensed microdata is involved.

Chronicle has zero NZ presence today. This issue authors the wave-1 packages; the nz_national profile issue is drafted and will be filed separately; wave 2 (take-up/validation surface) is #177; local geography is #178.

Sourcing note (NZ-specific): stats.govt.nz's SPA, treasury.govt.nz HTML, and catalogue.data.govt.nz bot-wall scripted clients, but the file CDNs are open (ird.govt.nz, msd.govt.nz, tenancy.govt.nz, Stats NZ /assets/Uploads/…) — pin those. Infoshare and the Aotearoa Data Explorer SPA are aggregator clues, not artifacts: pin the publisher release file, or a curated CSV extract with pinned cell geometry where only the SPA publishes the cut (Statbel precedent). Manual-download registration with the canonical URL recorded is the gov.wales-precedented fallback. Infoshare retires into ADE April 2027 — nothing here may pin an Infoshare URL.

Package inventory (margin → package)

Fifteen packages serve the 16 margins (the minimum-wage half of the earnings margin is not a chronicle fact — a statutory rate has no entity population and no honest aggregation; it is a rules parameter already encoded in rulespec-nz. Stated here so the margin accounting closes: 16 margins = 15 package families + 1 rules-side parameter).

# Package Artifact Parser Period Geography Margin(s) served
1 stats_nz/subnational_population_estimates_2025 S7 information-release xlsx (open /assets/Uploads/ CDN; direct-xlsx pattern confirmed on the Jun-2024 vintage) xlsx_used_range + sheets: calendar_year 2025, coverage 2025-06-30 region, REGC codes, vintage regc_2025 Age(5-yr)×sex×region ERP
2 stats_nz/national_population_estimates_2025 S8 national estimates release xlsx xlsx_used_range calendar_year 2025 country Single-year-age national ERP
3 stats_nz/census_2023_households_by_region S2 ArcGIS hub CSV export, regional-council layer delimited_text_full_rows calendar_year 2023, coverage 2023-03-07 region, regc_2023 Household composition and tenure×region (two record-set families)
4 stats_nz/census_2023_family_type S3 hub CSV (families/extended-families layer) delimited_text_full_rows calendar_year 2023 country Family type
5 stats_nz/census_2023_ethnicity_age_region O5 curated CSV extract from ADE (cross-tab not published as a hub layer) delimited_text_full_rows calendar_year 2023 region Ethnicity×age×region
6 ird/taxable_income_distribution_2025 I1 taxable-income-distribution-of-individuals-2025.xlsx xlsx_used_range + sheets: tax_year 2024 (ending-year label — #175) country Taxable income by band (count + $); age×income band (Tab 4)
7 ird/wage_salary_distribution_2025 I2 dataset xlsx (resolve direct URL at pin time) xlsx_used_range tax_year 2024 country Wage/salary distribution
8 ird/working_for_families_statistics_sept_2025 I3 working-for-families-statistics---sept-2025.xlsx xlsx_used_range + sheets: tax_year 2024 country, entity family WfF families + expenditure by credit type; recipient-family income distribution (the joint-income tripwire record set)
9 ird/student_loan_statistics_march_2026 I4 student-loan-statistics.xlsx xlsx_used_range month 2026-03 country Student loan borrowers
10 msd/benefit_fact_sheets_national_march_2026 M1 national benefit tables xlsx xlsx_used_range + sheets: month 2026-03 (quarter-end month; publisher quarter label in source_period_label — DWP precedent) country Main-benefit recipients by benefit × age
11 msd/benefit_fact_sheets_supplementary_march_2026 M1d W&I supplementary tables xlsx xlsx_used_range month 2026-03 statistical_scope, slug ids, vintage msd_wi_region (#175 ruling) + national total row AS recipients by W&I region
12 msd/nzs_vp_fact_sheet_march_2026 O4 NZS/VP fact-sheet xlsx xlsx_used_range month 2026-03 country NZS/VP recipients
13 msd/annual_report_benefit_expenses_2025 M2 Annual Report 2025 PDF, "Benefits or Related Expenses" schedule pdf_text_numbers fiscal_year 2024 (FY2024/25 → opening year) country, entity government, value_scale: 1000 Benefit expenditure $ by program
14 mbie/tenancy_bond_rents_tla_2026 H3 detailed-monthly-tla-tenancy-v2.csv (open CDN; CC BY 3.0 NZ, note in package header) delimited_text_full_rows with selected_rows pinning one month (PIPR lane) month (latest stable month) local_authority, TA codes, vintage ta_2025 (#175 ruling) Rent distribution by TA × bedrooms (median; geometric mean per #175's aggregation ruling)
15 stats_nz/qes_average_earnings_march_2026 S12 QES Mar-2026 release tables (open CDN, not Infoshare) xlsx_used_range month 2026-03 country; provenance_class: survey_aggregate + survey_instrument: Quarterly Employment Survey Average-earnings anchor (minimum wage → rules-side, see above)

All other packages are provenance_class: administrative except #1#5 (census) and #15 (survey_aggregate). Units: count, nzd (free-form — no vocabulary change).

Mechanics (all precedented — follow, don't reinvent)

  • Per package, the 6-file pattern: SOURCE_PACKAGE_ALIASES line (alias↔directory drift is test-enforced), artifact bytes under db/data/…, manifest.yaml, source_package.yaml, checklist row in docs/pe-nz-source-checklist.md (may be a follow-up commit — docs/** is outside this lane), re-measured tests/test_chronicle_bundle.py with the arithmetic in the commit body.
  • Commit rhythm per batch: pin artifacts → author packages → record checklist rows → re-measure bundle expectations. R2 publishes land under raw/nz/… (NZ source conventions: R2 country layout, tax-year labelling, geography rulings #175) in a separate credentialed publish-raw commit per batch — designed-red on raw_artifacts_have_r2 until it lands; the credential is a PolicyEngine Cloudflare account membership + wrangler login.
  • Multi-geography tables (Establish Arch data foundation #1, Add source-package facts for PE parity targets #3, Move Arch storage config to PolicyEngine #5, Add HHS ACF TANF source packages #14) use the whole-table orientation: one record set per publisher category, rows: carry geometry only, semantics in shared_filters/shared_constraints; declare sheets: on multi-sheet workbooks to cap source-cell cost.
  • A country regression test file tests/test_nz_targets.py (Belgium's test_belgium_targets.py precedent), grown per batch.
  • Gate per package: chronicle validate-packagechronicle build-suitereports/agent_acceptance.json all-green (watch for missing_raw_r2_link, missing_concept_alignment_evidence, selected_row_only_source_parse, missing_expected_constraints).

Acceptance

  • A consumer artifact built from these packages resolves every one of the 15 package families — each margin matched by period, geography level, and unit against the plan's margin table; the 16-margin accounting closes with the minimum-wage rules-side note.
  • Every fact traces to publisher bytes: manifest source_url + sha256 + raw/nz/… R2 URI, cell-geometry guards, concept_evidence_url per measure.
  • Deterministic checks green (source-package validation, source-cell preservation, consumer-contract validation, raw-facts boundary validation); both judges green.
  • No PolicyEngine-computed values enter the store (facts-only ADR): no uprating, no rate×count, no cross-source composition. Anything computed is a populace-side declaration.

Out of scope

Wave-2 take-up/validation packages (#177); local geography (#178); the nz_national profile (forthcoming issue); all populace-side consumption (tracked under microcosm#343).

Sizing

15 packages ≈ 5 PRs: (1) stats_nz demographic anchor — #1, #2, #15 first, because calibration initializes design weights at destination population scale; (2) census structure — #3#5, establishing the hub-CSV and ADE-curated-extract patterns; (3) IRD — #6#9, establishing the NZ tax-year convention in one review; (4) MSD — #10#13, carrying the W&I-region ruling and the one PDF package; (5) MBIE rents alone — #14, the big multi-geography whole-table.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions