Which cng-datasets version built each dataset? (0.8.0 now stamps it; earlier builds need an audit)

[cboettig]

Opening this here rather than in datasets because the decisions that remain are about how this repo generates and records builds. Would value your read on the audit scope in particular.

What changed upstream

cng-datasets 0.8.0 stamps every published parquet with the version that produced it, in the file's own key-value metadata rather than a sidecar, so it survives copies and syncs:

SELECT key, value FROM parquet_kv_metadata('s3://bucket/dataset/hex/h0=*/data_0.parquet');
-- cng_datasets_version  0.8.0
-- built_at              2026-09-21T23:06:45Z
-- gdal_version          3.13.0
-- duckdb_version        1.5.4
-- image                 (when CNG_IMAGE is set)

Covers the hex partitions, the merged data_0.parquet, the vector repartition output and the GeoParquet.

Why it was needed, and why it is sharper than it sounds

Generated manifests pin ghcr.io/boettiger-lab/datasets:latest, and that image is rebuilt on every push to main. So the code that produced a dataset was not necessarily a released version — it was whatever main happened to be when the pod pulled the image. Nothing recorded which.

Separately: every version-tagged image build had been failing since v0.5.0 (an empty {{branch}} in the tag template produced an invalid reference, which buildx rejected before pushing anything). Until today there were no versioned images at all, so pinning was not even possible. That is fixed, and :0.8.0, :0.8 and :latest all resolve now.

The audit

Two correctness bugs were fixed in 0.7.0 (2026-09-20). Both are undetectable from the output alone — right row count, right schema, right value range, job exits 0:

1. Multi-band sources silently read band 1 (datasets#214). The hex path passed the source to exactextract with no band index and labelled the result with --value-column regardless. This published 384,922,346 rows of annual forb & grass cover documented as perennial cover — two variables in the same 0–100 domain.

2. --h0-subset takes grid positions, not H3 base cell numbers (datasets#213, #218). Both numberings run 0–121, so a list computed the obvious way — from the H3 library — is accepted, and builds a different part of the world. In the measured case that was ~80% of a western-US dataset missing, with the job reporting success.

A dataset whose parquet carries no cng_datasets_version was built before 0.8.0 and falls in scope. Concretely, worth checking:

• any hex build from a multi-band source (bug 1 applies only to those);
• any build that passed --h0-subset / --h0-cells where the list came from the H3 library rather than the grid's own i column;
• ingests from roughly 2026-09-12 to 2026-09-20, which is when :latest carried 0.6.x.

Everything else from that window is a performance/reliability story rather than a correctness one — evictions, OOMs and slow runs, all of which failed rather than publishing bad data.

What I'd value your view on

1. Is the audit list above the right scope? You will know better than I do which recent ingests used multi-band sources or a hand-computed --h0-subset.
2. Should generated manifests pin :0.8.0 rather than :latest? Now possible. It makes a build reproducible and a rollback real, at the cost of updates no longer arriving automatically — a build would pick up a new version only on regeneration. That tradeoff is yours more than ours.
3. Should the version reach STAC as well as the parquet? The parquet stamp travels with the data, but a catalogue consumer will not open the file to find it.
4. Anything else worth stamping? The completion markers are the obvious next candidate, so a partly-failed or gap-filled fan-out shows whether it was assembled from more than one build.

Happy to implement any of (2)–(4) upstream; (1) is the one I cannot do from here.

[cboettig]

Audit run: the published RAP data is correct, and no other workflow is exposed

I ran the comparison against the live partitions rather than reasoning from the manifests. Result: rap-pfg-cover is perennial forb & grass, exactly as labelled. No remediation needed.

1. Which band is the published COG?

rap-pfg-cover-cog.tif and the 6-band source share dimensions, origin and pixel size exactly (214847 × 92304, EPSG:4326, 0.000269°), so this is a pixel-for-pixel comparison rather than a statistical one. Five windows across CONUS, 184,320 valid pixels:

	b1 AFG	b2 BGR	b3 LTR	b4 PFG	b5 SHR	b6 TRE
matching pixels	7.0%	11.7%	11.2%	100.00%	11.0%	10.3%

2. Does the published hex match it?

360 published cells with pfg > 5, sampled across all six CONUS partitions, re-aggregated with the same operation the pipeline ran (exact_extract mean, nodata 255):

candidate	cells	exact matches	mean abs diff
the published COG	360	360	0.0000
raw band 4 (PFG, perennial)	360	360	0.0000
raw band 1 (AFG, annual)	360	0	10.2 – 22.6

Exact to the last bit against band 4; nothing like it against band 1. The bad build described in datasets#214 was evidently rebuilt.

3. No other workflow is exposed

• Multi-band: I enumerated every hex step in this repo (cng-datasets raster with --output-parquet) and checked band counts on the real sources. RAP's vegetation-cover-v3-2025.tif is the only multi-band raster in the catalog. All three RAP hex steps read single-band COGs produced by a preprocess step — --band 4 for PFG; rap-arte and rap-iag come from separate single-band raw products (verified, 1 band each), so they were never at risk.
• h0 subsets: every committed subset is in correct grid positions — 12 14 20 50 71 78 (CONUS, 9 manifests), 105 (MTBS Alaska → base cell 6, 64.4°N 158.9°W ✓), 20 (WRC-2 dense → CONUS ✓). Nobody passed the base-cell list 9 19 20 21 34 36. Worth noting the Alaska near-miss: had 105 been read as a base cell it would have been position 57, in the South Pacific at 46°S — and the job would have reported success.

What this does not cover

The sampled cells come from the first row group of each partition, so they are three spatial clusters per run rather than a uniform random sample of 105.6M rows. Given 360/360 exact against one band and 0/360 against the other, that is not a close call — but it is a spot check, not a full verification.

The partitions carry no cng_datasets_version, as expected for anything built before 0.8.0. That is precisely the gap this issue is about: the check above took an afternoon of inference about which bands and which subsets, where a stamp would have answered it in one query. Going forward it will.