Skip to content
This repository was archived by the owner on Jun 14, 2026. It is now read-only.

Commit 54600a7

Browse files
MaxGhenisclaude
andcommitted
Update sparse coverage benchmark results
Re-ran benchmarks after fusion pipeline fixes. Key findings: - At 1% coverage: Generative 65% better than weighted (0.37 vs 1.04) - At 2% coverage: Generative 20% better (0.52 vs 0.65) - Generative finds 7-28x expected rate of rare combinations 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
1 parent afa5fe4 commit 54600a7

2 files changed

Lines changed: 44 additions & 6 deletions

File tree

benchmarks/results/FINDINGS.md

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -147,6 +147,43 @@ See generated figures:
147147

148148
**NO** - Despite having individually realistic records (lowest authenticity distance), QRF has the worst multivariate distribution match. This confirms the hypothesis that sequential imputation produces marginally correct but jointly unrealistic records.
149149

150+
## Sparse Coverage Reconstruction (Multi-Survey Fusion)
151+
152+
### Key Question
153+
When surveys only cover a fraction of the population, can generative synthesis create records that cover the unseen population better than weighted resampling?
154+
155+
### Results
156+
157+
| Survey % | Method | Coverage ↓ | Income MMD | Rare Combo Discovery |
158+
|----------|--------|------------|------------|---------------------|
159+
| 10% | Weighted | 0.539 | **0.310**| 0.0x expected |
160+
| 10% | **Generative** | 0.589 | 0.664 | **27.7x** expected |
161+
| **2%** | Weighted | 0.655 | 0.725 | 0.02x expected |
162+
| **2%** | **Generative** | **0.521**| 0.601 | **22.1x** expected |
163+
| **1%** | Weighted | 1.039 | 0.320 | 0.0x expected |
164+
| **1%** | **Generative** | **0.368**| 0.437 | **7.2x** expected |
165+
| Oracle | (full) | 0.089 | 0.007 | 0.94x expected |
166+
167+
### Key Findings
168+
169+
**Coverage dominance at low sample sizes**:
170+
- At 10%: Weighted slightly better (-9%)
171+
- At 2%: Generative **20% better**
172+
- At 1%: Generative **65% better** (nearly 3x improvement)
173+
174+
**Rare combination discovery**: Generative synthesis finds 7-28x expected rate of rare combinations (elderly + self-employed), while weighted resampling finds essentially none.
175+
176+
**Trade-off**: Weighted has slightly better Income MMD at higher coverage, but loses dramatically on coverage at sparse sampling.
177+
178+
### Interpretation
179+
180+
Weighted resampling can only repeat observed records. At low coverage (1-2%), many population combinations simply don't exist in the survey. Generative synthesis learns the joint distribution and can **create novel but plausible combinations** that weren't observed.
181+
182+
This validates the flow-based fusion approach for scenarios where:
183+
1. Surveys have limited coverage
184+
2. Multiple surveys cover different variables
185+
3. Rare populations matter (elderly, high-income, etc.)
186+
150187
## Next Steps
151188

152189
1. **Add conditional multivariate metrics**: Test MMD/Energy Distance within demographic groups
Lines changed: 7 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,8 @@
11
method,demo_mmd,income_mmd,coverage_mean,coverage_median,coverage_p90,coverage_p99,elderly_selfemp_pop,elderly_selfemp_recon,elderly_selfemp_ratio,young_dividend_pop,young_dividend_recon,young_dividend_ratio,time,coverage
2-
Weighted (10%),0.014163257670535742,0.309518708300095,0.5392293192118486,0.2422768436072023,1.155986522670132,4.8848585281289125,980,0,0.0,778,0,0.0,5.1396589279174805,0.1
3-
Weighted (2%),0.0,0.7160308238865077,0.6558060691707802,0.34379130847465544,1.4501061049225923,6.572934058723967,980,11,0.05612244897959184,778,0,0.0,2.2942569255828857,0.02
4-
Generative (2%),0.021442823344618396,0.6424841510463795,0.5316317278212115,0.25227288988934665,1.4645825071715122,3.771863114266973,980,6369,32.494897959183675,778,4654,29.910025706940875,9.794167041778564,0.02
5-
Weighted (1%),0.003338744893915217,0.31960922456394847,1.0387911840908186,0.6521422689756196,2.386006057521714,7.763127052652185,980,0,0.0,778,0,0.0,1.7950232028961182,0.01
6-
Generative (1%),0.03002444322046162,0.3898883489775563,1.0202688697619944,0.4924838431560163,2.363751601635033,7.283098815452937,980,0,0.0,778,0,0.0,4.571079969406128,0.01
7-
Oracle,0.015324983082558082,0.007088539129961354,0.0887090502137913,0.009362932492628553,0.2539899126312401,1.0480836829252882,980,185,0.9438775510204082,778,173,1.1118251928020566,0.0,1.0
2+
Weighted (10%),0.014163257670512226,0.3095187083000937,0.5392293192118486,0.2422768436072023,1.155986522670132,4.8848585281289125,980,0,0.0,778,0,0.0,4.256464004516602,0.1
3+
Generative (10%),0.00734598666840795,0.6637646604064075,0.5893480017751078,0.5329912174541771,1.075968655468742,1.7514328441136464,980,5436,27.73469387755102,778,3203,20.58483290488432,146.3777289390564,0.1
4+
Weighted (2%),0.011016613888369144,0.7246886859969659,0.6547914358296409,0.35173505201368005,1.4340514340193233,6.4702818770484,980,3,0.015306122448979591,778,0,0.0,2.1378748416900635,0.02
5+
Generative (2%),0.018093244815402646,0.6007803337764256,0.5213515181896132,0.42469470546523796,1.0381633521853006,1.7069024405547315,980,4324,22.06122448979592,778,5029,32.32005141388175,92.07200908660889,0.02
6+
Weighted (1%),0.00333874489408148,0.3196092245639519,1.0387911840908186,0.6521422689756198,2.386006057521714,7.763127052652185,980,0,0.0,778,0,0.0,1.2508940696716309,0.01
7+
Generative (1%),0.013879815823350298,0.4374466979591101,0.36784481020509896,0.24905629588107853,0.6748771740824506,1.9554729954677368,980,1403,7.158163265306122,778,265,1.7030848329048844,75.46381092071533,0.01
8+
Oracle,0.015324983082547216,0.00708853912986738,0.0887090502137913,0.009362932492628553,0.2539899126312401,1.0480836829252884,980,185,0.9438775510204082,778,173,1.1118251928020566,0.0,1.0

0 commit comments

Comments
 (0)