Continuous Population Register — Docker Prototype

A runnable proof-of-concept web platform for the Continuous Population Register and Modular Survey System design.

This is intentionally a conventional stack that is easy to operate on a small Ubuntu server:

• Flask + Gunicorn — web application
• PostgreSQL 16 + PostGIS — statistical register
• Separate PostgreSQL 16 Identity Vault — names, PINs and contacts
• Docker Compose — deployment
• Server-rendered HTML/CSS — one hand-written stylesheet, no Node.js build chain
• Survey Solutions REST integration — questionnaire sync, register prefill, assignments, interviews, change review and exports

What is implemented

The web UI currently supports:

• Dashboard and register counts
• Person creation and current person listing
• Separate Identity Vault records for names, PIN and contact details
• Bitemporal person version history and correction/supersession
• Dwelling creation, geocodes and addresses
• Household episodes linked to dwellings
• Person-to-household memberships
• Immutable demographic events
• Provenance/source catalogue
• Six-monthly update rounds
• Field case creation and contact-tier attributes
• Small authenticated JSON API
• Architecture diagram
• Documented design system, dark-mode support and a database-free UI preview
• Demo data and a two-database Docker deployment
• Survey Solutions connection/settings screen
• Survey Solutions questionnaire/version registry
• Register ↔ Survey Solutions variable mappings
• Assignment generation from register field cases with protected prefill variables
• Interview/status synchronisation and immutable raw API staging
• Proposed register-change queue with explicit approve/reject workflow
• Bitemporal application of accepted person/household/dwelling corrections
• Survey Solutions export job creation/status/download

The PostgreSQL schema also contains structures for ingestion/staging, validation, matching, metadata/questionnaires, PRN sample coordination, burden accounting, statistical snapshots, weighting, dissemination and audit. Those areas are database-ready skeletons, but not all have full web screens in this prototype.

Important status

This is a prototype / proof of concept, not a production national register. Before real deployment you would still need formal security review, proper user/role management, CSRF protection, TLS termination, secret management, hardened audit logging, database backup policy, penetration testing, matching workflows, data protection controls, disaster recovery and jurisdiction-specific legal/governance review.

Do not load real identifiable population data into an internet-facing deployment of this prototype without that work.

---

Ubuntu installation

1. Install Docker

On Ubuntu:

sudo apt update
sudo apt install -y ca-certificates curl
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
sudo chmod a+r /etc/apt/keyrings/docker.gpg

echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
  $(. /etc/os-release && echo \"$VERSION_CODENAME\") stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

sudo apt update
sudo apt install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
sudo systemctl enable --now docker

2. Clone your GitHub repository

git clone https://github.com/YOUR-USER/YOUR-REPO.git
cd YOUR-REPO

3. Configure secrets

cp .env.example .env
nano .env

Change at minimum:

• REGISTER_PASSWORD
• IDV_PASSWORD
• ADMIN_PASSWORD
• SECRET_KEY

If you will connect Survey Solutions, also set:

• SURVEY_SOLUTIONS_PASSWORD — the API password. The server URL and username are configured later in the web UI.

Generate a good Flask secret, for example:

openssl rand -hex 32

4. Start the platform

docker compose up -d --build

The web container applies the idempotent Survey Solutions integration migration on startup. This means an existing prototype database volume can be upgraded without docker compose down -v.

Check status:

docker compose ps

Watch logs:

docker compose logs -f web

5. Open the web application

From your browser:

http://YOUR_SERVER_IP:8080

Use the ADMIN_USER and ADMIN_PASSWORD values from .env.

The first startup automatically inserts a small fictional demo household so the interface is not empty.

---

Reset the demonstration databases

Warning: this deletes all prototype data.

docker compose down -v
docker compose up -d --build

---

Backup

./scripts/backup.sh

This creates separate PostgreSQL dumps for the statistical register and Identity Vault under ./backups/.

For production, store encrypted backups off-host and test restore procedures regularly.

---

Health check

curl http://localhost:8080/health

Expected response:

{"status":"ok","register_db":true,"identity_db":true}

---

API examples

The current API uses the same login session as the web interface.

Endpoints:

GET /api/v1/dashboard
GET /api/v1/persons
GET /api/v1/dwellings
GET /api/v1/survey-solutions/status

A later iteration should replace this with service/API authentication and scoped access tokens.

---

Repository structure

.
├── app/
│   ├── db.py
│   ├── web.py
│   ├── surveysolutions.py
│   ├── integration.py
│   ├── ss_web.py
│   ├── templates/            Jinja screens, plus _macros.html and _icons.html
│   └── static/               app.css, favicon and the architecture diagram
├── sql/
│   ├── 01_population_register.sql
│   ├── 01_identity_vault.sql
│   └── 02_surveysolutions_integration.sql
├── docs/
│   ├── INTERFACE.md
│   ├── ENUMERATOR_MANUAL.md
│   ├── SURVEY_SOLUTIONS_INTEGRATION.md
│   └── img/
├── scripts/
│   ├── backup.sh
│   ├── entrypoint.sh
│   └── preview_ui.py         renders every screen without a database
├── tests/
├── .github/workflows/
├── docker-compose.yml
├── Dockerfile
├── seed_demo.py
├── run.py
└── requirements.txt

---

Interface

The web interface is server-rendered Jinja plus a single hand-written stylesheet — no Node.js, no CSS framework, no build step. It is a navigation rail beside a content column, and every screen is a page header followed by panels containing a table, a form or a record detail list. Colours, spacing and type all come from custom properties in one place, status badges are coloured from a shared vocabulary so a given register status always looks the same, and the whole interface follows the operating system's light or dark appearance.

You can render every screen with representative sample data — including empty states and each status tone — without Docker or PostgreSQL:

python scripts/preview_ui.py     # writes ./_ui_preview

Then open _ui_preview/index.html in a browser. The same sample contexts back tests/test_templates.py, so a broken template or macro fails CI:

python -m unittest discover -s tests -p 'test_*.py'

See docs/INTERFACE.md for the design tokens, the component catalogue, the template macros, the accessibility rules and the checklist for adding a screen.

---

Architecture choices

Separate Identity Vault

The application runs two databases:

1. population_register — statistical/register content and operational metadata.
2. identity_vault — names, PINs, direct contact details and identity access audit.

They share only the opaque person_id. PostgreSQL cannot enforce a foreign key between two separate database services, so the application integration layer is responsible for maintaining the shared identifier.

Bitemporal records

Core version tables carry:

• valid_from, valid_to — when the fact was true in the real world.
• recorded_from, recorded_to — when that version was held by the system.

The database prevents destructive deletion of core history. The person correction screen demonstrates the intended pattern: close the transaction-time interval of the old version, then insert a new version.

Dwelling as geographic anchor

Households link to dwellings and person memberships link to the household/dwelling episode. Households can therefore form, split, merge or move without treating geography as part of the household identifier.

---

Push to GitHub

After extracting this project:

git init
git add .
git commit -m "Initial population register prototype"
git branch -M main
git remote add origin https://github.com/YOUR-USER/YOUR-REPO.git
git push -u origin main

Never commit your real .env file. It is already listed in .gitignore.

---

Survey Solutions integration

Survey Solutions is now integrated as the collection engine, while this PostgreSQL application remains the authoritative population register. The intended workflow is:

register case -> mapped prefill -> Survey Solutions assignment -> field interview
             -> API sync -> immutable staging -> proposed change review
             -> accepted bitemporal register update / immutable event

After starting the app:

1. Put SURVEY_SOLUTIONS_PASSWORD in .env and rebuild the web container.
2. Open Survey Solutions → Integration in the navigation rail, then Settings; enter the Headquarters URL, workspace and API username, and test the connection.
3. Open Questionnaires and synchronise imported questionnaires.
4. Configure variable mappings for the questionnaire.
5. Generate assignments from register field cases.
6. Synchronise interviews.
7. Review returned register corrections under Change queue before applying them.

The API password is read from an environment variable and is not stored in either database. Survey Solutions raw API responses are staged append-only. Survey content does not automatically become register content: only explicit mappings create proposed register changes.

See docs/SURVEY_SOLUTIONS_INTEGRATION.md for the questionnaire contract, mapping rules, operational workflow and current limitations.

Field manual for enumerators

docs/ENUMERATOR_MANUAL.md is the field-side counterpart to that document: the complete manual for the enumerators who work the cases in Survey Solutions Interviewer. It covers assignments and contact tiers, the prefilled roster and protected variables, the contact protocol, the core confirmation interview step by step, a catalogue of field situations (vacant dwelling, household split, death, emigration, refusal), the event and status vocabularies the register accepts, sync discipline, quality checks and data-protection rules.

---

Good next development steps

1. Add PostgreSQL-backed user/role management and row-level security.
2. Build administrative-feed upload and staging screens.
3. Add deterministic/probabilistic linkage review queues.
4. Extend Survey Solutions integration to create brand-new persons and fully automate membership move/split/merge proposals.
5. Implement reference-date snapshot generation and project-specific pseudonyms.
6. Add sample selection using permanent random numbers and burden rules.
7. Add production weighting/calibration workflows in R or Python.
8. Add disclosure-controlled publication endpoints.
9. Add a scheduler/outbox for high-volume Survey Solutions sync and retry handling.
10. Put the app behind Caddy/Nginx/Traefik with HTTPS and hardened headers.

License

MIT — see LICENSE.