DataPact

DataPact is a Django app that checks recurring data files against rules you define. You set the rules once, such as column types, allowed values, and null constraints. DataPact then checks every new file against them and points to the exact row and value that broke a rule, with a plain-English explanation.

Since A6, a signed-in user can also ask an AI model to explain a violation: whether the data is probably wrong or the contract is probably out of date, and what to do next. There are two engines on the same page:

• Local model: HuggingFaceTB/SmolLM2-1.7B-Instruct, run in the Django process with the Hugging Face transformers library. It is the model our 15-model benchmark chose. The weights download on first use (about 3.4 GB). It is on by default in development and off by default in production.
• Groq (optional): the same prompt sent to Groq's hosted openai/gpt-oss-20b. It needs a GROQ_API_KEY in .env; without one, the explain page shows Groq as not set up (with no button), and nothing is sent. Groq gets the dataset name, column, contract rule and row counts; from the file itself it gets only values on a short list of known status codes (such as AUDIT), or blanks for a NOT_NULL rule. Good answers are cached for an hour.

The answer is advice only and never changes the database. Details, privacy rules and what was verified are in README_AI.md.

INFO 490, Team 15, Fall 2026.

Team

Hriday Agarwal, Ashok Chacko, Tejas Jaggi, Connor Slattery

Where the project is

The repo holds the project through A6: the A1 data model, the A2 settings split and views, the A3 ORM/forms/navigation/visualisation work, the A4 APIs, charts, exports and deployment preparation, the A5 sign-in and access control, and the A6 text-intelligence features. All 294 tests pass on main (19 Oct 2026, after the A6 merges).

A4 part	What	Status
1	Internal chart APIs and two Vega-Lite charts (Tejas)	Done
2	External keyless API, Open Library (Connor)	Done
3	Reports page, CSV and JSON exports (Hriday)	Done
4.1	Static files, requirements, .gitignore (Ashok)	Done (in the repo)
4.2	Code cleanup and deployment readiness (Ashok)	Done (in the repo)
4.3	Pull the repository on PythonAnywhere	Server step; the repo points to an A5-era deployment (see below), current state not verified
4.4	Configure the PythonAnywhere web tab	Server step; the repo points to an A5-era deployment (see below), current state not verified

Deployment. Parts 4.1 and 4.2 are code and configuration, and they are in this repo. Parts 4.3 and 4.4 happen on PythonAnywhere, not in Git, so the repo cannot prove them. What the repo does show: A5 demos 2 and 3 (docs/demos/) call tjaggi.pythonanywhere.com, demo 4 is meant to be opened on that site, and the Google sign-in setup lists its callback URL, so a deployment existed during A5. We have not checked which commit that site runs now. None of the A6 changes have been verified on PythonAnywhere. There the local model is off by design (the host's disk is too small for it). The Groq button needs GROQ_API_KEY in the server's .env, and the server must be able to reach api.groq.com (a free PythonAnywhere account can only reach allowlisted sites); neither has been tested from the server.

A6 part	What	Where	Status
1	15-model benchmark (5 size categories x 3 models, 8 fixed inputs) and Route A prompting (Hriday); 3B+ follow-up with 4 larger models (Connor)	llm-test/ai_prototype.ipynb, llm-test/ai_prototype_3b_plus.ipynb, llm-test/README.md	Done, with results in llm-test/results_light/ and llm-test/results_3b_plus/
2.1	System description: data input, preprocessing, safety guardrails	README_AI.md	Done
2.2	Local Hugging Face model in a Django view (Tejas)	data_quality/ai/, data_quality/ai_views.py	Done; tested with the real model on a local GPU
2.3	Second AI feature through an external API: Groq (Tejas; cache and sample filter first added by Connor)	data_quality/ai/groq_client.py	Done; tested with fake API responses only. A real Groq call on the current code has not been verified.
3	Production reflection: size vs quality, cost and scaling, hybrid, own server	docs/notes/notes.txt, section "A6 - PART 3"	Done

Setup

These steps are for Windows PowerShell. You need Python 3 and Git.

1. Clone the repo.

git clone https://github.com/S1attaro/15_DataPact.git
cd 15_DataPact

2. Create and activate a virtual environment.

python -m venv venv
venv\Scripts\Activate.ps1

If PowerShell blocks the script, run Set-ExecutionPolicy -Scope Process Bypass and try again.

3. Install the packages.

pip install -r requirements.txt

This includes torch, transformers and accelerate for the A6 local model. PyPI's torch is CPU-only on Windows; requirements.txt has the one-line command for the NVIDIA (CUDA) build. A server that cannot fit these (such as PythonAnywhere) uses requirements-deploy.txt instead, and the local model stays off there. See README_AI.md.

4. Make your own .env file.

Copy-Item .env.example .env

Fill in your own values. Do not commit .env. Git ignores it. Settings read .env on startup; without a SECRET_KEY every manage.py command stops with a message telling you to create the file.

5. The repo already includes db.sqlite3 with demo data, so you can skip this step. To reset the database, run:

python manage.py migrate
python manage.py seed_demo

seed_demo clears the app tables before it loads the demo data.

Environment variables

.env.example lists these. The values in it are placeholders.

Name	Purpose
SECRET_KEY	Django secret key. Required. Make your own for .env. Production mode refuses keys that start with django-insecure-.
ALLOWED_HOSTS	Comma-separated hosts. Use localhost,127.0.0.1 locally.
API_KEY	Unused placeholder. The A4 external API (Open Library) is keyless, so nothing reads this. Kept so .env.example still matches base.py.
GOOGLE_CLIENT_ID / GOOGLE_CLIENT_SECRET	Optional. The OAuth client from the Google Cloud Console, used for Google sign-in (see "Google sign-in" below). Leave them blank and the site still runs; the Google button then shows a "not configured" note.
ACCOUNT_DEFAULT_HTTP_PROTOCOL	Optional, production only. Defaults to https so Google gets an https:// callback behind PythonAnywhere's proxy. Set it to http only to try production mode over plain http on your own machine.
DATABASE_NAME	Optional, development only. Set it to db.local.sqlite3 to work against a throwaway database (ignored by Git) instead of the demo db.sqlite3. Run python manage.py migrate once after setting it.
GROQ_API_KEY	Optional (A6 Part 2.3). Key for the "Explain with Groq" button (free at console.groq.com). Leave blank and the explain page shows Groq as not set up, with no button; nothing is sent. Groq gets the dataset, column, rule (as written in the contract, including its allowed values) and row counts. Values from the file are sent only when they are known status codes such as AUDIT or PENDING (at most 3, from SAFE_CODES in data_quality/ai/groq_client.py), or blanks for a NOT_NULL rule; numbers, names, emails, IDs and other text from the file are never sent. Answers are cached for an hour.
GROQ_MODEL	Optional (A6). Groq model to call. Defaults to openai/gpt-oss-20b.
AI_LOCAL_ENABLED	Optional (A6). Turns the "Explain with AI" local model on or off. On by default in development, off by default in production.
AI_LOCAL_MODEL_ID	Optional (A6). Hugging Face model to load. Defaults to HuggingFaceTB/SmolLM2-1.7B-Instruct, the model the benchmark notebook chose.
AI_LOCAL_DEVICE	Optional (A6). auto (GPU if there is one), cuda or cpu. The model needs about 4 GB of free GPU memory, or about 7 GB of free RAM on CPU.

To make a new secret key:

python -c "from django.core.management.utils import get_random_secret_key as g; print(g())"

Running

python manage.py runserver

Then open http://127.0.0.1:8000/datasets/.

manage.py uses the development settings (DEBUG = True). To run with the production settings (DEBUG = False, ALLOWED_HOSTS required):

python manage.py runserver --settings=datapact_project.settings.production

wsgi.py and asgi.py default to the production settings. DJANGO_SETTINGS_MODULE overrides either default.

Production mode serves static files through WhiteNoise, so collect them first:

python manage.py collectstatic --noinput --settings=datapact_project.settings.production

Skip that step and the pages still render, just without the content hash in the stylesheet URL. Development mode needs no collectstatic.

Pages

URL	View	Kind
/	home	Function-based, render()
/accounts/google/login/	django-allauth	Built in (starts the Google flow on POST)
/accounts/google/login/callback/	django-allauth	Built in (where Google sends the user back)
/accounts/logout/	django-allauth	Built in (POST signs the user out)
/datasets/manual/	dataset_manual	Function-based, HttpResponse
/datasets/render/	dataset_render	Function-based, render()
/datasets/overview/	DatasetOverviewView	Class-based, View
/datasets/	DatasetListView	Class-based, ListView
/datasets/<id>/	DatasetDetailView	Class-based, DetailView
/contracts/search/	ContractSearchView	Class-based, View (GET search + POST lookup)
/datasets/manage/	DatasetManageView	Class-based, View (GET filter + POST create)
/quality/	quality_history	Function-based, render()
/quality/run-outcomes.png	run_outcomes_chart	Function-based, PNG via HttpResponse
/charts/	charts_page	Function-based, render() (Vega-Lite charts)
/api/run-outcomes/	api_run_outcomes	Function-based, JsonResponse (chart API)
/api/run-volume/	api_run_volume	Function-based, JsonResponse (chart API)
/vega-lite/chart<n>.json	vega_chart_spec	Function-based, JsonResponse (chart spec)
/api/datasets/	dataset_api	Function-based, JsonResponse (dataset registry)
/api/lookup/	dataset_lookup_api	Function-based, JsonResponse (external API)
/violations/<id>/explain/	violation_explain	Function-based, render() (A6 AI; GET shows, POST explains with the local model or the Groq API)
/reports/	reports_view	Function-based, render()
/reports/export/validation-runs.csv	export_validation_runs_csv	Function-based, CSV via HttpResponse
/reports/export/validation-runs.json	export_validation_runs_json	Function-based, JsonResponse
/admin/	Django admin	Built in

API

/api/datasets/ returns the dataset registry as JSON (JsonResponse, Content-Type: application/json) instead of rendered HTML. Compare with /datasets/manual/, which returns the same underlying data as HttpResponse (Content-Type: text/html).

Optional filters via query parameters:

• ?owner=<username> — datasets owned by that user
• ?source_team=<team> — datasets from that source team (case-insensitive)

Example:

GET /api/datasets/?owner=cslat

External API Integration (A4 Part 2)

/api/lookup/?q= combines our internal Dataset registry with Open Library's public search API. Nothing from Open Library is stored; the external response is fetched, processed in memory, and returned alongside our own data.

Uses requests.get() with params=..., timeout=5, and .raise_for_status(). Handles timeouts and request failures with JSON error responses. Requires ?q=; returns a 400 error if missing.

Example: GET /api/lookup/?q=enrollment

Returns internal Dataset matches (name, description, or source_team containing the term) and up to 5 Open Library book results (title, author, first_publish_year) for the same term.

Alternative API-Use Demos (A5)

Four ways to consume /api/run-outcomes/ beyond a browser page rendered by Django. This is the public, unauthenticated endpoint for A5 (/api/datasets/ is now access-controlled, so the demos point here instead).

1. curl from the terminal: curl.exe http://127.0.0.1:8000/api/run-outcomes/
2. Python script using requests: docs/demos/demo2_requests.py
3. Pandas DataFrame load: docs/demos/demo3_pandas.py (pd.read_json + pass-rate calculation)
4. Browser fetch(): docs/demos/demo4_fetch.html (see below)

Response shape is a flat array: [{"outcome": "Passed", "runs": 4}, ...]

Run demo 2 or 3 with python docs/demos/demo2_requests.py (or demo3_pandas.py) while the dev server is running. pandas is only needed for demo 3, not added to requirements.txt since it is not a project dependency.

Screenshots: docs/screenshots/a5/demo1_curl.png, demo2_requests.png, demo3_pandas.png

Demo 4: browser fetch() (docs/demos/demo4_fetch.html)

A standalone HTML page that calls /api/run-outcomes/ with the browser's native fetch() API and renders the result as a table. Served through Django itself at /demos/fetch/ (added demo4_fetch_page in views.py and a route in urls.py) rather than opened as a local file:// page, since the browser blocks a fetch() from file:// to http:// as a cross-origin request and Django's dev server sends no CORS headers to allow it.

To run it: start python manage.py runserver, then visit http://127.0.0.1:8000/demos/fetch/. Open the browser dev console (F12) to see the logged fetch call and response.

Screenshot: docs/screenshots/a5/demo4_fetch.png

Templates

Every page extends one base template, so the navigation, stylesheet, page header and footer are written once.

data_quality/templates/data_quality/
  base.html                    site shell: <head>, CSS, top nav, page header, footer
  dataset_list.html            wireframe screen 1, the dataset registry
  dataset_detail.html          wireframe screen 2, one dataset, its contracts and (UX Stage 4) its violations
  dataset_overview.html        contract coverage per dataset
  includes/
    _empty_state.html          the shared "nothing here yet" block
    _status_badge.html         the shared DRAFT / ACTIVE / RETIRED pill

Blocks a page can fill in: title, extra_head, breadcrumbs, page_title, page_subtitle, page_actions, content.

UX Stage 4 (Tejas) gives the dataset page a Violations panel (#violations). It lists every violation recorded against any of the dataset's contract versions, newest run first. Each row shows:

• the run's file, date and contract version;
• the column and rule;
• the failed rows and the recorded resolution;
• an Explain link to the AI explanation page.

It leaves out sample values and messages. When the list is empty, it says either that no runs have been recorded yet, or how many runs found nothing, including any that ended in an error.

UX Stage 5 (Tejas) made the site shell in base.html easier to use with a keyboard, on a phone and at high zoom:

• Skip link: "Skip to main content" is the first Tab stop on every page and jumps to <main id="main">.
• Navigation: the nav is labelled ("Main") and marks the current page with aria-current="page" as well as the is-active colour.
• Upload: the item that is not built yet shows a "Not yet" tag instead of a hover tooltip.
• Menu button: signed in, below 1000px (62.5em) a Menu button folds the links and Log out away. A small script after the header adds it; without the script the bar wraps as before. Escape closes the menu and returns focus to the button.
• Wrapping: the top bar wraps instead of pushing the page sideways, and its breakpoints are in em, so they follow the browser's text size as well as its zoom.
• Contrast and focus: --ink-faint text and form-field borders now meet WCAG AA contrast, and every keyboard focus gets the 2px ring.

data_quality/tests_navigation.py covers these.

UX Stage 6, first increment (Tejas), tidied the shared components in datapact.css rather than any one page:

• Type steps: panel titles are sentence-case 1rem headings, a step above the small-capital column headers and fact labels, which now share one size (--text-label) and colour. Form labels are sentence case.
• Tables: column-header styling applies to thead only, so row headers (Reports, Quality History) read like the data they label. Number columns align their headers with their numbers.
• Links and buttons: every link has the site blue and a hover (--accent-strong), every button has a hover, and a plain button's edge uses --line-strong.
• Badges: red is kept for failed runs; a retired contract version uses the plain grey badge.

data_quality/tests_visual_foundation.py covers these.

UX Stage 6, second increment (Tejas), gave each kind of status its own colours and tightened the hierarchy:

• Status colours: a contract's lifecycle is no longer coloured like data health. Active is the accent's tint with a solid ring, Draft is white with a dashed edge, Retired is plain grey. Run outcomes (Reports, Quality History) are now badges: Passed green, Failed red, Error amber. Violation resolutions are badges too: Open red, Data issue and Contract updated plain grey. Every badge still names its state in words. Overview shows "None active" as plain text instead of a Draft badge.
• Hierarchy: panel titles are a step above the headings inside them, and the counts on Home ("At a glance") and Reports ("Totals") are set larger than their labels (.factlist--stats).
• Wording: labels, column headers and buttons are sentence case ("Source team", "+ Register dataset"); page names such as "Quality History" stay as they are.
• Busy buttons and press feedback: while the explain page waits for a model, its buttons fade and keep their resting colours under the pointer. Buttons give a small press (scale(0.97), 120ms), which prefers-reduced-motion turns off.

data_quality/tests_status_hierarchy.py covers these.

UX Stage 6, third increment (Tejas), finished the phone and touch details:

• Hover on touch screens: button hover fills and the table row highlight sit behind @media (hover: hover) and (pointer: fine), so a tapped button no longer keeps its hover look on a phone. The press, the busy fade and the keyboard focus ring do not depend on hover.
• Explain page: the recorded resolution uses the shared status badge. A value the app does not know shows its own text in the plain grey badge, and an empty one says "Not recorded".
• Quality History chart: the image keeps its title, axis labels and legend (A3 Section 4), with every word in the site's ink colours at 11pt or more and a visible baseline. The panel around it is headed "As a chart", so the page no longer repeats the chart's title.
• Scroll cue: a panel whose table is wider than the screen shows a soft shadow on the edge that has more to see. It is drawn with backgrounds only, so it appears only when the table really overflows and never covers a tap.

data_quality/tests_final_polish.py covers these.

dataset_list.html is written against a single context variable, datasets, so it does not depend on which kind of view rendered it. Any view can reuse it:

return render(request, "data_quality/dataset_list.html", {
    "datasets": Dataset.objects.select_related("owner"),
    "view_label": "Function-based view - render()",
})

view_label is optional. It fills the small caption under the page title that names the view kind, and falls back to naming the ListView when it is absent.

Each list has an empty state written as {% for %} ... {% empty %}, not as a separate {% if %} check. data_quality/tests.py asserts on the empty-state wording, so reword one and fix its test in the same commit.

Until A3 the CSS was inline in base.html, because runserver stops serving static files once DEBUG = False. It now lives in static/ and WhiteNoise serves it in production; see "Static files and UI" below.

Static files and UI

The stylesheet and the logo are real static assets, served through Django's staticfiles app in development and WhiteNoise in production.

data_quality/static/data_quality/
  css/datapact.css              the whole site stylesheet
  img/datapact-logo.svg         wordmark logo, also the favicon

They live at app level, namespaced under data_quality/, for the same reason the templates are: the app owns them, so it keeps them, and nothing in settings has to list a directory. base.html loads them with {% load static %} and {% static %} rather than a hard-coded /static/... path.

The CSS was inline in base.html until A3. It moved out for Section 3, which meant fixing the reason it was inline: runserver serves no static files once DEBUG = False. WhiteNoise now does, so the site looks the same in both modes.

Cache busting. In production, collectstatic renames each file to include a hash of its contents — datapact.css becomes datapact.e5ab6567e31e.css — and {% static %} emits that name. The response carries Cache-Control: max-age=315360000, public, immutable, so a browser can cache it for ten years; editing the CSS changes the hash, so the next visitor requests a different URL and sees the change immediately. There is no way to serve a stale stylesheet and no need to ever clear a cache by hand.

A full request/response transcript of both modes is in docs/a3_s3_static_evidence.txt.

Tests

python manage.py test data_quality

294 tests on main (19 Oct 2026): the A2 class-based views, the templates, the render() view, the settings split, ContractSearchView, DatasetManageView, the static-file and cache-busting checks, the home page, navigation and Quality History chart, the Reports page and CSV/JSON exports, the A4 chart APIs, Vega-Lite specification endpoints and charts page, Google sign-in, the A5 access control, and the A6 explain page (local model and Groq). The A6 tests use a fake local model and fake Groq responses, so they need no GPU, no key and no network. Run them before you open a pull request.

Google sign-in (Hriday)

"Continue with Google" is handled by django-allauth. It sits next to the username/password login, not instead of it, and /admin/ still takes a username and password.

How it works: the button is a POST form (allauth will not start the flow from a plain link, and the form carries a CSRF token). It sends the browser to Google, Google sends it back to /accounts/google/login/callback/, and allauth creates the User from the Google profile on first sign-in and logs them in. Only the profile and email scopes are requested and no Google token is stored. After signing in or out the user lands on the home page. A Google sign-in is not matched to an existing user by email address (SOCIALACCOUNT_EMAIL_AUTHENTICATION = False in settings/base.py, changed in the A5.1 integration): the seeded demo users are staff accounts (two of them superusers) with real @illinois.edu addresses, so matching by email would hand those accounts to anyone who controls the address. A new Google user gets their own ordinary account; one whose email collides with an existing user is sent to allauth's signup page instead. The reasoning is in the comment above that setting.

The button lives in one reusable partial, data_quality/templates/data_quality/includes/_google_button.html. Put it on a page with:

{% include "data_quality/includes/_google_button.html" %}

Include it without only, because it needs the request for the CSRF token. With no credentials in .env it prints "Google sign-in is not configured." instead of a button.

Get the credentials (once per team, in Google Cloud Console):

1. Create a project, then set up the consent screen (Google Auth Platform): audience External.
2. While the app is in Testing mode only the addresses under Audience, Test users can sign in. Add everyone who needs to, or publish the app (the profile and email scopes need no Google review) so anyone can.
3. Clients, Create client, type Web application. Add these Authorized redirect URIs:
   • http://127.0.0.1:8000/accounts/google/login/callback/
   • http://localhost:8000/accounts/google/login/callback/
   • https://tjaggi.pythonanywhere.com/accounts/google/login/callback/ for the deployed site
4. Put the Client ID and secret in .env as GOOGLE_CLIENT_ID and GOOGLE_CLIENT_SECRET. Never commit them. On PythonAnywhere, set them in the server's own .env.

Google matches redirect URIs exactly, including the trailing slash, and localhost is not 127.0.0.1.

URL order. In datapact_project/urls.py the project's own routes (data_quality.urls) come before allauth.urls. Django uses the first pattern that matches, so a project view on the same path as an allauth one, such as a custom accounts/login/, is not shadowed, while allauth still serves the Google login, its callback and the rest. A test guards the order.

On PythonAnywhere: pull the code, pip install -r requirements-deploy.txt (the same pins as requirements.txt minus torch, transformers and accelerate, which the host cannot fit; the local AI model stays off there), python manage.py migrate, python manage.py collectstatic, make sure the server's .env has both Google values, then reload the web app. Production settings send an https:// callback (ACCOUNT_DEFAULT_HTTP_PROTOCOL), because PythonAnywhere hands Django a plain-http request and Google would otherwise reject the callback with redirect_uri_mismatch.

After pulling this change locally, install the new package and create the allauth tables:

pip install -r requirements.txt
python manage.py migrate

migrate also changes the committed db.sqlite3. Restore it with git checkout -- db.sqlite3 unless the team decides to commit the migrated file for deployment.

A3, Section 2 & Section 5 (Hriday)

Section 2 - ORM Queries & Data Presentation (/contracts/search/)

Rubric item	Evidence
Search (GET + POST) + filter works	GET search by dataset name/status, and a POST lookup on Violation
Relationship-spanning query	dataset__name__icontains (Contract → Dataset), rule__contract__dataset__name__icontains (Violation, two hops)
Aggregations (total + grouping)	total contract count, plus values("status").annotate(total=Count("id"))
Template clarity & {% empty %}	empty-state message when a search has no matches

UX Stage 3 (Tejas) made the violation lookup easier to reach and to read, without changing what it finds:

• The lookup is still a POST. When there are open violations, a signed-in user sees a "Look up open violations" link under Home's count. It opens ?resolution=OPEN#violation-lookup, which only pre-selects Open in the form and says to press the button.
• The form posts to #violation-results, so the page opens on the answer.
• The results say how many violations matched and for what.
• Each row links to its dataset and names its validation run (file and date).
• Each Explain link has a name of its own for screen readers.
• When nothing has been recorded yet, the results say so.
• The explain page also names the run.

GET search filtered to one dataset (?q=Enrollment):

Same page after the POST violation lookup (aggregation panel + violation results):

Empty state when a search has no matches:

Section 5 - Forms & User Input (/datasets/manage/)

Rubric item	Evidence
GET form works	filters the dataset list by source team via ?team=...
POST form works	registers a new Dataset, success message shown, Post/Redirect/Get
CSRF correctly implemented	POST without a token is rejected (403); with the real token it succeeds (302)
CBV adapted for input handling	one DatasetManageView(View) implements both get() and post()

GET filter:

POST create (success message + new row):

CSRF enforcement:

A3, Section 1 & Section 4 (Tejas)

Section 1 - URL Linking & Navigation

The site now has a home page at /. It shows three counts (datasets, validation runs, open violations) and links to every real section of the app. The top navigation carries three working links - Datasets, Overview and Quality History - and every link in the project is built with {% url %} or get_absolute_url(), so no template hard-codes a path.

Dataset detail pages are keyed by primary key (/datasets/<int:pk>/), and the registry, overview, contract search and dataset manage pages all link to a dataset through {{ dataset.get_absolute_url }} rather than rebuilding that URL by hand. Dataset.get_absolute_url() is also what DatasetManageView redirects to after a successful POST, so "where does a dataset live?" is answered in exactly one place.

Section 4 - Data Visualization (/quality/)

Quality History charts how DataPact's validation runs have turned out. The numbers come from the ORM:

ValidationRun.objects.values("status").annotate(total=Count("id"))

One ValidationRun is one file checked against one contract version, so the chart counts runs, not rows. A status with no runs is kept at zero and the bar order comes from the model rather than the database, so the axis stays stable.

The chart itself is drawn by Matplotlib in data_quality/charts.py on the Agg backend, written to a BytesIO buffer and served straight from memory by a Django view at /quality/run-outcomes.png with content_type="image/png" - nothing is written to disk. Drawing goes through Figure/FigureCanvasAgg rather than pyplot, because pyplot's figure registry is process-wide and Django answers requests on threads. With an empty database the endpoint still returns a valid PNG that says so on its face. The page embeds the image through a reversed URL, gives it alt text generated from the same numbers, and repeats those numbers in a table so they are available without the image.

Screenshots: docs/screenshots/a3/s1_home.png, s1_navigation.png, s1_detail_via_link.png, s4_quality_history.png, s4_chart_endpoint.png.

A4, Part 3: Reports & Exports (Hriday)

Reports page at /reports/, built on ValidationRun.

Rubric item	Evidence
CSV export	export_validation_runs_csv - text/csv, header row, timestamped filename
JSON export + metadata	export_validation_runs_json - generated_at + record_count, json_dumps_params={"indent": 2}
Two grouped summaries	runs by outcome (reuses charts.run_outcome_counts()), runs by dataset (contract__dataset__name, annotate(total=Count("id")))
Totals line	dataset count, run count, failed-run count
Download buttons on the page	page_actions block links to both export URLs

The two export views share one queryset helper (_validation_runs_queryset), so the CSV and JSON can't drift out of sync with each other.

Reports page:

CSV export (real response headers + body):

JSON export (real response headers + body):

A4, Part 1: Internal Chart API & Vega-Lite Charts (Tejas)

/charts/ carries two Vega-Lite charts. Neither one has data in it: each specification points at an internal JSON API with data: {"url": ...}, so the charts show whatever is in the database at the moment the page is opened.

The two internal chart APIs

Both are plain GET endpoints that return clean JSON straight from the models, with no envelope, which is the shape Vega-Lite reads without any extra parsing.

Endpoint	Shape	Feeds
/api/run-outcomes/	[{"outcome": "Passed", "runs": 4}, ...]	the bar chart
/api/run-volume/	one object per validation run, with rows_checked, rows_failed, outcome	the scatter chart

These are separate from /api/datasets/, which lists dataset records. A bar chart needs an aggregate and a scatter needs one row per run, so neither chart could read that endpoint as it stands, and rewriting it would have changed an endpoint another part of the project depends on.

The two charts

• Bar - Validation Run Outcomes. The aggregated summary: how many runs ended passed, failed, or in error. A status with no runs stays on the chart at zero rather than disappearing.
• Scatter - Rows Checked vs Rows Failed. One point per validation run, coloured by outcome. It answers whether big files are the ones that break: on the current data they are not, which is the argument for checking every file rather than only the large ones.

Chart specifications

Each specification is published at its own endpoint and can be opened directly or pasted into the Vega-Lite editor:

/vega-lite/chart1.json     bar chart
/vega-lite/chart2.json     scatter chart

The page renders from those same URLs, so the published specification and the chart on screen cannot drift apart. The source for both lives in data_quality/vega.py.

The Vega, Vega-Lite and vega-embed libraries load from a CDN, so this page adds nothing to the static pipeline and needs no collectstatic step of its own.

Screenshots: docs/screenshots/a4/a4_p1_charts_page.png, a4_p1_chart1_spec.png, a4_p1_chart2_spec.png, a4_p1_api_run_outcomes.png, a4_p1_api_run_volume.png.

Project layout

15_DataPact/
  README.md
  README_AI.md          A6 AI features: workflow, guardrails, what was verified
  manage.py
  requirements.txt      everything, including torch/transformers for the local model
  requirements-deploy.txt  the same pins without torch/transformers/accelerate
  .env.example          placeholder values, safe to commit
  .gitignore
  db.sqlite3            demo database
  datapact_project/     root URLs, wsgi, asgi
    settings/           base.py (shared), development.py, production.py
  data_quality/         models, views, urls, charts, vega, forms, tests
    ai/                 A6: prompts, local model, Groq client
    static/
      data_quality/     css/datapact.css, img/datapact-logo.svg
    templates/
      data_quality/     base.html, page templates, includes/ partials
  staticfiles/          collectstatic output (generated, git-ignored)
  llm-test/             A6 Part 1 notebooks and their result CSVs/charts
  docs/
    notes/              notes.txt, with weekly updates from each teammate
    wireframes/v1/      wireframes PDF
    branching_strategy/ how we use branches and pull requests
    screenshots/        A2, A3, A4 and A5 browser screenshots

Git workflow

• No one commits directly to main.
• Each task gets its own branch, named name/task, like connor/env-example.
• Every change reaches main through a pull request, merged with a merge commit.
• Pull main before you start a new branch.

More detail is in docs/branching_strategy/branching_strategy.txt.