Self-service, hands-on labs that complement an upcoming dbt on Databricks course. The labs let you work through the course material at your own pace in your own workspace, taking you from a clean dbt project to production-grade models on Unity Catalog with optional deployment via Declarative Automation Bundles (DABs) and Lakeflow Jobs.
- Course companion. Each lab maps to a section of the dbt-on-Databricks course and is designed to be opened, read, and run end-to-end without an instructor.
- Self-service. Clone (or fork) the repo, fill in your own workspace values locally, and run. No shared environment, no shared state.
- Production-shaped, not toy. Models follow the staging + marts layering, use Unity Catalog with
dbt-databricks, and are deployable as Lakeflow Jobs via DABs. - Public-safe. Everything tracked here is portable across any Databricks workspace; workspace-specific config stays local (see
local_deployment/).
The first lab is the canonical Jaffle Shop dataset, modelled end-to-end with dbt-databricks against a SQL Warehouse. It is intentionally a first step: a small, well-known schema that lets you focus on getting the local dev loop, profiles, seeds, sources, staging/marts layering, tests, packages, and DAB deployment working before tackling harder modelling problems.
Future labs will build on this foundation with several deeper and more diverse examples of data modelling in Databricks, planned to cover patterns such as:
- Slowly Changing Dimensions (Type 1 / 2) using dbt snapshots
- Incremental models with merge / append / insert-overwrite strategies on Delta
- Streaming sources and CDC ingestion patterns (DLT / Lakeflow + dbt)
- Star and dimensional modelling beyond the Jaffle toy schema
- Data Vault style raw / business vault layers
- Wide event / semi-structured (JSON, variants) modelling
- Performance tuning: Z-ORDER, photon (partitioning, liquid clustering and materialization choice are covered in
module_03/) - Unity Catalog governance: row/column masking, tags, lineage-aware models
- Multi-environment promotion (dev / staging / prod) via DAB targets and CI
Standalone labs land as their own top-level project folder (e.g. jaffle_shop/). Multi-lab course
modules share one dbt project with a lab folder per exercise (e.g. module_03/, module_04/).
| Path | Purpose |
|---|---|
jaffle_shop/ |
Lab 01 — the dbt project: staging + marts models, seeds, macros, tests, packages. |
module_03/ |
Module 3 — Modeling with dbt on Databricks. Self-contained dbt project with shared silver seeds; Lab 3.5 (partitioning and liquid clustering) and Lab 3.6 (incremental insert-overwrite and MERGE). See the silver boilerplate README. |
module_04/ |
Module 4 — Testing and Data Quality with dbt on Databricks. Self-contained lab boilerplate: seeded fixtures with a catalog of known defects (deterministic pass/fail cases) and starter models with no generic tests yet; Lab 4.3 (singular SQL test). See the Module 4 README. |
local_deployment/ |
Workspace-specific DAB and deploy config. Only the README is tracked; everything else stays local. |
.cursor/rules/ |
Cursor agent guardrails: public-repo compliance, branch conventions, coding standards. |
- A Databricks workspace with Unity Catalog enabled
- A SQL Warehouse (Serverless or Classic)
- A catalog and schema you own (or the right to create one)
- Databricks CLI v0.218+ configured with a profile
- Python 3.10+
dbt-databricks>=1.11.7,<2.0.0
cd jaffle_shop
# 1. Copy the templates -- both copies are gitignored.
cp .env.template .env
cp profiles.yml.example profiles.yml
# 2. Open .env in your editor and fill in REAL values for every variable:
# DATABRICKS_HOST -> https://<your-workspace>.cloud.databricks.com
# DATABRICKS_HTTP_PATH -> /sql/1.0/warehouses/<your-warehouse-id>
# DATABRICKS_CATALOG -> <your-catalog>
# DATABRICKS_SCHEMA -> dbt_<first-initial><last-name> (best practices are to build models in development in a personal schema)
# DATABRICKS_TOKEN -> <your-pat> (or use OAuth, see below)
# profiles.yml reads these via env_var() at runtime, so you only fill in .env.
# 3. Install dbt and run.
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
set -a && source .env && set +a # export the env vars to dbt
dbt deps
dbt debug # confirms connection before you run anything
dbt seed --full-refresh --vars '{"load_source_data": true}'
dbt run
dbt testprofiles.yml and .env are gitignored. Never commit them.
The DAB bundle (databricks.yml at the repo root) is workspace-specific and therefore gitignored. See local_deployment/README.md for a placeholder bundle you can copy and fill in for your environment.
databricks bundle … needs a CLI profile that points at the same workspace you're deploying to. The recommended approach is a service principal with an OAuth client_id / client_secret -- no PATs, no expiring tokens. Add a profile to ~/.databrickscfg:
[dbt-ws-profile]
host = https://<your-workspace>.cloud.databricks.com
client_id = <service-principal-client-id>
client_secret = <service-principal-oauth-secret>If you'd rather use a PAT for a quick test, replace the OAuth pair with token = <your-pat>.
The root databricks.yml is thin: it include:s one job-resource file per deployable unit from
local_deployment/resources/ (all gitignored). Lab 01 is a single job; Module 3 is split into a
shared seed job (run once per workspace) and per-lab build jobs, because its silver seeds are
workspace-wide while gold/incremental builds are per-developer. See
local_deployment/README.md for the full layout.
From the repo root, pass --profile to every bundle command (or set DATABRICKS_CONFIG_PROFILE=dbt-ws-profile once for the shell):
databricks bundle validate --profile=dbt-ws-profile
databricks bundle deploy --profile=dbt-ws-profile
# Lab 01
databricks bundle run jaffle_shop_dbt_job --profile=dbt-ws-profile
# Module 3: seed the shared silver layer once, then build a lab
databricks bundle run module_03_seed_job --profile=dbt-ws-profile
databricks bundle run module_03_gold_job --profile=dbt-ws-profile
databricks bundle run module_03_incremental_job --profile=dbt-ws-profile
# Module 4: unlike Module 3, seeds are per-developer, so seed then build into your own schema
databricks bundle run module_04_seed_job --profile=dbt-ws-profile
databricks bundle run module_04_build_job --profile=dbt-ws-profileDatabricks does not offer official support for this content. For questions or bugs, please open a GitHub issue and the team will help on a best-effort basis.
Please report security issues per SECURITY.md.
See LICENSE.md and NOTICE.md. All third-party packages used by this lab are listed in jaffle_shop/packages.yml and jaffle_shop/requirements.txt; refer to each project's own license.