Skip to content

Library · Data Practice OS

CI/CD and environments

Reference guide · Published · Source reviewed

The answer: everything that matters lives in git and reaches production through a pipeline: models, tests, policies, warehouses, catalogs, grants, jobs. People get read access to prod. Only service principals get write access. This is Data as code, and it's a large part of what makes the platform auditable enough for AI.

Environments

Environment Purpose Data Who deploys
dev Per-developer sandboxes (dev_<user>_<domain>) Sampled or synthetic. Masked if derived from prod The developer, from their branch
ci Ephemeral schema per pull request (ci_pr_<n>) Same as test, with defer to prod for unchanged parents The pipeline
test Integration and UAT Masked production copy (zero-copy clone or shallow clone) The pipeline, on merge to main
prod Production Production The pipeline, on release tag with approval

Snowflake: one account with environment databases is fine to start with. Use separate accounts for prod when isolation or chargeback requires it. Databricks: separate workspaces for dev and prod sharing one Unity Catalog metastore, with catalogs per environment and workspace-catalog binding so dev can't write to prod.

Use real personal information in non-prod only with a documented exception. Clone, then apply masking policies by default (Governance, security and trust).

The pipeline

# .github/workflows/dbt-ci.yml (excerpt)
name: dbt-ci
on: pull_request
permissions: { id-token: write, contents: read }    # OIDC, no stored passwords
jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: pip install dbt-snowflake sqlfluff
      - run: sqlfluff lint models
      - name: Fetch prod manifest for state comparison
        run: ./scripts/get-prod-manifest.sh > prod-state/manifest.json
      - name: Slim CI - build only what changed, defer the rest to prod
        run: |
          dbt build --select state:modified+ --defer --state prod-state \
                    --target ci --vars "{schema_suffix: pr_${{ github.event.number }}}"
      - run: ./scripts/check-contracts.sh          # breaking contract changes need a version bump

Optional step, during migration waves only (draft, Tracing rules from legacy code to modern code). Run dbt parse and a strict trace report over the full manifest, not the slim-CI selection, because meta-only changes do not trigger state:modified (meta). Fail on a rule claimed without a test or an accepted decision. dbt parse needs no warehouse connection. A possible host for the meta checks is dbt_project_evaluator or a small custom script (verify). Not when: outside migration branches, or before the rules have ids.

On merge, the same job runs against test with a full build of the changed graph. Releases to prod are tagged and gated by an approval from the product owner.

Infrastructure as code

What Tool
Snowflake warehouses, databases, roles, grants, tags, policies, integrations Terraform (Snowflake provider)
Databricks workspaces, catalogs, schemas, grants, clusters policies, SQL warehouses Terraform (Databricks provider)
Databricks jobs, pipelines, model serving, apps Databricks Asset Bundles
dbt models, tests, docs dbt, via the pipeline above
Snowflake tasks, Cortex Search services, semantic views SQL migrations in the dbt project, or Terraform where supported (verify coverage)

The Platform team owns the Terraform modules (a domain gets a database or catalog, roles, a warehouse and a cost tag in one module call). Domain teams own their dbt projects and bundles. That split is the Paved road.

module "domain_road_safety" {
  source       = "./modules/snowflake-domain"
  domain       = "road_safety"
  environments = ["dev", "test", "prod"]
  warehouse    = { size = "SMALL", auto_suspend_s = 60 }
  cost_centre  = "CC-4412"
  owners_group = "grp-data-road-safety-owners"
}

Secrets and identity

  • CI authenticates by workload identity federation (GitHub OIDC). Snowflake uses key-pair or OAuth for service users; Databricks uses OAuth for service principals. Verify current federation options for your tenancy.
  • No personal credentials in pipelines. No long-lived tokens in repositories. Run secret scanning on every repository.
  • Each domain's deploy principal can write only to its own domain objects.

Separate compute per workload

Give CI, transformation, BI and AI their own warehouses or clusters, tagged by domain and environment. That's Workload isolation and FinOps in one move. A runaway CI build can't slow a minister's dashboard, and its cost shows up against the right team.

Pitfalls

  • Clicking in prod "just this once". It's drift on the next Terraform plan. Treat the UI as read-only in prod, and alert on manual DDL using access history or audit logs.
  • Full builds in CI. Slow and expensive. Use state:modified+ with --defer.
  • Forgotten ephemeral schemas. Drop ci_pr_* schemas when the PR closes. A nightly sweep catches the rest.
  • Test data with no masking. The quickest way to a privacy incident. Default to masked clones.

Lessons from platform options papers

Source: a platform team's draft options papers and platform docs (Oct-Nov 2026). All draft or proposed. None is an agreed decision; treat them as evidence, not standards.

  • A Zero-copy clone for each change. Following GitLab's data team, CI clones the production and preparation databases for a pull request, builds and tests the changed models and everything between them in the lineage, and drops the clones when the PR closes. A weekly job drops clones older than 14 days. Clone the layer schemas or the environment database, not the raw database, and use COPY GRANTS.
  • Compare the rebuild with the current output. Run MINUS in both directions plus a row count of both tables before cutover. A row count catches a mistyped join.
  • A test plan in every pull request, in the PR template.
  • Infrastructure deploys from CI with a service user only. See Infrastructure as code for the data platform.
  • CODEOWNERS and branch protection on shared macros, UDFs and load operations (Pipeline code - explicit vs metadata-driven).

Source links: Data Practice reading list.