Skip to content

Library · Data Practice OS

Data quality and observability

Reference guide · Published · Source reviewed

The answer: test the rules you know in the pipeline, monitor the things you don't know in production, and send every alert to the team that owns the data. Quality matters more once AI is a consumer. A dashboard with a bad number gets questioned. An agent with a bad number repeats it confidently. That's why quality evidence is part of What AI-ready means.

The pyramid

Level What Where Fails the build?
Schema Columns and types match the contract dbt model contracts Yes
Integrity Keys unique and not null, relationships hold dbt generic tests Yes
Business rules "A cancelled registration has no later renewal", "licence expiry after issue" dbt singular tests named after the rule Yes (error) or warn, as the contract says
Freshness Sources and products arrive on time dbt source freshness, platform monitors Alert
Volume and distribution Row counts, null rates, value distributions within normal bounds Platform-native monitors Alert
Reconciliation New equals old during migration Reconciliation jobs (Reconciliation, parallel run and cutover) Blocks cutover

Test what's cheap and certain in CI and build. Monitor what's statistical in production. Don't try to express "volume looks odd" as a hard test. It'll either never fire or always fire.

Platform-native monitoring

Snowflake: data metric functions (system DMFs such as NULL_COUNT, DUPLICATE_COUNT, FRESHNESS, ROW_COUNT, plus custom DMFs) are scheduled on tables. Results land in an event table, and alerts are built on them. Quality results surface in Snowflake Horizon Catalog.

ALTER TABLE prod_registrations.gold.fct_registration_event
  SET DATA_METRIC_SCHEDULE = 'USING CRON 0 7 * * * Australia/Sydney';

ALTER TABLE prod_registrations.gold.fct_registration_event
  ADD DATA METRIC FUNCTION SNOWFLAKE.CORE.NULL_COUNT ON (customer_key);

Databricks: pipeline expectations in Lakeflow (Connect, Declarative Pipelines, Jobs) Declarative Pipelines (warn, drop or fail per rule), plus Unity Catalog's data quality monitoring for freshness, completeness and anomaly detection on tables. Data profiling (formerly Lakehouse Monitoring) adds profile and drift metrics.

CREATE OR REFRESH STREAMING TABLE silver.traffic_detector_counts (
  CONSTRAINT valid_count  EXPECT (vehicle_count >= 0)      ON VIOLATION DROP ROW,
  CONSTRAINT has_detector EXPECT (detector_id IS NOT NULL) ON VIOLATION FAIL UPDATE
) AS SELECT * FROM STREAM(bronze.traffic_detector_raw);

The names and scope of these features change often. Check Snowflake data quality and Databricks data quality monitoring (names checked 6 Nov 2026) before you design around them.

dbt tests that name the rule

-- tests/registrations/reg_stat_017_no_renewal_after_cancellation.sql
-- REG-STAT-017: A registration cancelled on date D has no renewal effective after D.
select r.registration_id
from {{ ref('fct_registration_event') }} c
join {{ ref('fct_registration_event') }} r
  on r.registration_id = c.registration_id
where c.event_type = 'CANCELLATION'
  and r.event_type = 'RENEWAL'
  and r.effective_date > c.effective_date

Rule IDs link back to the extracted rule cards from the legacy system (Extracting business rules), so a failing test explains itself. Also put the rule id in the test's meta (verifies, test_kind) so a failing test's blast radius can be walked back to the legacy program (proposed, Tracing rules from legacy code to modern code).

Alert routing

  1. Every alert has an owner. It routes to the product's support channel from its contract, not to a central inbox nobody watches.
  2. Severity follows the contract. A breach of a contracted SLA pages the owner. A warning becomes a ticket.
  3. Tell consumers, automatically. When a gold product fails quality, mark it stale in the catalog, and have semantic layer and agent tools refuse or caveat answers from it. An AI that says "the registrations data for today is delayed" is better than one that answers wrongly.
  4. Use lineage for impact. Platform lineage plus OpenLineage across the boundary tells you which dashboards, shares and agents sit downstream of a failed table.

When not to over-invest

Don't buy a third-party observability tool on day one. Both platforms' native monitoring plus dbt tests covers most needs at the organisation's starting scale. Look again when you have many products across both platforms and need one cross-platform view.

Pitfalls

  • Alert fatigue. Hundreds of warn-level tests that always fail. Delete them or fix them. A test that's always red is noise.
  • Tests only on gold. Catching a bad CDC feed at gold is late. Put freshness and volume checks on bronze sources too.
  • Quality status invisible to AI. If agents can't see that a table is stale, they'll use it anyway. Expose quality status as metadata the tools read (Agents and MCP).

Lessons from platform options papers

Source: a platform team's draft options papers and platform docs (Oct-Nov 2026). All draft or proposed. None is an agreed decision; treat them as evidence, not standards.

  • Give each data product a tier (Data product tier, 1-3 from business impact and number of users). The tier sets monitoring, freshness thresholds and incident severity.
  • Have a rule for each kind of source schema change. A deleted or renamed table or column needs action. A new table or column needs none. A changed type or constraint needs a check. Before loading, run INFER_SCHEMA on the batch and compare the columns with the previous batch. Do not turn on schema evolution for raw tables, because it adds columns with no review. Note that COPY INTO ... SELECT ignores ERROR_ON_COLUMN_COUNT_MISMATCH.
  • Unconfirmed checks count as not passed. A pending check (row counts against the source, file hashes) is reported as pending, never as green.
  • Remove tables under control. A table is a removal candidate after 60 days with no queries, confirmed by a stakeholder. Give each removal a risk score (probability x impact) and two code-owner reviews, and run a quarterly orphan audit. Removing a dbt model does not drop its table. ACCESS_HISTORY (Enterprise Edition) shows who read what.
  • Alert on failed tasks with Snowflake task notifications, not by someone noticing the next morning.

Source links: Data Practice reading list.