Skip to content

Library · Data Practice OS

Target reference architecture

Architecture pattern · Published · Source reviewed

The architecture is eight layers. The rule that matters most: layers 2, 4 and 5 are portable, so the platform choice in layers 3, 6 and 7 can change without a rewrite.

These layers are the engine, not the purpose. The purpose is the From data to decisions - the decision stack: facts, meaning, knowledge, rules, context, decision and outcome. That page maps each decision layer onto the eight below. Two parts of it sit outside this diagram and still need a design: the workflow where decision authority lives, and the rule repository (Where business rules run after migration).

  Sources        1  Ingest              2  Store            3  Process           4  Model & transform
  ---------     ------------------     ---------------    ----------------     --------------------
  Core systems   CDC, batch, files,     Object storage     Engines:             dbt: bronze -> silver
  SaaS, IoT  ->  streams, APIs      ->  in open table  ->  Snowflake WH,    ->  -> gold data products,
  documents      (Openflow / Lakeflow   formats (Iceberg/  Databricks SQL,      contracts, tests
                 Connect / partners)    Delta) in AU       Spark, Snowpark
                                        region

  5  Semantic            6  Serve                    7  AI                          8  Govern & operate (across all)
  ----------------      ---------------------      ---------------------------    -------------------------------
  Metrics, dimensions,   BI (Power BI), SQL,         LLM functions in pipelines,    Catalog, classification, ABAC,
  synonyms - defined ->  APIs, sharing, apps,   ->   search/RAG, text-to-SQL,       lineage, quality, observability,
  once                   MCP endpoints               agents, ML - via AI gateway    FinOps, IaC, CI/CD

The layers

1. Ingest. Land data as it is, with Change data capture for operational databases and batch for the rest. Use the platform's managed connectors (Snowflake Openflow, Lakeflow (Connect, Declarative Pipelines, Jobs)) before writing custom code. Raw data lands in bronze, unmodified, with load metadata.

2. Store (portable). Tables in an Open table format in organisation-controlled object storage in an Australian region. Apache Iceberg is the interchange format (ADR-002: Apache Iceberg as the interchange table format). Platform-native tables are fine where nothing else needs to read them.

3. Process. The engines. This is where Snowflake and Databricks differ most, and where Workload isolation matters: separate compute for ingest, transform, BI and AI, each with its own cost line. See Choosing a platform - Snowflake, Databricks, or both.

4. Model and transform (portable). ELT in dbt, following the Medallion architecture layering (Medallion layers and modelling). Gold models are data products with contracts. dbt runs on both platforms, so this layer survives a platform change (ADR-003: dbt as the transformation standard).

5. Semantic (portable). Metric and dimension definitions declared once in the Semantic layer and fed to BI and to the AI text-to-SQL tools (ADR-006: Define metrics once in a semantic layer). This layer does the most to make AI answers correct.

6. Serve. Power BI for reporting. Governed SQL endpoints. Zero-copy sharing to other agencies and partners (Sharing data across platforms and agencies). Internal apps. Model Context Protocol endpoints so agents reach data through the same permissions as people.

7. AI. The six patterns in AI-native patterns. Every model call goes through an AI gateway (ADR-005: Governed model access through an AI gateway).

8. Govern and operate (cross-cutting). One Catalog of record per table (ADR-004: One catalog of record per table), tag-driven policy, lineage, quality and observability, cost attribution, and everything in Terraform and CI (CI/CD and environments).

How it maps to each platform

Layer Snowflake Databricks Portable choice
Ingest Openflow, Snowpipe, partner connectors Lakeflow Connect, Auto Loader Fivetran-class tools; landing files
Store Native tables; Iceberg tables (Snowflake- or externally managed) Delta (with UniForm for Iceberg readers); managed Iceberg Iceberg on organisation-owned storage
Process Virtual warehouses, Snowpark, Container Services SQL warehouses, Spark clusters, serverless n/a
Transform Dynamic Tables, dbt (incl. dbt Projects on Snowflake) Lakeflow Declarative Pipelines, dbt dbt
Semantic Semantic views Unity Catalog metric views dbt Semantic Layer
Serve Secure Data Sharing, Streamlit, Native Apps OpenSharing (Delta Sharing), Databricks Apps, Lakebase Power BI, APIs
AI Cortex AI SQL, Search, Analyst, Agents AI Functions, AI Search, Genie, Agent Bricks, Model Serving, Unity Gateway Claude via either, or Bedrock
Govern Horizon Catalog, Open Catalog Unity Catalog OpenLineage, Terraform

Verify specific features against the Snowflake and Databricks docs. The shape is stable; the product names are not.

Principles that shape the design

  1. Open storage, managed engines. Keep data in formats the organisation controls. Pay vendors for compute and convenience.
  2. One copy, many engines. Share through catalogs and open formats, not nightly extracts between platforms.
  3. Governance travels with the data. Classification tags are applied at bronze and inherited downstream.
  4. Everything is code. If it was configured by clicking, it does not exist.
  5. AI reads the same governed interfaces as people. No side doors, no service accounts with god-mode.
  6. Decisions leave a record. Anything that supports a decision writes a Decision record back into layer 4, so outcomes can be linked and evaluated (Decision records and provenance).

Not when

  • Do not build the interop layer (Iceberg, catalog federation) before a second engine needs it. On a single platform, native tables are simpler and often faster.
  • Do not stand up all eight layers before the first data product ships. Build the thin slice for one product end to end, then widen. See Roadmap and maturity.

Lessons from platform options papers

Source: a platform team's draft options papers and platform docs (Oct-Nov 2026). All draft or proposed. None is an agreed decision; treat them as evidence, not standards.

  • Layer 7 (AI) does not have to sit inside a data platform. Agents may run in Copilot Studio, Bedrock or AgentCore and reach governed data through MCP tools. The rules are the same wherever they run: scoped tools, identity propagated to each hop, and provenance in every result (Agents and MCP).
  • Two gateways. The AI gateway governs model calls, and a Tool gateway governs tool calls.
  • Containers vs content. Terraform makes databases, schemas, roles and integrations, and dbt makes tables and views, with one owner per object (Infrastructure as code for the data platform).
  • A curated schema for agents. One proposed option is a views-only schema that dbt manages, where each view has a short description for search and a full description for use. Semantic views are a natural fit.