Data.
How organisations turn trusted facts, shared definitions and clear business rules into better decisions.
Start reading
215 entries
What AI-ready means
Seven testable properties of AI-ready data as a quick triage, where the full readiness assessment takes over, and why "we have a lakehouse" is not the answer.
From data to decisions - the decision stack
Why the practice exists - to be the backbone of knowledge and decisions - and the seven layers, from facts to outcomes, that every important decision stands on.
Extracting business rules
Before converting code, mine the policies and calculations it implements into Given/When/Then specifications that become the tests for the new platform.
Choosing a platform - Snowflake, Databricks, or both
A workload-by-workload guide to when each platform wins, when to run both, what "both" really costs, and the scorecard to decide.
ADR-001: One platform by default, two only by workload
Start from a single data platform; add the second only where a named workload class clearly needs it, with a fixed split of responsibilities.
Decision inventory
A register of the important decisions the organisation makes - owner, volume, consequence of error, authority, what data, knowledge and rules each needs, and which AI pattern fits - used to choose what to build.
Decision record schema
A copyable schema for recording a system- or AI-supported decision - facts used, rule version, context sources, AI involvement, authority, human review and outcome - in YAML and as a table, with W3C PROV mappings.
ADR-002: Apache Iceberg as the interchange table format
Data that crosses engines or must outlive a platform choice is stored as Apache Iceberg in organisation-controlled object storage.
ADR-003: dbt as the transformation standard
All batch SQL transformation from silver to gold is written as dbt projects in git, on whichever platform runs it.
ADR-004: One catalog of record per table
Each table has exactly one owning catalog - the one its writing engine uses - and other engines read it through Iceberg REST or catalog federation.
ADR-006: Define metrics once in a semantic layer
Business metrics are defined once in the dbt Semantic Layer and generated or synced into Snowflake semantic views and Databricks metric views where native AI tools need them.
AI on Databricks
Which Databricks AI component fits which job - AI Functions, AI Search, Genie, Agent Bricks, serving and evaluation - and how to keep it governed.
AI on Snowflake - Cortex
Which Cortex service fits which job, how to keep it in-region and governed, and the patterns that work for your organisation.
Apache Iceberg as the interchange format
How Snowflake and Databricks share one copy of a table through Iceberg, and when that is worth the extra moving parts.
Catalogs and the catalog of record
Every table has exactly one owning catalog; the other platform federates to it. How to set that up with Horizon, Unity Catalog and Open Catalog.
CI/CD and environments
Dev, test and prod on both platforms, slim CI for dbt, infrastructure as code, and how a change gets from a branch to production.
Code conversion
Convert legacy SQL and ETL with vendor tools, finish the remainder with LLM help under tests, and land the result as dbt models rather than lifted procedures.
Cost control on Databricks
How Databricks bills (DBUs plus your own cloud), where the money goes, and the guardrails that keep it predictable.
Cost control on Snowflake
How Snowflake bills, where the money goes, and the guardrails that stop a consumption platform surprising you.
Data contracts
What goes in a contract, where it is enforced (CI, build and run time), how to version it, and the smallest version you can adopt this sprint.
Data engineering on Databricks
Ingestion, pipelines and orchestration on Databricks - Lakeflow Connect, Declarative Pipelines, Jobs, dbt and Asset Bundles - and which to use when.
Data engineering on Snowflake
Ingestion, transformation and orchestration on Snowflake - Openflow, Snowpipe, Dynamic Tables, dbt and Snowpark - and which to use when.
Data Practice reading list
Source documentation, standards and further reading supplied with Data Practice OS.
Data product canvas
One page that defines a data product - owner, consumers, contract, quality, access, semantics and AI use - ready to copy into the product's repo.
Data quality and observability
A test pyramid for data, platform-native monitoring on Snowflake and Databricks, freshness SLAs and alert routing that reaches the owner before the consumer notices.
Databricks at a glance
How the Databricks lakehouse is built, where it is strongest, where it is weak, and when it should be the organisation's answer.
dbt project standards
Project layout, conventions, materialisations and cross-platform rules so dbt stays the portable transformation layer across Snowflake and Databricks.
Decision modelling with DMN
Model a decision before you automate it - its requirements, inputs, knowledge sources and logic - and use decision tables where they beat both code and LLMs.
Decision records and provenance
What to record for every system-supported decision - facts used, rule version, context sources, authority, reviewer and outcome - modelled on W3C PROV, with retention and privacy designed in.
Decision register
Every practice decision in one list, with status rules that keep decision status apart from document and delivery status, and one accountable role per decision.
Governance on Databricks - Unity Catalog
Metastore, catalogs, grants, ABAC tags, row filters, column masks, lineage and audit on Databricks, set up as code for your organisation.
Governance on Snowflake - Horizon
Roles, tags, classification, masking, row access, lineage and audit in Snowflake, set up as code for the organisation's classification levels.
Governance, security and trust
Classification-driven access, privacy by design, AI risk assessment and auditability - implemented as policy-as-code on either platform.
Infrastructure as code for the data platform
Which tool makes which platform objects - Terraform, Snowflake DCM, Asset Bundles, dbt - with one owner per object, CI-only deploys and safe migration, from a platform team's Terraform vs DCM options paper.
Knowledge and decisions
The Data Practice handbook on modelling decisions, recording them and learning from the outcomes.
Medallion layers and modelling
What goes in bronze, silver and gold, how to model the gold layer, and the naming convention that works on both platforms.
Migration assessment
Inventory the legacy estate, measure usage, and give each workload a treatment - retire, rehost, replatform, refactor or rebuild - before converting a line of code.
Migration wave plan
Inventory and treatment for each legacy workload, grouped into waves by dependency, value and risk, with entry and exit criteria per wave.
Modernisation and migration
A source-agnostic migration method - assess, extract rules, choose a pattern per workload, convert with tooling and AI, reconcile, parallel-run, cut over - in waves.
Naming conventions
One job for each part of a name - database, schema, table - plus layer, model, role, stage and tag names, based on a platform team's proposed Snowflake conventions and mapped to Databricks.
Ontology, vocabularies and entity resolution
The meaning layer of the decision stack - a business glossary first, SKOS vocabularies for shared code lists, an ontology only where relationships need rules, and shared identifiers so one thing is one thing across systems.
Operating model
Who owns what - platform team, domain data product teams, enabling AI and data practice, governance forums - and the rituals and metrics that keep it running.
Options paper and fact check
A copyable options-paper structure with IDs, gate criteria, H/M/L x weight scoring, sensitivity checks, traceability and a separate fact-check table - adapted from a platform team's options-paper template and review practice.
Outcome feedback loops
Capture what happened after a decision, link it to the decision record, and feed it into evaluation sets, rule review and the decision inventory - so the practice learns instead of just running.
Pipeline code - explicit vs metadata-driven
When to generate dbt code from metadata and when to hand-write it - split by layer - with 13 conventions and measurable checks, from a platform team's pipeline-code options paper.
Platform selection scorecard
A weighted scoring matrix for Snowflake, Databricks or both, scored per workload class, with a worked example.
Reconciliation and cutover checklist
The checks that prove a migrated workload gives the same answers as the legacy - counts, checksums, aggregates, samples - and the cutover steps that follow.
Reconciliation, parallel run and cutover
Prove the new platform gives the same answers - counts, checksums, aggregates, samples - over a parallel run, then cut over consumers in waves and decommission with evidence.
Roadmap and maturity
A five-level maturity model and a staged roadmap - first 90 days, 6 months, 12 months, 24 months - with lighthouses chosen by decision and exit criteria measured in decision outcomes.
Security foundations, the first 90 days
Identity and classification first, then the platform skeleton, the controlled code path, data-plane and consumption security, then operate. Three signed go/no-go gates, and a mapping to Snowflake, Databricks and Microsoft.
Sharing data across platforms and agencies
Choosing between Snowflake Secure Data Sharing, OpenSharing (formerly Delta Sharing) and Iceberg for sharing within the organisation, with other agencies and with partners.
Snowflake at a glance
How Snowflake is built, where it is strongest, where it is weak, and when it should be the organisation's answer.
Start here - the map
What this practice is for - the backbone of knowledge and decisions - how the site is organised, and the ten things you should believe by the end of the path.
Target reference architecture
Eight layers from source to AI, which are portable across Snowflake and Databricks, and how they map onto each platform.
Traceability gap report and rule trace card
A per-wave report that turns the traceability gap queries into countable exit criteria, plus a one-row-per-P0-rule trace card an auditor can read (draft).
Tracing rules from legacy code to modern code
Make the business rule the hub - legacy code, modern dbt models, tests and decisions each link to rule ids with evidence and dates - so gap queries can gate every migration wave (draft).
When a knowledge graph earns its place
Use a knowledge graph when relationships are the question - legislation cross-references, vehicle lineage, asset networks - and count the costs first, including GraphRAG's indexing bill and a second permission model.
Where business rules run after migration
Keep the rule of record in a governed repository with effective and deploy dates; dbt tests, decision tables and engines are projections of it - from a platform team's executable-rules options paper.
Agent Bricks
The Databricks agent developer platform - build, deploy, trace and evaluate agents (with MLflow judges and expert feedback), governed through Unity Catalog and Unity Gateway.
AI-permission record
A record, separate from classification and privacy tags, of which AI purposes, services and environments are approved for a dataset, with redaction and training restrictions, approver, basis, review date and status (approved, prohibited, not assessed).
AI/BI Genie
Natural-language questions over curated tables, grounded in instructions, example SQL and Unity Catalog metric views. Databricks' text-to-SQL.
Amazon Bedrock
AWS managed access to foundation models including Anthropic Claude, with Australian-region options. Useful when AI runs outside the data platforms.
Amazon Bedrock AgentCore
AWS services for running agents - Runtime, Gateway (a managed MCP tool gateway), Memory and Identity. The platform team evaluated Gateway as the target front door for its MCP servers.
Anthropic Claude
Frontier LLM family available natively inside Snowflake Cortex, Databricks Model Serving and Amazon Bedrock - one model choice that does not force a platform choice.
Apache Iceberg
Open table format with broad engine support. The practical interchange format between Snowflake, Databricks and AWS services.
Apache Ossie (formerly Open Semantic Interchange)
Apache incubating project (formerly OSI, started by Snowflake, dbt Labs, Salesforce and others) for a vendor-neutral JSON/YAML semantic model specification, with a Python SDK and a compliance grader. No release yet - watch, do not depend on.
Apache Polaris
Open-source Iceberg REST catalog. Snowflake Open Catalog is the managed version.
Architecture decision record
A short document recording one significant decision - context, options, the choice and its consequences - so later readers know why, not just what.
Attribute-based access control
Access decided by attributes of the user and the data (role, classification tag, purpose) rather than per-table grants. Scales where role-per-table does not.
Break-glass account
An emergency admin account excluded from normal sign-in controls, for when identity services fail. Keep two, document them, monitor every use and test them on a schedule.
Business rules
The policies, calculations and validations buried in legacy code and ETL. Extracting them as testable specifications is the highest-leverage step in a migration.
Catalog
The system of record for what tables exist, where they live, who may use them and how they relate. In a lakehouse the catalog also brokers engine access to the files.
Change data capture
Streaming row-level inserts, updates and deletes from a source database's log, so the platform stays current without full reloads.
Characterisation (pinning) test
A test that records what the legacy system actually does, quirks included, rather than what it should do (after Feathers). It is kept even when a later decision changes the rule, so the old behaviour stays documented.
Cortex Agents and Snowflake Intelligence
Agents that orchestrate Cortex Analyst, Cortex Search and custom tools; Snowflake Intelligence is the chat front end for business users.
Cortex AI SQL functions
SQL functions such as AI_COMPLETE, AI_CLASSIFY, AI_EXTRACT and AI_AGG that call models over table rows, documents and images from a query.
Cortex Analyst
Natural-language questions to SQL, grounded in a semantic model or semantic view you define. Snowflake's text-to-SQL.
Cortex Search
Managed hybrid (vector + keyword) search service over text in Snowflake tables, kept fresh automatically. The retrieval half of RAG.
Data as code
Pipelines, tests, access policies and infrastructure in git, reviewed and deployed by CI/CD. Nothing important is configured by clicking.
Data classification
Labelling data by sensitivity (for example OFFICIAL and SENSITIVE in Australian government schemes, or Internal and Confidential in an enterprise) so access, masking and AI-use rules can be applied by policy rather than by hand.
Data contract
A machine-readable agreement between a producer and its consumers - schema, semantics, quality rules, SLA and ownership - checked in CI and at run time.
Data loss prevention (DLP)
Policies that detect sensitive information types (for example TFN or ABN) leaving approved boundaries. Start in audit mode to learn the false-positive rate, then move to block.
Data mesh
An operating model where domain teams own their data as products on a shared self-serve platform under federated governance. A way of organising people, not a technology.
Data observability
Monitoring freshness, volume, schema and distribution of data in production, so consumers hear about a problem from the platform, not from a wrong report.
Data product
A dataset (or set of them) owned by a named team, with a contract, documentation, quality checks and an SLA, published for others to consume.
Data product tier
A 1-3 rating from business impact and number of users that sets how much monitoring, freshness checking and incident response a data product gets.
Data residency
Where data is stored and processed. For government workloads this usually means Australian regions, and it extends to where an LLM call is processed.
Data security posture management (DSPM)
Continuous discovery of where sensitive data sits and who can reach it. Take a baseline on day 0 and compare against it to show posture actually improved.
Databricks AI (formerly Mosaic AI)
Databricks' AI tooling, no longer branded Mosaic AI - Model Serving (including Claude), Agent Bricks, AI Search, Unity Gateway, MLflow evaluation and fine-tuning - integrated with Unity Catalog.
Databricks AI Functions
SQL functions such as ai_query, ai_classify, ai_extract and ai_parse_document for running models over tables in batch.
Databricks AI Search (formerly Vector Search)
Managed vector index that syncs from Delta tables, with hybrid search and Unity Catalog permissions. Queried from SQL with vector_search() or through a managed MCP endpoint.
Databricks Apps
Host data and AI apps (Streamlit, Dash, React, etc.) inside the Databricks workspace under Unity Catalog permissions.
Databricks Asset Bundles
Databricks' infrastructure-as-code for jobs, pipelines, models and apps, deployed from CI.
Databricks Data Intelligence Platform
Lakehouse platform on open formats (Delta Lake, Iceberg) with Spark, SQL warehouses, ML and GenAI tooling, governed by Unity Catalog.
Databricks SQL
Serverless SQL warehouses for BI and analytics on the lakehouse.
dbt
SQL-first transformation framework - models, tests, docs, lineage - that runs on both Snowflake and Databricks. The portable layer of the stack.
dbt Semantic Layer (MetricFlow)
Metrics defined once in dbt and served to BI and LLMs across platforms. MetricFlow is open source.
dbt_project_evaluator
dbt Labs package that checks a project against dbt best practices (documentation coverage, primary-key tests, direct joins to sources) and returns countable violations for CI.
Decision intelligence
The discipline of starting from the decisions a business makes and working back to the data, knowledge, rules and AI that support them, then measuring decision outcomes rather than outputs.
Decision inventory
A register of the important decisions an organisation makes, each with an owner, volume, consequence of error, authority, the data, knowledge and rules it needs, and the AI pattern that fits. Used to choose what to build.
Decision provenance
The versioned link between a decision and the facts, rule versions, processing activities and responsible people or systems behind it, kept so the decision can be reviewed later.
Decision record
An append-only record of one system- or AI-supported decision - the facts used, rule version, context sources, AI involvement, authority, human review and, later, the outcome. Decision provenance made concrete. Not to be confused with an architecture decision record.
Delta Lake
Open table format, native to Databricks. UniForm lets Delta tables be read as Iceberg.
DMN (Decision Model and Notation)
The OMG standard for modelling decisions - decision requirements diagrams showing inputs, logic and knowledge sources, and decision tables with explicit hit policies that business owners can read and approve.
Dynamic Tables
Declarative tables defined by a query and a target freshness; Snowflake works out incremental refresh. Replaces much hand-written stream-and-task plumbing.
Effective dating
Reference data and rule parameters stored as dated versions (effective from/to), plus the date each version was deployed, so a past decision can be replayed with the rules known at the time.
ELT
Extract and load raw data first, transform inside the platform. The modern default; transformation logic lives in version-controlled SQL or Python.
Entity resolution
Deciding whether records in different systems refer to the same real-world thing (person, vehicle, road segment), and keeping the links, method and confidence as reviewable data rather than silent merges.
Equivalence test
A per-rule test that the modern implementation gives the legacy's answer on the same chosen inputs, usually fed by characterisation-test fixtures (in dbt, a unit test). It differs from reconciliation, which compares outputs on live data.
FAIRA
Foundational artificial intelligence risk assessment - one Australian state government's framework (v2.0.0, September 2026), a worked example of a government AI risk assessment, for transparency, accountability and risk of an AI use case. The AI governance policy requires a FAIRA or agreed equivalent; FAIRA LITE suits low-risk trials.
FastMCP
Python framework for building MCP servers. The platform team's pattern runs FastMCP servers on ECS Fargate with pooled database connections.
FinOps
Making cloud cost visible, attributable and owned by the teams that spend it. Consumption platforms make this a first-day concern.
Gate criterion
A criterion an option must pass. An option that fails a gate leaves the comparison, whatever its total. A gate needs a named policy, person or decision that requires it.
GitHub Actions
CI/CD for dbt builds, tests, policy checks and deployments across environments.
Golden record
The single, steward-approved version of an entity produced by master data management, published back to the catalogue with its lineage and history.
Human in the loop
A person reviews or approves AI output before it has effect. Required for decisions about people until evaluation evidence says otherwise.
Identity propagation
Carrying the end user's identity through every hop - client, gateway, MCP server, data store - usually by on-behalf-of token exchange, so access and audit reflect the person, not a service role.
Information asset owner
The information-management role (information asset custodianship in many government frameworks) accountable for one information asset - its classification, who gets access and retention. Supported by the information asset custodian.
Knowledge graph
Data stored or analysed as entities and typed relationships, usually following an ontology. Earns its place when the question is about relationships several hops deep; it is a technique, not a prerequisite for AI.
Lakebase
Managed Postgres integrated with the lakehouse, for low-latency operational reads and writes by apps and agents.
Lakebridge
Databricks' free migration toolkit - assessment, SQL and ETL transpilation, and reconciliation from legacy warehouses.
Lakeflow (Connect, Declarative Pipelines, Jobs)
Databricks' data engineering suite - managed ingestion connectors, declarative pipelines (formerly Delta Live Tables) with expectations, and orchestration.
Lakehouse
Warehouse-grade tables (ACID, schema, governance) on open files in object storage, so BI, engineering and ML share one copy of the data.
Lineage
The record of which sources, jobs and columns produced a dataset. Needed for impact analysis, audit, and for explaining an AI answer back to its evidence.
Masking and row policies
Column masking and row filtering applied by the platform at query time, driven by tags, so the same table serves different audiences safely.
Master data management
Matching and merging records about the same customer, asset or product across systems into a governed golden record, with a stewardship workflow. Start with one domain whose sources are already catalogued.
Medallion architecture
Bronze (raw, as landed), silver (cleaned, conformed) and gold (business-ready products) layers. A naming convention for refinement stages, not a law.
Microsoft Copilot Studio
Microsoft's agent builder, a likely consumer channel for staff. Its MCP support and Power Platform data policies constrain how tools are designed.
Migration pattern
The treatment chosen per workload - retire, rehost, replatform, refactor, rebuild - based on value and effort, not one strategy for the whole estate.
Migration traceability
An unbroken, queryable chain from each piece of modern code back to the business rule it implements, the legacy code that carried it, the tests that prove it and any decision that let it differ. The rule is the hub - legacy and modern code never link directly. Proposed practice.
MLflow
Open-source experiment tracking, model registry, tracing and GenAI evaluation. Managed in Databricks; usable anywhere.
One owner per object
Every platform object is created and changed by exactly one tool or repo. Terraform makes the containers (databases, schemas, roles, integrations); dbt fills them (tables, views). Two tools managing one object revert each other.
Ontology
A formal model of the concepts in a domain, their relationships and the constraints on them, which software can reason over. Goes beyond a glossary (definitions) and a vocabulary (managed code lists). Use one where relationships carry rules.
Open Data Contract Standard
Bitol's YAML standard for data contracts - schema, quality, SLA, ownership - with growing tool support.
Open table format
A specification (Apache Iceberg, Delta Lake) that turns files in object storage into tables with transactions, schema evolution and time travel that any compatible engine can read.
OpenLineage
Open standard for emitting lineage events from pipelines and engines into any catalog.
OpenSharing (formerly Delta Sharing)
Databricks' open protocol for sharing live tables and AI assets across platforms and clouds, including to non-Databricks consumers (OIDC federation or bearer tokens). Also an open-source project.
Options paper
A scored comparison of options for one decision - goal, criteria with weights and gates, evidence, totals, sensitivity, recommendation and traceability - written before an ADR records the choice.
Outcome feedback
Capturing what happened after a decision, linking it to the decision record, and feeding it into evaluation sets, rule review and decision outcome measures.
Parallel run
Running legacy and new pipelines side by side on live data for a defined period, reconciling every run, until the new one has earned cutover.
Paved road
The supported, templated, pre-governed way to do a common job. Teams may leave it, but staying on it is the easiest path.
Personal information
Information about an identified or reasonably identifiable individual (the usual definition in privacy law). Licence, registration and contact data are all personal information.
Platform team
The team that builds and runs the shared data platform as an internal product - paved roads, templates, guardrails - so domain teams can ship without tickets.
Power BI
The likely BI front end. Connects natively to both Snowflake and Databricks SQL; keep business logic in the semantic layer, not in each report.
Privacy impact assessment
A structured review of how a project collects, uses and discloses personal information against the applicable privacy principles, with the risks and mitigations recorded.
Privileged identity management (PIM)
Just-in-time, approved and time-limited activation of admin roles instead of standing admin rights. Every admin role on the data platform sits behind it.
RACI
Responsible, Accountable, Consulted, Informed. Each decision has exactly one Accountable role. Docs name roles, not people; a roster maps roles to people.
Raw replay copy
The loaded, checked, recoverable copy of source data from which every downstream layer can be rebuilt. Distinct from the landing files, which are the as-delivered copy.
Readiness gate
A pass/fail condition an AI use must meet before it is ready, whatever its maturity score. Five gates (rights and handling, access isolation, context sufficiency, fitness for purpose, operational control); the more restrictive result governs.
Reconciliation
Proving the new platform produces the same answers as the old one - row counts, checksums, aggregates and sampled records - before anyone switches off the legacy.
Rule authority
The principle that text extracted from a policy is evidence, not executable logic. A rule needs approval, applicability, effective dates and exception precedence before it is encoded; model-generated rules stay proposals until the owner validates them.
SCIM provisioning
System for Cross-domain Identity Management - the standard that pushes users and groups from the identity provider into each tool, so joiners, movers and leavers are handled centrally and tools never keep their own user model.
SnowConvert AI
Snowflake's free migration tooling - assesses and converts SQL, stored procedures and ETL from Oracle, SQL Server, Teradata and others, with AI-assisted conversion and validation.
Snowflake AI Data Cloud
Fully managed SQL-first data platform with separate storage and compute, strong governance and sharing, and an expanding AI layer (Cortex).
Snowflake Cortex AI
Managed LLMs and AI services inside Snowflake's governance boundary - AI SQL functions, Cortex Search, Cortex Analyst, Cortex Agents - billed on consumption.
Snowflake DCM Projects
Snowflake-native declarative change management - object definitions as SQL, no state file. Does not cover integrations, secrets, users or PrivateLink; several policy entities are preview.
Snowflake Horizon Catalog
Snowflake's built-in governance - tags, classification, masking and row policies, lineage, access history, data quality metrics - and its catalog for Iceberg interop.
Snowflake ML
Feature Store, Model Registry, notebooks and container-runtime training and inference inside Snowflake.
Snowflake Open Catalog (Apache Polaris)
Managed Apache Polaris - an Iceberg REST catalog that lets Snowflake, Spark, Databricks and other engines share Iceberg tables.
Snowflake Openflow
Managed ingestion service built on Apache NiFi with connectors for databases (CDC), SaaS, files and streams.
Snowflake Secure Data Sharing
Live, zero-copy sharing between Snowflake accounts, including listings across regions and to non-Snowflake consumers via reader accounts.
Snowpark
Python, Java and Scala DataFrames and functions that run inside Snowflake compute, for transformation and ML without moving data out.
Snowpark Container Services
Run your own containers (APIs, apps, open models, GPU jobs) inside Snowflake, next to the governed data.
Strangler fig
Replacing a legacy system piece by piece behind a stable interface, so value ships early and the old system shrinks rather than being switched off in one go.
Streamlit in Snowflake
Python data apps hosted inside Snowflake under its access controls - good for internal tools and AI front ends.
Technical debt
Shortcuts and obsolete designs whose ongoing cost - incidents, slow change, key-person risk - is the real business case for modernisation.
Terraform
Infrastructure as code. Both Snowflake and Databricks have mature providers for accounts, warehouses, catalogs, grants and policies.
Unity Catalog
Databricks' governance layer for tables, files, models, functions and agents - permissions, ABAC tags, lineage, auditing. Also an open-source project with an Iceberg REST interface.
W3C PROV
The W3C provenance vocabulary of entities, activities and agents. Use it to record which facts, rule versions and people or systems stand behind an AI-supported decision.
Workload isolation
Separating compute for ingestion, transformation, BI and AI so one workload cannot starve another, and each has its own cost line.
Zero-copy clone
A metadata-only copy of a database or schema (Snowflake CLONE, Delta shallow clone) used to build and test a change in isolation, then dropped.
Zero-copy sharing
Granting another account, workspace or agency live read access to data without copying it (Snowflake Secure Data Sharing, Delta Sharing).
Snowflake documentation
The source of truth for every Snowflake feature claim on this site.
Snowflake AI and ML features overview
Map of Cortex AI SQL, Search, Analyst, Agents and Snowflake ML, with current model and region availability.
Dynamic Tables
Declarative pipelines in Snowflake. Read before writing another stream-and-task chain.
Snowflake Horizon
Governance, classification, policies, lineage and Iceberg catalog capabilities in one place.
SnowConvert AI
Free assessment and code conversion into Snowflake. Run the assessment even if you are not sure Snowflake is the target.
Databricks documentation
The source of truth for every Databricks feature claim on this site.
Unity Catalog
Permissions, ABAC, lineage and the external-engine (Iceberg REST) story.
Databricks AI Search (formerly Vector Search)
How indexes sync from Delta tables and inherit permissions.
Databricks AI Functions
Batch LLM inference over tables from SQL.
Lakebridge
Databricks' assessment, transpilation and reconciliation toolkit.
Apache Iceberg
The table format that lets Snowflake and Databricks share one copy of data.
Apache Polaris
The open Iceberg REST catalog behind Snowflake Open Catalog.
Delta Lake
Databricks' native format, including UniForm for Iceberg readers.
Unity Catalog (open source)
The open-source Unity Catalog project and its multi-format, multi-engine ambitions.
dbt documentation
The portable transformation layer for both platforms - models, tests, contracts, docs.
dbt model contracts
Enforced schemas on published models - the cheapest data contract you can adopt today.
dbt Semantic Layer
Cross-platform metric definitions for BI and LLMs.
Open Data Contract Standard
A vendor-neutral YAML shape for contracts between producers and consumers.
OpenLineage
Emit lineage once, read it in any catalog.
MLflow
Tracing and evaluation for GenAI, registry for models - usable on either platform.
Data Mesh Principles and Logical Architecture
Zhamak Dehghani's original framing. Read for the organisational ideas; adopt them gradually.
Team Topologies
The vocabulary for platform teams, stream-aligned teams and enabling teams used in the operating model.
FinOps Foundation framework
Common language for making consumption cost visible and owned.
Policy for the responsible use of AI in government (DTA)
The Commonwealth baseline for accountable officials, transparency statements and AI use-case governance. A useful benchmark for any agency or enterprise.
Example government AI policy - Queensland AI governance policy (QGEA, v2.0.0)
Mandated in that state since August 2026. AI governance with regard to ISO 38507, and a FAIRA (or agreed equivalent) for AI solutions.
Example AI risk assessment - FAIRA framework (QGEA, v2.0.0)
The Foundational AI risk assessment framework and template, including FAIRA LITE for low-risk trials.
Example classification scheme - Queensland Information Security Classification Framework (QGISCF v6.0.0)
The OFFICIAL / SENSITIVE / PROTECTED labels and business impact levels that drive tag-based policy.
Example privacy regulator - Office of the Information Commissioner Queensland
One state's privacy principles and guidance on privacy impact assessments, including for AI.
Information Security Manual (ISM)
The control baseline behind IRAP assessments of cloud services.
Amazon Bedrock
Model access in Australian AWS regions when AI runs beside, not inside, the data platform.
GitLab enterprise data platform handbook
A public, working Snowflake + dbt platform - clone per change, tiers, schema-change rules, safe-data CI checks, a curated schema for agents. A platform team mapped 18 of its practices to its own plan.
dbt_project_evaluator
Turns dbt best practices into countable CI checks - documentation coverage, grain tests, direct joins to sources.
Decision Model and Notation (DMN)
The OMG standard for decision requirements diagrams and decision tables. Read the decision table and hit policy sections before designing rule logic.
W3C PROV overview
The family of W3C provenance specifications - entities, activities and agents. The vocabulary behind decision records.
SKOS Simple Knowledge Organization System reference
The W3C standard for controlled vocabularies and thesauri - preferred labels, synonyms, broader and narrower concepts, cross-scheme matches.
NIST AI Risk Management Framework (AI RMF 1.0)
Govern, map, measure, manage - a widely used structure for AI risk that complements FAIRA. Its "map" function starts from the context and decision an AI system supports. Verify the current version and profiles.
GraphRAG (Microsoft Research)
The reference implementation of graph-based retrieval over documents. Read the indexing cost notes before proposing it.
ISO 30401 Knowledge management systems
The international standard for establishing and maintaining a knowledge management system. Useful for the ownership and lifecycle of the knowledge layer. Verify the current edition.
Characterization testing (Michael Feathers)
The source of the pinning-test idea - record what the legacy system actually does, quirks included, and keep the test even after a decision changes the rule.
Bitemporal history (Martin Fowler)
Separates what was true (valid time) from when we recorded it (record time). Read before deciding how trace edges carry dates.
SEBoK - System Requirements Definition
Systems-engineering view of requirements traceability - trace to needs, check completeness, assess the impact of change, and fix the traceability model at the start of the project.
ISO/IEC/IEEE 29148:2018 Requirements engineering
The international standard for requirements processes and information items, including traceable requirements. Context only - the clause text is paywalled and was not read, and a revision (DIS) is pending. Verify any specific clause claim.
dbt meta config
Arbitrary key-values on models, columns, data tests and unit tests, compiled into manifest.json - where rule ids live in dbt. Changing meta does not trigger state:modified, so trace reports must read the whole manifest.