Skip to content

Library · Data Practice OS

Catalogs and the catalog of record

Reference guide · Published · Source reviewed

The answer: each table has one catalog of record: the catalog of the engine that writes it. That catalog holds the authoritative schema, the permissions and the lineage. The other platform federates to it and never re-registers its own copy. This is ADR-004: One catalog of record per table in practice.

In a two-platform estate the Catalog is more than documentation. It hands out credentials to the files, so whoever controls it controls access.

The options

Catalog Native to Speaks Iceberg REST Good for
Snowflake Horizon Catalog Snowflake Yes, for Snowflake-managed Iceberg tables (verify scope) Tables Snowflake writes
Unity Catalog Databricks Yes, at /api/2.1/unity-catalog/iceberg-rest: read/write for managed Iceberg tables, read-only for UniForm Delta and foreign Iceberg (Databricks Iceberg access) Tables Databricks writes, plus models, functions, volumes, agents
Snowflake Open Catalog (Apache Polaris) / Apache Polaris Neutral Yes (it is an Iceberg REST catalog) A shared catalog when several engines write Iceberg and neither platform should own it
AWS Glue Data Catalog AWS Yes Existing AWS analytics (Athena, EMR) in the estate

Our default is the writer's native catalog. Bring in Open Catalog or Polaris only when a third engine writes the shared tables too, or when governance demands a neutral catalog. Every extra catalog is another place permissions can drift.

Rules that keep it sane

  1. One writer, one catalog of record, per table. Record both in the Data contract (owner, platform, catalog).
  2. Read-only federation on the other side. Snowflake reads Unity Catalog tables through a catalog integration. Databricks reads Snowflake tables through Lakehouse Federation or catalog federation. Neither may write.
  3. Permissions mirrored from one source. Define access in Terraform from the same group model (Entra ID groups via SCIM to both platforms). Never hand-grant on one side. Tag-driven Attribute-based access control policies use the same tag names on both platforms (classification, contains_pii, domain).
  4. Classification lives with the owner. The owning catalog holds the authoritative Data classification tags. A nightly job compares tags across catalogs and fails loudly on a mismatch.
  5. Lineage stitched, not duplicated. Each platform captures its own lineage. Emit OpenLineage events from pipelines that cross the boundary, so the hand-off point stays visible.

Mirroring a tag policy

# terraform: the same classification tag, declared on both platforms from one variable
locals {
  classification_values = ["OFFICIAL", "OFFICIAL_SENSITIVE"]
}

resource "snowflake_tag" "classification" {
  database       = "GOVERNANCE"
  schema         = "TAGS"
  name           = "CLASSIFICATION"
  allowed_values = local.classification_values
}

resource "databricks_tag_policy" "classification" {   # governed tag; workspace-level provider
  tag_key     = "classification"
  description = "Security classification label"
  values      = [for v in local.classification_values : { name = v }]
}

databricks_tag_policy is a GA resource (tag_key, description, values = [{ name }]). It needs a workspace-level provider, or an account-level provider with provider_config { workspace_id = ... } (resource docs, checked 6 Nov 2026). Check the current Snowflake provider for snowflake_tag arguments. Governed tags and ABAC policies on Unity Catalog have been changing quickly, so pin provider versions.

When not to federate

  • When the consumer needs sub-second BI on the table at high concurrency. Federated reads can be slower than native ones. Measure first. If it really is too slow, materialise a native, derived table on the consumer side with a clear derived_from link. Never make a silent copy.
  • When the consumer needs the masking policy itself, not just the data. If the reading platform can't enforce an equivalent policy, publish a de-identified version instead.

Pitfalls

  • Catalog sprawl. Horizon, Unity Catalog, Glue, Open Catalog and a separate enterprise catalog (Purview, Collibra) can all end up describing the same table. Pick which one business users search, and feed it from the others.
  • Credential vending is privileged. The service principal behind a catalog integration can read everything it is granted. Scope it per domain, not one principal for all of prod.
  • Naming drift. prod.road_safety.crash_risk_segments in Unity Catalog and ROAD_SAFETY.GOLD.CRASH_RISK_SEGMENTS in Snowflake will confuse people. Agree one three-part naming convention (<env>.<domain>.<object>) and a mapping rule. See Medallion layers and modelling.

Worked example

Registrations and licensing data is written by Snowflake, so Horizon is its catalog of record. Road asset condition and traffic detector data is written by Databricks, so Unity Catalog owns it. A single data_product_register table, built from both catalogs' metadata, lists every gold product with its owner, writer and catalog. That register is what the enterprise catalogue and the AI agents search (Agents and MCP).