Library · Data Practice OS
ADR-004: One catalog of record per table
Decision record · Proposed · Source reviewed
Status: proposed.
Context
In a lakehouse the Catalog is more than a list of tables. It commits table changes, brokers storage credentials and enforces permissions. If two catalogs both think they own a table, writes can conflict and permissions diverge. Unity Catalog, Snowflake Horizon Catalog and Snowflake Open Catalog (Apache Polaris) (managed Apache Polaris) all speak the Iceberg REST protocol and can federate to each other to some degree.
Decision drivers
- Safe, single-writer table commits
- One place to set and audit permissions per table
- Data classification tags and policies that hold whichever engine reads the data
- End-to-end Lineage
Options considered
| Option | For | Against |
|---|---|---|
| A. Writer's native catalog owns the table; others read via federation/REST | Clear ownership; native features for the writer | Policies must be mirrored for readers. Federation features vary |
| B. Neutral external catalog (Polaris) owns every shared table | Vendor-neutral; one REST endpoint | Writers lose some native governance features; another service to run |
| C. Both catalogs write | None that survive scrutiny | Commit conflicts and divergent security. Rejected |
Decision
Option A. The engine that writes a table registers it in its own catalog, and that catalog is the catalog of record: Unity Catalog for Databricks-written tables, Snowflake Horizon for Snowflake-written tables. The other platform gets read-only access through Iceberg REST or catalog federation. Option B stays available for tables shared with third-party engines.
Classification tags are the source of truth for access. They are applied in the catalog of record and replicated to the reading platform by automation, never by hand.
Consequences
- Each data product's contract names its catalog of record (Data product canvas).
- The Platform team builds and tests tag and policy sync between catalogs. Until that exists, shared tables carry only OFFICIAL data. SENSITIVE data is shared only after the sync is proven (Governance, security and trust).
- Cross-catalog lineage is stitched with OpenLineage where native lineage stops at the platform edge.
- Current federation capability is tracked in Catalogs and the catalog of record. Verify against vendor docs.
Revisit when
- The vendors converge on cross-catalog policy enforcement
- A neutral catalog becomes the natural owner for most shared tables