Skip to content

Library · Data Practice OS

Governance on Databricks: Unity Catalog

Reference guide · Published · Source reviewed

Unity Catalog (UC) is the single governance layer for everything in Databricks: tables, volumes (files), views, functions, models, vector indexes and agents. If a workspace still uses the legacy Hive metastore, migrating it to UC is step zero of any modernisation.

The object model

metastore (one per region)            e.g. ap-southeast-2
  catalog    tmr_prod / tmr_test / tmr_dev, plus sandbox and shared catalogs
    schema   registrations_gold, network_silver, ...
      table | view | volume | function | model | vector index
  • One metastore per region. Keep it in Sydney.
  • Catalogs per environment (and optionally per sensitivity), bound to specific workspaces so production data is reachable only from production workspaces.
  • External locations and storage credentials map catalogs to S3 buckets per environment. Only the Platform team creates them.

Who: groups and service principals

Federate identities from the organisation's identity provider via SCIM. Grant to groups, never to individuals. Run production jobs as service principals, not as people.

GRANT USE CATALOG ON CATALOG tmr_prod TO `grp-analysts-registrations`;
GRANT USE SCHEMA, SELECT ON SCHEMA tmr_prod.registrations_gold TO `grp-analysts-registrations`;
GRANT MODIFY ON SCHEMA tmr_prod.registrations_silver TO `sp-registrations-prod`;

What: tags, classification and ABAC

Tag columns and tables with the mandated security classification and personal-information type. Use data classification to suggest tags, and have stewards confirm them. Then write policies against tags, not tables. UC's Attribute-based access control policies (governed tags plus policies defined at catalog or schema level) apply a mask or row filter wherever a tag appears.

-- The masking function, kept in a governance schema
CREATE FUNCTION governance.policies.mask_pii(val STRING)
RETURNS STRING
RETURN CASE WHEN is_account_group_member('grp-pii-full') THEN val ELSE '***MASKED***' END;

-- Direct form: attach to a column
ALTER TABLE tmr_prod.licensing_silver.licence_holder
  ALTER COLUMN licence_no SET MASK governance.policies.mask_pii;

-- Row filter: regional staff see only their region
CREATE FUNCTION governance.policies.region_filter(region STRING)
RETURN is_account_group_member('grp-all-regions')
    OR is_account_group_member(concat('grp-region-', lower(region)));

ALTER TABLE tmr_prod.licensing_silver.licence_holder
  SET ROW FILTER governance.policies.region_filter ON (region);

Verify the current status of tag-driven ABAC policies (some capabilities were in preview at the time of writing) in the Unity Catalog documentation. Fall back to column masks and row filters attached by Terraform where needed.

Lineage, audit and monitoring

  • Lineage is captured automatically to column level for SQL, Python, pipelines and notebooks, including into models and dashboards. Use it for impact analysis before changing a gold table.
  • System tables (system.access.audit, system.access.table_lineage, system.billing.usage) are queryable. Build the privacy-audit and access-review reports on them.
  • Data quality monitoring watches freshness and anomalies on gold tables. See Data quality and observability.

AI objects are governed too

Models, vector indexes, functions used as agent tools, and serving endpoints are all UC securables. An agent can retrieve only rows and chunks its identity can read. Index curated, classified text, and grant the index the same way you grant the table. See AI on Databricks.

External engines and sharing

  • Snowflake and other engines read UC tables through the Iceberg REST catalog endpoint, using credential vending. Row filters and masks are not enforced by the external engine. Expose SENSITIVE data externally only as filtered, masked gold views or tables. See Catalogs and the catalog of record.
  • OpenSharing (formerly Delta Sharing) shares live tables with other organisations. Record each share with an owner, purpose and expiry, and review it quarterly.

Put it in code

Use Terraform (the Databricks provider) for the metastore assignment, catalogs, external locations, grants, masks and filters. Asset Bundles deploy workloads, not permissions. Grants made through the UI are treated as drift.

Pitfalls

  • Workspace-local Hive tables left behind after a UC migration, readable outside governance.
  • Granting ALL PRIVILEGES to make something work.
  • Personal access tokens for automation. Use OAuth with service principals.
  • One catalog for all environments. It makes isolation by workspace binding impossible.

Worked example

Licensing data sits in tmr_prod.licensing_silver. Classification proposes tags, and stewards confirm them in a pull request. Terraform attaches the mask_pii mask and the region_filter row filter. The Customer Service Centre agent runs with a service principal that holds only the masked, filtered gold view. A quarterly job queries system.access.audit for every read of licence_no by grp-pii-full, as evidence for the privacy officer. Compare with Governance on Snowflake - Horizon and the practice-wide model in Governance, security and trust.