Skip to content

Library · Data Practice OS

Sharing data across platforms and agencies

Reference guide · Published · Source reviewed

The answer: share live and don't send extracts. Pick the mechanism by who the consumer is:

Consumer Use
Another Snowflake account (another agency, a partner) Snowflake Secure Data Sharing (direct share or private listing)
Another Databricks workspace or metastore Databricks-to-Databricks OpenSharing (formerly Delta Sharing)
Any other engine, or the consumer is unknown OpenSharing (open protocol), or read access to an Iceberg table via REST catalog
The organisation's own other platform Not "sharing" at all: Iceberg plus catalog federation (Apache Iceberg as the interchange format)
A person who wants a file A governed export from a gold product, logged, with an expiry. This is the exception, not the pattern

When live sharing is right

  • Recurring feeds to other agencies, partner organisations or research partners, such as aggregated crash statistics or road network attributes.
  • Data a consumer queries ad hoc, where today's process emails a CSV every month.
  • Anywhere you need to revoke access instantly. A share can be withdrawn; an emailed extract can't.

When it isn't

  • Unaggregated personal information. Live sharing makes it easier to share, not safer. Licence holder records, addresses and enforcement data need a privacy impact assessment, an information sharing agreement and usually de-identification first. See Governance, security and trust.
  • Consumers outside Australia, or on non-Australian regions. Cross-region replication for listings moves data. Check Data residency obligations before you enable auto-fulfilment.
  • When the consumer needs a stable contract you can't promise. Sharing a table nobody owns makes every schema change an outage for someone else. Only share gold Data products with a published Data contract.

Snowflake: share a gold product

CREATE SHARE road_safety_stats_share;
GRANT USAGE ON DATABASE road_safety TO SHARE road_safety_stats_share;
GRANT USAGE ON SCHEMA road_safety.gold TO SHARE road_safety_stats_share;

-- Share a secure view, not the base table: it fixes the contract and hides internals
GRANT SELECT ON VIEW road_safety.gold.v_crash_stats_by_lga TO SHARE road_safety_stats_share;

ALTER SHARE road_safety_stats_share ADD ACCOUNTS = <consumer_org>.<consumer_account>;

Sharing to a consumer account in another region or cloud goes through a listing with replication. Verify the current mechanism and costs in the Snowflake sharing docs.

Databricks: share through Unity Catalog

CREATE SHARE road_network_share;
ALTER SHARE road_network_share ADD TABLE prod.road_assets.road_segments_public;

CREATE RECIPIENT partner_agency;      -- open sharing: token-based recipient
GRANT SELECT ON SHARE road_network_share TO RECIPIENT partner_agency;

Databricks now calls Delta Sharing OpenSharing; the open protocol and open-source clients are unchanged. Prefer OIDC federation for open recipients: the recipient authenticates against their own identity provider (with MFA), instead of holding a long-lived bearer token that you issue. Where a bearer token is unavoidable, rotate it and expire it when the agreement ends.

The documented way to create an OIDC recipient is in Catalog Explorer, not SQL: OpenSharing > New recipient, type Open, authentication OIDC Federation. Then add a federation policy with the recipient IdP's issuer URL (for example Entra ID or Okta) and the subject, group or application claims allowed in. It supports user-to-machine access from tools such as Power BI and Tableau, and machine-to-machine access from the Python and Spark clients (OIDC recipients, checked 6 Nov 2026; no preview label). See also OpenSharing.

Rules

  1. Share views or curated tables, never raw. The shared object is the contract.
  2. One register of shares. Record every share, recipient, agreement reference and review date in the data product register. Review shares quarterly, as you would review access.
  3. Aggregate or de-identify by default. Apply minimum cell-size rules (for example, suppress counts under 5) to any statistical product leaving the organisation.
  4. Audit consumer access. Both platforms log share consumption. Feed those logs into the same monitoring as internal access.
  5. Costs land on the right side. With Snowflake sharing the consumer pays for their own compute. With OpenSharing to open recipients, the provider pays egress for cross-region reads. Factor this into the agreement (Cost control on Snowflake, Cost control on Databricks).

Pitfalls

  • Sharing as a substitute for interop. OpenSharing from Databricks into the organisation's own Snowflake works, but Iceberg plus federation is simpler and keeps one catalog of record. Use sharing across organisational boundaries.
  • Forgotten recipients. A partner project ends and the token lives on. Put an expiry on every recipient.
  • Schema change without notice. Version shared views (v_crash_stats_by_lga_v2) and run the old and new versions together for an agreed notice period.

Worked example

A partner agency and the organisation both need the same incident data. The organisation publishes v_crash_stats_by_lga (aggregated, suppressed under 5, OFFICIAL) as a Snowflake share to the agency's account in the same region. The council partner programme gets road_segments_public from Databricks through OpenSharing, with recipient credentials that expire at the end of the funding agreement. No CSVs are emailed.