Skip to content

Library · Data Practice OS

Apache Iceberg as the interchange format

Reference guide · Published · Source reviewed

The answer: when a table has to be read by both platforms, store it once as Apache Iceberg in the organisation's own S3 bucket (Sydney region), and let the engine that writes it own it in its catalog. The other engine reads it through an Iceberg REST catalog. Don't copy gold tables between platforms on a schedule. That's how you end up with two versions of the truth and two storage bills.

This is the mechanical half of ADR-002: Apache Iceberg as the interchange table format. The ownership half is in Catalogs and the catalog of record.

When to use it

  • A gold Data product built on one platform is consumed by the other. For example, Databricks computes crash-risk features and Snowflake serves them to Power BI and Cortex Analyst.
  • You want an exit option: the data stays readable by any Iceberg engine (Athena, Trino, Spark, DuckDB) even if a platform contract changes.
  • You land large raw or semi-structured data (traffic detector feeds, sensor streams) once, and more than one engine needs it.

When not to

  • Single-platform tables. If only one platform ever touches a table, use its native managed storage. Snowflake standard tables and Databricks managed Delta tables are simpler, and the platform optimises them for you.
  • Small, hot, heavily updated tables. Frequent row-level updates create many small files and delete files on Iceberg. Keep operational-style tables native, and publish a periodic Iceberg snapshot if the other side needs it.
  • Anything needing platform-only features that don't yet carry across Iceberg. Check row access policies, some masking behaviours, and clustering or liquid clustering. Policies are enforced by the engine you query through, not by the files.

The two shapes

Writer How it is stored How the other side reads it
Snowflake Snowflake-managed Iceberg table on an external volume (Snowflake is the catalog) Databricks reads it as a foreign Iceberg table through Unity Catalog catalog federation to Snowflake Horizon (read-only)
Databricks Managed Delta with UniForm (Iceberg metadata generated), or a Unity Catalog managed Iceberg table Snowflake reads through a catalog integration pointing at Unity Catalog's Iceberg REST endpoint

As of 6 Nov 2026, Unity Catalog managed Iceberg tables carry no preview label and need Databricks Runtime 16.4 LTS or later. Managed Iceberg materialized views are still in Public Preview, and foreign Iceberg tables (including Snowflake-managed ones) are read-only in Databricks (Databricks Iceberg docs). Both directions move quickly, so check the Snowflake Iceberg docs and the Databricks page again before you design against a feature.

Snowflake writes, Databricks reads

-- One external volume per environment, pointing at the organisation's bucket in ap-southeast-2
CREATE EXTERNAL VOLUME tmr_lake_prod
  STORAGE_LOCATIONS = ((
    NAME = 's3-syd'
    STORAGE_PROVIDER = 'S3'
    STORAGE_BASE_URL = 's3://org-lake-prod/gold/'
    STORAGE_AWS_ROLE_ARN = 'arn:aws:iam::<account>:role/snowflake-lake-prod'
  ));

CREATE ICEBERG TABLE gold.registrations.active_registrations_daily (
  snapshot_date     DATE,
  vehicle_class     STRING,
  region_code       STRING,
  active_count      NUMBER(12,0)
)
  CATALOG = 'SNOWFLAKE'
  EXTERNAL_VOLUME = 'tmr_lake_prod'
  BASE_LOCATION = 'registrations/active_registrations_daily';

Databricks writes, Snowflake reads

-- Databricks: make a Delta table readable as Iceberg
ALTER TABLE prod.road_safety.crash_risk_segments SET TBLPROPERTIES (
  'delta.enableIcebergCompatV2' = 'true',
  'delta.universalFormat.enabledFormats' = 'iceberg'
);

-- Snowflake: one catalog integration to Unity Catalog's Iceberg REST endpoint
CREATE CATALOG INTEGRATION uc_prod
  CATALOG_SOURCE = ICEBERG_REST
  TABLE_FORMAT = ICEBERG
  REST_CONFIG = (
    CATALOG_URI = 'https://<workspace-host>/api/2.1/unity-catalog/iceberg-rest'
    CATALOG_NAME = 'prod'          -- the Unity Catalog catalog
    CATALOG_API_TYPE = PRIVATE     -- if reached over private connectivity
    ACCESS_DELEGATION_MODE = VENDED_CREDENTIALS
  )
  REST_AUTHENTICATION = (          -- a Databricks service principal; secret from a vault
    TYPE = OAUTH
    OAUTH_TOKEN_URI = 'https://<workspace-host>/oidc/v1/token'
    OAUTH_CLIENT_ID = '<client_id>'
    OAUTH_CLIENT_SECRET = '<oauth_secret>'
    OAUTH_ALLOWED_SCOPES = ('all-apis')
  )
  ENABLED = TRUE;

The endpoint is /api/2.1/unity-catalog/iceberg-rest. Unity Catalog's older /iceberg endpoint is read-only and marked legacy (Databricks Iceberg access, checked 6 Nov 2026). The OAuth block follows Snowflake's Unity Catalog tutorial. Databricks' own example uses TYPE = BEARER with a PAT or a service principal's OAuth token instead. Prefer the OAuth client-credentials form, because it avoids a long-lived personal token.

You can then expose tables to Snowflake users individually, or through a catalog-linked database, which syncs the remote catalog's namespaces. Catalog-linked databases are generally available (no preview label as of 6 Nov 2026), and Snowflake documents a bidirectional Unity Catalog setup for them. They are writable by default (ALLOWED_WRITE_OPERATIONS = ALL), and a drop in Snowflake then removes the table and its data from Unity Catalog too. Where Unity Catalog is the catalog of record, link read-only (CREATE DATABASE (catalog-linked)):

CREATE DATABASE uc_prod_linked
  LINKED_CATALOG = (
    CATALOG = 'uc_prod',
    ALLOWED_NAMESPACES = ('road_safety'),
    ALLOWED_WRITE_OPERATIONS = NONE   -- read-only: the owning platform writes
  );

Through Unity Catalog's Iceberg REST API, managed Iceberg tables are read/write for external engines, while managed and external Delta tables with Iceberg reads enabled (UniForm) and foreign Iceberg tables are read-only. Foreign tables need REFRESH FOREIGN TABLE to pick up new snapshots (Databricks Iceberg access, Snowflake REST catalog integration). In the other direction, Databricks reads Snowflake-managed Iceberg through a Snowflake Horizon foreign catalog, reading files directly from storage and falling back to JDBC federation for tables it cannot read directly (Snowflake catalog federation).

Pitfalls

  • Two writers, one table. Never let both engines write the same Iceberg table. Commits will conflict, and nobody owns quality. One writer per table, always. See ADR-004: One catalog of record per table.
  • Policies don't travel with files. Masking and row filters are enforced by the engine you query through. If PII-bearing columns are exposed through Iceberg, the reading platform needs its own equivalent policy, or the column must be excluded or tokenised before publishing. See Governance, security and trust.
  • Bucket access is the back door. Anyone with S3 read on the bucket bypasses both catalogs. Lock the bucket to the platform roles only, and log access.
  • Small files and snapshot growth. Schedule compaction and snapshot expiry on the owning platform. Iceberg will not tidy itself.
  • Egress and duplicate compute. Same-region reads avoid egress, but you still pay both platforms' compute. Make the cross-platform read path a costed, deliberate choice (FinOps).

Worked example

The road safety team builds crash_risk_segments in Databricks from crash records, traffic volumes and road asset condition. It's a Spark-heavy job with ML scoring. The table is UniForm-enabled and lives in Unity Catalog, which is its catalog of record. Snowflake reads it through uc_prod, joins it to the governed road_asset dimension, and serves it to Power BI and to a Cortex Analyst semantic view used by regional planners. Data is stored once, in Sydney, with one owner.

Lessons from platform options papers

Source: a platform team's draft options papers and platform docs (Oct-Nov 2026). All draft or proposed. None is an agreed decision; treat them as evidence, not standards.

  • Do not make raw data Iceberg by default. the platform team scored managed tables 58/63 against 49 for Snowflake-managed Iceberg for the raw layer. The landing files in a versioned organisation-owned bucket are already an open copy. Move one source at a time, on a named trigger (ADR-002: Apache Iceberg as the interchange table format).
  • Iceberg in your own bucket changes who protects the data. There is no Snowflake Fail-safe, so use S3 versioning. Snowflake needs write and delete rights on the bucket: keep that prefix separate from the read-only landing location. Snowflake does not delete orphan files. Compaction and S3 request charges apply.
  • Check data types early. VARIANT needs Iceberg v3, and XML in Iceberg is undocumented and untested in that platform team's work. Snowflake external tables cannot read XML at all. Verify in the Iceberg data types doc.
  • Avoid Delta Lake external tables in Snowflake. Snowflake says that feature will be deprecated.
  • Anyone with bucket access bypasses masking. Open files mean the bucket policy is the access control. Classify before you open.