Library · Data Practice OS
ADR-002: Apache Iceberg as the interchange table format
Decision record · Proposed · Source reviewed
Status: proposed.
Context
If gold data sits only in a proprietary format inside one platform, the second engine (or the next platform, in ten years) must copy it to read it. Copies drift, double storage cost and break Lineage. Both platforms now read and write open formats: Snowflake supports managed and externally catalogued Apache Iceberg tables, and Databricks writes Delta Lake with UniForm metadata for Iceberg readers, plus managed Iceberg tables.
Decision drivers
- One physical copy of shared data
- Data in organisation-owned storage in an Australian region (Data residency)
- Broad engine support: Snowflake, Databricks, AWS services, open-source engines
- Low performance and feature cost for the platform that writes the table
Options considered
| Option | For | Against |
|---|---|---|
| A. Native formats only, copy across | Best native performance and features | Copies, drift, double cost, lock-in |
| B. Delta Lake everywhere | Native to Databricks; mature | Snowflake reads it through Iceberg compatibility (UniForm) rather than natively |
| C. Iceberg for interchange, native elsewhere | Broadest engine support; supported by both vendors' catalogs | Some platform features need checking per table type |
| D. Iceberg for everything | Maximum portability | Gives up native-only features where nothing else reads the table. Not worth it |
Decision
Option C. Tables that are read by more than one engine, or are published as data products to more than one engine, are stored as Iceberg (Delta with UniForm counts if the writer is Databricks) in Organisation-controlled object storage in an Australian region. Platform-internal staging and scratch tables may use native formats.
The raw layer is the exception. Raw tables are the Raw replay copy: every layer is rebuilt from them, so recovery and one point of access control matter most. Keep them as native managed tables, and keep the original landing files in versioned organisation-owned object storage as the open copy that any engine can read. Move a raw source to Iceberg one source at a time, and only when a trigger occurs:
- A named consumer must read that raw data with another engine (for example SageMaker or Athena).
- A data owner requires the queryable raw copy to sit in an organisation-owned bucket.
- Another agency or system must receive the data in an open format.
Evidence from a platform options paper
A platform team's Options paper: raw data storage - Snowflake managed tables vs Iceberg (draft, Oct 2026, not agreed)
scored managed tables at 58 of 63 against 49 for Snowflake-managed Iceberg, 40 for external tables and 30 for
Glue-catalogued Iceberg. Managed tables win on recovery (Time Travel and Fail-safe), on security (Snowflake is the
only path to the data) and on data types (XML into VARIANT). Iceberg in an organisation-owned bucket has no Fail-safe: the organisation must
protect the files itself, Snowflake needs write and delete rights on the bucket, orphan files are the organisation's cost, and
VARIANT needs Iceberg v3. Open access was the only criterion where managed tables scored low. Even weighted 3,
they still came first. This ADR originally named "long-lived system-of-record history" as Iceberg. The platform team evidence
narrows that to data that actually crosses engines.
Consequences
- The catalog question matters: who commits to each table is decided in ADR-004: One catalog of record per table.
- Engineers must know the feature gaps per table type on each platform. The handbook keeps the current list (Apache Iceberg as the interchange format). Verify against vendor docs before relying on any feature.
- Storage accounts, encryption keys and lifecycle rules are organisation-owned infrastructure managed in Terraform, not platform defaults.
- Small-file compaction and snapshot expiry are owned by the writing platform.
Revisit when
- A required feature is unavailable on Iceberg tables for a critical workload
- The Iceberg spec (v3 and later) or vendor support changes the trade-offs