Skip to content

Library · Data Practice OS

Databricks at a glance

Reference guide · Published · Source reviewed

In one line: Databricks is the open, engineering-first Lakehouse. Your data sits in your own cloud storage in open formats, and one platform handles SQL, streaming, data science and custom AI. The price is more choices to make and more to operate.

How it is built

Layer What it does What you manage
Storage Delta Lake (and Iceberg) tables in your S3 buckets, or in Databricks-managed storage Buckets, encryption keys, table maintenance (largely automated by predictive optimisation)
Compute Serverless or classic clusters for jobs and notebooks, serverless SQL warehouses for BI, GPU compute for AI Cluster policies, serverless versus classic, sizes, auto-termination
Control plane Workspaces, jobs, notebooks, Unity Catalog, Databricks AI (formerly Mosaic AI), MLflow Workspace layout, catalog design, identity federation

Because the data is in open formats in your own account, other engines (including Snowflake, Athena and EMR) can read it. Unity Catalog governs it all: tables, files, models, functions, vector indexes and agents.

Strengths

  • Engineering at scale. Spark plus Photon handles large batch and streaming workloads. Lakeflow (Connect, Declarative Pipelines, Jobs) provides managed ingestion, declarative pipelines with expectations, and orchestration. See Data engineering on Databricks.
  • Data science and ML. Notebooks, a feature store, MLflow tracking and registry, GPU training and model serving are first-class, not add-ons.
  • Custom AI and agents. Agent Bricks (the Databricks agent developer platform), AI Search (formerly Vector Search), MLflow evaluation and Unity Gateway form the most complete in-platform toolkit for building your own agents. See AI on Databricks.
  • Unstructured data. Images, PDFs, video and sensor data at volume, processed in Python next to the tables.
  • Openness. Delta Lake with UniForm, Iceberg, an open-source Unity Catalog and OpenSharing (formerly Delta Sharing) reduce lock-in.
  • Migration tooling. Lakebridge assesses, transpiles and reconciles legacy warehouse code.

Weak spots

  • Operational surface. There are more knobs: cluster types, runtimes, policies and workspaces. A thin Platform team can get lost. Serverless compute narrows this gap a lot; default to it.
  • Pure BI concurrency. Databricks SQL is competitive, but SQL-only teams often find Snowflake simpler to run and tune.
  • Cross-organisation sharing is good (OpenSharing, formerly Delta Sharing), but the Snowflake sharing and listing ecosystem is more established with other agencies.
  • Sprawl. Notebooks are easy to create and hard to govern. Without standards (CI/CD and environments) production logic ends up in someone's personal folder.

When to choose Databricks

Choose Databricks as the primary, or only, platform when most of these are true:

  1. Significant streaming or high-volume engineering: traffic sensors, vehicle telematics, connected-vehicle data.
  2. Data science and ML are core: forecasting, risk models, computer vision on road imagery.
  3. Custom agents or AI applications are planned, beyond "chat with the data".
  4. The team includes Python and Spark engineers, or will hire them.
  5. Keeping data in open formats in the organisation's own AWS account is a hard requirement.

Not when the workload is almost all SQL reporting with a small SQL-skilled team and a strong need to share data with other agencies. That points to Snowflake. If both lists are long, see Choosing a platform - Snowflake, Databricks, or both and ADR-001: One platform by default, two only by workload.

Worked example

The traffic and network operations data is a natural fit. STREAMS loop-detector counts, travel-time feeds and incident logs arrive continuously. Engineers need minute-level aggregates, and data scientists want congestion forecasting models.

Kinesis / Kafka --> Lakeflow pipeline (bronze streaming table)
                --> silver: cleaned, deduplicated, mapped to network links (expectations enforced)
                --> gold:   link_travel_time_15min  (Delta, UniForm-enabled for Iceberg readers)
                --> MLflow-tracked forecasting model, served via Model Serving

The gold travel-time product is readable from Snowflake through Iceberg, so the corporate Power BI estate can use it without a copy. See Apache Iceberg as the interchange format.

Region and certification

Databricks on AWS runs in Asia Pacific (Sydney, ap-southeast-2). Databricks has been assessed at IRAP PROTECTED in AWS ap-southeast-2, Google Cloud australia-southeast1 and the Azure Australian regions (Databricks IRAP trust page, which also offers the assessment report). IRAP workloads require the compliance security profile on the workspace, and Databricks publishes a per-feature list of what is available under it in ap-southeast-2. That list currently includes serverless compute (SQL warehouses, jobs, notebooks and Lakeflow pipelines), Model Serving (foundation models pay-per-token, external models and custom models), AI Functions including ai_parse_document, Vector Search, Genie and ABAC. Serverless base environment version 5 or later will soon be required for IRAP workloads. See Databricks IRAP and the compliance security profile. Databricks warns that features may be listed before they are released, so test each one in the workspace.

Foundation model availability also depends on the workspace's cross-geo settings. In ap-southeast-2, pay-per-token currently serves several Claude models (for example Opus 5.5, Opus 5, Sonnet 5 and Haiku 5.5) without cross-geo routing, while other models, including Claude Sonnet 5.5, the open models and the databricks-gte-large-en and databricks-bge-large-en embedding models, need cross geography routing enabled (model availability by region, checked 6 Nov 2026). The list changes often, so check it for the model you plan to use, and choose an embedding model that runs in-region before you build Vector Search on SENSITIVE data. See Governance, security and trust.