Skip to content

Library · Data Practice OS

Cost control on Databricks

Reference guide · Published · Source reviewed

Databricks bills in DBUs (Databricks Units) per second of compute, at a rate that depends on the compute type: jobs, all-purpose, SQL, serverless, model serving. On classic compute you also pay AWS directly for the EC2 instances, storage and networking. Serverless bundles the infrastructure into the DBU rate. Budget for both lines, or the comparison with Snowflake will be wrong.

Where the money goes

Cost driver Typical waste Guardrail
All-purpose (interactive) clusters Running production jobs; left on overnight Jobs on job or serverless compute; auto-termination at 30 minutes or less
Classic clusters Oversized, on-demand, no spot instances Cluster policies with size caps, spot instances for workers, the latest runtime
SQL warehouses Large size, long auto-stop Serverless SQL, auto-stop at 5-10 minutes, scaling limits
Pipelines Continuous mode where triggered would do Triggered mode for anything not truly real-time
Model serving Provisioned throughput left running, no scale-to-zero Scale-to-zero for low traffic, pay-per-token for exploration
AI Functions batch Re-running inference over whole tables Process new rows only, and persist results
AI Search (formerly Vector Search) Endpoints sized for peak, indexes over everything Index curated subsets, right-size endpoints
Storage (AWS) Unvacuumed history, duplicate copies Predictive optimisation, retention settings, one copy shared via Iceberg

Guardrails to set on day one

  • Cluster policies (in Terraform) cap instance types, node counts and auto-termination, and enforce tags. Most users should not be able to create unrestricted clusters.
  • Serverless budget policies attribute serverless usage to a team or project tag.
  • Budgets with alerts per workspace and per tag.
  • Compute per workload: separate SQL warehouses for BI and for dbt, separate job compute per pipeline, and separate serving endpoints per app. This is Workload isolation that also gives clean cost lines.
resource "databricks_cluster_policy" "standard_jobs" {
  name = "standard-jobs"
  definition = jsonencode({
    "autotermination_minutes" : { "type" : "range", "maxValue" : 30, "defaultValue" : 20 },
    "num_workers"             : { "type" : "range", "maxValue" : 8 },
    "aws_attributes.availability" : { "type" : "fixed", "value" : "SPOT_WITH_FALLBACK" },
    "custom_tags.cost_centre" : { "type" : "unlimited", "isOptional" : false }
  })
}

Make it visible

system.billing.usage, joined to system.billing.list_prices, gives DBU spend by workspace, SKU, job, warehouse, endpoint and tag. Add AWS Cost Explorer data (tagged consistently) for classic compute and storage. Build one FinOps dashboard covering both and review it weekly.

SELECT u.usage_date,
       u.custom_tags['cost_centre'] AS cost_centre,
       u.sku_name,
       SUM(u.usage_quantity * p.pricing.default) AS est_list_cost
FROM system.billing.usage u
JOIN system.billing.list_prices p
  ON u.sku_name = p.sku_name
 AND u.usage_start_time >= p.price_start_time
 AND (p.price_end_time IS NULL OR u.usage_start_time < p.price_end_time)
WHERE u.usage_date >= date_sub(current_date(), 90)
GROUP BY ALL
ORDER BY est_list_cost DESC;

AI-specific cost habits

  • Use Agent Bricks or evaluation runs to compare models on quality and cost, and choose the cheapest one that passes (Evaluating AI systems).
  • Use pay-per-token endpoints for prototypes, and provisioned throughput only when volume and latency justify it.
  • Unity Gateway usage tracking attributes token spend to teams. Set rate limits per consumer.
  • Batch inference through ai_query in a scheduled job is far cheaper than real-time calls for enrichment.

Commercial

Committed-use contracts reduce DBU rates, and some can be bought through the AWS Marketplace and count toward an AWS commitment. Wait for three to six months of real usage before committing, and check for any existing whole-of-government or enterprise arrangement.

Pitfalls

  • Comparing platform quotes without the AWS line for classic compute and storage.
  • Data scientists on GPU clusters with no auto-termination.
  • Continuous pipelines for data that arrives daily.
  • Many small workspaces, each with its own idle warehouses.

Worked example

The network operations team ran loop-detector pipelines in continuous mode and a GPU cluster for forecasting experiments that never terminated. Switching the non-real-time aggregates to triggered mode every 15 minutes, adding a 30-minute auto-termination policy, and moving dashboards to a serverless warehouse with a 10-minute auto-stop cut monthly spend by about 60%. The genuinely real-time feeds stayed continuous.

Compare with Cost control on Snowflake. Verify current DBU rates, serverless pricing in Sydney and model serving prices with Databricks or the procurement arrangement.