Skip to content

Library · Data Practice OS

AI-native patterns

Reference guide · Published · Source reviewed

"AI-native" means AI is part of how data is processed and used, not a chatbot bolted onto a report. Six patterns cover most of the work. Pick the simplest pattern that solves the problem: a pipeline function is cheaper and safer than an agent.

The six patterns, simplest first

1. LLM functions in pipelines

Run a model over rows or documents inside the pipeline: classify complaint emails, extract fields from scanned forms, summarise inspection notes, translate, score sentiment. Output lands in a governed table like any other column. See LLM functions in SQL.

  • Snowflake: Cortex AI SQL functions (AI_CLASSIFY, AI_EXTRACT, AI_COMPLETE and others). Databricks: Databricks AI Functions (ai_query, ai_extract and others).
  • Use when: volume is high, the task is narrow, and you can evaluate on a labelled sample.
  • Worked example: classify road-defect reports from the public by asset type and urgency. Route the results to the right district.

2. Search and RAG over documents

Index policies, procedures, technical standards and correspondence. Answer questions with citations. See Retrieval-augmented generation and RAG and enterprise search.

3. Text-to-SQL over governed data

Business users ask questions in plain English, and the system writes SQL against curated data products. See Text-to-SQL.

  • Snowflake: Cortex Analyst over semantic views. Databricks: AI/BI Genie over curated tables and metric views.
  • This only works on a good Semantic layer. Expect poor answers on raw tables. See The semantic layer.
  • Use when: questions are about metrics and dimensions you have defined, for users who would otherwise wait for a report.
  • Worked example: "How many heavy-vehicle registrations lapsed in the Wide Bay district last quarter, compared with the year before?"

4. Agents

An LLM that plans, calls tools (text-to-SQL, search, APIs) and checks its work to complete a multi-step task. See AI agent and Agents and MCP.

  • Snowflake: Cortex Agents and Snowflake Intelligence and Snowflake Intelligence. Databricks: Agent Bricks, the Databricks agent developer platform. Either way, expose tools through Model Context Protocol so other assistants can reuse them.
  • Use when: the task needs both structured and unstructured data, or several steps. Not when a single query or search would do.
  • Constrain tightly: scoped tools, run as the user, step and cost budgets, Human in the loop for actions.
  • Worked example: an asset-planning assistant that pulls condition data for a corridor, retrieves the relevant maintenance standard, and drafts an intervention summary for an engineer to review.

Separate read, recommend, approve and execute. For decision support and any agent that acts, grant these four permissions separately, and verify each one:

Permission The AI may Typical holder
Read Retrieve the facts, rules and case state it needs, as the user The assistant, scoped to the user's access
Recommend Propose an outcome, citing the facts and the rule version The assistant
Approve Accept the recommendation An officer whose delegation covers it, never the AI by default
Execute Change a system of record or trigger an action A workflow step after approval, under its own scoped identity

An identity that can recommend must not be able to approve or execute. A request above the officer's delegation routes to someone who holds it. This is the context sufficiency gate in AI readiness assessment (proposed; not an agreed decision).

5. Predictive ML

Classical models are still the right tool for forecasting and risk scoring: crash-risk hotspots, pavement deterioration, demand forecasting, fraud signals. See Feature store and Model registry.

  • Databricks: the deepest toolset (MLflow, feature engineering, GPUs). Snowflake: Snowflake ML (Feature Store, Model Registry, container runtime).
  • Use when: you have labelled history and a numeric or categorical target. Do not use an LLM for this.

6. AI-assisted engineering

AI helps build and run the platform itself: generating dbt models and tests, documenting columns, converting legacy code, explaining reconciliation diffs, triaging data-quality alerts. This often pays back fastest. See Code conversion.

Cross-cutting rules for every pattern

  1. Evaluation set first. Before building, write 30-100 representative questions or inputs with expected answers. That set is the release gate (LLM evaluation, Evaluating AI systems).
  2. Model calls through an AI gateway, using approved, in-region models (ADR-005: Governed model access through an AI gateway).
  3. Same permissions as a person. AI queries run with the user's identity or a scoped role.
  4. Guardrails on inputs and outputs, proportionate to the risk tier.
  5. Trace everything. Prompts, tool calls, retrieved context and outputs are logged for audit and improvement.
  6. Cost per answer is a metric. Track it from the pilot onward.

Choosing between platforms for AI

If both platforms are in play, a reasonable default (ADR-001: One platform by default, two only by workload):

  • Business-user Q&A, RAG over governed content, LLM functions in SQL pipelines → wherever the data products live (often Snowflake in a split).
  • Custom agents with a heavy engineering and evaluation lifecycle, predictive ML, large-scale unstructured processing → Databricks.

Anthropic Claude is available on both, so the model does not force the choice.

Status of these rules

Rule 2 (model calls through a gateway) rests on ADR-005: Governed model access through an AI gateway, which is proposed, not accepted. Treat it as the recommended standard until the ADR is approved (Decision register). For agents that run outside the data platforms, see the platform team lessons in Agents and MCP.