Skip to content

Library · Data Practice OS

ADR-005: Governed model access through an AI gateway

Decision record · Proposed · Source reviewed

Status: proposed.

Context

Teams want to use LLMs over the organisation data. Without a decision, they will each hold API keys to different providers, send data to unknown regions and have no shared logging. Anthropic Claude and other models are available inside Snowflake Cortex AI, through Databricks Model Serving (Databricks AI (formerly Mosaic AI)) and through Amazon Bedrock, so the choice of model does not force the choice of platform.

Decision drivers

  • Data stays in the governance boundary that already protects it
  • Processing in Australian regions (Data residency), especially for SENSITIVE data and Personal information
  • One audit trail of prompts, responses, users and cost
  • Freedom to change models without rewriting applications
  • Evidence of quality before release (LLM evaluation)

Options considered

Option For Against
A. Direct provider APIs per team Fast to start Keys sprawl; no shared logging, guardrails or residency control. Rejected
B. In-platform models only (Cortex / Model Serving) Data never leaves the platform; inherits platform RBAC The model list depends on what the platform offers in-region
C. Central gateway, preferring the in-platform route One policy point; models can be swapped; full logging Gateway configuration and ownership needed

Decision

Option C:

  1. Model calls go to the platform holding the data first: Cortex for Snowflake-resident data, Databricks Model Serving and Unity Gateway (formerly AI Gateway) for Databricks-resident data. Bedrock serves workloads outside both.
  2. All calls are logged with user, use case id (AI use case intake and risk triage), model, tokens and cost. Usage is enforced through the platform's gateway or AI gateway controls.
  3. Approved models list per classification level, maintained by the AI governance group.
  4. Routing is pinned to Australia and set explicitly. Snowflake accounts set CORTEX_ENABLED_CROSS_REGION = 'AWS_AU' (Sydney has no native Claude models, so DISABLED would rule Claude out). Bedrock uses Australian geographic inference profiles (au. model IDs). Databricks cross-geo processing stays off. Wider routing (APJ, global) may be approved for OFFICIAL, non-personal data per use case, never for SENSITIVE data or personal information. Defaults are not trusted: new Snowflake organisations default to ANY_REGION. Verify each model's availability under the chosen routing in the Cortex model availability and Databricks foundation model pages before committing a use case.
  5. Guardrails (PII redaction, topic limits, output checks) are configured per use case.

Consequences

  • Some models will not be available in Australian regions at first. Use cases wait or use an approved in-region model, and the trade-off is recorded.
  • Evaluation sets are a release gate for every production use case (Evaluating AI systems).
  • Cost per use case is visible, which feeds FinOps.
  • Gateway patterns are maintained in AI gateway and model access.

Revisit when

  • Vendors change their in-region model availability or residency terms
  • Government, regulator or corporate AI policy changes its requirements

Amendment from options-paper evidence (proposed)

  • Embedded model calls are in scope. Managed memory, knowledge-base and document services make their own model calls. Each use case inventories them and confirms their model and region, even though they do not pass through the gateway.
  • Residency distinguishes geography from globality. In-country cross-region routing (for example Bedrock au. profiles, Cortex AWS_AU) may be allowed for SENSITIVE data after a dated check. Global routing is not.
  • Until this ADR is accepted, pages that cite it say "(proposed)". See Decision register.