Library · Data Practice OS
Modernisation and migration
Reference guide · Published · Source reviewed
Modernisation is not moving the old warehouse into a new one. It means retiring the Technical debt, keeping the Business rules, and landing the result as AI-ready data products. The legacy source is not decided yet, so this method is source-agnostic. Each source type adds its own wrinkles (see the table at the end).
The method
Assess -> Extract rules -> Choose pattern -> Convert -> Reconcile -> Parallel run -> Cut over -> Decommission
(what) (why) (per workload) (how) (prove) (earn trust) (switch) (actually)
1. Assess. Inventory sources, ETL jobs, reports, consumers, volumes, schedules and owners. Find dead code and unused reports. Typically 20-40% of an estate is retired at this step, which is the cheapest migration there is. See Migration assessment. Run SnowConvert AI and Lakebridge assessments: both are free and both size the conversion.
2. Extract business rules. Mine the calculations, validations and policies out of stored procedures, ETL mappings and application code into Given/When/Then specifications with source citations. These become the acceptance tests for the new pipelines. This is the highest-leverage step, and AI-assisted extraction makes it tractable. See Extracting business rules.
3. Choose a Migration pattern per workload.
| Pattern | When | Example |
|---|---|---|
| Retire | No consumers, or superseded | Unused monthly extracts |
| Rehost (lift as-is) | Stable, low-value, short remaining life | Archive reporting tables |
| Replatform (convert code, keep the design) | Sound design, legacy dialect | SQL Server procs to Snowflake SQL or Spark SQL |
| Refactor (redesign as dbt data products) | High value, heavy reuse, AI use planned | Registration and licensing marts |
| Rebuild (from extracted intent) | Logic tangled in application code, e.g. CA Gen | Fee calculation rules |
Most estates end up mostly replatform + refactor, with a long tail of retire.
4. Convert. Use vendor tooling for the mechanical bulk and LLM-assisted conversion for the rest, always against the tests from step 2. Target dbt models, not stored procedures. See Code conversion.
5. Reconcile. Prove equivalence: row counts, column checksums, business aggregates and sampled record-level diffs. Automate it, because it runs on every load during the parallel run. Differences are triaged as defect, legacy bug (documented and accepted) or intended change. See Reconciliation, parallel run and cutover. To trace each rule from legacy to modern code and its tests, see Tracing rules from legacy code to modern code (draft).
6. Parallel run. Run old and new side by side on live data for an agreed period, for example two month-end cycles, reconciling every run. Exit criteria are agreed up front with data owners.
7. Cut over, then decommission. Repoint consumers, freeze legacy writes, keep read-only access for an agreed tail, then switch it off. If the legacy is not switched off, the migration has added a platform rather than replaced one.
Waves, not big bang
Group workloads into waves of 6-12 weeks. Each wave delivers working data products to real users. Sequence by value x readiness, with dependencies first. The first wave should be valuable but not mission-critical, so the method can be proven. Use Migration wave plan.
Apply the Strangler fig pattern: new consumers go to the new platform from day one, and Change data capture keeps it current from the legacy source while consumers move across.
Where AI helps (and where it does not)
| Helps a lot | Needs care | Do not |
|---|---|---|
| Summarising and documenting legacy code | Converting complex procedural logic: test everything | Let an LLM decide a reconciliation difference is "fine" |
| Drafting Given/When/Then rules with citations | Inferring intent from badly named code: confirm with SMEs | Skip the human review of P0 rules |
| Generating dbt tests from rules | Column-mapping suggestions | Convert without a test oracle |
| Explaining reconciliation diffs |
Notes by source type
| Source | What to watch |
|---|---|
| CA Gen / DB2 mainframe | Logic is in action blocks, not SQL. Extract rules from generated code and models. Use Change data capture from DB2 for coexistence. Watch EBCDIC, packed decimals and date conventions |
| SQL Server / Oracle EDW + SSIS / Informatica | Best tool support: SnowConvert AI (SQL, SSIS, Informatica PowerCenter) and Lakebridge (SQL, SSIS; no Informatica). Procedures and ETL mappings convert well. Reports often encode metrics, so move those into the semantic layer |
| Hadoop / existing lake | Data may already be Parquet. Converting to Iceberg or Delta can be in place. Hive and Spark jobs map naturally to Databricks or Snowpark |
The handbook section on migration has the detail: Migration assessment, Extracting business rules, Code conversion, Reconciliation, parallel run and cutover.