Work / Migration engineering
Migration accelerators from Splunk and Snowflake to ClickHouse
I built two migration accelerators, Splunk to ClickHouse and Snowflake to ClickHouse. Each one runs a migration end to end: discover what exists, map and convert it, deploy it, load the data, test it, and report on it.
- Type
- Independent build
- Role
- Builder
- Focus
- Migration engineering
The workflow
- Connect: link to the source system using its connection details.
- Discover: fetch every object, SPL query and lookup table in the source, and let the user select what to migrate.
- Lineage: build a lineage of the selected objects, showing how they relate to each other.
- Column mapping: compare the current column mapping side by side for each object. A wrongly mapped data type can be changed, and every change requires a name and a justification, so there is a record of who changed what and why.
- Code conversion: show the original Splunk code next to the converted ClickHouse SQL.
- Deployment: take the ClickHouse connection details on the same page and push the table DDLs to ClickHouse. Right after deployment, a single Load data button loads the data.
- Testing: run two kinds of test cases. One set checks the DDLs and the converted code. The other reconciles data between Splunk and ClickHouse.
- Migration report: close the migration with a report.
The Snowflake version
The Snowflake to ClickHouse accelerator follows the same workflow, with Snowflake as the source system.
What I care about in it
- Lineage before conversion, so the order and dependencies of a migration are visible before any code changes.
- An audit trail on every manual data type override, so the migration stays explainable after the fact.
- Testing and reconciliation inside the workflow, rather than as a separate exercise after launch.