Work / Migration engineering

Migration accelerators from Splunk and Snowflake to ClickHouse

I built two migration accelerators, Splunk to ClickHouse and Snowflake to ClickHouse. Each one runs a migration end to end: discover what exists, map and convert it, deploy it, load the data, test it, and report on it.

Type
Independent build
Role
Builder
Focus
Migration engineering

The workflow

  1. Connect: link to the source system using its connection details.
  2. Discover: fetch every object, SPL query and lookup table in the source, and let the user select what to migrate.
  3. Lineage: build a lineage of the selected objects, showing how they relate to each other.
  4. Column mapping: compare the current column mapping side by side for each object. A wrongly mapped data type can be changed, and every change requires a name and a justification, so there is a record of who changed what and why.
  5. Code conversion: show the original Splunk code next to the converted ClickHouse SQL.
  6. Deployment: take the ClickHouse connection details on the same page and push the table DDLs to ClickHouse. Right after deployment, a single Load data button loads the data.
  7. Testing: run two kinds of test cases. One set checks the DDLs and the converted code. The other reconciles data between Splunk and ClickHouse.
  8. Migration report: close the migration with a report.

The Snowflake version

The Snowflake to ClickHouse accelerator follows the same workflow, with Snowflake as the source system.

What I care about in it

  • Lineage before conversion, so the order and dependencies of a migration are visible before any code changes.
  • An audit trail on every manual data type override, so the migration stays explainable after the fact.
  • Testing and reconciliation inside the workflow, rather than as a separate exercise after launch.
Source to ClickHouse, end to end01Connect02Discover03Lineage04Map columns05Convert06Deploy and load07Test and reconcile08ReportSplunk or SnowflakeClickHouse
Next case studyIngestion, deployment and monitoring for a data platform shared by several business units