Work / Platform engineering

Ingestion, deployment and monitoring for a data platform shared by several business units

Working with Finance and Retail business units, I built metadata-driven streaming ingestion, automated deployments, and the monitoring that tells teams when data is late or wrong.

Client
Reliance Industries
Company
Celebal Technologies
Role
Data consultant
Focus
Platform engineering
50%fewer manual deployment approvals
4+business units on the monitoring framework
95%+pod delivery commitments maintained

Metadata-driven Kafka ingestion

A Kafka ingestion pipeline driven by metadata, built together with several teams, so bringing in a new source is a configuration change rather than a new pipeline.

Deployments through Databricks Asset Bundles

I implemented Databricks Asset Bundles with Git workflows, which reduced manual deployment approvals by 50%.

Ingestion trend monitoring

A centralised framework for monitoring ingestion trends, adopted by more than four business units.

Daily data quality reports

Automated daily data quality reports, delivered to the teams that need them through Microsoft Teams.

Delivery

I worked across Finance and Retail business units and helped keep pod delivery commitments above 95%.

Metadata in, pipelines outMetadatasource configKafkaFinanceRetailMonitoring
Next case studyAutomated IntelOps: monitoring that runs itself