Work / Platform engineering
Ingestion, deployment and monitoring for a data platform shared by several business units
Working with Finance and Retail business units, I built metadata-driven streaming ingestion, automated deployments, and the monitoring that tells teams when data is late or wrong.
- Client
- Reliance Industries
- Company
- Celebal Technologies
- Role
- Data consultant
- Focus
- Platform engineering
50%fewer manual deployment approvals
4+business units on the monitoring framework
95%+pod delivery commitments maintained
Metadata-driven Kafka ingestion
A Kafka ingestion pipeline driven by metadata, built together with several teams, so bringing in a new source is a configuration change rather than a new pipeline.
Deployments through Databricks Asset Bundles
I implemented Databricks Asset Bundles with Git workflows, which reduced manual deployment approvals by 50%.
Ingestion trend monitoring
A centralised framework for monitoring ingestion trends, adopted by more than four business units.
Daily data quality reports
Automated daily data quality reports, delivered to the teams that need them through Microsoft Teams.
Delivery
I worked across Finance and Retail business units and helped keep pod delivery commitments above 95%.