Work / Engineering leverage
Automated IntelOps: monitoring that runs itself
On an on-premise Hortonworks Data Platform, I built Automated IntelOps. It checks every job about once a minute, diagnoses failures from the logs, raises ServiceNow tickets by priority, and shows everything on one Grafana dashboard.
- Company
- Infosys
- Role
- Specialist programmer (data)
- Focus
- Engineering leverage
90%+manual monitoring tasks automated
~70%lower manpower requirement
~20%better data accuracy
~3 hrsof reporting saved per day
The problem
Each business unit had two or three people monitoring jobs across production and DR, on an on-premise Hortonworks Data Platform. The routine was entirely manual:
- Every job had to be looked up by hand in Apache Ambari, and the exact job name had to be typed in for the search to work.
- Even once a job was found, its status was hard to read and understand.
- When a job failed, the same person had to log in to the jump server, dig into the task level, and diagnose the failure by hand. Slow and tedious, every time.
What Automated IntelOps does
- Runs roughly every minute, pulls the latest status of every job, and pushes it to a Grafana dashboard.
- Shows, for each job: name, upstream and downstream jobs, average run time, current running time, status, whether the job is active, the error code of the failed task, and the ServiceNow ticket number if it failed.
- On a failure, reads the config file to find the current script and its logs, checks what went wrong, and raises a ServiceNow ticket through the ServiceNow API, based on the job's priority.
- Sends scheduled daily email reports to stakeholders.
The outcome
More than 90% of the manual monitoring work was automated. The manpower needed for monitoring dropped by about 70%, removing manual steps improved data accuracy by about 20%, and reporting effort fell by about three hours a day. It won an Infosys hackathon, recognised for its automation coverage and its monitoring dashboard.