3+ years of experience in related engineering disciplines
Experience monitoring production AI or machine learning systems
Strong programming skills in Python and Bash
Proficiency in Linux administration
Hands-on experience with Docker and Kubernetes
Familiarity with cloud platforms (AWS, Azure, GCP)
Experience with observability tools like Prometheus and Grafana
Responsibilities
Design and implement monitoring frameworks for AI applications
Monitor model performance, prediction quality, and availability
Detect and analyze data and model drift
Build dashboards and alerts for AI system health
Develop SLIs, SLOs, and operational metrics for AI services
Implement logging, tracing, and telemetry for applications
Collaborate with teams to improve model reliability
Benefits
Hybrid/remote work options
Collaborative team environment
Opportunities for continuous learning and professional development
Engagement with cutting-edge AI technologies
Contributions to high-impact projects in AI systems
Full Job Description
Job Title: AI Monitoring Engineer
Job Summary
We are seeking an AI Monitoring Engineer to design, implement, and manage monitoring solutions for AI, machine learning, and Generative AI systems. The ideal candidate will ensure the availability, reliability, performance, and quality of AI services by developing observability frameworks, monitoring model behavior, detecting anomalies, and automating alerts and incident response. This role collaborates closely with AI Engineers, ML Engineers, Data Scientists, DevOps Engineers, and Platform Engineers to maintain high-performing AI systems in production.
Key Responsibilities
Design and implement monitoring frameworks for AI applications, machine learning models, and AI infrastructure.
Monitor model performance, prediction quality, latency, throughput, and availability.
Detect and analyze data drift, model drift, concept drift, and inference anomalies.
Build dashboards and alerts for AI system health using observability tools.
Develop Service Level Indicators (SLIs), Service Level Objectives (SLOs), and operational metrics for AI services.
Implement logging, distributed tracing, and telemetry for AI applications.
Configure automated alerting and incident response workflows.
Monitor GPU, CPU, memory, storage, and network utilization for AI workloads.
Analyze AI system performance trends and recommend optimization strategies.
Collaborate with AI Engineers and Data Scientists to improve model reliability and operational performance.
Support root cause analysis (RCA) for production incidents and implement preventive measures.
Maintain monitoring documentation, operational runbooks, and reporting dashboards.
Ensure compliance with monitoring, security, governance, and operational standards.
Required Qualifications
Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, Data Science, or a related field.
3+ years of experience in Monitoring Engineering, Site Reliability Engineering (SRE), DevOps, Platform Engineering, MLOps, or Cloud Operations.
Experience monitoring production AI or machine learning systems.
Strong programming skills in Python and scripting with Bash.
Experience with Linux administration.
Hands-on experience with Docker and Kubernetes.
Experience with AWS, Microsoft Azure, or Google Cloud Platform.
Strong understanding of distributed systems, cloud infrastructure, and networking.
Experience with observability tools such as Prometheus, Grafana, OpenTelemetry, ELK Stack, Datadog, Splunk, or New Relic.
Experience with SQL and time-series databases for operational reporting.
Preferred Qualifications
Experience with machine learning monitoring platforms such as Evidently AI, Arize AI, Fiddler AI, WhyLabs, or Azure AI monitoring capabilities.
Experience monitoring Large Language Models (LLMs) and Generative AI applications.
Knowledge of AI evaluation metrics, prompt monitoring, hallucination detection, and Retrieval-Augmented Generation (RAG) observability.
Experience with MLOps platforms such as MLflow, Kubeflow, SageMaker, Vertex AI, or Azure Machine Learning.
Familiarity with vector databases and model serving platforms.
Experience monitoring GPU-enabled infrastructure and AI inference systems.
Knowledge of Responsible AI, fairness monitoring, and AI governance.
Technical Skills
Python
Bash
SQL
Linux
Docker
Kubernetes
AWS / Azure / Google Cloud Platform
Prometheus
Grafana
OpenTelemetry
ELK Stack
Datadog
Splunk
New Relic
MLflow
Kubeflow
SageMaker
Vertex AI
Azure Machine Learning
Evidently AI
Arize AI
Fiddler AI
WhyLabs
PostgreSQL
Redis
Soft Skills
Strong analytical and troubleshooting skills
Excellent communication and collaboration
Attention to detail
Problem-solving mindset
Ownership and accountability
Continuous learning and improvement
Nice to Have
Experience monitoring LLMs, AI agents, and Retrieval-Augmented Generation (RAG) systems
Knowledge of AI evaluation frameworks and model quality assessment
Experience implementing automated anomaly detection and predictive alerting
Familiarity with FinOps and AI infrastructure cost monitoring
Cloud, observability, or Kubernetes certifications
Key Performance Indicators (KPIs)
AI service availability and uptime
Model inference latency and throughput
Alert accuracy and mean time to detect (MTTD)
Mean time to resolve (MTTR) production incidents
Model drift detection and response time
Dashboard coverage for AI services
Reduction in production incidents and false alerts
Infrastructure resource utilization and optimization
