AI Observability Engineer

We are seeking an experienced AI Observability Engineer to design, implement, and manage enterprise-scale observability solutions for Machine Learning (ML), Generative AI, and Large Language Model (LLM) platforms. This role will be responsible for establishing end-to-end visibility into AI workloads by monitoring model performance, inference reliability, system health, latency, cost, data quality, drift, and user experience.

The ideal candidate will have strong expertise in modern observability ecosystems including Prometheus, Grafana, Thanos, and data visualization technologies, along with experience analyzing telemetry and operational data to improve the reliability, performance, and scalability of AI systems in production

Responsibilities

• Build and maintain observability solutions for AI/ML applications and platform services.
• Develop dashboards and alerts using Grafana and Tableau.
• Monitor AI service health, latency, throughput, error rates, and reliability metrics.
• Create and maintain metrics using Prometheus and PromQL.
• Analyze operational and time-series data to identify trends and performance issues.
• Define and monitor SLIs, SLOs, and service health indicators.
• Partner with AI/ML, Platform, and SRE teams to improve production reliability.
• Support incident troubleshooting and root cause analysis

Required Qualifications

Experience

• Proven years of experience in observability, monitoring, analytics, or telemetry engineering.
• Proven years of experience creating operational dashboards and visualizations.
• Experience supporting production-scale applications in cloud or distributed environments.
• Experience working with large-scale metrics and monitoring systems.

Preferred Skills

• Experience monitoring AI/ML or GenAI workloads.
• Familiarity with OpenTelemetry.
• Knowledge of model performance and data drift concepts.
• Experience working in Agile environments using Jira.

We are an Equal Opportunity Employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex (including pregnancy, sexual orientation, or gender identity), national origin, citizenship status, age, disability, genetic information, protected veteran status, or any other characteristic protected by applicable law.

https://www.e-verify.gov/sites/default/files/everify/posters/IER_RighttoWorkPoster.pdf