Req ID: 386102
We are currently seeking a
Observability & Evaluation Engineer to join our team in
Charlotte,
North Carolina (US-NC),
United States (US).
Job Duties and Responsibilities:Observability & Evaluation Engineer will build telemetry, tracing, dashboards, evaluation suites, alerts, service objectives, runbooks, and readiness evidence for Tachyon agent releases. This role ensures production AI systems can be monitored, evaluated, improved, and supported with clear operational visibility.
Key Responsibilities- Implement observability and telemetry for LLM-powered applications, agents, tools, and platform services.
- Build evaluation suites for agent behavior, prompt quality, response quality, retrieval performance, latency, reliability, and safety signals.
- Develop dashboards, alerts, traces, metrics, service objectives, and reporting for production readiness.
- Work with platform engineers and Product Owners to define monitoring requirements and evaluation metrics.
- Automate evidence collection for release readiness, operational reviews, and governance checkpoints.
- Create runbooks and support documentation for priority agent releases.
- Analyze production behavior and recommend improvements to reliability, performance, and quality.
Required Qualifications- 7+ years of engineering experience with observability, monitoring, test automation, platform operations, or AI/ML systems.
- 5+ years of strong hands-on Python experience.
- 5+ years of Experience with dashboards, metrics, alerts, traces, logs, SLOs, and production monitoring.
- 5+ years of Understanding of LLM evaluation, prompt evaluation, RAG evaluation, or AI quality assessment approaches.
- 5+ years of Experience working in Agile engineering teams and production support environments.
Required Skills / Knowledge- Python, telemetry, tracing, monitoring, dashboards, alerting, SLOs, evaluation frameworks, test automation, and production operations.
- Understanding of LLMs, agents, RAG, prompt performance, retrieval quality, latency, and reliability metrics.
- Experience with observability tools and open telemetry concepts.
Preferred Qualifications- Experience with GenAI observability, AI evaluation tools, ML monitoring, or platform reliability engineering.
- Experience in regulated environments with evidence and readiness documentation.
- Kubernetes, cloud platforms, and CI/CD experience.
Expected Outcomes- Operational dashboards and evaluation suites for priority agent releases.
- Clear readiness evidence, alerts, SLOs, and runbooks.
- Improved quality, reliability, and trust in production Agentic AI systems.
#LI-NorthAmericaNTT DATA provides a reasonable range of compensation for U.S.-based positions. The starting pay range for this role is $96,800.00 - $145,200.00. Actual compensation will depend on a number of factors, including the candidate's relevant experience, technical skills, and other qualifications.
This position may also be eligible for incentive compensation based on individual and/or company performance.
This position is eligible for company benefits including medical, dental, and vision insurance with an employer contribution, flexible spending or health savings account, life and AD\&D insurance, short and long term disability coverage, paid time off, employee assistance, participation in a 401k program with company match, and additional voluntary or legally-required benefits.
