Role description

Job Summary

We are seeking an experienced AI Quality Engineer to validate and assure the quality of Generative AI applications RAG systems AI agents APIs and data pipelines The ideal candidate will have expertise in test automation AI evaluation frameworks prompt testing hallucination detection and performance validation to ensure AI solutions are accurate reliable safe and productionready

Key Responsibilities

Design and implement automated testing frameworks for AIGenAI applications

Validate LLM RAG and Agentbased systems for functionality accuracy reliability and safety

Develop and execute prompt testing retrieval testing and response quality validation

Perform hallucination testing guardrail validation and adversarialsecurity testing

Build automated API and integration test suites for AI services and data pipelines

Establish AI evaluation metrics and quality benchmarks using tools such as LangSmith and Galileo

Conduct regression performance load and scalability testing for AI solutions

Validate data quality retrieval accuracy and endtoend RAG workflows

Collaborate with AI engineers data engineers and platform teams to identify and resolve quality issues

Integrate test automation into CICD pipelines and support continuous quality monitoring

Required Skills

Strong proficiency in Python

Test Automation Framework Development

API Testing REST APIs Microservices

AI Evaluation Frameworks LangSmith Galileo

Prompt Testing Validation

RAG Testing Retrieval Validation

Hallucination Detection Response Evaluation

Guardrail Validation Responsible AI Testing

Regression Testing

Performance Load Testing

SQL and Database Testing

Git Jenkins CICD Pipelines

Preferred Skills

MongoDB and Oracle Databases

LangGraph AI Agent Testing

Linux and Shell Scripting

OpenShift Container Platforms

Security Adversarial Testing

Knowledge of Vector Databases and Embedding Validation

Experience with AI observability and monitoring tools