Job Title AI Evaluation Engineer Location Hybrid / Remote
Employment Type Full-time
Job Summary We are seeking an AI Evaluation Engineer to design, implement, and maintain evaluation frameworks for AI and machine learning systems, with a focus on Large Language Models (LLMs) and generative AI applications. The ideal candidate will develop robust evaluation methodologies, assess model performance across quality and safety dimensions, and collaborate with AI engineers, data scientists, and product teams to improve AI system reliability and user experience.
Key Responsibilities - Design and implement evaluation frameworks for AI, machine learning, and generative AI systems.
- Develop automated and manual evaluation pipelines to assess model quality and performance.
- Define evaluation metrics for accuracy, relevance, factuality, consistency, completeness, latency, and user satisfaction.
- Create benchmark datasets, test suites, and evaluation scenarios for AI models.
- Evaluate LLMs and AI applications for hallucinations, bias, toxicity, fairness, robustness, and safety.
- Measure Retrieval-Augmented Generation (RAG) performance, including retrieval quality and response grounding.
- Conduct A/B testing and comparative evaluations of models, prompts, and AI workflows.
- Analyze evaluation results and provide actionable recommendations for model improvement.
- Collaborate with AI engineers, data scientists, product managers, and QA teams throughout the AI development lifecycle.
- Monitor production AI systems and identify performance degradation, data drift, and model drift.
- Document evaluation methodologies, findings, and best practices.
- Ensure compliance with organizational AI governance, privacy, security, and responsible AI policies.
Required Qualifications - Bachelor's degree in Computer Science, Artificial Intelligence, Data Science, Statistics, Mathematics, Software Engineering, or a related field.
- 3-5+ years of experience in AI, machine learning, software engineering, data science, or model evaluation.
- Strong understanding of machine learning concepts and evaluation methodologies.
- Experience evaluating AI or generative AI applications.
- Proficiency in Python and SQL.
- Experience with REST APIs and cloud-based applications.
Preferred Qualifications - Master's degree in AI, Machine Learning, Data Science, Computer Science, or a related field.
- Experience evaluating Large Language Models (LLMs) and Retrieval-Augmented Generation (RAG) systems.
- Knowledge of Responsible AI, AI safety, and AI governance principles.
- Experience with MLOps and model lifecycle management.
- AI or cloud certifications (AWS, Azure, Google Cloud).
Technical Skills - Python
- SQL
- Machine learning fundamentals
- LLM evaluation methodologies
- Prompt engineering
- RAG evaluation
- AI benchmarking techniques
- Statistical analysis
- Data visualization (Power BI, Tableau, Matplotlib)
- Git and CI/CD
- REST APIs
- JSON
- AI evaluation frameworks (DeepEval, Ragas, LangSmith, Promptfoo)
- MLflow
- Docker and Kubernetes
- Cloud platforms (AWS, Azure, Google Cloud)
Soft Skills - Analytical thinking
- Critical thinking
- Problem-solving
- Strong communication skills
- Technical documentation
- Collaboration
- Attention to detail
- Time management
- Continuous learning
Preferred Experience - Generative AI applications
- Conversational AI and chatbots
- Enterprise AI platforms
- Retrieval-Augmented Generation (RAG)
- Machine learning model validation
- AI-powered SaaS applications
- Regulated industries such as healthcare, finance, or insurance
Success Metrics - AI evaluation coverage
- Benchmark quality and completeness
- Improvement in model accuracy and reliability
- Reduction in hallucinations and unsafe responses
- Evaluation pipeline automation
- Detection of model regressions before release
- Production AI performance and stability
- Stakeholder satisfaction
- Compliance with Responsible AI and governance standards
Nice-to-Have Skills - Explainable AI (XAI)
- Model monitoring and observability tools
- Synthetic data generation
- Vector databases
- LangChain or similar AI orchestration frameworks
- Data annotation tools
- Experiment tracking platforms
- A/B testing methodologies
- AI red teaming
- Familiarity with standards such as NIST AI Risk Management Framework (AI RMF) or ISO/IEC 42001
