Job Title: Language Model EvaluatorJob Summary We are seeking a Language Model Evaluator to assess, benchmark, and improve the quality, accuracy, safety, and reliability of Large Language Models (LLMs) and Generative AI applications. The ideal candidate will evaluate model performance using automated metrics, human evaluations, and structured testing methodologies while collaborating with AI Engineers, Data Scientists, Prompt Engineers, and Product teams to enhance model behavior and user experience.
Key Responsibilities - Design and execute evaluation frameworks for Large Language Models (LLMs) and Generative AI applications.
- Assess model outputs for accuracy, relevance, factual consistency, reasoning quality, coherence, fluency, and completeness.
- Evaluate model performance across diverse domains, languages, and real-world use cases.
- Develop benchmark datasets, evaluation prompts, and test scenarios for model validation.
- Measure and monitor hallucinations, bias, toxicity, harmful content, and safety risks.
- Perform prompt testing, regression testing, and comparative evaluations across different models and versions.
- Analyze model performance using quantitative metrics and human evaluation methodologies.
- Collaborate with AI Engineers and Prompt Engineers to improve prompts, retrieval pipelines, and model responses.
- Evaluate Retrieval-Augmented Generation (RAG) systems, AI agents, and tool-calling workflows.
- Document evaluation findings, recommendations, and quality improvement plans.
- Support model release decisions through structured quality assurance processes.
- Stay current with emerging LLM evaluation techniques, benchmarks, and Responsible AI practices.
Required Qualifications - Bachelor's or Master's degree in Computer Science, Artificial Intelligence, Data Science, Computational Linguistics, Statistics, or a related field.
- 2-5 years of experience in AI evaluation, NLP, machine learning, quality assurance, or data analysis.
- Strong understanding of Large Language Models (LLMs), Generative AI, and Natural Language Processing (NLP).
- Experience creating evaluation datasets, benchmarks, and test cases.
- Strong analytical skills with the ability to identify quality issues and failure patterns.
- Programming experience in Python and SQL.
- Familiarity with data analysis libraries such as Pandas and NumPy.
- Experience using spreadsheets, visualization tools, or notebooks for analysis.
- Excellent written communication and attention to detail.
Preferred Qualifications - Experience with LLM evaluation frameworks such as DeepEval, Ragas, LangSmith, Promptfoo, OpenAI Evals, or similar tools.
- Experience evaluating Retrieval-Augmented Generation (RAG), AI agents, and function-calling workflows.
- Knowledge of prompt engineering, model fine-tuning, and reinforcement learning from human feedback (RLHF) concepts.
- Familiarity with AI safety, red teaming, adversarial testing, bias detection, and Responsible AI principles.
- Experience with multilingual model evaluation.
- Knowledge of cloud AI platforms and MLOps workflows.
- Experience with annotation tools and human-in-the-loop evaluation processes.
Technical Skills - Python
- SQL
- Pandas
- NumPy
- Jupyter Notebook
- Large Language Models (LLMs)
- Natural Language Processing (NLP)
- Prompt Engineering
- DeepEval
- Ragas
- LangSmith
- Promptfoo
- OpenAI Evals
- Git
- REST APIs
- AWS / Azure / Google Cloud Platform (preferred)
Soft Skills - Strong analytical and critical thinking skills
- Exceptional attention to detail
- Excellent written and verbal communication
- Curiosity and a quality-first mindset
- Collaboration across technical and non-technical teams
- Ability to document findings clearly and objectively
Nice to Have - Experience evaluating multimodal AI models (text, image, audio, and video)
- Knowledge of AI governance, regulatory compliance, and model risk management
- Experience with statistical analysis and experimental design
- Familiarity with A/B testing and user experience evaluation
- Contributions to AI evaluation research or open-source projects
Key Performance Indicators (KPIs) - Evaluation coverage across models, prompts, and use cases
- Accuracy and consistency of evaluation results
- Reduction in hallucinations and factual errors
- Detection rate of safety, bias, and quality issues
- Turnaround time for evaluation cycles
- Improvement in benchmark scores across model releases
- Quality of evaluation documentation and actionable recommendations
- Support for successful production model releases
Location Hybrid / Remote / On-site (as applicable)
Employment Type Full-time
