Job TitlePrompt EvaluatorLocationHybrid / Remote
Employment TypeFull-time
Job SummaryWe are seeking a Prompt Evaluator to assess, analyze, and improve the quality of prompts and AI-generated responses for Large Language Model (LLM) applications. The ideal candidate will evaluate prompt effectiveness, response quality, accuracy, safety, and consistency while collaborating with AI engineers, prompt engineers, product teams, and data scientists to optimize AI system performance and user experience.
Key Responsibilities- Evaluate prompts and AI-generated responses for quality, relevance, accuracy, completeness, and consistency.
- Assess responses for factual correctness, reasoning quality, and adherence to instructions.
- Identify issues such as hallucinations, bias, toxicity, prompt injection, and unsafe outputs.
- Develop evaluation rubrics, scoring guidelines, and quality benchmarks for prompt performance.
- Conduct manual and automated evaluations across diverse use cases and domains.
- Compare prompt variations and perform A/B testing to identify optimal prompt strategies.
- Analyze evaluation results and provide actionable recommendations to improve prompts and model outputs.
- Collaborate with Prompt Engineers, AI Engineers, Data Scientists, and Product Managers to refine AI applications.
- Create benchmark datasets and test scenarios for prompt evaluation.
- Document evaluation findings, trends, and best practices.
- Support Responsible AI initiatives by ensuring outputs meet ethical, safety, and compliance standards.
- Monitor prompt performance after deployment and recommend continuous improvements.
Required Qualifications- Bachelor's degree in Computer Science, Artificial Intelligence, Data Science, Linguistics, English, Psychology, Human-Computer Interaction, or a related field.
- 2-5+ years of experience in AI evaluation, quality assurance, content evaluation, prompt engineering, or a related role.
- Strong written communication and analytical skills.
- Experience working with Generative AI or Large Language Models (LLMs).
- Ability to evaluate content objectively using defined quality criteria.
Preferred Qualifications- Experience evaluating prompts for conversational AI or enterprise AI applications.
- Knowledge of prompt engineering techniques and LLM behavior.
- Familiarity with Responsible AI, AI safety, and model evaluation concepts.
- Experience with AI evaluation tools and benchmarking frameworks.
- AI or cloud certifications are a plus.
Technical Skills- Prompt engineering fundamentals
- LLM evaluation methodologies
- Generative AI concepts
- AI response quality assessment
- Prompt optimization
- Python (basic to intermediate)
- SQL (basic)
- JSON
- REST APIs
- AI evaluation frameworks (DeepEval, Ragas, LangSmith, Promptfoo)
- Spreadsheet and data analysis tools (Excel, Google Sheets)
- Data visualization (Power BI, Tableau) - preferred
- Git - preferred
Soft Skills- Critical thinking
- Strong analytical skills
- Excellent written communication
- Attention to detail
- Problem-solving
- Collaboration
- Curiosity and continuous learning
- Time management
- Decision-making
Preferred Experience- Large Language Model (LLM) applications
- Conversational AI and chatbots
- Retrieval-Augmented Generation (RAG) systems
- AI content evaluation
- Human-in-the-loop (HITL) workflows
- AI quality assurance
- Enterprise AI products
Success Metrics- Prompt effectiveness and response quality scores
- Reduction in hallucinations and unsafe outputs
- Improvement in factual accuracy and instruction adherence
- Evaluation throughput and consistency
- Detection of prompt-related issues before production
- Stakeholder satisfaction
- Benchmark coverage and quality
- Continuous improvement in AI performance
Nice-to-Have Skills- Experience with AI red teaming
- Knowledge of AI safety and alignment techniques
- Familiarity with NIST AI Risk Management Framework (AI RMF) or ISO/IEC 42001
- Experience evaluating multilingual AI responses
- Data annotation and labeling
- Experiment design and A/B testing
- Basic understanding of machine learning concepts
- Experience with vector databases and RAG evaluation
