Job Title AI QA Engineer Location Hybrid / Remote
Employment Type Full-time
Job Summary We are seeking an AI QA Engineer to ensure the quality, reliability, safety, and performance of AI-powered applications and machine learning solutions. The ideal candidate will design and execute testing strategies for AI systems, including large language model (LLM) applications, generative AI features, machine learning models, and AI-driven workflows. This role requires expertise in traditional software quality assurance combined with an understanding of AI model behavior, prompt engineering, and automated testing.
Key Responsibilities - Design and implement test strategies for AI and machine learning applications.
- Develop and maintain automated test frameworks for AI-powered systems.
- Validate LLM responses for accuracy, consistency, relevance, and safety.
- Create and manage test datasets, prompts, and evaluation benchmarks.
- Test AI features for functionality, usability, performance, security, and reliability.
- Identify, document, and track defects using bug management tools.
- Perform regression, integration, API, end-to-end, and performance testing.
- Evaluate AI systems for hallucinations, bias, toxicity, and prompt injection vulnerabilities.
- Collaborate with AI engineers, data scientists, product managers, and developers to resolve issues.
- Analyze model outputs and recommend improvements to prompts or workflows.
- Monitor AI application performance after deployment.
- Ensure compliance with organizational quality standards and AI governance policies.
Required Qualifications - Bachelor's degree in Computer Science, Information Technology, Software Engineering, or a related field.
- 3-5+ years of experience in software quality assurance or test automation.
- Experience testing web, mobile, or API-based applications.
- Familiarity with AI, machine learning, or generative AI applications.
- Strong understanding of software testing methodologies and QA best practices.
- Experience with Agile or Scrum development environments.
Preferred Qualifications - Experience testing LLM-based or generative AI applications.
- Knowledge of AI evaluation frameworks and benchmarking techniques.
- Familiarity with prompt engineering and AI model behavior.
- AI or cloud certifications (AWS, Microsoft Azure, Google Cloud, etc.).
- Experience with MLOps or AI deployment pipelines.
Technical Skills - Manual and automated testing
- Test case design and execution
- API testing (Postman, REST Assured)
- Automation tools (Selenium, Playwright, Cypress)
- Programming (Python, Java, JavaScript, or C#)
- SQL and database validation
- Git and CI/CD pipelines
- Test management tools (Jira, TestRail, Zephyr)
- Performance testing (JMeter, k6)
- AI model evaluation techniques
- Prompt engineering
- LLM testing and validation
- AI safety testing (hallucinations, bias, toxicity, prompt injection)
- Basic machine learning concepts
- JSON and REST APIs
Soft Skills - Analytical thinking
- Strong attention to detail
- Problem-solving
- Communication and documentation
- Collaboration with cross-functional teams
- Critical thinking
- Continuous learning
- Time management
Preferred Experience - AI-powered enterprise applications
- Generative AI products
- Machine learning platforms
- SaaS applications
- Conversational AI or chatbot testing
- Cloud-native applications
Success Metrics - Defect detection rate
- Test automation coverage
- AI response quality and accuracy
- Reduction in production defects
- Test execution efficiency
- AI safety and compliance metrics
- Release quality
- Customer satisfaction
- Mean time to detect (MTTD) and resolve (MTTR) issues
Nice-to-Have Skills - LangChain or similar AI orchestration frameworks
- AI evaluation frameworks (e.g., DeepEval, Ragas, LangSmith)
- Vector databases
- Docker and Kubernetes
- MLOps tools
- Cloud platforms (AWS, Azure, Google Cloud)
- Data annotation and synthetic data generation
- Security testing for AI systems
- Accessibility testing
- Experience with Retrieval-Augmented Generation (RAG) systems
