Job Title AI Testing Specialist Location Hybrid / Remote
Employment Type Full-time
Job Summary We are seeking an AI Testing Specialist to ensure the quality, reliability, security, and performance of AI-powered applications, machine learning models, and generative AI solutions. The ideal candidate will develop and execute comprehensive testing strategies for AI systems, validate model outputs, assess AI-specific risks, and collaborate with cross-functional teams to deliver high-quality AI products.
Key Responsibilities - Design and execute test strategies for AI, machine learning, and generative AI applications.
- Create and maintain test plans, test cases, and test data for AI features and workflows.
- Validate AI model outputs for accuracy, consistency, relevance, factuality, and reliability.
- Evaluate AI systems for hallucinations, bias, toxicity, fairness, and robustness.
- Perform functional, regression, integration, API, end-to-end, performance, usability, and security testing.
- Test prompt-based applications and optimize prompts for consistent results.
- Develop automated testing frameworks for AI applications and APIs.
- Verify data quality, preprocessing pipelines, and model inputs.
- Conduct stress, load, and scalability testing for AI services.
- Identify, document, prioritize, and track defects using bug management tools.
- Collaborate with AI engineers, data scientists, software developers, product managers, and UX teams.
- Monitor production AI systems and support continuous quality improvement.
- Prepare test reports, quality metrics, and release recommendations.
- Ensure compliance with organizational AI governance, security, privacy, and regulatory requirements.
Required Qualifications - Bachelor's degree in Computer Science, Information Technology, Software Engineering, Data Science, or a related field.
- 3-5+ years of experience in software testing, QA, or AI testing.
- Strong understanding of software testing methodologies and quality assurance principles.
- Experience testing APIs, web applications, and cloud-based systems.
- Familiarity with AI, machine learning, and generative AI concepts.
- Experience working in Agile or Scrum environments.
Preferred Qualifications - Experience testing Large Language Model (LLM) applications.
- Knowledge of prompt engineering and AI evaluation methodologies.
- Experience with Responsible AI practices and AI governance.
- AI, cloud, or software testing certifications.
- Experience with MLOps workflows and model lifecycle management.
Technical Skills - Manual and automated testing
- Test planning and execution
- API testing (Postman, REST Assured)
- Automation frameworks (Selenium, Playwright, Cypress)
- Programming (Python, Java, JavaScript, or C#)
- SQL and database validation
- Git and CI/CD tools
- Test management tools (Jira, TestRail, Zephyr)
- Performance testing (JMeter, k6, LoadRunner)
- AI model evaluation techniques
- Prompt engineering
- LLM testing and validation
- AI safety testing (hallucinations, bias, toxicity, prompt injection, jailbreak resistance)
- Data validation and preprocessing verification
- JSON, REST APIs, and cloud platforms (AWS, Azure, Google Cloud)
Soft Skills - Analytical and critical thinking
- Strong attention to detail
- Problem-solving
- Effective communication
- Collaboration across multidisciplinary teams
- Documentation and reporting
- Adaptability
- Time management
- Continuous learning
Preferred Experience - AI-powered enterprise applications
- Conversational AI and chatbots
- Generative AI products
- Machine learning platforms
- Retrieval-Augmented Generation (RAG) systems
- SaaS and cloud-native applications
- Healthcare, finance, retail, or other regulated industries
Success Metrics - Test coverage and automation coverage
- Defect detection and prevention rate
- AI response quality and reliability
- Reduction in production defects
- Model evaluation accuracy
- Compliance with AI quality and governance standards
- Release readiness and stability
- Customer satisfaction and user experience
- Test execution efficiency
Nice-to-Have Skills - AI evaluation frameworks (DeepEval, Ragas, LangSmith, Promptfoo)
- LangChain or similar AI orchestration frameworks
- Vector databases
- Docker and Kubernetes
- MLOps tools (MLflow, Kubeflow, SageMaker)
- Explainable AI (XAI) concepts
- Data annotation and synthetic data generation
- Accessibility testing
- Security testing for AI systems
