4-6 years of experience in quality engineering, particularly with AI-powered applications and microservices
Proficiency in Python and relevant AI/ML testing libraries (e.g., PyTest, DeepEval)
Experience with adversarial testing and safety evaluations
Familiarity with CI/CD workflows and automated testing frameworks
Bachelor's degree in Computer Science, AI, IT, or Software Engineering
ISTQB Foundation Level certification or equivalent
Certification in AI/ML testing or Responsible AI preferred
Responsibilities
Develop and execute evaluation frameworks for AI agents focusing on Responsible AI metrics
Design synthetic testing environments to stress-test agents under real-world conditions
Implement adversarial testing and prompt injection methodologies to identify vulnerabilities
Integrate evaluation pipelines into CI/CD workflows for continuous quality assurance
Collaborate with policy and legal teams to ensure compliance with regulatory standards
Analyze evaluation results and provide insights to development teams
Manage bug detection and tracking using Jira, focusing on performance issues
Benefits
Flexible work hours
Opportunities for professional development and certifications
Collaborative work environment
Access to cutting-edge technology and tools
Health and wellness programs
Full Job Description
Lead Quality Engineer Primary Skills
Test Data Management, Jenkins, Regression Testing, Postman, Selenium, Functional Testing, BDD - Behavior Driven Development, SoapUI, System Integration testing, Application Testing, Pixel Testing, Cross Browser Testing, Zephyr, TDD - Test Driven Development, Accessibility Testing (JAWS), Visual Testing, Cucumber
Specialization
Quality Engineering - Product Applications & Microservices: Lead Quality Engineer
Job requirements
Experience Range: with at least 4 to 6 years of experience in quality engineering, including hands-on testing and evaluation of AI-powered product applications and microservices Key Responsibilities: 1. Develop and execute robust evaluation frameworks for AI agents, focusing on Responsible AI and safety metrics such as bias, toxicity, and alignment failures 2. Design synthetic testing environments and simulation strategies to stress-test agents under diverse real-world conditions 3. Implement adversarial testing, red-teaming, and prompt injection methodologies to identify vulnerabilities and ensure agent robustness 4. Integrate evaluation pipelines into CI/CD workflows for continuous quality assurance and rapid iteration 5. Collaborate with policy and legal teams to ensure agent evaluations meet regulatory compliance and ethical standards 6. Analyze evaluation results and provide actionable insights to development teams, prioritizing safety and risk mitigation 7. Configure and execute automated test suites using Python and relevant AI/ML testing libraries (e.g., PyTest, DeepEval, Ragas, Giskard) 8. Manage bug detection, documentation, and tracking using Jira, focusing on performance degradations and alignment failures Required Skills: 1. Python programming 2. PyTest 3. DeepEval 4. Ragas 5. Giskard 6. Langsmith 7. Google Agent Development Kit (ADK) 8. Vertex AI 9. Safety evaluations (red teaming, adversarial testing) 10. Prompt injection testing 11. API testing for AI/ML systems 12. Data analysis with Pandas and SQL Preferred Skills: 1. Gemini models 2. Hugging Face 3. Garbage-in/Garbage-out (GIGO) testing tools 4. Data poisoning techniques 5. AI ethics principles and policy background 6. Visualization tools for evaluation results Desired Qualifications: 1. Bachelor's degree in Computer Science, Artificial Intelligence, Information Technology, or Software Engineering 2. ISTQB Foundation Level certification or Certified Software Tester (CST) 3. Certification in AI/ML testing or Responsible AI (e.g., Google Cloud Certified - Professional Machine Learning Engineer)
About Brillio
Brillio is a technology consulting and technology services company focused on the implementation of digital technologies.
Brillio Careers
There has never been a more opportune time to join Brillio—the forefront of innovation and digital transformation.
Opportunities Await
Brillio offers a plethora of job opportunities aimed at harnessing the potential of professionals eager to drive change. With a focus on technology and consulting, Brillio is not just a company; it's a vibrant ecosystem conducive to growth and professional development.
Transform Your Career
Embark on a journey of growth and leadership at Brillio. This is where industry expertise meets digital innovation to transform businesses and industries. Brillio stands out in the marketplace, providing a unique blend of technology solutions and strategic insights.
Join a Diverse and Inclusive Team
Brillio is committed to diversity and inclusion, ensuring a wide range of perspectives that fuel innovation and creative solutions. The team at Brillio, enriched with diverse skills and backgrounds, collaborates to deliver excellence and drive success.
Internship Programs and Employment
Kickstart your career with Brillio's internship programs, designed to offer hands-on experience and a gateway to full-time employment. Brillio is dedicated to nurturing young talent, providing them with the necessary tools and guidance to excel.
Innovate and Lead
At Brillio, every team member is an innovator and leader. The company fosters a culture of continuous learning and improvement, encouraging each individual to push boundaries and contribute to groundbreaking projects.
Networking and Professional Development
Networking at Brillio opens doors to influential connections, mentorship, and collaborative opportunities. Brillio supports career advancement through professional development, offering training in leadership, diversity, and specific job skills.
Benefits and Culture
Brillio is known for its dynamic culture and the comprehensive benefits it offers. Team members enjoy a supportive environment that values their well-being and professional growth.
Explore Job Positions
Brillio is hiring! Explore the various positions available that match your skills and interests. Brillio seeks passionate, curious, and solution-driven professionals ready to make an impact.
Prepare for Your Interview
Brillio helps candidates succeed by providing resources on how to effectively prepare your resume and ace the interview process. Stand out as a prospective employee by demonstrating your knowledge and enthusiasm for Brillio's mission.
Stay Connected with Brillio Careers
Keep up to date with career tips, industry insights, and the latest job openings at Brillio. Personalize your job search and stay informed about new and exciting opportunities within the company.
APPLY TO BRILLIO JOBS
Read Careers Blog
Stay ahead in your career with insights from Brillio's experts. Discover articles and advice tailored to advancing your career and enhancing your professional journey.
Job Alert Emails
Subscribe to receive customized job alerts and the latest news from Brillio. Discover the rewarding career opportunities that await and how Brillio can be the key to your professional future.