Owns the quality engineering function for an agentic conversational AI platform - serving as the guardian of the customer experience across every voice and chat interaction. This role spans two mandates: hands-on assurance that every release is effective and defect-free before it reaches end users, and building, scaling, and maintaining the evaluation infrastructure that makes quality continuous, measurable, and increasingly self-improving.
The role plans and directs the development, application, and maintenance of quality standards for conversational AI applications and their supporting systems and owns the automated evaluation platform - synthetic and adversarial agent-to-agent conversation testing, DeepEval test suites, proprietary LLM-as-judge evaluators, and deterministic rule-based checks - as the system of record for shipping model, prompt, and agent changes with confidence. Because LLM systems are non-deterministic, this role designs test plans and evaluation rubrics that turn ambiguous notions of quality into clear, defensible metrics, and correlates those metrics with real-world performance and user experience.
The role is weighted toward quality infrastructure and evaluation engineering (~70%), balanced with hands-on manual and exploratory QA (~30%), with an explicit mandate to advance the platform toward recursive self-improvement: a closed-loop quality flywheel in which evaluation findings automatically generate new test cases, refine judge rubrics, surface model and prompt improvement candidates, and re-validate fixes - so the evaluators, and the agents they evaluate, get better with every conversation.
To perform this job successfully, an individual must be able to perform each essential duty satisfactorily. The requirements listed below are representative of the knowledge, skill, and/or ability required. Reasonable accommodations may be made to enable qualified individuals with disabilities to perform the essential duties.
This job description is a summary of the primary duties and responsibilities of the job and position. It is not intended to be a comprehensive or all-inclusive listing of duties and responsibilities. Contents are subject to change at the company's discretion.
Work Experience- Required - 3 years Software; quality assurance and testing methodologies; and defect-tracking tools. Requires conversational AI testing experience (intent accuracy, fallback handling, escalation logic)
- Preferred - Familiarity with AI/machine learning and natural language processing concepts, particularly for conversational AI or chatbot testing.
Knowledge Skills and Abilities (KSAs)- Understanding of the software development lifecycle and API/integration testing.
- Ability to design and execute detailed test cases across multiple scenarios.
- Strong attention to detail and quality standards.
- Ability to collaborate and communicate effectively with engineers and product teams to resolve defects.
- Documentation and reporting of test outcomes.
- Ability to balance speed of testing with thoroughness and accuracy while maintaining consistency and discipline in the testing process.
Job Duties- Develops and implements quality standards, tests plans, and methods for inspection, testing, and evaluation across the conversational AI platform and supporting systems.
- Defines functional and non-functional test strategies, including conversation flow accuracy, response quality, latency, and fallback and escalation behavior.
- Designs and executes test cases covering user scenarios, edge cases, and failure points, including regression, negative, usability, and integration testing.
- Builds and maintains automated test scripts and frameworks, and perform manual testing where appropriate.
- Identifies, documents, and tracks defects through resolution, and confirm fixes.
- Tests the product both as intended and as misused to anticipate incorrect usage and reduce operational and compliance risk.
- Maintains test documentation and reports on quality assurance activities, defect patterns, and release readiness.
- Performs other duties as assigned
- Complies with all policies and standards
The above statements describe the general nature and level of work only. They are not an exhaustive list of all required responsibilities, duties, and skills. Other duties may be added, or this description amended at any time.
Remains knowledgeable on current federal, state and local laws, accreditation standards or regulatory agency requirements that apply to the assigned area of responsibility and ensures compliance with all such laws, regulations and standards.
Physical and Environmental DemandsThe physical demands described here are representative of those that must be met by an employee to successfully perform the essential functions of this job. Reasonable accommodations may be made to enable individuals with disabilities to perform the essential functions.
Sedentary Work- Exerting up to 10 pounds of force occasionally (occasionally means activity or conditions exist up to 1/3 of the work day), and/or, a negligible amount of force frequently (frequently means activity or condition exists from 1/3 to 2/3 of the work day) to lift, carry, push, pull, or otherwise move objects, including the human body. Sedentary work involves sitting most of the time but may involve walking or standing for brief periods of time. Jobs are sedentary if walking and standing are required only occasionally and all other sedentary criteria are met.
Normal routine involves no exposure to blood, body fluid or tissue and as part of their employment, incumbents are not called upon to perform or assist in emergency care or first aid.
The incumbent has no occupational risk for exposure to communicable diseases .
Because the incumbent works within a healthcare setting, there may be occupational risk for exposure to hazardous medications or hazardous waste within the environment through receipt, transport, storage, preparation, dispensing, administration, cleaning and/or disposal of contaminated waste. The risk level of exposure may increase depending on the essential job duties of the role.
