Who We're Looking ForWe are looking for a highly detail-oriented QA Engineer to own testing, release validation, and AI conversation quality across the Alivo platform.
You will be in charge of developing and running our automated test suite as well as (where needed) manually validating new features, identifying edge cases, evaluating user experience, and ensuring releases meet a high standard before reaching customers.
A core part of this role is evaluating the quality of AI-driven phone and chat conversations. You will help determine whether our AI communicates naturally, accurately, and effectively while identifying opportunities to improve customer interactions and overall product quality.
This role is critical to our success. You will serve as the final quality gate before releases ship and help define what great AI-powered customer communication should look like.
If you thrive in fast-paced environments, enjoy uncovering problems before customers do, and have strong instincts for quality, this role is for you.
Responsibilities- Own Product Testing
- Develop, run, and maintain automated test suites
- Manually test new features, workflows, and user experiences before release
- Perform exploratory testing to identify bugs, usability issues, and edge cases
- Validate functionality across staging and production-like environments
- Execute structured test plans and release checklists to ensure consistent quality
- Verify bug fixes and regression testing across product releases
- Drive Product Quality & Release Readiness
- Review feature implementations and release candidates before deployment
- Partner with Engineering to identify risks and improve release confidence
- File clear, reproducible bug reports with detailed documentation
- Help establish and improve manual testing processes as the company grows
- Serve as a trusted quality gate for production releases
- Own AI Conversation Quality
- Evaluate AI-powered phone and chat conversations for quality and effectiveness
- Assess whether conversations are natural, accurate, helpful, and aligned with customer goals
- Review objection handling, conversational flow, and customer experience outcomes
- Identify failure modes, edge cases, hallucinations, and off-script behavior
- Help define quality standards and acceptance criteria for AI-powered interactions
- Build AI Evaluation Processes
- Develop and maintain conversation evaluation datasets and testing scenarios
- Create scoring rubrics and quality frameworks for measuring AI performance
- Track AI quality over time and identify regressions or performance changes
- Provide actionable recommendations that improve prompts, workflows, and AI behavior
- Collaborate Cross-Functionally
- Work closely with Engineering, Product, Customer Success, and Leadership
- Provide feedback that improves both product quality and customer outcomes
- Help prioritize quality improvements based on customer impact and business goals
- Support a culture of quality throughout the organization
RequirementsQualifications- What You Bring
- 2-5+ years of QA, software testing, product operations, or similar experience
- Strong attention to detail and a methodical approach to testing
- Excellent written and verbal communication skills
- Ability to identify edge cases, failure modes, and customer experience issues
- Strong problem-solving and analytical thinking skills
- Experience creating clear bug reports and testing documentation
- Comfortable working in a fast-paced, evolving startup environment
- Who You Are
- Highly detail-oriented and naturally curious
- A strong critical thinker who enjoys finding problems before customers do
- Comfortable making judgment calls about quality and user experience
- Organized, reliable, and highly accountable
- Able to communicate findings clearly and objectively
- Passionate about creating exceptional customer experiences
- Bonus Points For
- Experience evaluating AI, LLM, or conversational AI systems
- Experience with test management or bug tracking tools such as Linear, Jira, or TestRail
- Experience with API testing tools such as Postman
- Basic SQL, scripting, or technical troubleshooting skills
- Experience in customer-facing, sales, or support roles
- Exposure to AI-powered products or machine learning systems
- Background in SaaS products or high-growth startup environments
Benefits- Company Sponsored Insurance Offerings
- Long Term Financial Benefits
- Time to Recharge
- Unlimited paid time off
- Paid sick time
