About the roleWe are hiring for a QA Engineer, (AI), who will be responsible for validating both: Deterministic systems (APIs, backend logic, UI flows) and Non-deterministic AI systems (LLM outputs, prompt behavior, edge cases). We are building an AI-first product where core functionality depends on prompt engineering and LLM behavior. This position requires someone to design, execute, and automate testing strategies for LLM-based features, and AI-driven workflows. Additionally, they will test end-to-end product flows across frontend, backend, and APIs.
What you'll do- Test end-to-end product flows across frontend, backend, and APIs of an event based system
- Design structured test cases for prompt-based features
- Build and maintain test datasets
- Define evaluation criteria for AI outputs
- Run regression testing when prompts or models change• Automate repeatable tests
- Track quality metrics over time (output quality, failure rate, latency issues)
Qualifications- 3+ years in QA / software testing, preferably for LLM based product
- Strong Python skills (Automation, LLM, Evaluations)
- Experience testing APIs and full-stack systems
- Ability to think in edge cases and break systems intentionally
- Experience working with LLM-based products
- Experience testing prompt engineering products
Why Join Us?- Comprehensive health, dental, and vision coverage.
- 401(k) with company funded match.
- Mission-driven company with a 45+ year legacy and industry-leading products trusted by thousands of lenders.
- Collaborative, supportive team known for long tenure, deep expertise, and a culture built on trust and integrity.
- Help modernize and evolve the leading platform in private lending technology.
- Beautiful San Mateo office with a tight-knit culture.
- Competitive compensation and room for professional growth.
- Performance-based bonuses.
- Generous PTO.
- Modern workspace with on-site amenities.
- Awesome company Christmas party.
The pay range for this role is:
120,000 - 140,000 USD per year (Hybrid, San Mateo, CA)
