Location: Salt Lake City, UT In this role, you will build automation, evaluate AI-driven behavior, validate responsible AI controls, and identify risks early so our products are reliable, scalable, production-ready, and trusted by millions of users. Key Responsibilities - Become a Subject Matter Expert on the AI platform and possess a deep understanding of system interactions, upstream/downstream dependencies, and data flows. - Design, develop, and maintain automated test frameworks for AI platform services, APIs, agents, prompt-based workflows, and RAG-enabled applications. - Develop automated AI evaluation suites and regression benchmarks that continuously measure model behavior and detect quality degradation before release. - Build and execute functional, integration, end-to-end, regression, performance, and reliability tests for AI-driven systems and services. - Define evaluation strategies, quality metrics, and acceptance criteria for AI-generated outputs, including accuracy, relevance, consistency, grounding, safety, and business value. - Validate responsible AI controls, permissions, guardrails, data handling practices, and business rules that protect customer trust. - Partner with Engineering, Product, and Support throughout the software lifecycle to drive risk-based testing strategies, influence quality to ensure production readiness. - Integrate automated testing, AI evaluations, and release validation into CI/CD pipelines. - Investigate, triage, and communicate defects, quality issues, and production incidents, driving root cause analysis and continuous improvement. - Utilize observability tools, logs, metrics, and traces to investigate production issues and perform root cause analysis. - Establish and promote testing standards, automation patterns, and AI quality practices that improve reliability and delivery speed. Required Qualifications - 3+ years of experience in software testing, quality engineering, test automation, or software development. - Hands-on experience developing automated tests, testing tools, or quality frameworks using Python, Selenium and Playwright. - Experience testing APIs, microservices, distributed systems, backend services, or event-driven architectures. - Strong experience testing AI-enabled applications using technologies such as LLMs, LangChain, LangGraph, or similar platforms. - Hands-on experience evaluating AI-generated outputs using datasets, scoring rubrics, golden test sets, benchmarking frameworks, or automated quality checks. - Experience testing RAG systems, including retrieval quality, embeddings, grounding, citations, context accuracy, and streaming responses. - Experience working with SQL and NoSQL technologies, including relational, document, key-value, or vector databases. - Experience integrating automated testing into CI/CD pipelines and modern software delivery practices. - Experience investigating production issues and using incident and defect data to improve system quality and reliability. - Strong understanding of test automation, performance testing, risk-based testing, release validation, and responsible AI principles. - Excellent collaboration and communication skills, with the ability to clearly articulate quality risks and acceptance criteria. Preferred Qualifications - Experience with AWS, Kubernetes, Docker, and cloud-native architectures. - Experience testing event-driven systems using Kafka or similar messaging platforms. - Familiarity with application security testing and OWASP Top 10 principles. - Proficiency in Python and at least one additional programming language such as Java, Ruby, or Go. - Experience with reliability engineering practices, including observability, production readiness reviews, incident analysis, SLOs, and continuous improvement. - Experience conducting performance, load, stress, scalability, or resilience testing.