AI Operations Analyst

As myFICO expands its AI-powered product features, the Scores B2C team is hiring an AI Operations Analyst to review how our AI features talk to consumers about their credit, identify wrong or unsafe answers, and turn those findings into a better product. What You'll Contribute • Review a risk-weighted sample of AI chat conversations and Personalized Credit Insights summaries, and flag inaccurate, unsupported, off-topic, or unsafe responses. • Categorize member feedback, guardrail blocks, and escalations into a failure-mode taxonomy, and track how it changes over time. • Own the golden regression dataset by adding labeled examples from real production misses, keeping labels consistent, and documenting the rubric behind them. • Label data to help measure and improve automated evaluators, including double-labeling a share of items to measure reviewer agreement. • Publish a weekly quality report covering intent coverage, resolution and containment, refusal and guardrail rates, safety flags, top failure modes, and recommended fixes. • Escalate flagged content to Legal, Compliance, and Trust & Safety with clear severity ratings, and follow each item through to resolution. • Use SQL and dashboards to understand what members ask, where the experience falls short, and where quality is drifting. • Partner with Product, the Applied AI Engineer, and QA engineers on review tooling, rubrics, and sampling methods, and verify fixes after each release. What We're Seeking • Bachelor's degree in a related field, or equivalent experience, and demonstrated experience in a quality, analytical, or operations role reviewing customer conversations, such as CX or QA, support operations, compliance QA, or conversation design. • Strong SQL skills and comfort building and reading dashboards in a BI tool such as Tableau. • Sound judgment and close attention to detail, with the ability to apply a written rubric consistently and explain your decisions. • Strong written communication, including clear failure descriptions and concise summaries for leadership. • Comfort working with sensitive consumer data in a regulated environment, and the discretion that requires. • Experience working in Agile/Scrum teams. • Experience in consumer finance, credit, banking, fintech, or other regulated environments (FCRA, ECOA, UDAAP), knowledge of credit reports and FICO Scores, and experience with conversational AI or LLM-based products, data labeling, rubric design, inter-rater agreement, failure-mode taxonomies, basic Python and Jupyter, or Legal and Compliance escalation processes is a plus. Our Offer to You • An inclusive culture strongly reflecting our core values: Act Like an Owner, Delight Our Customers and Earn the Respect of Others. • The opportunity to make an impact and develop professionally by leveraging your unique strengths and participating in valuable learning experiences. • Highly competitive compensation, benefits and rewards programs that encourage you to bring your best every day and be recognized for doing so. • An engaging, people-first work environment offering work/life balance, employee resource groups, and social events to promote interaction and camaraderie. • The targeted base pay range for this role is: $105,000 to $165,000 with this range reflecting differences in candidate knowledge, skills and experience.