AI Quality and Evaluation Lead, Application Development

Reporting To:

Assistant Vice President IT, Application Development

Full-Time/Part- Time:

Full-time

Salary Range:

$80,021-$128,034

Posting Date:

August 21, 2026

Closing Date:

September 21, 2026

Hours of Work:

8:30 a.m. - 5:00 p.m.

Grade:

Office Location:

14.4

Toronto

Great location! Steps away from the main public transit station

What we offer:

Highly competitive compensation package which includes, base salary, bonus, benefits, and career advancement opportunities!

*Eligibility for benefits is dependent on the terms of employment

The Opportunity:

The AI Quality and Evaluation Lead is accountable for the evaluation strategy, quality standards, release gates, guardrails, and continuous monitoring used across First National AI Factory solutions.

This role defines what good enough to ship means for AI enabled mortgage workflows, and converts business policy, risk appetite, user expectations, and production evidence into measurable acceptance criteria. Working within the AI Platform Squad, leads hands on evaluation for residential underwriting use cases with the mandate to evolve a scalable quality capability as the AI Factory grows, including reusable test assets, automated continuous evaluation, consistent scorecards, and clear accountability between product squads and shared quality teams.

How you will contribute:

• Own and continuously evolve the AI Factory quality and evaluation strategy, including risk tiers, standard metrics, acceptance thresholds, representative test suites, release evidence, monitoring expectations, and continuous improvement practices across product squads.

• Define use case specific quality measures with the Product Manager, the Residential Underwriting Business SME, the AI Engineering Lead, QA Engineers, Risk, Compliance, and Security, covering policy adherence, factual grounding, completeness, precision, recall, false positives, false negatives, confidence, explainability, user impact, and appropriate escalation.

• Own the methodology for evaluation harnesses, golden datasets, adversarial tests, regression suites, human review protocols, and automated scoring workflows across document intelligence, retrieval, summarization, agent tool use, deterministic calculations, and multi step workflows, with the AI Platform Squad building and operating the underlying evaluation infrastructure.

• Establish the quality and evaluation gates within First National production readiness standards, jointly with the AI Engineering Lead who owns engineering and operational readiness, including UAT and parallel run evidence, minimum performance thresholds, guardrail validation, privacy and access control checks, rollback criteria, kill switch readiness, runbooks, and approval records.

• In collaboration with Information Security conduct failure mode testing for prompt injection, data leakage, unsupported claims, policy deviations, tool misuse, unsafe write back, inappropriate automation, model degradation, and other AI specific operational risks.

• Monitor production quality using evaluation results, model and prompt versions, drift, latency, user feedback, exception rates, override patterns, incidents, and token or platform cost, and define the triggers for investigation, rollback, retraining, prompt changes, or workflow redesign.

• Create defect taxonomies and root cause practices that distinguish model, retrieval, prompt, data, rules, integration, user experience, and process issues, and work with product and engineering teams to prioritize durable fixes.

• Evolve the function from hands on evaluation of the first use cases to a federated operating model, with reusable scorecards, automated continuous evaluation, trained squad level QA practices, oversight of AI Quality Analysts as volume grows, independent model and vendor benchmarking, and audit ready quality evidence.

The experience you need:

• Bachelor's degree in computer science, engineering, data science, statistics, mathematics, information systems or a related discipline.

• 7 plus years of progressive experience in quality engineering, software testing, model validation, machine learning, data science, risk analytics, AI governance, or related technology disciplines.

• Experience designing evaluation frameworks and release criteria for production AI or generative AI systems, including RAG, document intelligence, agents, model and prompt lifecycle management, observability, and human in the loop controls.

• Strong analytical and technical skills with Python, SQL, test automation, data analysis, statistical methods, and evaluation tooling such as MLflow, Ragas, DeepEval, or comparable frameworks.

• Working knowledge of accuracy and retrieval metrics, confidence scoring, sampling, experiment design, regression testing, drift monitoring, adversarial testing, red teaming, guardrails, privacy, security, and operational risk controls.

• Ability to translate business policy and risk appetite into measurable acceptance criteria, and to explain quality, risk, and trade off implications clearly to technical, business, control, audit, and senior stakeholders.

• Experience building or scaling a quality capability through reusable standards, automated test assets, continuous monitoring, governance evidence, coaching, and leadership of analysts or cross functional quality communities, including exposure to model risk, credit risk, or independent validation environments.

• Experience in financial services, mortgage lending, lending operations, servicing, broker channels, third party partnerships, or regulated technology environments is preferred.

Relationships:

External: Engages implementation partners, cloud providers, data and AI platform providers, model providers, and specialist vendors on evaluation methods, independent benchmarking, quality evidence, and knowledge transfer to First National.

Internal: Works within the AI Platform Squad while maintaining assessment independence from the delivery teams whose work it gates.

Working Environment and Physical Demands Analysis:

• Office environment
• Periods of high volume with tight timelines
• Long periods of stationary position/sitting
• Prolonged periods of repetitive movement (i.e. using a keyboard and mouse)
• Long periods of time in viewing a computer screen
• Multi-tasking may include speaking to customers on a telephone call while looking up information on a computer program.