Artificial Intelligence & Model Risk Auditor

The position at a glance In accordance with Section 165 of the Dodd-Frank Act (DFA), BNP Paribas has established an Intermediate Holding Company (IHC) over its U.S. subsidiaries. Furthermore, BNP Paribas has established a Combined U.S. Operations (CUSO) and IHC governance framework to support capital and liquidity risk management, regulatory reporting, and compliance risk management. The Inspection Générale North America (IGNA) Hub is part of BNP Paribas' global Internal Audit group (Inspection Générale, or "IG"). It is staffed by highly qualified individuals who conduct risk-based reviews to provide reasonable assurance to senior management regarding the adequacy of internal controls and to recommend improvements in controls and efficiency. Within IGNA, a team dedicated to the audit of Enterprise Risk Management covers several business lines and corporate functions including Model and Artificial Intelligence Risk Management. Candidate Success Factors: Candidates will be assessed against the following four performance drivers, which determine how individual impact is evaluated across the Americas platform: * Results and Impact * Impacts the division and influences peers and the team * Demonstrates sound judgment when making highly complex, high-impact decisions * Requires limited guidance when making the most complex decisions * Drives outcomes that have a meaningful effect on the team or department * Leadership and Collaboration * Builds trust with department leaders * Assumes a leadership role in large projects, processes, or programs for the team * Client, Customer and Stakeholder Focus * Builds relationships with a range of intermediate and senior colleagues or clients * Interacts regularly with management and department leaders * Demonstrates the ability to persuade and influence stakeholders * Compliance Culture and Conduct * Takes full responsibility for personal actions and demonstrates courage when confronting problems and conflicts * Is regarded as a person of high moral character, upholds corporate values, and demonstrates high ethical standards Responsibilities: The Artificial Intelligence & Model Risk Auditor will play a key role in supporting the internal audit team in independently assessing the effectiveness of governance and risk management frameworks. A strong background in frameworks for auditing models is a key requirement for this position. Specific Artificial Intelligence and Model Risk Audit Responsibilities include: * Apply frameworks for auditing models, including criteria such as robustness, fairness, interpretability, alignment, compliance, and human-in-the-loop assessments. * Conduct audits across the model lifecycle, from training through deployment and monitoring, to assess compliance with quality, performance, fairness, and risk management standards. * Identify model vulnerabilities, including bias, fairness violations, harmful hallucinations, and security risks, and recommend remediation strategies. * Assess model performance metrics, including accuracy, precision, recall, F1 score, calibration, robustness, and fairness metrics; measure hallucination rates in LLMs; quantify bias and fairness; evaluate confidence scores; and perform stability analyses. Specific Audit Responsibilities include: * Conduct multiple complex and concurrent audit assignments, including related scoping, planning, evaluation, and reporting. * Write audit reports containing value-added recommendations. * Understand and assess the risks applicable to the areas of audit responsibility through continuous monitoring activities. * Manage time and priorities while conducting audits. * Review and prepare risk assessments for audit units. * Develop and maintain relationships with heads and managers of business lines and functions as part of continuous risk monitoring activities. * Validate the adequacy and appropriateness of remediation or resolution plans addressing regulatory and internal audit recommendations. * Investigate serious incidents or cases of fraud, when required. * Deliver clear presentations to senior management and IG management. * Keep abreast of developments in auditing standards and key regulations governing compliance and operational risk management. * Adopt and promote strong ethical values. * Participate in various committees, as assigned, to report on and provide information about current and emerging business issues to senior management. Minimum Required Qualifications: * Bachelor's or master's degree in Mathematics, Statistics, Computer Science, Financial Engineering, or another related quantitative field. * At least three years of professional experience in applied data science, machine learning engineering, or AI research, specifically involving LLMs and traditional machine learning models. * Programming experience in statistical or mathematical languages such as Python or R, as well as hands-on experience with version control platforms such as GitHub. * Experience with machine learning and NLP frameworks such as TensorFlow, PyTorch, and scikit-learn, and with relevant libraries such as pandas, Polars, and NumPy. * Strong communication and synthesis skills, critical-thinking capabilities, and a passion for learning. * Proficiency in English* is required; proficiency in French is preferred. *Given that the vast majority of our clients, both internal and external, are based outside Quebec and Canada, specific language requirements apply. Preferred Qualifications: * Professional certification, such as the CIA designation. * Good knowledge of key U.S. and Canadian banking regulations affecting the areas in scope. * Ability to analyze model design, training methods, and data pipelines. What's in it for you In addition to competitive compensation, we offer flexible benefits, including a family and spousal insurance program, a defined contribution pension plan, and paid volunteer days. Hybrid work arrangements, including remote work of up to 40%, and flexible working hours are available for most positions. BNP Paribas provides excellent training and personal development programs, as well as opportunities for career advancement within the company and internationally. To find out more about our range of benefits, [1] click here What you need to know * We review applications as they are received, so do not wait to apply. * BNP Paribas is committed to accessibility and inclusion. If you require accommodation during the recruitment process, please contact accessibility [at] us.bnpparibas (dot) com. * You must be legally eligible to work in the Greater Montreal area and, if applicable, hold a valid work or study permit. Physical presence in BNP Paribas' offices is an essential function of this position.