Job DescriptionWhat's the opportunity?We're seeking a seasoned Staff Software Engineer to join the RBC Borealis AI Platform team and own the end-to-end lifecycle of machine learning systems-from experimentation and validation through to high-throughput production serving at scale. You'll be the technical anchor for operationalizing vision language models and document processing systems that handle thousands of documents per minute, setting the bar for reliability, observability, and engineering excellence across our AI platform.
You'll lead the design and evolution of our scalable document processing platform-a production system that combines event-driven architecture, vision language models, and cloud-native infrastructure to extract intelligence from financial documents at enterprise scale:
- ML System Operationalization: Own the production lifecycle of LLM and computer vision models, from integration and validation to serving, monitoring, and continuous improvement at 1000+ documents/minute throughput
- Platform Architecture: Design resilient microservices using FastAPI and event-driven patterns with Apache Kafka, ensuring 99.5%+ reliability for mission-critical financial document processing
- Scalable Infrastructure: Build and optimize Kubernetes-native workloads with KEDA-based autoscaling (3-50 replicas dynamically), PostgreSQL/MongoDB data layers, and S3 object storage with lifecycle management
- Observability & Reliability: Establish comprehensive monitoring, alerting, and SRE practices that provide deep visibility into model performance, system health, and business metrics across distributed services
This is a rare opportunity to shape the foundation on which Canada's largest financial institution runs its most critical AI workloads, working directly with leading researchers in machine learning while having access to rich, massive datasets and the computational resources to support groundbreaking innovation
Your responsibilities include:Technical Leadership & ML Engineering
- Architect production ML pipelines that seamlessly integrate vision language models, OCR engines, and document extraction services into scalable, fault-tolerant systems
- Drive technical decisions on complex distributed systems challenges involving data consistency, exactly-once processing semantics, and sub-500ms API response times
- Collaborate closely with ML researchers to translate cutting-edge models in computer vision, NLP, and reinforcement learning into production-ready services
- Set engineering standards for model serving, A/B testing, feature flags, and gradual rollouts that enable safe, data-driven experimentation at scale
Platform Development & Innovation
- Build sophisticated retry mechanisms with exponential backoff, circuit breakers, dead-letter queues, and fallback strategies that ensure system resilience
- Implement advanced event-driven patterns across Kafka topics (ingestion, processing, callbacks, DLQ) with precise consumer group management and lag-based autoscaling
- Develop reusable frameworks and libraries for async processing, template-based document parsing, and callback orchestration that accelerate team productivity
- Lead the evaluation and adoption of emerging AI technologies, ensuring alignment with enterprise security, compliance, and data governance requirements
Cross-Functional Collaboration
- Partner with data scientists and ML researchers to understand model requirements, performance characteristics, and integration patterns for production deployment
- Work with process engineers and business stakeholders to translate financial document processing needs into robust, scalable technical solutions
- Foster strong relationships across platform, infrastructure, and security teams to deliver end-to-end capabilities that span multiple domains
- Mentor engineers on distributed systems design, event-driven architecture, ML ops best practices, and cloud-native development patterns
Strategic Problem Solving
- Navigate ambiguity in complex technical challenges, from Kafka partition strategies to LLM provider selection to autoscaling configurations
- Identify and mitigate architectural risks before they impact production, using techniques like chaos engineering, load testing, and failure mode analysis
- Provide clear, data-driven recommendations to engineering leadership on infrastructure investments, technology choices, and platform roadmap priorities
- Drive continuous improvement in system performance, cost efficiency, and developer experience through metrics-driven iteration
You're our ideal candidate if you have:- 5-8+ years of software engineering experience with 3+ years focused on ML systems, data platforms, or high-scale distributed systems
- Deep expertise in Python and production-grade API frameworks (FastAPI, Flask, or similar) with strong software design principles
- Proven track record operationalizing ML models in production-you've integrated LLMs, vision models, or similar AI services into scalable systems
- Strong hands-on experience with event-driven architectures using Apache Kafka, RabbitMQ, or cloud-native messaging platforms
- Production experience with both SQL (PostgreSQL) and NoSQL (MongoDB, DynamoDB) databases, understanding tradeoffs and optimization strategies
- Expert-level knowledge of containerization (Docker) and Kubernetes/OpenShift orchestration, including custom resources, operators, and autoscaling
What's in it for you?- Become part of a team that thinks progressively and works collaboratively. We care about seeing each other reach full potential;
- A comprehensive Total Rewards Program including bonuses and flexible benefits, competitive compensation, commissions, and stock options where applicable;
- Leaders who support your development through coaching and managing opportunities;
- Ability to make a difference and lasting impact from a local-to-global scale.
Additional Job DetailsAddress:407 8 AVE SW:CALGARY
City:Calgary
Country:Canada
Work hours/week:37.5
Employment Type:Full time
Platform:TECHNOLOGY AND OPERATIONS
Job Type:Regular
Pay Type:Salaried
Posted Date:2026-04-24
Application Deadline:2026-07-16
Note: Applications will be accepted until 11:59 PM on the day prior to the application deadline date above
RBC is presently inviting candidates to apply for this existing vacancy. Applying to this posting allows you to express your interest in this current career opportunity at RBC. Qualified applicants may be contacted to review their resume in more detail.
