Distributed Foundation Model Training 

• Design and execute large‑scale training using: 

• FSDP 

• ZeRO‑3 

• Tensor parallelism 

• Pipeline parallelism 

• Apply 

• Mixed precision (bf16, fp8) 

• Gradient checkpointing 

• Partner closely with ML systems teams while retaining architectural ownership 

Architecture Innovation 

• Explore and implement novel model designs, including:

• MoE routing strategies

• Multimodal fusion architectures

• SSM / hybrid architectures

• Design architectures with KV cache efficiency and inference implications in mind

What Success Looks Like 

• Training remains stable as models scale in size and complexity 

• Architectural decisions are principled and defensible 

• Models converge faster and generalize better due to architecture and optimisation choices 

• Failure modes are understood, not mysterious 

• The organization develops true in‑house foundation model expertise 

Required Experience & Skills 

Strongly Required 

• Deep theoretical and practical understanding of modern deep learning 

• Hands‑on experience training large models from scratch 

• Ability to reason about optimization, not just tune hyperparameters 

• Comfort operating in ambiguous, research‑driven environments 

Critical Technical Skills 

• Transformer internals and attention mechanisms

• Optimisationalgorithms and training dynamics

• Scaling laws and compute/data tradeoffs

• Distributed training strategies and mixed precision

• Architecture innovation for large, real‑world models

Common Problems You’ll Be Solving  

• Why training diverges at scale 

• How optimizer dynamics interact with architecture 

• When scaling laws break down 

• The real tradeoffs between data, compute, and model design 

What we offer 

We offer a generous compensation and benefits package (in addition to the base salary), including: 

• Salary range $185,000.00 - $280,000.00  It is not typical for offers to be made at or near the top of the range. The actual salary will be determined based on experience and other job-related factors. 

• Annual bonus opportunity 

• Insurance coverage (medical, dental, vision, life, and disability) 

• Paid time off 

• Paid holidays 

• Company contribution to the RRSP (Registered Retirement Savings Plan)

• Equity awards for certain positions and levels

• Remote and/or hybrid work available depending on the position

All compensation and benefits are subject to the terms and conditions of the underlying plans or programs, as applicable, and may be amended, terminated, or replaced from time to time.