Experience training large models in multi-node GPU environments
Deep understanding of parallelism strategies
Experience debugging cross-layer issues in production ML systems
Strong ownership mindset for critical infrastructure
Proven track record of improving large-scale systems' performance
Responsibilities
Scale distributed training on large GPU clusters through various parallelism techniques
Optimize communication patterns and manage gradient synchronization
Enhance checkpointing, fault tolerance, and job recovery processes
Profile and resolve performance bottlenecks involving compute, networking, and storage
Improve experiment reproducibility and orchestration workflows
Maximize hardware utilization and training throughput
Collaborate with teams to align model architecture with system capabilities
Benefits
401(k) plan with 6% salary matching
Generous health, dental, and vision insurance for employees and dependents
Unlimited paid time off
Visa sponsorship and relocation stipend for applicants moving to SF
Small and highly focused team environment
Full Job Description
About the role
As a Research Engineer on the Pretraining Systems team, you will design and operate the distributed infrastructure that trains Magic's long-context models at scale.
This role focuses on large-scale model training across massive GPU clusters. You will work at the boundary between deep learning and distributed systems, ensuring that training runs are performant, reliable, and reproducible under extreme scale.
Magic's long-context models create non-trivial systems challenges: sustained memory pressure, communication overhead across thousands of devices, long-running jobs that must survive failures, and efficient sequence packing under hardware constraints. You will own the systems that make large-scale pretraining stable and fast.
What you'll work on
Scale distributed training across large GPU clusters (data, tensor, pipeline parallelism)
Optimize communication patterns and gradient synchronization
Improve checkpointing, fault tolerance, and job recovery systems
Profile and eliminate performance bottlenecks across compute, networking, and storage
Improve experiment reproducibility and orchestration workflows
Increase hardware utilization and training throughput
Collaborate with Kernels and Research to align model architecture with systems realities
What we're looking for
Strong software engineering and distributed systems fundamentals
Experience training large models in multi-node GPU environments
Deep understanding of parallelism strategies and performance trade-offs
Experience debugging cross-layer issues in production ML systems
Strong ownership mindset and ability to operate critical infrastructure
Track record of improving performance or reliability of large-scale systems
Compensation, benefits, and perks (US):
Annual salary range: $275K - $550K
Equity is a significant part of total compensation, in addition to salary
401(k) plan with 6% salary matching
Generous health, dental and vision insurance for you and your dependents
Unlimited paid time off
Visa sponsorship and relocation stipend to bring you to SF, if possible
