MS or PhD in Computer Science, Machine Learning, or a related field
Experience with PyTorch distributed training, including FSDP and Megatron-style parallelism
Ability to debug loss spikes, divergence, and throughput regressions at scale
Experience in building and balancing data mixtures and tokenization pipelines
Publications in top-tier conferences like NeurIPS, ICML, or ICLR (desirable)
Familiarity with low-precision training techniques like FP8 (desirable)
Experience with custom CUDA or Triton kernels (desirable)
Responsibilities
Conduct research on model architecture, training algorithms, optimizers, and data pipelines
Lead independent small projects while contributing to flagship training efforts
Design scaling-law experiments to inform data, architecture, and hyperparameter choices
Scale training across thousands of accelerators with robust fault tolerance
Develop development tools to enhance research team efficiency
Work across the technology stack from kernel throughput to high-level model design
Consider downstream serving costs alongside model quality
Benefits
Opportunity to lead your own projects
Access to substantial computational resources for hypothesis testing
Focus on foundational research rather than fine-tuning existing models
Collaborative environment with a focus on impactful AI research
Encouragement to apply even if all qualifications are not met, valuing mindset as much as experience
Full Job Description
Research Scientist / Engineer, Pre-training
San Francisco | Permanent DeepRec has partnered with a frontier AI research lab building new model architectures, not fine-tuning someone else's. Pre-training is the foundation every product and post-training effort builds on, and the team concentrates on a handful of large training efforts rather than spreading thin. The Opportunity
You'll run controlled experiments on large models and let the data settle the arguments. You'll have the compute to test real hypotheses at scale, plus room to lead your own projects alongside the flagship runs. What You'll Do
Research model architecture, training algorithms, optimisers and data pipelines
Lead small projects independently while contributing to flagship training runs
Design scaling-law experiments that set data, architecture and hyperparameter choices
Scale training across thousands of accelerators with sound fault tolerance
Build the dev tooling that keeps the research team moving quickly
Work across the stack, from kernel throughput to high-level model design
Design for downstream serving cost as well as model quality
What You'll Bring
Essential
MS or PhD in CS, ML or a related field
PyTorch distributed training, including FSDP and Megatron-style tensor or pipeline parallelism
Debugging loss spikes, divergence and throughput regressions at scale
Building and balancing data mixtures and tokenization pipelines
Desirable
Publications at NeurIPS, ICML, ICLR or similar venues
FP8 or other low-precision training
Custom CUDA or Triton kernels, or SSM-style and other non-standard architectures
We encourage you to apply even if you don't meet every requirement. The right mindset matters as much as the right CV. Apply Now
If you want your experiments to shape the models themselves, we'd like to hear from you. Send your CV and we'll share the full brief in a confidential conversation.
