Experience training models on large GPU clusters with Kubernetes
Proficiency in building or managing complex distributed systems
Experience working with large datasets (terabytes to petabytes)
Familiarity with large-scale data processing tools
Strong commitment to observability and metric collection
Ability to transition from research code to production-quality systems
Passion for running experiments and investigating results
Responsibilities
Implement new ideas and innovations in model training
Scale experiments utilizing large GPU clusters
Build infrastructure to allow rapid iteration
Analyze failures to identify and resolve issues
Facilitate multimodal representation learning
Process and evaluate large datasets efficiently
Support reinforcement learning initiatives
Benefits
Collaborative team environment in San Francisco
Opportunities for professional growth and innovation
Focus on impactful projects at large scale
Emphasis on building and executing novel systems
Flexibility in approaches to problem-solving
Full Job Description
We9re looking for a research engineer to help us train increasingly capable models across enormous and diverse datasets.
You9ll work across the boundary between research and engineering: implementing new ideas, scaling experiments across large GPU clusters, building the systems that let us iterate quickly, and figuring out why things aren9;t working. The work spans large-scale model training, multimodal representation learning, reinforcement learning, data processing, evaluation, and the infrastructure required to support all of it.
You might be a good fit if you:
Have trained models across large GPU clusters and are comfortable working with Kubernetes
Have built or operated complex distributed systems
Have worked with multi-terabyte or multi-petabyte datasets
Are comfortable with large-scale data processing tools
Care deeply about observability and collect enough metrics to understand what every part of a system is doing
Are comfortable moving between research code and production-quality systems
Like running experiments, getting surprising results, and digging in until you understand why
Move quickly and reach for simple approaches before complicated ones
Nice to have:
Experience with JAX
Experience writing CUDA kernels or otherwise optimizing GPU workloads
Low-level Linux or kernel programming experience
Experience with large-scale video or multimodal datasets
Experience building training or evaluation infrastructure
Experience with distributed training
Experience deploying models into real-world systems, especially robotics
We care much more about what you9ve built than any specific credential. We9re a small, fast-moving team working together in person in San Francisco. If you9re excited about architecting novel systems at unprecedented scale, we9d love to talk.
