5-7 years of experience in managing large GPU fleets
Expertise in Slurm or Kubernetes for ML orchestration
Proven track record with high-scale ML data loaders
In-depth knowledge of ML hardware, storage, and networking stacks
Familiarity with the Nvidia GPU ecosystem
Responsibilities
Own and manage GPU compute fleets
Facilitate easy access and maximization of GPU utilization for researchers
Optimize ML data loading and storage in distributed environments
Orchestrate inference fleets for robots
Ensure real-time performance within latency constraints
Benefits
Cutting-edge technology environment
Opportunity to work with large-scale, innovative robotics projects
Collaboration with top researchers in the field
Access to advanced training and development resources
Full Job Description
About the Role
Generalist trains very large robot foundation models. This requires utilizing very large numbers of the latest generation GPU hardware and infrastructure (currently Nvidia) to run distributed training jobs and researcher experiments. We have extreme requirements on storage and data loading infrastructure that requires maximizing cloud infrastructure and custom solutions.
You will also own inference infrastructure. For our robots this is a fleet of on-prem GPUs attached to robots that have extreme real-time and latency budgets in compute constrained environments.
You'll be responsible for:
Owning our GPU compute fleets
Ensure our GPUs are easy for researchers to use and maximally utilized
Optimizing and improving ML data loading transport and storage in highly distributed fully utilized environments.
Orchestration of robot inference fleets
You might thrive in this role if you:
Have managed large fleets of GPUs doing large-scale, long-term, highly distributed training runs or inference
Deep experience in Slurm or Kubernetes for ML workload orchestration
Have build high-scale ML data loaders and preparation systems
Deeply understand every layer of the ML hardware, storage, and networking stacks