Qualifications

• 5-7 years of experience training large models with real cost implications on compute allocation.
• Proven ability to design empirical experiments that test competing hypotheses.
• Expertise in scaling laws to leverage small-scale data for large-scale applications.
• Deep knowledge of modern training stacks and distributed training over large GPU clusters.
• Strong engineering skills, capable of writing code and analyzing performance metrics.
• A history of delivering contributions in real-world models, evidenced by publications or production systems.

Responsibilities

• Conduct research on model architecture and scaling for improved efficiency.
• Establish benchmarks and run experiments to validate scalable ideas.
• Define training parameters at scale, including learning rates and resource allocation.
• Investigate and implement new model architectures like mixture-of-experts and hybrid attention.
• Identify and troubleshoot instability in large model runs, addressing loss spikes and system failures.
• Work with multimodal models, integrating various types of data from the outset.

Benefits

• Opportunity to work on cutting-edge research with direct production impact.
• Access to a high-performance computing environment for large model training.
• Collaboration with interdisciplinary teams across multimodal projects.
• Opportunity to publish findings in influential research platforms.
• Flexibility to define innovative training methodologies and architectures.