About the roleEvals builds the internal platform that teams across Magic use to evaluate the performance of first-party and third-party models. The team supports pre-training, post-training, data, inference, and product, and sits on the critical path of many of the company's most important decisions.
As a Member of Technical Staff on Evals, you will build both the platform and the evaluations themselves. You'll develop infrastructure for large-scale evaluations, data ablations, and dataset quality analysis, while designing and validating the methodologies used to measure model performance.
Sweating the details matters on this team. Many benchmarks, papers, and open-source evaluation frameworks contain subtle bugs or flawed assumptions that lead to misleading conclusions. We care deeply about correctness, reproducibility, and measurement quality.
Evals are essential to the success of the company. By building trustworthy evaluation systems, you will help Magic make better research decisions, build better datasets, and ship better products.
What you'll work on- Build and maintain the internal evals platform used across Magic
- Design, implement, and validate eval tasks for pre-training, post-training, reinforcement learning, inference, and product systems
- Develop infrastructure for running large-scale evaluations
- Build systems to measure dataset quality and identify opportunities to improve training data
- Improve evaluation correctness, reproducibility, and reliability
- Audit and improve upon public benchmarks, evaluation methodologies, and open-source implementations
- Partner with research, data, inference, and product teams to define metrics that accurately reflect model quality
- Build tooling and frameworks that enable teams across Magic to make decisions based on trustworthy measurements
What we're looking for- Strong software engineering fundamentals
- Experience building production systems, internal platforms, or developer infrastructure
- Exceptional attention to detail and a high bar for correctness
- Experience working with machine learning systems, evaluation frameworks, data infrastructure, or research tooling
- Ability to reason critically about benchmarks, metrics, and experimental methodology
- Strong intuition for measurement quality and experimental design
- Experience designing, implementing, or operating systems that run at scale
- Strong debugging and investigative skills
- Comfortable navigating ambiguity and determining whether a measurement is actually capturing the behavior it claims to measure
- Skepticism toward results that cannot be reproduced, validated, or explained
- Track record of owning technical projects end-to-end
- Excitement about helping researchers and engineers make better decisions through trustworthy measurements
Compensation, benefits, and perks (US)- Annual salary range between $200K - $550K depending on experience
- Equity is a significant part of total compensation, in addition to salary
- 401(k) plan with 6% salary matching
- Generous health, dental, and vision insurance for you and your dependents
- Unlimited paid time off
- Visa sponsorship and relocation support for candidates moving to San Francisco
- A small, fast-moving, highly collaborative team working on frontier AI systems
