About the RoleJoin a small, highly technical team of researchers and engineers - including International Olympiad medalists and published AI researchers - at an early-stage startup building high-quality benchmarks to evaluate frontier AI agents on realistic, domain-specific workflows. As a
Research Engineer, Benchmarks, you'll own the design and implementation of evaluations that frontier labs and enterprise customers rely on to measure real-world agent performance. This is a critical, high-ownership role at the intersection of research rigor and engineering execution.
The company operates in the
AI/ML evaluation and reinforcement learning infrastructure space, providing a platform for building, running, and scaling RL environments and post-training datasets. The team is based in
San Francisco, CA and works on-site.
Visa sponsorship is available.What You'll Do- Design, implement, and own the quality of internal benchmarks for evaluating frontier agents on domain-specific tasks.
- Partner with subject-matter experts to define realistic workflows and tasks for domain-specific evaluations.
- Build reliable infrastructure to run models and agents against benchmark tasks at scale.
- Develop metrics and statistical analyses that measure benchmark difficulty, reliability, and failure modes.
- Validate that benchmark performance correlates with real-world evaluations, customer needs, and frontier lab expectations.
- Write clear documentation and benchmark reports that make results legible and credible to technical audiences.
What We're Looking ForRequired- 2-4 years of experience in research engineering, ML engineering, or related roles - with a focus on building and delivering AI benchmarks, evaluation infrastructure, or agent environments.
- Demonstrated experience designing, implementing, and running benchmarks or evaluation environments for AI agents or large language models.
- Strong proficiency in Python, Docker, and Linux environments for building research or production infrastructure.
- Experience building and operating infrastructure to reliably run AI models or agents against benchmark or evaluation tasks at scale.
- Experience developing metrics, statistical analyses, or validation studies to assess benchmark difficulty, reliability, and real-world correlation.
- Experience collaborating with subject-matter experts to translate domain workflows into benchmark tasks and evaluation criteria.
- Experience analyzing workflows across diverse technical or business domains to inform task design.
- Strong technical writing skills - able to produce benchmark reports and documentation for research and engineering audiences.
Nice to Have- Published papers or technical blog posts on AI benchmarking, model evaluation, or model failure modes.
- Experience with reinforcement learning training pipelines, data generation, or RL agent evaluation.
- Background at frontier AI labs, research institutions, or involvement in widely used public benchmark projects.
Traits We Value- Deep curiosity about how workflows operate across varied domains.
- Sharp attention to detail - a habit of spotting subtle inconsistencies and edge cases in task design.
- Ability to reason from first principles about task design, scoring, and failure modes.
- Comfort thriving in unstructured problem spaces and working independently in a fast-paced, early-stage environment.
- Excellent communication skills for collaborating across time zones and with technical teams.
Compensation & Benefits- Salary: $150,000 - $250,000 USD annually, depending on experience.
- Early-stage equity participation.
- Visa sponsorship available.
LocationThis is an
on-site role based in
San Francisco, CA, United States. Candidates must be willing and able to work from the office. Fully remote arrangements are not available for this position.
