Hands-on research experience in synthetic data pipelines and evaluations.
Experience building environments or benchmarks for AI agents at a leading AI lab.
1-2 years in a fast-growing or AI-focused startup with mid-level or senior experience.
Proficiency in simulating software systems for agent tool use and identifying benchmark flaws.
Degree from a selective university or equivalent entrepreneurial experience.
High-level competition experience in mathematics, programming, or debate is a plus.
Responsibilities
Design synthetic environments with realistic data for AI agents.
Build evaluation systems to assess advanced AI agents.
Develop benchmarks that reflect real-world performance.
Identify and address evaluation failure modes like reward hacking.
Simulate software tools to support agent interactions at scale.
Benefits
Base salary range of $200,000 to $300,000 annually.
Commission-style bonuses tied to delivered environments.
Founding equity included in compensation package.
Visa sponsorship available for qualified candidates.
Full Job Description
About the Role
Build evaluation environments and benchmarks that measure whether AI agents can perform real-world company work and meet or exceed human performance. This founding engineering role combines agent evaluation research with production readiness, helping establish reliable evidence of agent capability. What You'll Do
Design synthetic company environments with realistic email, records, and files for agents to use over extended periods.
Build graders and evaluation systems capable of assessing advanced agents, including cases where human reviewers may lack the relevant expertise.
Develop benchmarks that correlate with real-world production performance.
Find and address evaluation failure modes, including reward hacking and benchmark gaming.
Simulate software tools such as inboxes and CRMs with enough fidelity to support agent interactions at scale.
What We're Looking For
Hands-on research experience in synthetic data pipelines, benchmarks, evaluations, or model post-training; this is a research-oriented role, not a pure infrastructure engineering position.
Experience building environments or benchmarks used to train or evaluate agents at a frontier AI lab, and implementing graders or evaluation systems for AI agents.
At least 1 to 2 years at a fast-growing or AI-focused startup, plus mid-level or senior engineering or research experience.
Experience simulating software systems for agent tool use, identifying benchmark flaws, and detecting and fixing reward hacking in evaluation systems.
A degree from a highly selective university or an equivalent credential, such as founding a company or participating in a recognized founder program.
High-level competition experience in mathematics, programming, debate, or games is a plus.
Compensation & Benefits
Base salary is $200,000 to $300,000 annually, with up to $350,000 for exceptional candidates. Additional commission-style bonus of up to $200,000 or more is tied to the value of delivered environments. Meaningful founding equity is included. Visa sponsorship is available. Location
On-site five days per week in San Francisco, California, United States.
