About the RoleLead the data quality team at an early-stage AI evaluation company, building the strategy and systems used to assess and improve training data for AI agents. Your work will help ensure data is realistic, reliable, and useful for training.
What You'll Do- Lead development of systems to evaluate tasks across reinforcement learning environments, synthetic data, benchmarks, and domain-specific workflows.
- Define quality standards, metrics, experiments, and processes for assessing agent outputs.
- Develop scalable synthetic data validation methods, including failure analysis, task mutation checks, and trajectory audits.
- Partner with research engineers, domain experts, and data vendors to diagnose issues and improve data generation workflows.
- Turn research insights into production tools, dashboards, validation pipelines, and feedback loops.
- Mentor research engineers and promote technical rigor and clear communication.
What We're Looking For- At least 5 years of research or engineering experience building AI or machine learning data evaluation and quality systems.
- Experience leading technical teams or projects from problem definition through implementation and iteration.
- Advanced proficiency in Python, Docker, and Linux, with a technical education or background.
- Strong understanding of AI evaluations, post-training, and the qualities that make agent training data realistic, learnable, diverse, and reliable.
- Experience building evaluation infrastructure, benchmarks, synthetic data pipelines, or validation workflows.
- Ability to translate domain expertise and research insights into scalable review systems and production data pipelines.
- Strong written communication and comfort working independently in an early-stage startup environment.
Compensation & BenefitsSalary range: $150,000 to $180,000 annually. Visa sponsorship is available.
LocationOn-site in San Francisco, California, United States.
