About the RoleOwn complex data and evaluation programs for AI research teams and internal partners. This role brings together technical program management, data operations, and vendor coordination to deliver high-quality data that improves LLM and agent evaluation.
What You'll Do- Lead data and evaluation programs from scoping and requirements gathering through production, quality assurance, delivery, and retrospectives.
- Turn ambiguous technical requests into specifications with clear milestones, owners, dependencies, and acceptance criteria.
- Improve data quality processes using quantitative signals and qualitative review, including task coverage, difficulty, and reward distributions.
- Manage external vendors throughout delivery and scale reliable production pipelines.
- Improve workflows, documentation, automation, and cross-functional handoffs for data production and evaluation.
- Work with research, platform, and marketplace teams to turn findings into product improvements.
What We're Looking For- At least 3 years of experience owning complex technical programs in AI/ML, data operations, support engineering, applied research, forward-deployed engineering, or a related cross-functional environment.
- Experience as a Technical Program Manager or Strategic Program Lead at a startup, translating ambiguity into executable plans and maintaining momentum as requirements change.
- Strong quantitative and qualitative judgment, with the ability to assess dataset quality using aggregate metrics and individual examples.
- Experience improving data quality procedures, managing vendors, scaling delivery pipelines, and documenting or automating workflows.
- Background supporting AI research teams, technical enterprise customers, or frontier AI labs with urgent, changing requirements.
- Familiarity with RLHF, reinforcement-learning environments, agent evaluations, human-in-the-loop data pipelines, annotation, or expert data collection is valuable.
- Strong written and verbal communication skills, sound tradeoff judgment, and the ability to work independently in a fast-paced startup environment.
Compensation & BenefitsBase salary is $130,000 to $225,000 USD annually. Visa sponsorship and relocation support are available for strong full-time candidates.
LocationOn-site in San Francisco, California. The role is also open to remote candidates who can maintain 70% to 80% time zone overlap with San Francisco or Singapore.
