3+ years managing complex technical programs in AI/ML or related fields.
Proven ability to translate ambiguous technical requirements into actionable plans.
Strong judgment in assessing dataset quality and metrics.
Experience managing vendor relationships and improving delivery pipelines.
Background in supporting AI research teams or technical enterprise customers.
Familiarity with reinforcement learning and human-in-the-loop data processes is a plus.
Ability to work independently in a fast-paced environment.
Responsibilities
Manage data and evaluation programs from inception to delivery.
Translate technical requests into clear specifications with milestones and criteria.
Enhance data quality processes using quantitative and qualitative metrics.
Oversee external vendors throughout the delivery lifecycle.
Streamline production workflows and improve documentation and automation.
Collaborate with cross-functional teams to apply program insights to product enhancements.
Benefits
Visa sponsorship and relocation support for strong candidates.
Full Job Description
About the Role
Own complex, time-sensitive data and evaluation programs for AI research teams and internal partners. This role brings together technical program management, data operations, and vendor coordination to turn evolving requirements into reliable programs that improve AI and agent evaluation. What You'll Do
Manage data and evaluation programs from scoping and requirements gathering through production, quality assurance, delivery, and retrospectives.
Translate ambiguous technical requests into specifications with clear milestones, owners, dependencies, and acceptance criteria.
Improve data quality processes through quantitative metrics and qualitative review, including analysis of task coverage, difficulty, and reward distributions.
Manage external vendors across the delivery lifecycle and scale delivery pipelines.
Improve production workflows, documentation, automation, and cross-functional handoffs.
Partner with research, platform, and marketplace teams to turn program learnings into product improvements.
What We're Looking For
At least 3 years of experience owning complex technical programs in AI/ML, data operations, support engineering, applied research, forward-deployed engineering, or a related cross-functional field.
Experience turning ambiguous technical requirements into executable plans and maintaining data quality through metrics and direct inspection.
Strong judgment assessing dataset quality, moving between aggregate measures and individual examples.
Experience managing vendors, improving delivery pipelines, and documenting or automating workflows and handoffs.
Background supporting AI research teams, technical enterprise customers, or other teams with urgent, changing requirements.
Familiarity with RLHF, reinforcement-learning environments, agent evaluation, human-in-the-loop data pipelines, annotation, or expert data collection is valuable.
Comfort working independently in a fast-paced startup environment.
Compensation & Benefits
Base salary is $130,000 to $225,000 USD annually. Visa sponsorship and relocation support are available for strong full-time candidates. Location
On-site in San Francisco, California. The role is also open to remote candidates who can maintain 70% to 80% time zone overlap with San Francisco or Singapore.
