5+ years in AI/ML data operations or production, including 2+ years managing project managers in a distributed environment.
Strong understanding of LLM training processes and evaluation methodologies.
Experience managing teams against throughput and quality targets with advanced spreadsheet and SQL skills.
Proven track record of on-time delivery across multiple data collection programs for enterprise customers.
Experience managing hourly and freelance staff across regions and languages.
Proven experience with Agile, Scrum, or Kanban methodologies for workflow management.
Ability to produce clear communication and guidelines for multilingual audiences.
Responsibilities
Ensure all active programs have accountable Project Managers and meet delivery targets without escalation.
Source, onboard, and develop new Project Managers to maintain program capacity and start all programs on time.
Identify and correct quality issues mid-program through retraining and guideline updates.
Standardize processes and implement software improvements to enhance program efficiency.
Proactively surface risks and manage escalations between project managers and cross-functional teams.
Benefits
Collaborative work environment with a focus on professional development.
Opportunities for continuous learning in AI and data operations.
Flexibility to work across different time zones with a diverse team.
Engagement in innovative AI projects impacting real-world applications.
Full Job Description
Key Responsibilities
PM Performance
Outcome: Every active program has accountable Project Manager(s), and every PM carries a workload within the agreed span. Programs hit on-time delivery and first-pass acceptance targets without escalation; each PM is reviewed monthly against a scorecard of throughput, quality, and cost-per-task.
PM Hiring, Onboarding, and Development
Outcome: PM pool capacity keeps pace with signed demand: new PMs are sourced, onboarded, and running their first program within the agreed ramp window, all programs start on time, and PM attrition is below threshold.
Quality Interventions Across Programs
Outcome: Quality dips are caught mid-program through QA loops and corrected via retraining of annotator pools or guideline updates. Repeated misses by a PM or annotator pool lead to documented remediation or replacement. Issues are proactively discovered.
Process Standardization & Software Improvements
Outcome: Programs launch from shared playbooks, guideline templates, and dashboard standards rather than being rebuilt per engagement. Every post-mortem produces documented improvements that lead directly into our custom software stack, and time from program handoff to first delivery declines quarter over quarter.
Escalation and Cross-Functional Interface
Outcome: Risks surface to Technical Program Managers early enough to be managed. Escalations between the PM pool, Quality, Talent, and Delivery are resolved within agreed timelines.
Qualifications
People management in AI data operations: 5+ years in AI/ML data operations or production, including 2+ years directly managing project managers or team leads in a distributed, multi-time-zone contractor environment.
LLM knowledge: Strong understanding of LLM training processes (pre-training, SFT, RLHF) and evaluation methodologies (human-in-the-loop, red teaming), and of what drives quality and throughput in annotation workflows.
KPI-driven management: Has run teams against throughput, quality (accuracy, IAA, gold-set), and cost-per-task targets; advanced proficiency with spreadsheets and dashboards, and able to use SQL to extract and analyze performance data.
Delivery track record: Has sustained on-time delivery and acceptance targets across multiple concurrent data collection or evaluation programs for enterprise or research lab customers.
Contractor workforce operations: Has hired, ramped, performance-managed, and offboarded hourly and freelance staff across regions and languages.
Methodology: Proven track record using Agile, Scrum, or Kanban to manage complex workflows across a portfolio of programs.
Communication: Writes clear, unambiguous guidelines and feedback for multilingual audiences and communicates status, risk, and tradeoffs crisply to leadership.
Preferred Skills
Fluency in multiple human languages.
Experience with multilingual data deliveries (pre-training, SFT, RLHF, machine translation, multimodal, etc.), especially in rare-resource languages
Experience with data annotation platforms (e.g., Label Studio, SuperAnnotate) and project management tooling (e.g., Jira).
Background in ML engineering, computer science, or data science.
