B.S./M.S. in a quantitative field or equivalent experience.
6+ years in quality programs for ML/LLM systems.
Proven ability to design and run quality frameworks.
Strong grasp of measurement and reliability methods.
Experience with rater and annotator operations.
Hands-on with labeling/eval tooling and automation processes.
Understanding of data governance and compliance.
Responsibilities
Ensure quality-driven workflows for every Applied AI deliverable.
Own technical tooling and quality infrastructure for automated metrics.
Manage operational quality across multiple concurrent programs.
Document delivery sign-offs and manage go/no-go decisions.
Lead customer quality remediation and dispute resolution.
Benefits
Opportunity to work with cutting-edge AI technologies.
Collaborative environment emphasizing quality and precision.
Professional development in advanced quality assurance techniques.
Access to data-driven decision-making tools and analytics.
Engagement with cross-functional teams including engineering and product.
Full Job Description
Key Responsibilities
Ensure Workflows Are Built for Quality
Outcome: Every Applied AI deliverable ships with measured quality: rater agreement meets agreed thresholds per dimension, rubric scores are reproducible across raters, and reported quality figures hold up under customer audit.
Own Technical Tooling & Quality Infrastructure
Outcome: Quality metrics for every active program (defect rate, acceptance rate, rework, reviewer reliability, throughput, SLA adherence) are available without manual compilation, and systemic quality issues are detected from workflow data before a customer reports them.
Own Operational Quality at Scale
Outcome: Concurrent programs meet their quality targets within agreed cost and throughput. Acceptance rates hold at or above target, rework declines over time, and standard QC on standardized workflows is executed and interpreted by Production staff without Quality Manager involvement by the end of the second quarter in role.
Delivery sign-off: Every deliverable has a documented go/no-go decision before shipping, based on programmatic QA and a readiness memo with flags. Post-sign-off quality escapes stay below an agreed threshold, and each escape results in a documented process change.
Own Customer Quality Remediation
Outcome: Customer quality disputes are closed with quantitative analysis, normally at first response. Re-adjudication completes within SLA, recurring defect classes decline release over release, and annotator integrity issues are detected internally before they reach a deliverable.
Qualifications
Education: B.S./M.S. in a quantitative field (CS, Statistics, Data Science, Engineering, Linguistics, HCI) or equivalent practical experience.
Applied AI / evaluation experience: 6+ years building or operating quality programs for ML/LLM systems (data labeling, benchmark creation, model evaluation, or human-in-the-loop pipelines).
Quality systems leadership: Proven ability to design and run end-to-end quality frameworks (rubrics, sampling plans, QC gates, acceptance criteria, escalation paths) across multiple concurrent programs.
Measurement & statistics: Strong grasp of reliability and agreement methods (e.g., Krippendorff's , Cohen's ), power/sample-size intuition, error analysis, and reporting for executive stakeholders.
Rater/annotator operations: Experience with calibration, adjudication, targeted retraining, drift monitoring, and integrity/fraud detection in high-throughput human evaluation programs.
Tooling & automation: Hands-on with labeling/eval tooling (e.g., Label Studio or similar) and designing deterministic validations; comfort partnering with engineering/research to implement quality instrumentation.
Data governance: Strong understanding of dataset/version management, traceability, privacy/compliance constraints, and audit-ready documentation for deliverables.
Stakeholder management: Excellent written and verbal communication; able to translate ambiguous research goals into measurable quality targets and negotiate tradeoffs with Applied AI, product, and external partners.
Preferred Skills
Fluency in multiple human languages.
Proficiency in a programming language and experience working with modern AI frameworks.
