Reality Labs Product Data Operations Program Manager - AI Data and Evaluation

Reality Labs Product Data Operations Program Manager - AI Data and Evaluation Own a portfolio of annotation and evaluation programs across Meta Reality Labs' perception and generative AI experiences, running each through established workflows and shared tooling so every delivery can be traced, reproduced, and handed off • Partner with research scientists, engineers, and technical program managers to translate model and evaluation requirements into task designs, rubrics, rater guidelines, and acceptance criteria • Operate, maintain, and extend existing program tooling, including task configurations, data pipelines, queries, dashboards, and automated reports, with the help of AI coding assistants • Define how quality is measured for each program: break down what a correct answer means into verifiable checks, set the acceptance rules a delivery must pass, and decide how rater disagreement is resolved, rather than applying one-size-fits-all metrics • Design, configure, and operate AI and automation across the annotation pipeline, including pre-annotation, LLM and VLM judges, and quality triage, and measure where they meet the bar before they replace a human step • Build and maintain evaluation practices, including gold-standard datasets, model versus human comparison, human preference evaluation for generative models, and the accuracy thresholds that determine whether data and AI outputs meet the bar • Analyze operational data to identify quality and throughput trends, building the dashboards and leadership-facing reporting that drive decisions, and stating the coverage behind every quality number • Drive resolution of risks and blockers across vendor capacity, tooling, data availability, and model training schedules • Manage external annotation vendors, including rater training, throughput, quality feedback, and per-unit cost. Document the systems and decisions you own so that any program can be handed to another program manager without loss Minimum Qualifications • Currently has, or is in the process of obtaining a Bachelor's degree in a directly related field, or equivalent practical experience. Degree must be completed prior to joining Meta • Bachelor's degree in a directly related field, or equivalent practical experience • 4+ years of experience in program management, operations, or technical operations within an AI, ML, research, or data-driven organization • Analytical experience using data to identify trends, diagnose operational issues, and influence program direction • Experience building or deploying automation or AI solutions in production workflows • Experience in communicating and influencing multiple cross-functional stakeholders and leadership • Experience solving problems and breaking down ambiguous issues into component parts to develop solutions • Familiarity with AI/ML concepts, applications, and industry trends sufficient to engage with technical teams and inform operational decisions Preferred Qualifications • Experience in AR/VR, wearables, or consumer electronics, or with perception data such as hand, body, or eye tracking • Experience running human data annotation programs at scale, including task design, rater guidelines, and quality measures for tasks with no gold standard • Experience using AI coding agents to build or maintain operational tooling, reports, or workflows • Demonstrated ability to integrate AI tools to optimize/redesign workflows and drive measurable impact (e.g., efficiency gains, quality improvements) • Experience owning program budgets, cost forecasting, or per-unit cost tracking • Experience adhering to and implementing responsible, ethical AI practices (e.g., risk assessment, bias mitigation, quality and accuracy reviews) • Experience designing evaluations for generative AI systems, including rubrics, human preference studies, or LLM-as-judge methods • Working familiarity with how computer vision and machine learning models are trained and evaluated, including benchmark datasets, ground truth, accuracy, and agreement measurement, and a high-level understanding of model limitations • Demonstrated ongoing AI skill development (e.g., prompt/context engineering, agent orchestration) and staying current with emerging AI technologies • Experience in writing SQL to analyze large datasets