Legal and Compliance Operations is seeking an Automation Program Manager to help us automate high-volume review at scale across Legal and Compliance workflows. Our workflows run on a composite of LLM-based decisioning, rules-based logic that encodes policy directly, and signals drawn from reporter history, content matching and behavioral patterns. Automation now carries a substantial share of enforcement volume, which means its quality determines both the legal risk we mitigate and the experience of the rights holders, regulators and people who depend on these decisions. As an Automation Program Manager, you will own the measured quality of assigned automations. Half of the role is forward motion: taking an automation from where it performs today to where the operational plan requires, through iteration on prompts, rules and signal thresholds. The other half is regression response: when reliability, correctness, precision or recall degrades, you find out why and drive it back. An effective candidate in this role is rigorous about measurement, comfortable working across all three automation types, and able to tell a genuine regression apart from a shift in what is arriving.
Responsibilities
Hill-climb reliability (prompt, iterate and improve), correctness, precision and recall on assigned automations across Legal and Compliance Workflows
• Iterate on LLM components, including prompts, task decomposition, routing, and abstention and fallback behavior
• Diagnose and improve rules-based components, including coverage, ordering and precedence, conflicts between rules, rules firing on the wrong population, and rules that stop matching after a policy or schema change
• Tune signal-based components, including thresholds, signal freshness and availability, and the precision cost of upstream signal drift
• Investigate regressions to isolate whether the cause is model, rule, signal, upstream data, policy change, or a shift in the incoming population, and distinguish real degradation from volume mix shift
• Drive regressions to a measured recovery that holds across a following measurement period, and document what degraded, why it was not caught earlier, and what detection would have caught it
• Build and maintain offline eval and gold standard data sets, coordinating expert-review sampling with the reviewers who hold the ground truth
• Build regression detection that surfaces degradation before the next reporting cycle, and close measurement coverage gaps so automations that are live are also measured
• Build scripts, notebooks and internal tooling so program managers and operations leads can run measurement independently
• Document what was tried, what moved the metric, and what did not, so subsequent workflows start further along
Minimum Qualifications
• 2+ years of relevant experience in data annotation, labeling, quality assurance, or content review operations, including time spent close to automated decision output
• Experience with guideline accuracy and inter-rater agreement, including diagnosing why qualified reviewers disagree
• Working knowledge of Python and SQL, with experience pulling a dataset, computing precision and recall, and segmenting results to locate where loss is concentrated
• Experience reading and reasoning about conditional logic to trace how a decision was reached
• Experience working with large operational datasets in a notebook environment
• Investigative approach to metric movement, identifying cause rather than the explanation that fits
• Experience communicating results clearly, including reporting a metric that has not moved or a regression that is not yet explained
Preferred Qualifications
• Experience in adversarial or fraud-adjacent domains where the reviewed population actively shifts
• Experience with rules engines, decision tables, routing logic or classifier thresholds
• Experience coordinating with vendor or cross-functional reviewer pools
• Demonstrated ongoing AI skill development (e.g., prompt/context engineering, agent orchestration) and staying current with emerging AI technologies
• Demonstrated ability to integrate AI tools to optimize/redesign workflows and drive measurable impact (e.g., efficiency gains, quality improvements)
• Experience or expertise in legal and compliance operations, or trust and safety
• Experience iterating on prompts with measured before-and-after results
• Experience adhering to and implementing responsible, ethical AI practices (e.g., risk assessment, bias mitigation, quality and accuracy reviews)
