Apple

Annotation Data Scientist, Evaluation Integrity (Siri)

Apple$130K — $155K *
Consumer Technology
5 - 7 years of experience
Job Overview by Ladders

Qualifications

  • Bachelor's or Master's degree in a quantitative field or equivalent experience.
  • 5+ years in human-annotated datasets or human-in-the-loop evaluation for ML/NLP.
  • 5+ years of Python experience for data processing and analysis.
  • Experience designing annotation schemas or rubrics for ML data.
  • Proven management of dataset curation efforts and annotator performance monitoring.

Responsibilities

  • Design and manage human-in-the-loop annotation projects.
  • Evaluate agentic user personae and agent-to-agent conversations.
  • Calibrate annotators and ensure high-quality data output.
  • Analyze and convert annotator judgments into actionable insights.
  • Collaborate on custom annotation tooling with software engineers.

Benefits

  • Opportunity to contribute to cutting-edge human-computer interaction technologies.
  • Collaborative work environment across cross-functional teams.
  • Engagement in high-impact projects that influence product decisions.
  • Access to ongoing professional development and training resources.
  • Potential for projects with large-scale datasets and various locales.
Full Job Description
Play a part in the ongoing revolution in human-computer interaction. Siri is evolving - and the way we evaluate it has to evolve with it. Join the Evaluation Integrity team to help build the trusted quality signal behind every Siri release. Within the Siri evaluation organization, the Human Evaluation sub-team is responsible for answering the question: can we trust our evals? We do that by designing human-in-the-loop (HITL) annotation tasks that scrutinize every moving part of an agentic evaluation - the simulated user agent, the conversation it has with Siri, and the automated evaluators that grade the exchange. This role sits at the intersection of data science, human annotation engineering, and evaluation methodology, and is instrumental in turning human judgment into a rigorous, reproducible signal that directly informs pre-ship model and product decisions. As an Annotation Data Scientist on the Evaluation Integrity team, you will design and run HITL annotation projects that evaluate the quality and authenticity of agentic user personae, the validity of agent-to-agent conversations, and the reliability of LLM-as-judge and rule-based evaluators against Siri's product specifications. You will own annotation initiatives end-to-end; from rubric design and tooling, through annotator calibration, to data science analysis that turns annotator judgments into actionable signal for modeling, planning, and product teams.

Minimum Qualifications

Bachelor's or Master's degree in a quantitative or related field such as Data Science, Computer Science, Linguistics, Statistics, or Cognitive Science, or equivalent job-related experience.

5+ years of hands-on experience working with human-annotated datasets or human-in-the-loop evaluation methodologies for machine learning, natural language processing, or large language model systems.

5+ years of experience using Python for data processing, analysis, and prototyping, including experience with libraries such as pandas, Jupyter, and at least one data visualization library.

Experience designing, implementing, and communicating annotation schemas, rubrics, or ontologies for machine learning training or evaluation data.

Experience managing multiple concurrent dataset curation efforts, including scoping work, iterating on guidelines, coordinating with in-house or vendor annotators, and monitoring annotator performance metrics such as accuracy, throughput, and inter-annotator agreement.

Experience specifying or designing custom annotation tooling in collaboration with software engineers.

Preferred Qualifications

Experience evaluating LLM-powered or agentic systems, including familiarity with LLM-as-judge methodologies, rubric-based grading, or trajectory and tool-call evaluation.

Familiarity with statistical methods that address accuracy and variability in human annotation data, such as inter-annotator agreement, Cohen's or Fleiss' kappa, Krippendorff's alpha, or bootstrapping.

Data-querying experience with SQL, Spark, or similar, and comfort working with large, complex, real-world datasets.

Experience building pre-ship evaluation pipelines for conversational or assistant products.

Experience with prompt engineering, or with designing simulated user personae for agent evaluation.

Experience running annotation programs across multiple locales or at large scale.

Excellent written and verbal communication skills, with the ability to explain technical topics clearly to data scientists, engineers, annotators, and cross-functional partners.

Proven ability to collaborate effectively across functions and drive projects of varying sizes and scopes - knowing when to dive deep and when to delegate.

About Apple

Apple is a corporation that designs, manufactures, and markets mobile communication and media devices, personal computers, portable digital music players, and sells a variety of related software, services, peripherals, networking solutions, and third-party digital content and applications. Apple provides many products and services, including iPhone; iPad; iPod; Mac; Apple TV; a portfolio of consumer and professional software applications; the iOS and OS X operating systems; iCloud; and accessories, service, and support offerings. It sells its products worldwide through its retail stores, online stores, direct sales force and third-party cellular network carriers, wholesalers, retailers, and value-added resellers to the consumer and also sells third-party iPhone, iPad, Mac and iPod compatible products, including application software and accessories through its online and retail stores. Introduced in 1984, the Macintosh was the first widely sold personal computer with a graphical user interface (GUI). That feature and others such as an improved floppy drive design and a low-cost hard drive that made data retrieval faster helped Apple cultivate a reputation for innovation. Apple was named as the most admired company in the United States in 2008 and in the world from 2008 to 2012 by the Fortune magazine. The company was founded by Steven Paul Jobs, Steve Wozniak, and Ronald Gerald Wayne on April 1, 1976, and is headquartered in Cupertino, California.

Apple Careers

Join Apple, a place where extraordinary people gather to do their best work. Our ever-expanding global team is at the forefront of innovation and leadership in the tech industry. At Apple, we're not just building products—we're crafting the kind of wonder that revolutionizes entire industries. It's the diversity of our people and their ideas that inspires the innovation that runs through everything we do, from amazing technology to industry-leading environmental efforts. Work You’ll Do Embark on a journey with Apple’s market-leading team to help some of the world’s most influential companies navigate their path to digital mastery with cutting-edge technology and services. Transform industries and touch lives with your unique ideas at Apple. Here, you’ll lead through a unique position at the intersection of technology, creativity, and robust industry expertise. Collaborate with a global team of professionals who are at the top of their game in technology and design. Apple isn’t just a company, it’s a community of innovators and passionate thinkers. Introducing the Apple Innovation and Leadership Initiative We are building a market-leading team to drive our efforts in delivering groundbreaking solutions and services. At Apple, job opportunities are abundant, offering you the chance to explore diverse roles from engineering to marketing, all designed to empower your career growth. Do Innovative Work Join the largest group of creative and technical experts in the world—professionals dedicated to redefining what’s possible through technology and innovation. Deliver targeted solutions through a depth and breadth of expertise that’s unmatched, driving forward our commitment to excellence and leadership in every project we undertake. Be Part of a Great Team Engage in a wide range of projects utilizing Apple’s technology and resources. Harness the unparalleled capabilities, global scale, and joint solution development that only Apple can offer. Future-Proof Your Career Advance your career with limitless opportunities at Apple. Go as far as your ambition takes you with unmatched training, development, and certification support. Explore Discover how Apple is leading the way in tech innovation: [With iOS] Businesses can streamline operations and enhance customer interactions... READ MORE Smart home technology integration that sets the standard for convenience and security... READ MORE The Apple Experience Our combined service capabilities, global scale, and joint solution development help clients overcome challenges and lead transformation in their industries. Clients worldwide look to Apple for new strategies and solutions that drive growth and innovation in the digital era. Stay Connected Join Our Team Search open positions that match your skills and interests. We look for passionate, curious, creative, and solution-driven team players. Whether you’re seeking a professional role, an internship, or a leadership position, Apple offers a variety of employment opportunities. SEARCH APPLE JOBS Keep Up to Date Stay ahead with career tips, insider perspectives, and industry-leading insights you can put to use today—all from the people who work here. READ CAREERS BLOG Job Alert Emails Personalize your subscription to receive job alerts, latest news, and insider tips tailored to your preferences. See what exciting and rewarding opportunities await at Apple, a company committed to diversity, innovation, and leadership. Explore job opportunities, employment benefits, and the culture of growth and innovation at Apple. Prepare your resume, hone your interview skills, and ready yourself for a career at one of the most prestigious companies in the world. Join us in pushing the boundaries of what is possible.
Learn more about Apple
Size
154,000 employees
Market Cap
$2,074.3 billion
Industry
Net Income
$63.9 billion
Founded
1976
5 Year Trend
+11.5%
Revenue
$294.1 billion
NASDAQ

Similar Jobs

More Jobs at Apple

More Consumer Technology Jobs

Find similar Annotation Data Scientist, Evaluation Integrity (Siri) jobs: