About the Role The Quick Data team is hiring a Data Scientist II to lead evaluation and benchmarking data development for Quick features. In this role, you will design the datasets, metrics, and evaluation frameworks that show whether Quick meets customer needs. Your insights will shape product direction and model improvements.
Key job responsibilities
Design and build evaluation and benchmark datasets
Define quality metrics and evaluation frameworks, including human evaluation, model-based evaluation (LLM-as-a-Judge), and checks on how well the two correlate
Run human annotation programs from start to finish: write guidelines, measure inter-annotator agreement, and control data quality
Analyze model and agent performance to find gaps, root causes, and the highest-impact improvements
Partner with science, engineering, and product teams to turn evaluation results into launch decisions and roadmap priorities
Present findings and recommendations to senior leadership
BASIC QUALIFICATIONS
- 5+ years of data querying languages (e.g. SQL), scripting languages (e.g. Python) or statistical/mathematical software (e.g. R, SAS, Matlab, etc.) experience
- 4+ years of data scientist experience
- Experience as a leader and mentor on a data science team
PREFERRED QUALIFICATIONS
- 2+ years of data visualization using AWS QuickSight, Tableau, R Shiny, etc. experience
- Experience managing data pipelines
- Experience evaluating LLMs, RAG systems, or agentic AI applications
The base salary range for this position is listed below. Your Amazon package will include sign-on payments and restricted stock units (RSUs). Final compensation will be determined based on factors including experience, qualifications, and location. Amazon also offers comprehensive benefits including health insurance (medical, dental, vision, prescription, Basic Life & AD&D insurance and option for Supplemental life plans, EAP, Mental Health Support, Medical Advice Line, Flexible Spending Accounts, Adoption and Surrogacy Reimbursement coverage), 401(k) matching, paid time off, and parental leave. Learn more about our benefits at https://amazon.jobs/en/benefits.
USA, CA, Santa Clara - 183,000.00 - 247,600.00 USD annually