Amazon's Customer Experience and Business Trends (CXBT) is looking for an Applied Scientist to help shape the science of evaluating agent-driven builder experiences on AWS. As builders increasingly delegate goals to autonomous coding agents, the agent's experience becomes the builder's experience - and measuring it rigorously is an open research problem. At Amazon Benchmarking, we are developing new scientific methodologies to evaluate how autonomous agents perceive, reason about, and build on AWS.
This role sits at the intersection of GenAI, agent evaluation, and developer-experience research. You will design principled measurement methodologies that treat autonomous agents as instruments, capturing behavioral evidence rather than self-reported outcomes and develop rigorous, reproducible techniques for reasoning about why agents make the architectural decisions they do. Evaluating those decisions demands a scientist who understands cloud architecture and services in depth: to reason about the choices an agent makes, you must understand the trade-offs it is navigating. Our team's mission is to constantly challenge the status quo and drive innovation to improve customer experience.
This role requires an expert in the areas of GenAI and machine learning with a keen interest in research. The ideal candidate will have experience with deep learning models for natural language processing tasks, agentic/tool-using systems, and the evaluation of autonomous agents, along with a deep working knowledge of cloud architecture and services.
A successful candidate will have a passion for clarity, be a self-starter comfortable with ambiguity, have strong attention to detail, be able to work in a fast-paced and entrepreneurial environment and be driven by a desire to innovate. Given the complex nature of the questions we are working on, the ideal candidate will be relentlessly curious and be able to/be interested in having the business conversation underlying the data.
Key job responsibilities
- Develop novel scientific methodologies for evaluating autonomous, tool-using agents, advancing the state of the art in agent behavioral measurement and developer-experience research.
- Design rigorous, reproducible experiments that yield causal insight into agent decision-making, controlling for confounds and ensuring findings are grounded in observed behavior rather than self-report.
- Apply deep knowledge of cloud architecture and services to reason about the architectural decisions agents make - the service trade-offs, integration constraints, and deployment considerations that shape what gets built.
- Develop verification and synthesis methods that ensure only evidence-grounded conclusions survive, advancing techniques for reliability and faithfulness in agent evaluation.
- Design evaluation methodologies and metrics that quantify the quality of agent-produced outcomes across dimensions such as correctness, robustness, and security posture.
- Publish and present your work at internal and external scientific venues in the fields of ML.
- Collaborate with scientists, engineers, and product managers to design and implement science-focused systems.
- Proficiency in model development, validation and implementation for NLP applications.
- Excellent verbal and written communication and presentation skills, ability to convey rigorous mathematical concepts and considerations to non-experts.
A day in the life
As a scientist on the team, you will be involved in every aspect of the process from idea generation, business analysis and scientific research, through development and deployment of advanced models and agentic systems, giving you a real sense of ownership. You'll research, design, and build new methodologies to evaluate the experience of autonomous agents building on AWS - developing the measurement science that lets us reason rigorously about what agents do and why. Besides theoretical analysis and innovation, you will work closely with talented engineers and put your algorithms and models into practice.
About the team
Customer Experience and Business Trends (CXBT) is an organization made up of a diverse suite of functions dedicated to deeply understanding and improving customer experience, globally. We are a team of builders that develop products, services, ideas, and various ways of leveraging data to influence product and service offerings - for almost every business at Amazon - for every customer (e.g., consumers, developers, sellers/brands, employees, investors, streamers, gamers).
Our approach is based on determining the customer need, along with problem solving, and we work backwards from there. We use technical and non-technical approaches and stay aware of industry and business trends. We are a global team, made up of a diverse set of profiles, skills, and backgrounds - including: Scientists, Product Managers, Software Developers, Computer Vision experts, Solutions Architects, Business Intelligence Engineers, Business Analysts, Risk Managers, and more.
BASIC QUALIFICATIONS
- 3+ years of building models for business application experience
- PhD, or Master's degree and 4+ years of CS, CE, ML or related field experience
- Experience in patents or publications at top-tier peer-reviewed conferences or journals
- Experience programming in Java, C++, Python or related language
- Experience in any of the following areas: algorithms and data structures, parsing, numerical optimization, data mining, parallel and distributed computing, high-performance computing
- Hands-on experience (academic or industrial) building ML models for natural language processing tasks using GenAI
PREFERRED QUALIFICATIONS
- Deep working knowledge of cloud architecture and services (e.g., compute, storage, databases, auth, networking) and the trade-offs involved in composing them into production systems
- Experience with agentic/tool-using systems, LLM-based agents, or the evaluation of autonomous agents
- Experience using cloud-based services and big data processing technologies such as AWS Redshift, EMR, Spark, and Sagemaker
- Prior work experience as a research scientist or applied scientist at a consumer product company
- Excellent problem-solving ability. Demonstrable track record of dealing well with ambiguity, prioritizing needs, and delivering results in a dynamic environment
- Experience in professional software development
The base salary range for this position is listed below. Your Amazon package will include sign-on payments and restricted stock units (RSUs). Final compensation will be determined based on factors including experience, qualifications, and location. Amazon also offers comprehensive benefits including health insurance (medical, dental, vision, prescription, Basic Life & AD&D insurance and option for Supplemental life plans, EAP, Mental Health Support, Medical Advice Line, Flexible Spending Accounts, Adoption and Surrogacy Reimbursement coverage), 401(k) matching, paid time off, and parental leave. Learn more about our benefits at https://amazon.jobs/en/benefits.
USA, WA, Seattle - 142,800.00 - 193,200.00 USD annually
