We're looking for an experienced Psychometrician to serve as the measurement science backbone of our certification programs. In this role, you'll ensure that every assessment we build is valid, reliable, fair, and defensible - at global scale. You'll work at the intersection of workforce capability, assessment design, and data-driven quality assurance, partnering closely with subject matter experts, instructional designers, and program managers to translate complex technical competencies into rigorous, scalable assessments.
n your first 90 days, you'll complete a psychometric audit of existing assessments, establish quality benchmarks, and begin embedding rigorous measurement practices into our item development and review workflows. Within six months, you'll be the go-to expert on assessment validity and reliability across the program - and your fingerprints will be on every certification we launch.
This is a high-impact individual contributor role with significant influence over program strategy and quality standards.
Key job responsibilities
- Lead the psychometric design and validation of certification assessments across role-based and specialty-based programs, including written, practical, and performance-based formats
- Design and audit scalable Job Task Analyses (JTAs) in partnership with subject matter experts and operations stakeholders to establish defensible content blueprints
- Apply classical test theory (CTT) and item response theory (IRT) to evaluate item and test quality; conduct standard-setting studies (e.g., Angoff, Bookmark) to establish and validate cut scores
- Develop and govern item development, review, and validation workflows across global content contributors, ensuring consistency and rigor
- Analyze assessment data to identify bias, differential item functioning (DIF), and reliability issues; recommend and implement remediation strategies
- Maintain independence between training and certification development to protect assessment integrity
- Build and maintain psychometric reporting frameworks that surface actionable insights on pass rates, item performance, and certification quality
- Partner with third-party assessment vendors on test development, delivery, and quality assurance; hold vendors accountable to psychometric standards
- Communicate complex psychometric concepts clearly to non-technical stakeholders, including senior leadership
A day in the life
Your morning starts with a fresh cup of coffee and a dataset that's already asking questions. You're reviewing item-level statistics from a recent pilot, flagging a handful of underperforming items. Mid-morning, you're deep in a standard-setting session with SMEs, guiding them through an Angoff exercise with patience and precision. After lunch, you're presenting DIF analysis results to the program team - translating p-values into plain language without losing the rigor. You close the day refining the content blueprint for an upcoming JTA. It's measurement science at its most applied. You love it.
About the team
The DC Certifications team is redefining how work readiness is measured across AWS data centers. Rather than tracking training completion, we deliver outcome-based certifications that validate whether a Builder can confidently do the job.
Our program assesses competencies tied to specific data center tasks, using practical and simulated environments to maintain high, consistent standards. We automate assessment delivery to keep the process simple, while ensuring independence and rigor through qualified assessors. Behind the scenes, we provide the tooling and data solutions that help sites plan and manage their workforce's capabilities.
BASIC QUALIFICATIONS
- Knowledge of data analysis languages (e.g., R, Stata, Python, etc.) or other statistical software
- Master's degree in Psychometrics, Educational Measurement, Industrial-Organizational Psychology, or a related field
- 5+ years of experience in psychometrics, assessment development, or credentialing programs
- Demonstrated experience with JTA facilitation, item development, and standard-setting methodologies
- Experience working with large-scale, high-stakes certification or licensure programs
PREFERRED QUALIFICATIONS
- Ph.D. in econometrics, statistics, industrial engineering, operations research, optimization, data mining, analytics, or equivalent quantitative field
- Experience in technical or workforce credentialing programs (e.g., trades, engineering, operations)
- Familiarity with performance-based and simulation-based assessment design
- Experience managing vendor relationships in an assessment or credentialing context
- Ability to operate effectively in a fast-paced, globally distributed environment
The base salary range for this position is listed below. Your Amazon package will include sign-on payments and restricted stock units (RSUs). Final compensation will be determined based on factors including experience, qualifications, and location. Amazon also offers comprehensive benefits including health insurance (medical, dental, vision, prescription, Basic Life & AD&D insurance and option for Supplemental life plans, EAP, Mental Health Support, Medical Advice Line, Flexible Spending Accounts, Adoption and Surrogacy Reimbursement coverage), 401(k) matching, paid time off, and parental leave. Learn more about our benefits at https://amazon.jobs/en/benefits.
USA, VA, Herndon - 148,700.00 - 201,200.00 USD annually
USA, WA, Seattle - 148,700.00 - 201,200.00 USD annually
