What You'll Be DoingYou'll manage the full assessment development lifecycle at Outsmart - from item authoring through field testing, standard setting, scoring, and reporting. You'll work closely with psychometricians, faculty, product managers, and engineers to build rigorous, scalable assessments. As an AI-forward organization, you'll also help shape how AI tools are responsibly woven into our development and QA workflows.
Who You Are- Master's degree (Ph.D. preferred) in educational measurement, assessment , or a related field
- 3+ years managing operational, high-stakes assessment lifecycles end-to-end
- Proven experience leading standard-setting studies and managing diverse SME panels
- Experience managing external vendors for field tests and pilot studies
- Working knowledge of validity, reliability, and fairness standards (e.g., AERA/APA/NCME)
- Familiarity with cognitive complexity frameworks (DOK, Bloom's)
- Strong communicator who can translate psychometric concepts for non-technical audiences
- Bonus: knowledge of CTT, IRT, CDM, DIF,, AI-assisted item generation, or accreditation reviews
What You'll Own- End-to-end assessment development operations for assigned assessments
- SME network - recruitment, training, calibration, and quality
- Standard-setting studies - method selection, panel facilitation, and final reporting
- Vendor and external partner relationships, SLAs, and security protocols
- Field testing and pilot operations
- Scoring operations for constructed-response and performance-based assessments
- Project timelines, risk identification, and stakeholder communication
- Core assessment documentation - specs, blueprints, training materials, technical reports
What Helps You Succeed- You thrive in ambiguity and are energized by building from scratch
- You balance psychometric rigor with startup pragmatism - you know when to ship and iterate
- You communicate with unusual clarity across technical and non-technical audiences
- You take ownership without being asked and surface risks early
- You're excited about AI's role in assessment and know where validation is non-negotiable
- You care deeply about fairness and the learners behind every score
Salary Range: $120K-$180K
Perks- Competitive salary, equity, and benefits
- Healthcare for you and your dependents (medical, dental, and vision)
- Parental leave for any full-time employee (birthing, non-birthing, and adoptive parents)
- 15 PTO days per year, a winter break, and 12+ paid company holidays per year
- Remote work days
- Lunch provided in-office daily
- Working with super talented people who want to make a difference in the world
