Lead Assessment Developer

Outsmart

$120K — $180K *
Education, Government & Non-Profit
Less than 5 years of experience
Job Overview by Ladders

Qualifications

  • Master's degree (Ph.D. preferred) in educational measurement, assessment, or related field.
  • 3+ years managing operational, high-stakes assessment lifecycles end-to-end.
  • Proven experience leading standard-setting studies and managing diverse subject matter expert (SME) panels.
  • Experience managing external vendors for field tests and pilot studies.
  • Working knowledge of validity, reliability, and fairness standards (e.g., AERA/APA/NCME).
  • Familiarity with cognitive complexity frameworks (DOK, Bloom's).
  • Strong communicator who translates psychometric concepts for non-technical audiences.

Responsibilities

  • Manage end-to-end assessment development operations for assigned assessments.
  • Recruit, train, calibrate, and ensure quality of the SME network.
  • Facilitate standard-setting studies, including method selection and final reporting.
  • Manage vendor and external partner relationships, including SLAs and security protocols.
  • Oversee field testing and pilot operations.
  • Handle scoring operations for constructed-response and performance-based assessments.
  • Manage project timelines, risk identification, and stakeholder communication.

Benefits

  • Competitive salary and equity.
  • Healthcare coverage for employees and dependents (medical, dental, vision).
  • Parental leave available for all full-time employees after 6 months.
  • 15 PTO days per year, winter break, and 12+ paid company holidays.
  • Remote work flexibility.
  • Daily in-office lunch provided.
  • Collaborate with talented colleagues focused on making a positive impact.
Full Job Description
What You'll Be Doing

You'll manage the full assessment development lifecycle at Outsmart - from item authoring through field testing, standard setting, scoring, and reporting. You'll work closely with psychometricians, faculty, product managers, and engineers to build rigorous, scalable assessments. As an AI-forward organization, you'll also help shape how AI tools are responsibly woven into our development and QA workflows.

Who You Are
  • Master's degree (Ph.D. preferred) in educational measurement, assessment , or a related field
  • 3+ years managing operational, high-stakes assessment lifecycles end-to-end
  • Proven experience leading standard-setting studies and managing diverse SME panels
  • Experience managing external vendors for field tests and pilot studies
  • Working knowledge of validity, reliability, and fairness standards (e.g., AERA/APA/NCME)
  • Familiarity with cognitive complexity frameworks (DOK, Bloom's)
  • Strong communicator who can translate psychometric concepts for non-technical audiences
  • Bonus: knowledge of CTT, IRT, CDM, DIF,, AI-assisted item generation, or accreditation reviews


What You'll Own
  • End-to-end assessment development operations for assigned assessments
  • SME network - recruitment, training, calibration, and quality
  • Standard-setting studies - method selection, panel facilitation, and final reporting
  • Vendor and external partner relationships, SLAs, and security protocols
  • Field testing and pilot operations
  • Scoring operations for constructed-response and performance-based assessments
  • Project timelines, risk identification, and stakeholder communication
  • Core assessment documentation - specs, blueprints, training materials, technical reports


What Helps You Succeed
  • You thrive in ambiguity and are energized by building from scratch
  • You balance psychometric rigor with startup pragmatism - you know when to ship and iterate
  • You communicate with unusual clarity across technical and non-technical audiences
  • You take ownership without being asked and surface risks early
  • You're excited about AI's role in assessment and know where validation is non-negotiable
  • You care deeply about fairness and the learners behind every score

Salary Range: $120K-$180K

Perks
  • Competitive salary and equity
  • Healthcare for you and your dependents (medical, dental, and vision)
  • Parental leave for any full-time employee who has been with us for at least 6 months (birthing, non-birthing, and adoptive parents)
  • 15 PTO days per year, a winter break, and 12+ paid company holidays per year
  • Remote work days
  • Lunch provided in-office daily
  • Working with super talented people who want to make a difference in the world

Similar Jobs

More Jobs at Outsmart

  • Senior Applied AI Engineer
    $190K — $250K *
    Los Angeles, CA 90011 (Los Angeles County)
    Information Technology
    In-Person
  • Senior Applied AI Engineer
    $190K — $250K *
    San Francisco, CA 94112 (San Francisco County)
    Consumer Technology
    In-Person
  • Principal Product Manager
    $150K — $250K *
    San Francisco, CA 94112 (San Francisco County)
    Education, Government & Non-Profit
    In-Person
  • Principal Product Manager
    $150K — $250K *
    Lehi, UT 84043 (Utah County)
    Education, Government & Non-Profit
    In-Person
  • Principal Product Manager
    $150K — $250K *
    Los Angeles, CA 90011 (Los Angeles County)
    Education, Government & Non-Profit
    In-Person

More Education, Government & Non-Profit Jobs

Find similar Lead Assessment Developer jobs: