7+ years of software engineering experience in complex programs and contracts.
Bachelor's degree in Computer Science or related field, or equivalent experience.
Proficiency in InspectAI, Python, and MongoDB.
Experience with Jupyter notebooks and data visualization tools.
Strong skills in statistical analysis, data cleaning, and validation.
Responsibilities
Design and implement benchmark suites for AI model performance evaluation.
Develop benchmarks for agentic workflows and multi-step task performance.
Curate and validate evaluation datasets, including adversarial sets.
Produce scoring frameworks and visualizations for evaluation results.
Translate raw evaluation data into actionable insights for stakeholders.
Benefits
Collaborative work environment with cutting-edge technology.
Opportunities for professional development and skill enhancement.
Access to advanced tools and resources for project execution.
Full Job Description
Upcoming Tasking
AI Model Benchmarks - Design, implement, and maintain benchmark suites that measure model performance against defined evaluation criteria, including dataset curation, scoring, and results reporting.
Agentic AI Benchmarks - Develop benchmarks and test scenarios that evaluate agentic workflows and multi-step task performance, including tool-use, task completion, and reliability metrics.
Evaluation Dataset Development - Curate, validate, and version evaluation datasets and test scenarios, including any adversarial or edge-case sets needed to stress agent behavior.
Metrics, Analysis, and Reporting - Produce scoring frameworks, visualizations, and written analyses that translate raw evaluation results into actionable findings for stakeholders.
Required Qualifications/Experience
Seven years experience as a SWE, in programs and contracts of similar scope, type, and complexity is required. Bachelor's degree in Computer Science or related discipline from an accredited college or university is required. Four years of additional SWE experience on projects with similar software processes may be substituted for a bachelor's degree.
Preferred Qualifications/Experience
InspectAI
Python
MongoDB
Jupyter notebooks
Data visualization tools
Statistical analysis
Data cleaning and validation
Cloud or containerized environments
Git-based version control
Collaboration tools such as Jira or Confluence
Experience operating in airgapped or constrained environments
