Salary range: $400,000 - $650,000/year + benefits
About the role: As Scientific Lead for Embedded Assessments, you will scope, design, and lead the assessments carried out by a team of 3-5 evaluators embedded at a frontier AI lab. We expect each engagement to last 1-3 months, during which you'll work on-site at the host lab to assess the coverage of its internal safety stack. This involves identifying the most important open questions about model safety, auditing the lab's existing evaluations to find gaps in coverage, and designing experiments to investigate risks and failure modes that haven't been examined. You'll own the scientific content of each engagement's public report. Between engagements, you'll lead practice evaluations on public models, train other scientific staff, and work with our engineering team to improve our tooling.
Core responsibilities:- Refine the scientific scope of the assessment based on information gained while embedded.
- Review the output of already-conducted experiments or evals within the host organization with the aim of auditing coverage.
- Design and help execute experiments with the aim of evaluating safety practices of the host organization.
- Lead the scientific input into the final report that will be the culmination of the embedded assessment.
It is crucial for this work to happen in person, since much of the benefit of embedded evals comes from working physically within the host company's facilities.
Preferred qualifications:We are looking for candidates with an excellent track record of scientific research, ideally in an area adjacent to AI alignment. Other qualities we value for this role:
- Experience building AI evaluations, such as benchmarks, eval suites, or red-teaming work.
- Experience leading or managing researchers, ideally small, talented teams in a scrappy environment.
- Demonstrated ability to produce high-quality results while prioritizing under uncertainty and tight deadlines.
- Writing ability: both technical writing and the ability to communicate results to less technical team members as well as the general public.
- Ability to quickly incorporate a lot of technical context, in the form of code or other technical artifacts, and begin making progress towards a goal.
We are hiring at all levels of experience and would encourage those enthusiastic about the role who do not meet all of the qualifications to apply. We are located in San Francisco and excited to work together in-person. We are open to sponsoring international visas.
