5-7 years of experience in threat modeling or dual-use technologies
Proven ability to assess misuse risks in AI systems
Strong analytical skills with experience in evaluating model outputs
Expertise in translating technical evidence into policy guidelines
Hands-on experience with model data and evaluation results
Ability to collaborate across disciplines and adapt to changing priorities
Responsibilities
Build and update threat models as AI capabilities evolve
Define appropriate model responses to legitimate and harmful uses
Create training examples and evaluations for model safety
Analyze model outputs and deployment data for misuse identification
Collaborate with cross-functional teams to apply findings to model training
Benefits
Relocation support offered
Hybrid work model with three days in-office
Optional remote work on Thursdays and Fridays
Full Job Description
About the Role
As AI systems become more capable, the same advances that expand their legitimate uses can also lower barriers to harm. Emerging misuse risks range from AI assistance with conventional weapons development and the misuse of AI-enabled robotics to fraud, impersonation, and other forms of abuse that people can carry out at scale using AI agents. You'll build threat models for how people could misuse AI in digital and physical settings, assess what access to AI changes in practice, and use that evidence to define how models should respond.
This role is based in San Francisco. We offer relocation support and work from the office three days a week, with optional work from home on Thursdays and Fridays.
In this role, you will:
Build and update threat models as capabilities evolve. Assess what people could plausibly do, what expertise and resources they would need, and how AI changes those barriers.
Define when models should help, limit assistance, or refuse in situations where the same capability has legitimate and harmful uses.
Turn expert safety judgment into instructions, training examples, and evaluations that teach models when to help, when to limit assistance, and when to refuse.
Turn threat models into evaluation scenarios and grading guidance. Analyze model outputs, red-team findings, and deployment data to identify where models enable misuse or block legitimate work.
Partner with research, engineering, security, and product teams to apply findings to model training, deployment decisions, and monitoring.
You might thrive in this role if you:
Have hands-on experience building, testing, or assessing dual-use technologies, or a track record of threat modeling grounded in how people use and misuse real systems.
Demonstrate deep practical experience in one relevant domain and the judgment to know when to bring in experts from others.
Are able to assess whether a capability demonstrated in a test materially changes what someone could do outside it.
Can translate technical evidence and uncertainty into clear policy boundaries and measurable evaluation criteria.
Have experience analyzing gold sets, evaluation results, and training data to spot inconsistent human labels, assess how annotator disagreement affects measured performance, and decide what to investigate next.
Are comfortable working hands-on with model data and evaluation results, including inspecting examples, analyzing failure patterns, assessing data quality, and distinguishing policy failures from grader, model, or system failures.
Use empirical evidence to develop and refine safety policies and safeguards.
Work well across disciplines and revises their judgment as capabilities and evidence change.
Enjoy fast-paced, collaborative research environments where priorities shift as models, evidence, and risks change.
Relevant Publications
Introducing the Model Spec
Managing frontier AI risk: From training to deployment
GPT-6.2 System Card (Astra and Sol)
OpenAI Deployment Safety Hub
Safety at every step
About OpenAI
OpenAI is an artificial intelligence research laboratory consisting of the for-profit corporation OpenAI LP and its parent company, the non-profit OpenAI Inc. The company was founded in 2015 by a group of technology leaders, including Elon Musk, Sam Altman, Greg Brockman, Ilya Sutskever, and John Schulman. OpenAI's mission is to develop and promote friendly AI for the betterment of humanity. The company has developed a number of cutting-edge AI technologies, including GPT-3, a language processing system that can generate human-like text. OpenAI has received funding from a number of high-profile investors, including LinkedIn co-founder Reid Hoffman and venture capitalist Peter Thiel.