About the RoleEvaluation is the bottleneck in healthcare AI - you can't ship what you can't measure. You'll build the systems that tell us whether our models are safe, accurate, and ready for real patients: evaluation frameworks, synthetic data pipelines, automated benchmarks, and LLM-as-judge systems. This is a high-leverage engineering role where your work directly gates what goes to production.
What You'll Do- Design and build evaluation frameworks for LLM safety, clinical accuracy, and conversational quality
- Develop synthetic data generation pipelines to stress-test models across diverse clinical scenarios
- Build automated and human-in-the-loop evaluation pipelines at scale
- Create benchmarks, metrics, and LLM-as-judge systems for healthcare tasks and conversational experience
- Analyze failure modes and translate findings into actionable model improvements by collaborating with the LLM post-training team
- Collaborate with research, engineering, and clinical teams to define and raise the quality bar
What You BringMust-Have- MS or PhD in CS or related field
- 5+ years in ML engineering, evaluation systems, or applied ML
- Strong software engineering skills - Python, PyTorch, and production-quality code
- Hands-on experience with LLM evaluation, benchmarking, or synthetic data generation
- Comfort building robust data analysis and evaluation infrastructure, not just running experiments
- Experience with UI/UX and front-end development toolkits such as Streamlit, Gradio, React, etc.
Nice-to-Have- Experience in healthcare AI
- Experience with RL/RLVR/RLHF or safety evaluation
Please be aware of recruitment scams impersonating Hippocratic AI. All recruiting communication will come from [redacted].com email addresses. We will never request payment or sensitive personal information during the hiring process.
