5+ years of software engineering experience in production AI/ML systems.
Experience in building evaluation frameworks and LLM-based applications.
Strong understanding of modern LLMs and evaluation methodologies.
Proficient in Python for building reliable, production-grade systems.
Ability to tackle ambiguous technical problems independently.
Excellent communication skills, both verbal and written.
Bonus: Experience with LLM-as-judge and human evaluation systems.
Responsibilities
Own evaluations across FurtherAI by building frameworks to measure AI agent performance.
Develop high-quality datasets and benchmarks in collaboration with engineers and domain experts.
Design evaluations for complex agent behaviors and multi-step task completions.
Create automated grading systems for accurate real-world quality assessments.
Integrate evaluations into development processes for continuous improvement.
Benchmark models based on accuracy, latency, cost, and reliability for informed decision-making.
Transform production failures and feedback into new evaluation cases for agent quality enhancement.
Benefits
Fully covered health, dental, and vision benefits.
Unlimited PTO for work-life balance.
Outstanding in-office culture with daily meals provided.
Team events to foster collaboration and camaraderie.
Pre-tax commuter benefits for ease of travel.
Full Job Description
What You'll Do
Own evaluations across FurtherAI: Build the evaluation frameworks and infrastructure that measure the accuracy, quality, and reliability of our AI agents across workflows, Assistant, document processing, and computer use.
Build high-quality datasets: Develop golden datasets, benchmarks, and regression suites, working closely with Forward Deployed Engineers and domain experts to capture real-world insurance tasks and edge cases.
Evaluate complex agent behavior: Design evaluations for multi-step agent trajectories, tool use, reasoning, and end-to-end task completion - not just individual LLM responses.
Build automated grading systems: Develop LLM-as-judge evaluations, deterministic graders, human evaluation workflows, and rubrics that accurately reflect real-world quality.
Make development evaluation-driven: Integrate offline and online evaluations into development and release processes so engineers can measure improvements, catch regressions, and ship confidently.
Drive model evaluation and selection: Continuously benchmark frontier and open-weight models across accuracy, latency, cost, and reliability. Use results to inform model selection, upgrades, and production releases.
Build continuous improvement loops: Turn production failures, customer feedback, and agent traces into new evaluation cases and measurable improvements in agent quality.
Representative Projects
Build an end-to-end evaluation framework for FurtherAI agents, measuring success across multi-step workflows, tool calls, and complex insurance tasks.
Create a library of golden datasets and benchmarks for common insurance workflows, reusable across customer implementations and out-of-the-box AI skills.
Build an automated benchmarking system that evaluates new frontier and open-weight model releases and identifies the best models for different tasks.
Develop production evaluation pipelines that detect quality regressions, identify failure patterns, and turn them into actionable improvements.
What We're Looking For
5+ years of software engineering experience, with strong hands-on experience building production AI/ML systems.
Experience building evaluation frameworks, LLM-based applications, model benchmarking systems, or applied AI infrastructure.
Strong understanding of modern LLMs, agent architectures, evaluation methodologies, and model performance tradeoffs.
Strong Python engineering skills and the ability to build reliable, production-grade systems.
Ability to independently own ambiguous technical problems, from research and experimentation through production.
Clear communicator - both in person and in writing.
Bonus: Experience with LLM-as-judge, human evaluation systems, agent trajectory evaluations, or large-scale benchmarking.
Bonus: Experience with open-weight models, fine-tuning, post-training, or model routing.
Bonus: Background in B2B SaaS and 0-to-1 experience.
Above all: drive, grit, and ownership. If you excel here, other requirements are secondary.
You can look forward to the following benefits:
Fully covered, best-in-class health, dental, and vision benefits
Competitive compensation, meaningful stock options, company 401(k)
Unlimited PTO
Outstanding in-office culture in the heart of San Francisco, 5x per week
