We're looking for a Principal Applied Scientist to lead the design and build of an evaluation framework for an enterprise planner-based agent. You will work directly with the client's engineering team to benchmark their agent across planner models, executor LLMs and inference optimization levers. Every proposed change, whether a model upgrade, a harness change or a routing strategy, becomes a controlled, statistically sound experiment with a measurable dollar outcome.
This is a research-heavy, customer-facing role with a significant build component. You will own both the experimental method and the tooling that makes it repeatable: a scenario-driven user simulator, isolated experiment execution, statistical comparison, and automated insights in a locally hosted dashboard the client can adopt as their own. The work runs in three overlapping streams (evaluation pipeline, grading and statistics, dashboard and experiment campaign) and ends in a recommendation the client can act on.
What You'll DoEvaluation Pipeline & Simulation- Design the end-to-end evaluation workflow for a planner-executor agent: scenarios, user simulation, configuration, execution, grading, comparison and insights
- Define the scenario schema, grading spec and validators with the client's team, who own scenario creation, so their data plugs directly into the framework (each scenario has a goal, persona, starting state and expected end state)
- Build an LLM-based user simulator with a hidden goal and a stop rule that holds realistic multi-turn conversations, and check its realism against individual production logs
- Build isolated, parallel experiment execution: config files, one run per config and scenario, and a harness adapter that routes all model traffic through the client's existing LLM gateway
- Package optimization levers as plug-ins and implement a shortlist of 3 to 4 for v1, such as cascade-on-failure, cache-aware model switching and prompt caching
- Tag every model call at the gateway and send traces to a trace store, so tokens, cache usage, latency and cost are attributed to each request
Grading, Statistics & Insights- Grade each run against the expected end state as the primary signal, with optional LLM judges for answer and plan quality
- Compute the core metrics: verified success, cost per success, p95 latency, and failures traced to pipeline stage
- Compare experiments on the same scenarios with paired methods: confidence intervals, win/loss/tie per scenario, and a non-inferiority test that a cost lever does not drop quality past an agreed margin. Say clearly when a result is inconclusive
- Convert differences into projected dollar savings at production volume using real deployment costs, not just list token prices, and plot the cost-versus-quality frontier
- Build an automated insight engine where a statistics layer computes every number and an LLM only writes them up, so each ranked finding links back to its evidence
Dashboard, Experiments & Readout- Build the locally hosted dashboard (Launch, Live, Results, Compare, Insights and Traces views), API-first with one-command local deployment so it can be integrated into the client's platform later
- Run the staged campaign: production baseline, model screen, lever screen, then confirm the best combination
- Analyze generalization by scenario category, optionally alongside a small data-analytics anchor set
- Deliver a final recommendation with confidence intervals and dollar impact, plus the runbook and integration guide that support go/no-go decisions
Collaboration & Technical Leadership- Partner with the client's engineers and leadership to agree on scenarios, margins and success criteria, and explain findings to both scientists and business stakeholders
- Write clear technical reports and leadership summaries that translate results into business impact
- Set the experimental standards for the pod and mentor other engineers on rigorous agent evaluation, including AI-assisted development tools like Claude Code
What You'll Bring- Experience with LLM and agent evaluation methods, such as programmatic state-based verifiers, LLM-as-judge or agent-as-judge, trajectory-level scoring and user simulation
- Experience building or evaluating planner-executor or other multi-step, tool-using agent architectures in production, not just prototypes
- A strong background in experimental design and statistical inference, including paired designs, non-inferiority testing and power analysis, and the judgment to call a result inconclusive
- Experience with multi-objective optimization or cost-aware ML, including Pareto frontier analysis and converting results into projected dollar savings
- Familiarity with LLM routing, model cascades or model selection, and an understanding of LLM inference internals and cost levers (prompt and plan caching, context compaction, reasoning budgets, cache-aware model switching)
- Experience with observability and tracing for LLM systems, including attributing tokens, cache usage, latency and cost per request
- The ability to build production-quality research tooling: Python services, containerized and sandboxed execution, and data pipelines
- Prior customer-facing or consulting experience where you scoped and delivered research engagements
- The ability to explain technical findings clearly to non-technical audiences and translate them into business impact, and to go a level deeper when someone pushes on your answer instead of staying abstract
Helpful Extras and Unique Skills- Publications or open-source contributions in LLM evaluation, agents or efficient inference
- Experience building internal dashboards or web UIs that a client team can adopt as their own
- Hands-on use of an LLM gateway or tracing stack, such as LangFuse or LangSmith, to attribute cost and latency at the request level
- Experience in a regulated or high-stakes enterprise domain where evaluation rigor carries real consequences
- A point of view on cost-quality tradeoffs gained from running them in production rather than reading about them
Our salary range is $200,00 - $300,000
