5-7 years of experience in robotics, machine learning, or model evaluation with a focus on evaluation design and analysis.
Proven track record in managing and developing technical teams, including setting priorities and providing feedback.
Strong experimental judgment to define success and assess improvements meaningfully.
Technical fluency to collaborate with researchers on model checkpoints and data quality.
Proficiency in Python or similar tools for data analysis and workflow improvement.
Excellent communication skills to translate ambiguous requests into actionable plans.
Responsibilities
Lead and develop the evaluation team, managing priorities and fostering team growth.
Collaborate with researchers to create evaluation plans with clear hypotheses and success criteria.
Design evaluations that reflect real-world conditions, measuring success and quality.
Standardize evaluation processes to ensure repeatability and traceability of results.
Analyze failures to distinguish between model and hardware issues, recommending focused investigations.
Maintain regression coverage and report on model performance across tasks and conditions.
Enhance evaluation tools and workflows through data analysis and scripting to streamline processes.
Benefits
Opportunity to work at the forefront of robotics and AI technology.
Collaborative environment with researchers and engineers.
Professional development through team management and technical leadership.
Hands-on role with significant impact on product readiness and deployment.
Access to cutting-edge evaluation tools and methodologies.
Full Job Description
The Role
Dyna Robotics is building general-purpose robots that perform useful work in real-world environments. We're looking for a Model Evaluation Lead to lead the team responsible for understanding how our robot policies perform and whether new model versions deliver meaningful improvements.
You'll manage and develop the evaluation team, own evaluation planning and execution, and work closely with researchers to define what we test, how we measure it, and what we learn. This is a hands-on leadership role combining people management, experimental rigor, and technical judgment. Your work will help researchers understand model behavior, prioritize improvements, and assess readiness for deployment.
What You'll Do
Lead and develop the evaluation team. Manage the team's priorities, performance, and growth. Establish clear ownership, provide regular coaching and feedback, and build onboarding and training practices that help team members execute evaluations and investigate issues independently.
Work closely with researchers. Translate research questions into evaluation plans with clear hypotheses, success criteria, and deliverables. Partner with researchers to compare model checkpoints, assess new capabilities, and balance recurring evaluations with urgent experiments and available robot capacity.
Design meaningful evaluations. Develop task definitions, scoring rubrics, and test scenarios that reflect real operating conditions. Measure task success, output quality, cycle time, human interventions, and recovery behavior. Choose appropriate trial counts and account for variability when interpreting results.
Make results repeatable and trustworthy. Standardize setup, resets, scoring, and intervention rules. Ensure results are traceable to model versions, robot configurations, test conditions, logs, and videos. Coordinate with hardware and operations partners to resolve issues that affect evaluation quality.
Turn failures into actionable findings. Review robot behavior, logs, and video with the team and researchers. Distinguish model failures from hardware, software, setup, or execution issues. Identify recurring failure patterns and recommend focused investigations, additional data collection, or annotation improvements.
Own regression coverage and reporting. Maintain consistent benchmarks and held-out test scenarios across tasks and conditions. Identify regressions, track progress across model versions, and communicate what improved, what remains unreliable, and where evidence is still limited. Provide clear recommendations that inform deployment decisions.
Improve evaluation tools and workflows. Use data analysis and lightweight scripting to improve reporting and reduce manual work. Partner with engineering on evaluation tooling, automation, and dashboards. Shorten the time from a new checkpoint to useful findings while preserving the quality of the evidence.
What You'll Bring
Experience in robotics, machine learning, applied research, or model evaluation, with direct ownership of evaluation design and analysis.
Demonstrated experience managing and developing a technical team, including setting priorities, giving feedback, and building effective working practices.
Strong experimental judgment: you can define success, control relevant variables, identify gaps in test coverage, and assess whether an apparent improvement is meaningful.
Technical fluency to work directly with researchers on model checkpoints, generalization, data quality, and experimental tradeoffs.
Comfort using Python or similar tools to analyze results, inspect logs, and improve evaluation workflows.
Clear communication and strong ownership. You can turn an ambiguous request into an executable plan, surface problems early, and explain findings to both research and operations teams.
Bonus Points For
Evaluating robot manipulation policies, embodied AI, autonomous systems, or other machine learning systems operating in the physical world.
Building regression suites, benchmark datasets, failure taxonomies, or evaluation tools, and connecting findings to model development or deployment readiness.
