Lead Research Engineer – AI Evaluation & Benchmarking

Job Description

Thomson Reuters Labs is seeking a Lead Research Engineer to lead the design and delivery of production-ready software for artificial intelligence (AI) model evaluation and benchmarking. You will build scalable evaluation infrastructure that enables research and product teams to compare large language models (LLMs), measure quality, cost, and latency, and make informed decisions about model selection and development.
You will own the engineering approach from early prototypes through reliable, maintainable tools, collaborating closely with research scientists, engineering teams, and product partners while helping shape technical direction and engineering delivery.

Key Responsibilities

• Lead technical delivery and engineering planning: Partner with program and research leaders to define priorities, break projects into delivery stages, estimate effort and cost, recommend staffing and required skills, identify dependencies, and adapt plans to available resources.
• Design and build scalable evaluation infrastructure: Extend existing shared tools and services, design new components where needed, and develop production-quality Python software, APIs, and reusable evaluation frameworks. Deliver useful initial versions, validate them with users, and improve them through iterative development.
• Turn research methods into working software: Collaborate with scientists to build automated evaluation pipelines connecting model APIs, test datasets, and agreed scoring methods. Implement reliable workflows for comparing LLMs and other AI models using evaluation datasets relevant to products and customer needs.
• Ensure reliable, reproducible model comparisons: Maintain repeatable experiments, separate evaluation data from model development, and make results traceable to evaluation inputs and methods. Measure model quality, operating cost, and latency to support product teams in validating model performance.
• Build practical tools for researchers and product teams: Develop clear user interfaces, documentation, evaluation reports, and diagnostic capabilities. Enable scientists and product teams to run evaluations, understand failures, interpret results, and extend tools without depending on the original developer.
• Own production reliability, performance, and responsible data use: Integrate automated testing, deployment, and monitoring from the beginning. Address performance, scalability, and operating costs, establish service maintenance and support ownership, and work with data specialists to implement access controls, permitted-use rules, and retention requirements.
• Collaborate, mentor, and guide engineering decisions: Partner with foundational research, data engineering, platform engineering, and product teams to align interfaces, responsibilities, and delivery plans. Resolve technical dependencies, mentor engineers, communicate trade-offs and delivery risks, and recommend changes to scope or technical approach when necessary.

Required Qualifications

• 8-10+ years of relevant software engineering experience, including building and maintaining production systems involving machine learning or complex data.
• Strong hands-on Python programming skills, including API design, automated testing, and working with structured and unstructured data.
• Technical leadership and end-to-end delivery experience, including architecture decisions, engineering planning, staffing and effort estimation, project execution, and adapting priorities as requirements change.
• Experience collaborating with research scientists or machine learning practitioners to translate experiments and research methods into working software.
• Experience integrating or testing machine learning models, interpreting evaluation results, and applying findings to engineering decisions. Understanding of language models and the ability to recognize unreliable comparisons or test-data leakage.
• Cloud engineering experience using AWS, Azure, or a comparable platform, including distributed workflows, automated deployment, and monitoring.
• Working knowledge of SQL and experience developing software that processes structured and unstructured data.
• Strong collaboration, communication, problem-solving, and mentoring skills, with experience coordinating technical delivery across research, engineering, and product stakeholders.
• Bachelor's degree in Computer Science, Engineering, or a related field, or equivalent practical experience.

Preferred Qualifications

• Experience building LLM evaluation and benchmarking pipelines, including automated scoring methods and human assessment workflows.
• Familiarity with experiment tracking, dataset versioning, and reproducible evaluation processes.
• Experience in natural language processing, information retrieval, or other applications involving complex text.
• Experience reducing model inference or operating costs while maintaining evaluation quality and system performance.
• A track record of developing reusable software libraries, frameworks, or tools adopted by other engineering teams.
• Research contributions or open-source software contributions.

#LI-TH1

This posting is for proactive recruitment purposes and may be used to fill current openings or future vacancies within our organization.

What’s in it For You?

• Hybrid Work Model: We’ve adopted a flexible hybrid working environment for our office-based roles while delivering a seamless experience that is digitally and physically connected.
• Flexibility & Work-Life Balance: Flex My Way is a set of supportive workplace policies designed to help manage personal and professional responsibilities, whether caring for family, giving back to the community, or finding time to refresh and reset. This builds upon our flexible work arrangements, including work from anywhere for up to 8 weeks per year, empowering employees to achieve a better work-life balance.
• Career Development and Growth: By fostering a culture of continuous learning and skill development, we prepare our talent to tackle tomorrow’s challenges and deliver real-world solutions. Our Grow My Way programming and skills-first approach ensures you have the tools and knowledge to grow, lead, and thrive in an AI-enabled future.
• Industry Competitive Benefits: We offer comprehensive benefit plans to include flexible vacation, two company-wide Mental Health Days off, access to the Headspace app, retirement savings, tuition reimbursement, employee incentive programs, and resources for mental, physical, and financial wellbeing.
• Culture: Globally recognized, award-winning reputation for inclusion and belonging, flexibility, work-life balance, and more. We live by our values: Obsess over our Customers, Compete to Win, Challenge (Y)our Thinking, Act Fast / Learn Fast, and Stronger Together.
• Social Impact: Make an impact in your community with our Social Impact Institute. We offer employees two paid volunteer days off annually and opportunities to get involved with pro-bono consulting projects and Environmental, Social, and Governance (ESG) initiatives.
• Making a Real-World Impact: We are one of the few companies globally that helps its customers pursue justice, truth, and transparency. Together, with the professionals and institutions we serve, we help uphold the rule of law, turn the wheels of commerce, catch bad actors, report the facts, and provide trusted, unbiased information to people all over the world.

Eligible office location(s) for this role include one or more of the following: New York City, San Francisco, Los Angeles, and/or Irvine, CA; McLean, VA; Washington, DC. The base compensation range for the role in any of those locations is $146,800 USD - $272,600 USD. For any eligible US locations, unless otherwise noted, the base compensation range for this role is $127,400 USD - $236,600 USD. For Ontario, Canada, the base compensation range for this role is $140,000 CAD - $175,000 CAD. Base pay is positioned within the range based on several factors including an individual’s knowledge, skills and experience with consideration given to internal equity. Base pay is one part of a comprehensive Total Reward program which also includes flexible and supportive benefits and other wellbeing programs. This role may also be eligible for an Annual Bonus based on a combination of enterprise and individual performance.