OverviewHow do you know an AI product is actually getting better, and how do you prove it, at scale, before millions of people feel the difference? We build and operate the offline evaluation platform that gates Microsoft Copilot's quality: teams across Copilot depend on us to run their scenarios against the product, score the responses, and produce the scorecards that decide what ships. We do this at the scale of one of the world's largest AI products, inside a strict enterprise compliance boundary. As Copilot becomes a fleet of autonomous agents, the way we measure quality has to be reinvented, and you will be one of the engineers building it.
As a Senior Software Engineer on the Copilot evaluation team, you will be a key engineer on a platform designed to be AI-first and agentic-first, where autonomous agents are not just measured, but help operate and extend the system itself. You will design and deliver major components of the platform and partner closely with senior and principal engineers on the systems that keep evaluation trustworthy across a deep chain of dependencies we drive far outside their normal operating envelope, hundreds of calls per job, under relentless and growing load. You will build the capabilities that let us evaluate agentic Copilot experiences, from multi-turn trajectories and tool selection to end-to-end task outcomes. This opportunity will allow you to deepen your expertise in large-scale agentic systems and reliability engineering, broaden your impact across one of the world's most visible AI products, and grow toward technical leadership on problems the industry has not solved yet.
Responsibilities- Partners with engineers and stakeholders across the Copilot organization to turn quality questions into rigorous, reproducible evaluations, building the scenarios, metrics, and scorecards that gate what ships.
- Designs and builds major components of an AI-first, agentic-first evaluation platform where autonomous agents are first-class operators, delivering services, pipelines, and tooling that agents can run, extend, and reason about, with the observability and guardrails that make agent-driven operation trustworthy.
- Contributes to the platform's hardest challenge, delivering trustworthy results across a deep chain of dependencies driven far outside their normal operating envelope, implementing graceful degradation, intelligent retry, and dependency-aware gating so reliability holds under sustained, growing load.
- Builds the capabilities that make agentic Copilot experiences measurable end to end, from multi-turn trajectories and tool selection to task outcomes and modalities such as UX and voice, including the simulation, scraping, and scoring that capture agent behavior.
- Writes and reviews extensible, maintainable code and mentors less-experienced engineers, contributing to modernization of the evaluation runtime and platform so it scales with fast-growing demand while improving cost and latency.
- Acts as a Designated Responsible Individual (DRI) for a system that gates releases, helping instrument the reliability, throughput, and dependency health of the platform and working on-call so the results teams depend on stay correct and timely.
- Stays current with agentic systems, large-scale AI evaluation, and autonomous operations, adopting new tools and patterns and sharing what works to raise the bar for how the team measures quality.
QualificationsRequired Qualifications:- Bachelor's Degree in Computer Science or related technical field AND 4+ years technical engineering experience with coding in languages including, but not limited to .Net, Java, JavaScript, Rust or Python
- OR equivalent experience.
Other Requirements:Ability to meet Microsoft, customer and/or government security screening requirements are required for this role. These requirements include but are not limited to the following specialized security screenings:
- Microsoft Cloud Background Check: This position will be required to pass the Microsoft Cloud background check upon hire/transfer and every two years thereafter.
Preferred Qualifications:- Master's Degree in Computer Science or related technical field AND 6+ years technical engineering experience with coding in languages including, but not limited to, .Net, Java, JavaScript, Rust or Python
- OR Bachelor's Degree in Computer Science or related technical field AND 8+ years technical engineering experience with coding in languages including, but not limited to, .Net, Java, JavaScript, Rust or Python
- OR equivalent experience.
- Experience with large-scale distributed systems and reliability engineering, especially building dependable services that operate across many interacting dependencies under heavy or unusual load.
- Experience building AI-first or agentic systems, including systems, APIs, or tooling intended to be operated by autonomous agents, with the observability and guardrails that make agent-driven operation trustworthy.
- Experience with AI/ML evaluation, measurement, or benchmarking systems, such as LLM-based grading, metric design, or large-scale data and quality pipelines.
Software Engineering IC4 - The typical base pay range for this role across the U.S. is USD $119,800 - $234,700 per year. There is a different range applicable to specific work locations, within the San Francisco Bay area and New York City metropolitan area, and the base pay range for this role in those locations is USD $160,200 - $261,000 per year.
Certain roles may be eligible for benefits and other compensation. Find additional benefits and pay information here:
https://careers.microsoft.com/us/en/us-corporate-pay
This position will be open for a minimum of 5 days, with applications accepted on an ongoing basis until the position is filled.
