Microsoft is seeking a Principal Software Engineer to own the technical vision of the Copilot offline evaluation platform. This AI-first, agentic-first system gates Copilot quality and operates at scale within a strict enterprise boundary. You will architect evaluation strategies for agentic experiences—from multi-turn trajectories and tool selection to end-to-end task outcomes—and define how quality is measured across the product. You will design the services, pipelines, and tooling that autonomous agents can run, extend, and reason about, with observability and guardrails that make agent-driven operation trustworthy. You will tackle the platform’s hardest challenge: delivering trustworthy results across deep dependency chains under sustained, heavy load, designing for graceful degradation, intelligent retry, and dependency-aware gating. You will define end-to-end evaluation for agentic experiences and build simulations, data collection, scraping, and scoring capabilities to measure agent behavior. You will mentor engineers across teams, drive modernization of the evaluation runtime, and scale the platform to meet growing demand while improving cost and latency. You will stay at the frontier of agentic systems, large-scale AI evaluation, and autonomous operations, bringing new trends to the Copilot organization. The role is hybrid in the United States with locations including New York, Mountain View, and Redmond.

Microsoft is seeking a Principal Software Engineer to own the technical vision of the Copilot offline evaluation platform. This AI-first, agentic-first system gates Copilot quality and operates at scale within a strict enterprise boundary. You will architect evaluation strategies for agentic experiences—from multi-turn trajectories and tool selection to end-to-end task outcomes—and define how quality is measured across the product. You will design the services, pipelines, and tooling that autonomous agents can run, extend, and reason about, with observability and guardrails that make agent-driven operation trustworthy. You will tackle the platform’s hardest challenge: delivering trustworthy results across deep dependency chains under sustained, heavy load, designing for graceful degradation, intelligent retry, and dependency-aware gating. You will define end-to-end evaluation for agentic experiences and build simulations, data collection, scraping, and scoring capabilities to measure agent behavior. You will mentor engineers across teams, drive modernization of the evaluation runtime, and scale the platform to meet growing demand while improving cost and latency. You will stay at the frontier of agentic systems, large-scale AI evaluation, and autonomous operations, bringing new trends to the Copilot organization. The role is hybrid in the United States with locations including New York, Mountain View, and Redmond.