Consulting/Principal Site Reliability Engineer

About the Role

We're looking for a Consulting/Principal Site Reliability Engineer to guide our observability and reliability initiative for Government Systems, running on Azure Government. This is an advanced individual-contributor role for someone with a proven track record leading an organization through a transition to SLO-based reliability engineering - you've done this before, and you know what it takes to get development teams bought in. This is a hands-on SRE role, not a software development role: you'll work alongside other SREs on the initiative, guiding development teams on telemetry and instrumentation strategy rather than writing that code yourself. You'll operate as the go-to expert on Datadog and observability practice, helping define meaningful SLOs and error budgets and making data-driven reliability decisions. You'll also help train and mentor junior SREs, working within priorities set by the SRE Manager.

Eligibility: This role requires U.S. citizenship and the ability to obtain and maintain a Public Trust clearance.
What You'll Do

• Guide the observability and reliability initiative, working alongside other SREs to set the technical direction for how the organization measures and improves reliability

• Lead the transition to SLO-based reliability engineering, building on prior experience doing this elsewhere

• Guide development teams on telemetry and instrumentation strategy for their applications, without doing the application development work yourself

• Serve as the primary technical authority on Datadog, from dashboarding to alerting to SLO tracking

• Recommend service level objectives in partnership with Product and Development teams

• Act as a senior escalation point during incidents

• Help improve and evolve the broader SRE framework

• Build disaster recovery plans, including advanced fault injection scenarios

• Advise on SRE training curriculum and content, and help set and follow priorities established by the SRE Manager

Tech Stack You'll Work With

• Cloud: Microsoft Azure Government

• Server Environment: Linux and Windows

• CI/CD: Azure DevOps Pipelines, GitHub Actions

• Infrastructure as Code: Terraform

• Observability: Datadog (primary)

• AI-assisted development: GitHub Copilot

What We're Looking For
Observability (core focus)

• Significant, hands-on experience with Datadog - dashboards, alerting, SLO/error-budget tracking, and telemetry pipelines

• Proven experience leading an organization's transition to SLO-based reliability engineering, not just supporting one

• Track record guiding development teams on instrumentation and telemetry strategy (advisory, not hands-on application coding)

• Deep technical understanding of observability techniques across the full stack; able to bring clarity to complex incidents or performance issues

• Able to build templated observability dashboards and configuration as code so others can implement quickly

• Able to set and influence appropriate SLOs and Error Budgets org-wide

• Comfortable operating across both Linux and Windows server environments

Incident Management

• Experience preparing and assessing SREs for on-call readiness, including mentoring and identifying training needs

• Skilled at running post-mortems where participants feel safe to contribute; fosters a culture of learning from failure

• Comfortable acting as a senior on-call escalation point and guiding restoration efforts

• Experience partnering with SRE and Development leads to identify and fix root causes of excessive alerting and pager load

Design for Reliability

• Advanced understanding of systems design, including high availability, deployment, and recovery techniques, plus failure modes and system behaviors

• Specialized depth in one or more areas such as denial-of-service protection or containerization, with the ability to consult others on it

• Able to make resilience and recoverability decisions backed by load testing data and fault injection results

• Experience documenting SRE design practices and contributing to a broader SRE framework

Disaster Recovery

• Experience contributing to DR plans, including recovery priorities, procedures, and guiding others through runbooks

• Able to design and lead DR practice scenarios and incident response exercises, identifying knowledge gaps and improvements

• Experience planning chaos engineering scenarios to simulate faults across memory, network, I/O, or distributed systems

Platforms and Automation

• Able to build engineering and architecture consensus for new standard components and services on the Paved Road, including platforms and CI/CD

• Experience promoting adoption and standardization, with contributions back via inner sourcing

• A champion for Paved Road adoption, driving efficiency by removing rework and silos

Reliability Culture

• Creates and shares SRE best practices and training material

• Comfortable coaching other SREs and identifying development gaps

• Able to spot and eliminate large-scale toil

AI-Driven Development (desired)

• Experience using GitHub Copilot to accelerate troubleshooting, automation, or reliability tooling

• Interest in helping define best practices for AI-assisted development within an SRE context

Behavioral Competencies

• Collaboration and Teamwork

• Customer & External Focus

• Solves Problems and Analyzes Issues

• Learning Agility

• Builds Relationships

• Develops Others

U.S. National Base Pay Range: $118,300 - $219,800. Geographic differentials may apply in some locations to better reflect local market rates.This job is eligible for an annual incentive bonus.
We know your well-being and happiness are key to a long and successful career. We are delighted to offer country specific benefits. Click here to access benefits specific to your location.