About the RoleWe're looking for a Consulting/Principal Site Reliability Engineer to guide our observability and reliability initiative for Government Systems, running on Azure Government. This is an advanced individual-contributor role for someone with a proven track record leading an organization through a transition to SLO-based reliability engineering - you've done this before, and you know what it takes to get development teams bought in. This is a hands-on SRE role, not a software development role: you'll work alongside other SREs on the initiative, guiding development teams on telemetry and instrumentation strategy rather than writing that code yourself. You'll operate as the go-to expert on Datadog and observability practice, helping define meaningful SLOs and error budgets and making data-driven reliability decisions. You'll also help train and mentor junior SREs, working within priorities set by the SRE Manager.
Eligibility: This role requires U.S. citizenship and the ability to obtain and maintain a Public Trust clearance.
What You'll Do- Guide the observability and reliability initiative, working alongside other SREs to set the technical direction for how the organization measures and improves reliability
- Lead the transition to SLO-based reliability engineering, building on prior experience doing this elsewhere
- Guide development teams on telemetry and instrumentation strategy for their applications, without doing the application development work yourself
- Serve as the primary technical authority on Datadog, from dashboarding to alerting to SLO tracking
- Recommend service level objectives in partnership with Product and Development teams
- Act as a senior escalation point during incidents
- Help improve and evolve the broader SRE framework
- Build disaster recovery plans, including advanced fault injection scenarios
- Advise on SRE training curriculum and content, and help set and follow priorities established by the SRE Manager
Tech Stack You'll Work With- Cloud: Microsoft Azure Government
- Server Environment: Linux and Windows
- CI/CD: Azure DevOps Pipelines, GitHub Actions
- Infrastructure as Code: Terraform
- Observability: Datadog (primary)
- AI-assisted development: GitHub Copilot
What We're Looking ForObservability (core focus)- Significant, hands-on experience with Datadog - dashboards, alerting, SLO/error-budget tracking, and telemetry pipelines
- Proven experience leading an organization's transition to SLO-based reliability engineering, not just supporting one
- Track record guiding development teams on instrumentation and telemetry strategy (advisory, not hands-on application coding)
- Deep technical understanding of observability techniques across the full stack; able to bring clarity to complex incidents or performance issues
- Able to build templated observability dashboards and configuration as code so others can implement quickly
- Able to set and influence appropriate SLOs and Error Budgets org-wide
- Comfortable operating across both Linux and Windows server environments
Incident Management- Experience preparing and assessing SREs for on-call readiness, including mentoring and identifying training needs
- Skilled at running post-mortems where participants feel safe to contribute; fosters a culture of learning from failure
- Comfortable acting as a senior on-call escalation point and guiding restoration efforts
- Experience partnering with SRE and Development leads to identify and fix root causes of excessive alerting and pager load
Design for Reliability- Advanced understanding of systems design, including high availability, deployment, and recovery techniques, plus failure modes and system behaviors
- Specialized depth in one or more areas such as denial-of-service protection or containerization, with the ability to consult others on it
- Able to make resilience and recoverability decisions backed by load testing data and fault injection results
- Experience documenting SRE design practices and contributing to a broader SRE framework
Disaster Recovery- Experience contributing to DR plans, including recovery priorities, procedures, and guiding others through runbooks
- Able to design and lead DR practice scenarios and incident response exercises, identifying knowledge gaps and improvements
- Experience planning chaos engineering scenarios to simulate faults across memory, network, I/O, or distributed systems
Platforms and Automation- Able to build engineering and architecture consensus for new standard components and services on the Paved Road, including platforms and CI/CD
- Experience promoting adoption and standardization, with contributions back via inner sourcing
- A champion for Paved Road adoption, driving efficiency by removing rework and silos
Reliability Culture- Creates and shares SRE best practices and training material
- Comfortable coaching other SREs and identifying development gaps
- Able to spot and eliminate large-scale toil
AI-Driven Development (desired)- Experience using GitHub Copilot to accelerate troubleshooting, automation, or reliability tooling
- Interest in helping define best practices for AI-assisted development within an SRE context
Behavioral Competencies- Collaboration and Teamwork
- Customer & External Focus
- Solves Problems and Analyzes Issues
U.S. National Base Pay Range: $118,300 - $219,800. Geographic differentials may apply in some locations to better reflect local market rates.This job is eligible for an annual incentive bonus.
We know your well-being and happiness are key to a long and successful career. We are delighted to offer country specific benefits. Click here to access benefits specific to your location. 
