Position Summary: About the TeamOur Site Reliability Engineering team is the execution engine behind the reliability, availability, and performance of distributed store technology powering thousands of retail and pharmacy locations nationwide. We operate across pharmacy platforms, Point of Sale (POS) systems, handheld devices, store servers, dispensing systems, and edge computing infrastructure - spanning hybrid cloud and on-premises environments at massive fleet scale.
Our engineering philosophy is grounded in five pillars:
Detection,
Prevention,
Recovery,
Learning Loops, and
Developer Experience (DevX).
Our operating principle is the
reliability covenant: our success is not measured by incident response volume - it is measured by the reliability capability we transfer to the engineering teams we serve.
Your success in this role is measured by what the engineering teams in your domain can do independently after working with you, not by how indispensable you become to them. An SSE who has enabled a development team to detect, respond to, and learn from production failures without SRE involvement has delivered the highest-value outcome this role can produce.
We track
operational toil as an engineering metric. Engineers at this level are expected to identify recurring manual work, eliminate it through automation, document the reduction, and treat toil accumulation as a reliability risk - not as a sign of operational expertise.
About the RoleAs a
Senior Software Engineer - SRE, you independently own the reliability posture of an assigned engineering domain. You are not waiting for direction - you are setting it for your domain. You design the alerting strategy, own the SLO health, lead incident command for production issues, facilitate postmortems, tune anomaly detection models, and partner directly with engineering domain owners to shift reliability left into design.
You are a technical mentor to SE-level engineers and an escalation resource during active incidents. You have the technical depth to diagnose complex distributed system failures, the data instincts to distinguish genuine anomalies from noise in ML-generated signals, and the organizational skills to drive reliability practice adoption in teams that did not necessarily ask for SRE involvement.
Scope: Domain ownership - you operate independently and influence adjacent engineering teams.
The Environment You Are JoiningThis role exists inside an active SRE transformation. The domain-based SRE ownership model you will operate within is in its early stages. Some of the toolchains you will work with are being built in parallel with the operational work. Engineering domain owners are simultaneously learning what SRE can offer them.
This is an honest description of the role, not a caveat. The opportunity is to shape a domain's reliability posture from the ground up in a large-scale, consequential technology environment. The engineering decisions you make will affect pharmacy dispensing, prescription fill workflows, and store operations across thousands of locations. If you are energized by the combination of technical depth and organizational building, the scope here is significant. If you are looking for a mature, fully-defined SRE environment where the frameworks and toolchains already exist, this role will feel like a different challenge.
Success in this role requires
patience alongside technical rigor: you will demonstrate value before you demand process change, build credibility before you expect adoption, and earn trust with engineering domain owners through partnership rather than mandate.
The operating environment includes an
edge computing fleet deployed directly inside store locations - unattended nodes where deployment blast radius is geographic and fleet-wide, not functional and service-scoped. You will develop a fleet operations mindset: the primary failure mode in this environment is deployment and configuration propagation, not service logic.
What You Will DoDetection & Observability- Own SLI/SLO health for your assigned domain end-to-end; monitor error budget burn rates and drive proactive burn-down actions before incidents reach end users or pharmacy patients
- Design multi-signal alerting strategies that go well beyond threshold alerts: burn rate alerting, composite health signals, and anomaly-based detection using time-series models - and validate their output against production ground truth
- Tune and maintain ML-based anomaly detection models in production: adjust sensitivity thresholds, evaluate false positive rates against alert fatigue metrics, and iterate on model configurations based on incident data
- Build Critical User Journey (CUJ)-anchored dashboards that surface end-to-end business flow health - not just individual service metrics
- Drive alert signal-to-noise improvement across the domain; own suppression policies during deployments and maintenance windows to protect on-call quality
- Identify and instrument unmonitored external dependencies - third-party APIs, downstream services, data providers - using tools like Prometheus Blackbox Exporter, OpenTelemetry, or custom health probes
- Contribute to the observability platform design: understand the hot/warm/cold tier architecture (Apache Kafka 12 ClickHouse 12 cold storage) and contribute domain-specific data models and SLI schemas
Prevention & Reliability Engineering- Lead Production Readiness Reviews (PRR) for services in your assigned domain; own the readiness gate sign-off and be accountable for what makes it into production on your watch
- Design and execute fault injection experiments at service level using tools such as LitmusChaos, Chaos Toolkit, or Gremlin; validate blast radius assumptions before rollouts reach production
- Partner with engineering domain owners on reliability requirements during architecture design and sprint planning - reliability is designed in, not bolted on
- Own dependency risk mapping for your domain: catalog third-party API failure modes, shared infrastructure failure paths, and chain-wide blast radius scenarios
- Apply a fleet operations mindset to change management: for any configuration or deployment change touching the edge fleet, assess deployment blast radius by node cohort, review the rollback procedure, and contribute to the go/no-go decision on high-risk changes
- Understand progressive rollout strategies - canary cohorts, staged fleet expansion, blast radius budgets - and apply them in high-risk deployment reviews
Incident Response & Recovery- Serve as the primary on-call Technical Incident Commander (IC) for domain incidents; drive structured bridge calls from detection to resolution using established incident command frameworks
- Author and maintain P0/P1 runbooks with validated, step-by-step remediation procedures; own a quarterly review and dry-run testing cadence to ensure runbooks are accurate when they are actually needed
- Lead post-incident reviews using structured root cause analysis: causal chain documentation, origin layer classification, and contributing factor identification - not just a timeline of what happened
- Serve as the real-time escalation point and technical decision support for SE engineers during active incidents
- Pursue Technical Incident Commander (TIC) certification; qualify as a cross-domain IC candidate available to lead major incidents beyond your assigned domain
Learning Loops & Continuous ImprovementLearning Loops at this level is
systems engineering for organizational memory - not postmortem administration.
- Facilitate domain-level postmortems with rigor and structure: timestamped timelines, contributing factor taxonomy (origin layer + failure pattern classification), systemic findings, and action items with owners, due dates, and measurable definitions of done
- Measure learning velocity: track not just whether action items close, but whether the incident class frequency decreases after a fix is applied. Report the ratio of incident classes that recurred versus incident classes that were resolved systemically. This is your primary learning loop quality metric
- Identify recurring incident patterns across your domain and drive architectural or process changes that eliminate the root cause - not suppress the symptom
- Close the prevention loop: translate postmortem findings directly into PRR requirement updates, new monitoring coverage, runbook improvements, or development-team practices that prevent recurrence. A postmortem that doesn9t change something upstream hasn9t finished
- Mentor SE-level engineers on SRE craft - not just on what to do, but on why the learning loop exists: connect the postmortem to the PRR, the PRR to the design review, the design review to the runbook, and the runbook back to the next incident
Developer Experience & Automation (DevX)- Eliminate toil systematically: identify and automate manual operational work in your domain using Python, Go, or Bash. Track toil volume eliminated per quarter - time recovered, manual touchpoints removed, and error modes eliminated through automation
- Build self-service observability tooling that enables development teams to own their own service health dashboards and SLO status pages - reducing SRE as a dependency for basic operational visibility
- Track and report DORA metrics - Deployment Frequency, Lead Time for Change, MTTR, and Change Failure Rate - for your assigned domain on a quarterly basis; use the data to target reliability investments where they will have the most impact
- Partner with development teams to shift reliability practices left: reliability checklists in design reviews, SLI instrumentation in definition-of-done, runbook templates that developers can author and own
- Contribute to the team9s shared SRE tooling library; write reusable, well-documented automation modules for common operational patterns
Organizational InfluenceThis is a core function of the SSE role - not a soft skill addendum.
- Drive reliability practice adoption without mandate: you will work with engineering teams that have operated independently for years. Adoption of SLO culture, PRR gates, and chaos engineering practices depends entirely on your ability to build credibility through demonstrated value, not through authority you don9t have
- Establish working relationships with engineering domain owners; understand their delivery pressures, their quality concerns, and what they need from SRE to say yes to new reliability requirements
- Identify the smallest, highest-value reliability practice that a skeptical engineering team will adopt first - land that win, document the outcome, and use it to earn the trust needed for the next one
- When reliability practices are not being adopted, diagnose the real reason: unclear value, unclear ownership, too much friction, or wrong timing - and adapt the approach rather than escalating to mandate
Required Qualifications- 5+ years of experience in SRE, DevOps, platform engineering, or related production-systems roles
- 3+ years operating cloud-native distributed systems at production scale with active on-call responsibility
- Demonstrated experience as an on-call Incident Commander (IC) for P1 or P2 incidents - structured bridge leadership, not just participant involvement
- Experience tuning and validating time-series anomaly detection models in a production observability context - this is a Required qualification, not a preferred one; anomaly-based detection is a core function of this role
- Strong programming proficiency in at least one of Python, Go, or Java at production quality - capable of writing operational tooling that other engineers will rely on
- Hands-on experience designing SLIs, SLOs, and managing error budgets for customer-facing or business-critical services
- Deep observability platform experience: Prometheus, Grafana, OpenTelemetry, and at least one log aggregation solution (Loki, Splunk, Elasticsearch)
- Organizational influence without authority: demonstrated track record of driving SRE practice adoption in engineering teams that did not initially request SRE involvement - this is a Required qualification; technical depth alone is insufficient at this level in a transformation environment
- Fleet-scale deployment awareness: familiarity with progressive rollout strategies, blast radius management, and configuration drift as a reliability risk in large unattended node deployments
- Strong cloud platform expertise: AWS, Microsoft Azure, or Google Cloud Platform (GCP)
- Advanced Kubernetes operational experience: debugging, resource management, networking policies, and workload failure modes. Experience with AI-assisted tooling and development.
Preferred Qualifications- Experience owning Production Readiness Reviews or service launch gates
- Hands-on chaos or fault injection experience using LitmusChaos, Chaos Toolkit, or Gremlin
- TIC (Technical Incident Commander) certification or equivalent structured incident command training
- Experience operating distributed systems in retail, pharmacy, healthcare, or other operationally sensitive environments where failures have direct patient or customer impact
- LLM integration for operational use cases (alert summarization, runbook suggestion, incident triage assistance) - design or implementation experience
- Experience with streaming data platforms: Apache Kafka, Redpanda, Apache Flink, or ksqlDB
- Familiarity with analytical databases for observability workloads:
