Staff Software Engineer - Resiliency
The role is preferred as Hybrid working in Austin TX, Houston TX or Denver CO, we will consider virtual for the right candidate.
Position responsibilities
- Own the resiliency strategy and reference architecture for the AWS estate: set availability targets by service tier, define the standard patterns for multi-AZ and multi-region operation, and drive existing services toward them
- Own the decision framework for active-active versus active-passive versus warm-standby and pilot-light topologies, including the real tradeoffs in cost, operational complexity, and achievable RTO/RPO, and the migration path for services that need to move between them
- Lead business continuity and disaster recovery end to end: RTO/RPO target-setting, automated failover design, regular DR exercises and game days, recovery runbook ownership, and translating exercise findings into durable architectural change
- Run AWS Well-Architected Reviews across service teams with emphasis on the Reliability and Operational Excellence pillars, and convert findings into a prioritized, tracked remediation backlog rather than a report that sits on a shelf
- Drive resiliency into the container platform: ECS/Fargate and EKS workload placement, capacity headroom, failure-domain isolation, graceful degradation, and recovery behavior under partial-region impairment
- Drive resiliency into the data tier so stateful workloads recover as predictably as stateless ones, covering Aurora and RDS multi-AZ and Global Database topologies, DynamoDB global tables, ElastiCache, and S3 cross-region replication, with clear expectations for failover time, replica lag, and data-loss boundaries
- Encode resilience in automation: IaC-defined failover paths, self-healing and auto-remediation, drift detection, and continuous fault injection so recovery capability is exercised routinely instead of documented once
- Own the reliability signal set for core infrastructure: SLOs and error budgets, health-check and failover-trigger design, dependency mapping, and blast-radius analysis across control-plane and data-plane signals
- Raise the quality bar on incident response for availability events, lead resiliency-focused postmortems, and convert repeat failure modes into architectural fixes and platform guardrails
- Operate effectively across a complex organizational context, translating continuity and availability requirements from engineering, product, and business stakeholders into concrete AWS design and validated recovery capability
Required skills
- 12+ years of engineering experience, with at least 7 as the primary architect or technical owner of infrastructure resiliency and high-availability architecture at scale
- Deep hands-on AWS production experience in large multi-account, multi-region environments, including VPC and networking design, Route 53, ALB/NLB, Global Accelerator, and the IAM and account-boundary decisions that constrain failover
- Demonstrated ownership of highly available architectures where multi-AZ is the baseline and multi-region is real for critical workloads, with credible judgment about when active-active is warranted and when active-passive or warm-standby is the better tradeoff
- Track record owning BC/DR strategy end to end: setting RTO/RPO targets, designing and automating regional failover, and running genuine recovery exercises rather than tabletop-only reviews
- Expert command of the AWS Well-Architected Framework, particularly the Reliability pillar, with experience leading formal reviews and driving the resulting remediation to completion
- Production depth with container platforms on AWS (ECS/Fargate, EKS, or both): scaling behavior, placement and failure domains, deployment safety, and degradation characteristics under failure
- Strong data-tier resiliency experience with Aurora or RDS (multi-AZ, Global Database, failover behavior, replica lag) and at least one of DynamoDB global tables, ElastiCache replication, or S3 cross-region replication
- Deep production experience designing and operating IaC at scale, Terraform preferred (CDK acceptable), with strong opinions on module strategy, state management, and guardrail enforcement across many accounts and environments
- Expert command of CI/CD for infrastructure: pipeline design, drift detection, plan/apply workflows, and self-service patterns that serve engineering teams safely at scale
- Hands-on experience with chaos engineering and fault injection in production environments, including failure-injection tooling and game-day operations
- Strong grounding in observability for infrastructure: SLO and error-budget design, health-check strategy, dependency mapping, and state-of-the-fleet visibility
- Cross-functional leadership, comfortable as a peer with senior engineering, product, and finance leaders on availability, redundancy cost, and recovery-capability conversations
- Comfortable with the coordination work of a recently combined company: divergent AWS estates, in-flight unification, and the political work that comes with consolidation
Preferred qualifications
- Hands-on experience with AWS resiliency tooling such as Resilience Hub, Route 53 Application Recovery Controller, Elastic Disaster Recovery, or Fault Injection Service
- AWS Solutions Architect Professional certification, or equivalent demonstrable depth
- Hands-on experience with infrastructure orchestration tooling beyond raw Terraform (Terragrunt, Atlantis, Spacelift, env0, Crossplane, or similar)
- Background in SRE or platform reliability with strong instincts for SLO design, error budget policy, and toil reduction
- Experience modeling the cost of redundancy (standby capacity, cross-region data transfer, replication overhead) and defending those tradeoffs to business stakeholders
- Experience post-M&A integrating infrastructure and resiliency practices across two or more legacy AWS estates
- Experience in regulated industries, or other domains with mission-critical workloads and strict continuity requirement
- Medical, dental, vision insurance with Employer paid health premiums
- Open PTO Policy
- 401(k) with up to 6% company matching and immediate vesting
- 12 Weeks Paid Parent Leave
- Flight Training Rewards
Pay is based upon candidate experience and qualifications, as well market and business considerations: Summary Pay Range: $159,000.00 to $195,000.00
Jeppesen ForeFlight - EOE including Disability/Vets | Pay Transparency | E-Verify Participant