About the RoleWe're looking for a Site Reliability Engineer (SRE) to drive reliability for RL Infra end-to-end. RL training is unlike a typical batch training job: it interleaves rollout generation, environment or tool interactions, reward scoring, and policy weight updates in a continuous loop, often across long-tailed trajectories with unpredictable latency. Keeping this loop healthy - and keeping it healthy for many concurrent tenants sharing the same underlying clusters - is the core of this role.
You'll work alongside the engineers building the RL training and rollout systems, and with the research teams running experiments on top of them, to make every layer of the stack more robust and resilient: from the rollout and inference-serving layer, through weight synchronization between trainers and samplers, to the schedulers deciding how GPUs are shared across simultaneous jobs.
This is a role with real ownership. You'll be trusted to set reliability priorities, push back on changes that put stability or correctness at risk, and drive incidents affecting RL workloads to resolution without waiting for permission.
What You'll Do- Define and own end-to-end reliability for RL Infra, from CI/CD flows through production observability and incident response.
- Develop Service Level Objectives for distributed RL training systems, balancing job completion reliability and rollout/scheduling latency against development velocity.
- Design and implement monitoring and observability across the full RL loop - rollout generation, environment and tool execution, reward computation, and weight synchronization between trainers and samplers - so failures and slowdowns are caught close to their source.
- Drive incident response for RL Infra platform issues, ensuring rapid recovery, thorough incident reviews, and systematic improvements that prevent recurrence.
- Harden multi-tenant isolation and resource scheduling so that LoRA-based workload co-scheduling maximizes utilization without compromising reliability or data separation.
- Reduce the impact of long-tailed rollouts and stale or off-policy data on training stability, working with research and infra teams to bound staleness and recover cleanly from stuck or slow trajectories.
- Build and harden checkpointing and recovery for long-running RL jobs, so trainer, sampler, or environment failures cost minutes of progress, not hours.
- Collaborate with security teams to address production vulnerabilities across the RL Infra stack.
Skills and QualificationsMinimum Qualifications- Bachelor's degree or equivalent experience in computer science, engineering, or a similar field.
- Experience in distributed systems, cloud infrastructure, or site reliability engineering.
- Proficiency writing software to solve reliability problems, including building tooling and automation.
- Experience with production incident response, postmortems, and systematic reliability improvement.
- Strong communication skills and a track record of coordination across engineering and research teams.
Preferred Qualifications- Deep experience operating production cloud services at scale (e.g., public cloud platforms, internal cloud services).
- Background in distributed training frameworks and how infrastructure failures - stuck rollouts, stale weights, environment flakiness - surface in training behavior.
- Direct experience with RL-specific infrastructure: rollout/inference serving, reward pipelines, or asynchronous training systems that overlap generation with policy updates.
- Track record building checkpoint and recovery systems for long-running distributed jobs.
- Expertise in Kubernetes at scale: deploying, operating, debugging, and tuning clusters handling heterogeneous GPU workloads.
Logistics- Location: This role is based in San Francisco, California.
- Compensation: Depending on background, skills and experience, the expected annual salary range for this position is $350,000 - $475,000 USD.
- Visa sponsorship: We sponsor visas. While we can't guarantee success for every candidate or role, if you're the right fit, we're committed to working through the visa process together.
- Benefits: Thinking Machines offers generous health, dental, and vision benefits, unlimited PTO, paid parental leave, and relocation support as needed.
