7+ years in infrastructure, SRE, or platform engineering, ideally in inference, ML, or HPC environments.
Hands-on experience with Kubernetes at production scale, including debugging.
Strong understanding of tail latency and its relationship with utilization.
Experience managing vendor relationships, particularly in resolving production issues.
Proven track record in building effective observability practices that identify real problems.
Experience with on-call duties during significant incidents, with lessons learned to share.
Desire to be the first infrastructure hire in a growing company.
Responsibilities
Own the inference control plane, modifying and extending it as needed.
Manage the gateway and load balancer, focusing on model placement and request routing.
Oversee end-to-end observability, including per-request tracing and alerting.
Conduct capacity planning for distributed traffic across open-weight models.
Act as the technical liaison with ASIC partners for operational issues.
Lead the integration of pre-fill architecture on new hardware platforms.
Establish and grow the on-call and incident response practice from the ground up.
Benefits
Opportunity to shape the infrastructure of a growing company from the ground up.
Hands-on role with direct impact on production and performance.
Chance to build and lead a team in a critical area of the business.
Engagement with cutting-edge technology and diverse hardware platforms.
Collaboration with ASIC engineering teams to solve complex issues.
Full Job Description
About the role
You'll own the infrastructure layer of our inference cloud end-to-end. Today that means the control plane, the gateway in front of our ASIC fleet, and the observability stack that tells us where every millisecond goes. Over the next 6-8 months, it will grow into a heterogeneous fleet: ASICs for decode, GPUs for pre-fill, and the physical-layer ownership that comes with it.
The first six months are hands-on: k8s manifests, dashboards, oncall, and a direct line to our ASIC partner's engineering team when production behaves strangely. The team grows under you from there.
What you'll do:
Own the inference control plane. At the moment, it's built on configuration provided by our ASIC partner; you'll be the person who understands it deeply enough to modify, extend, and eventually replace pieces of it.
Own the gateway and load balancer that fronts the fleet. Model placement, request routing, and tail-latency engineering live here, driven by live utilization and per-model SLOs.
Own observability end-to-end. Per-request tracing from OpenRouter ingress through to the accelerator, with p50/p95/p99 dashboards, SLOs, and alerting that wakes the right person.
Run capacity planning against a real, distributed traffic mix across the open-weight models we serve.
Own the operational side of the ASIC partnership. Most weird production issues route through their engineering team until we build that expertise in-house, and you'll be our technical face in those conversations.
Bring up the pre-fill side of our disaggregated architecture on a second hardware platform as it comes online. Different vendor, different fabric, different kernels.
Build the on-call and incident response practice from zero. Hire and grow the team underneath you.
What we need from you:
7+ years in infrastructure, SRE, or platform engineering, with at least some of it at a serious inference, ML, or HPC shop.
Hands-on with Kubernetes at production scale - not just deploying, but debugging the weird stuff.
Strong instincts for tail latency. You think about p99 and utilization as the same problem, not different ones.
Comfortable owning a vendor relationship where the vendor's bugs are now your production issues.
Track record of building observability practices that actually catch problems, not just generate dashboards.
Have been on-call through real incidents and can talk about what you learned.
Want to be the first infra hire at something early, not the tenth at something big.
NIce-to-Haves:
Experience operating non-NVIDIA accelerators in production - TPUs, ASICs, or alternative GPU vendors.
Background with model-serving stacks (vLLM, TGI, TensorRT-LLM, SGLang).
Network fabric experience at data-center scale (RoCE, InfiniBand).
Have hired and managed an infra team before.
Comfort at the hardware boundary - firmware, drivers, thermals - for when the roadmap takes us there.
