THE ROLEBaseten is building its own GPU infrastructure for large-scale inference. As we move into large scale, high-density NVIDIA systems, the hardest failures are intermittent, cross-layer, and difficult to prove: RoCE congestion, InfiniBand stalls, ECN/DCQCN mis-tuning, bad optics, RNIC issues, host kernel stalls, GPU driver problems, and workload symptoms that look like network problems, but are not.
We are hiring a Lead Software Engineer to build a first-class observability and root-cause analysis system for GPU fabrics. This is a hard distributed systems problem, not a dashboarding problem. The system will collect high-volume signals from switches, hosts, active probes, and inference services; reduce and correlate them in real time; understand topology and service ownership; and produce actionable diagnosis while an incident is still unfolding.
This role sits at the boundary between networking and inference software. RDMA data paths, GPUDirect transfers, prefill/decode disaggregation, KV cache movement, request routing, and workload backpressure can all create fabric symptoms or hide real fabric failures. The goal is to tell an operator, quickly and with evidence, whether an incident is caused by the fabric, host, NIC, GPU, RDMA path, scheduler, or serving layer - and what to do next.
EXAMPLE INITIATIVES- Real-time telemetry engine - Build the ingestion, reduction, storage, and query path for high-cardinality fabric, host, GPU, and workload telemetry.
- Service-aware fabric diagnosis - Build collectors, probes, and topology-aware correlation to detect latency, drops, stalls, congestion, bad paths, and degradation.
- Software-aware RDMA diagnosis - Tie network behavior to RDMA operations, GPUDirect paths, KV cache transfers, prefill/decode disaggregation, and request latency.
RESPONSIBILITIES- Own Baseten's GPU fabric observability and root-cause analysis architecture.
- Build telemetry pipelines across switches, NICs, hosts, GPUs, Kubernetes, and inference services.
- Model topology, flow paths, service ownership, and failure domains.
- Separate true fabric faults from host, NIC, GPU, kernel, driver, RDMA, scheduler, and workload failures.
- Create clear operator workflows for triage, remediation, and post-incident learning.
REQUIREMENTS- Staff-level or senior staff-level experience building production infrastructure software.
- Strong distributed systems background, especially streaming systems, telemetry pipelines, diagnostics, or control-plane software.
- Experience building systems that process high-volume, high-cardinality, noisy operational data.
- Understanding of networking fundamentals and high-performance networks
- Ability to work with low-level infrastructure signals and build practical correlation, anomaly detection, or root-cause analysis systems.
BENEFITS- Competitive compensation, including meaningful equity.
- 100% coverage of medical, dental, and vision insurance for employee and dependents
- Flexible PTO policy including company wide Winter Break (our offices are closed from Christmas Eve to New Year's Day!)
- Paid parental leave
- Fertility and family-building stipend through Carrot
- Company-facilitated 401(k)
- Exposure to a variety of ML startups, offering unparalleled learning and networking opportunities.
Apply now to embark on a rewarding journey in shaping the future of AI! If you are a motivated individual with a passion for machine learning and a desire to be part of a collaborative and forward-thinking team, we would love to hear from you.
