5-7 years of experience in AI and GPU infrastructure
Strong expertise in Kubernetes operations and debugging
Proficient in Python backend development and modern frontend technologies
Solid understanding of distributed training fundamentals
Experience with observability tools like Prometheus and Grafana
Familiar with cloud platforms, preferably GCP
Passionate about advancing AI accessibility and democratization
Responsibilities
Design and manage Kubernetes-based training orchestration across multiple clusters
Create and maintain Helm charts for reproducible training stacks
Develop control-plane agents to manage pod status and cluster synchronization
Implement autoscaling and scheduling for diverse GPU hardware
Run a GitOps workflow for all changes via pull requests
Build monitoring tools for real-time log and performance analysis
Interface with various training and inference systems within cloud infrastructure
Benefits
Significant equity alongside competitive salary
Flexible work environment with remote options
Full visa sponsorship and relocation assistance
Allocated budget for professional development
Opportunity to attend team off-sites and industry conferences
Ability to influence the direction of decentralized AI development
Full Job Description
Role Impact
You'll help build our hosted training platform - the product that lets users launch LoRA and full fine-tuning runs on managed GPU clusters with a single API call or a few clicks. The role spans the developer-facing platform and the underlying Kubernetes-based training infrastructure that runs the jobs.
Core Technical Responsibilities
Hosted Training Infrastructure
Design and operate Kubernetes-based training and inference orchestration across multi-cluster, multi-cloud GPU fleets
Build and maintain Helm charts that compose trainers, inference servers, environment servers, and supporting services into reproducible "Training stacks"
Develop the Python control-plane agents that watch pods, report run state to the platform, and keep clusters in sync
Implement scheduling and autoscaling for heterogeneous hardware (H100/H200/B200) using KEDA, LeaderWorkerSet, taints/tolerations, and gang scheduling
Run a tight GitOps workflow - every change ships through PRs, Helm values, and CI
Build node-local model caches, checkpoint pipelines, and shared storage for fast cold starts
Operate the observability stack (Prometheus, Grafana, Loki, DCGM) and make GPU cluster debugging fast
Platform Development
Build the developer-facing surfaces for hosted training: job submission, live run monitoring, logs, metrics, model/adapter management, comparisons
Develop FastAPI backend services and REST APIs that bridge the platform to running clusters
Ship product UI in Next.js / React / TypeScript with shadcn, Tailwind, tRPC, and TanStack Query
Research Bridge
Interface with the RL trainer, inference servers, and environment servers running inside our clusters
Productize new training capabilities (new model architectures, RL algorithms, modes)
Technical Requirements
We're looking for engineers who are fluent across three areas - you don't need to be the world's best at any one, but you should have real depth in all three and a clear point of view on how they connect.
AI & GPU Landscape
Strong working knowledge of the modern AI stack - open model families, finetuning techniques (LoRA, QLoRA, full FT, RLHF/RLAIF), inference engines (vLLM, SGLang, TensorRT-LLM)
Familiarity with GPU hardware tradeoffs (H100 / H200 / B200, NVLink, interconnects, memory hierarchy) and what they mean for training and inference workloads
Understanding of distributed training fundamentals (data/tensor/pipeline/expert parallelism, NCCL, multi-node scheduling)
Awareness of what's happening at the frontier - new models, training methods, infra patterns - and the ability to translate that into product decisions
Linux fundamentals: networking, namespaces, performance tuning
Programming & Platform
Strong Python backend development (FastAPI, async, SQLAlchemy)
Comfortable building Python control-plane agents that talk to Kubernetes APIs
Modern frontend development (TypeScript, React/Next.js, Tailwind, shadcn) - enough to ship product surfaces end-to-end
REST and tRPC API design
Experience building developer tools, dashboards, and live-monitoring UIs
What We Offer
Cash compensation $150K-$300K with significant equity
Flexible work arrangement (remote or San Francisco office)
Full visa sponsorship and relocation support
Professional development budget for courses and conferences
Regular team off-sites and conference attendance
Opportunity to shape the future of decentralized AI development
Growth Opportunity
You'll join a team of experienced engineers and researchers working on cutting-edge problems in AI infrastructure. We believe in open development and encourage team members to contribute to the broader AI community through research and open-source work.
We value potential over perfection - if you're passionate about democratizing AI development and have experience in either platform or infrastructure development (ideally both), we want to talk to you.
Ready to help shape the future of AI? Apply now and join us in our mission to make powerful AI models accessible to everyone.
