About the RoleWe're hiring a Software Engineer to build and scale the distributed production inference systems that serve Inkling, Inkling-Small, and Tinker in production. You'll own the systems that turn trained models into fast, reliable, cost-efficient services - from request routing and batching to multi-node serving and GPU utilization at scale.
This is a systems-heavy, production-first role. You'll work closely with research and infrastructure teams to translate rapidly evolving model architectures into serving systems that meet real-world latency, throughput, and reliability requirements, and you'll be on the front line when production inference systems need to scale, recover, or improve.
What You'll Do- Design, build, and operate distributed infrastructure for large-scale model serving, including request routing, load balancing, batching, and multi-node coordination
- Optimize inference latency and throughput in production, including work on KV cache management, continuous batching, speculative decoding, and quantization
- Build and maintain high-concurrency serving systems with strong uptime, low tail latency, and deep observability
- Benchmark, tune, and extend inference engines to support new model architectures as they move from research into production
- Partner with research and infrastructure teams to translate emerging model designs into production-ready serving systems
- Build tooling for tracing, debugging, and resolving issues across the serving stack, from orchestration down to GPU kernels
- Participate in on-call rotation to support production inference systems
Skills & Qualifications- 3+ years of experience building and operating distributed systems in production
- Strong systems programming skills in Python, C++, Rust, or similar languages
- Experience with production infrastructure at scale: reliability, observability, and performance under real-world load
- Solid understanding of networking, concurrency, and distributed systems fundamentals
Preferred Qualifications- Experience with LLM inference engines such as vLLM, SGLang, or TensorRT-LLM
- Familiarity with GPU programming (CUDA) or low-level performance optimization
- Experience with model parallelism, tensor/pipeline parallelism, or other distributed inference techniques
- Track record of operating large-scale production systems with strict latency and uptime requirements
- Experience with Kubernetes or similar orchestration systems for GPU workloads
- Contributions to open-source ML systems or inference infrastructure projects
Logistics- Location: This role is based in San Francisco, CA.
- Compensation: Depending on background, skills and experience, the expected annual salary range for this position is $350,000 - $500,000 USD (placeholder - verify against current internal bands before publishing).
- Visa sponsorship: We sponsor visas. While we can't guarantee success for every candidate or role, if you're the right fit, we're committed to working through the visa process together.
- Benefits: Thinking Machines offers generous health, dental, and vision benefits, unlimited PTO, paid parental leave, and relocation support as needed.
