AI Hardware + Software Company (Stealth) | San Francisco, CA | HybridAbout the Role We are seeking a founding-team Senior Inference Engineer to join our core SpeechLLM lab. You will be responsible for building and optimizing high-throughput, ultra-low-latency inference engines for large language models and foundational speech models. This is a rare opportunity to define the architecture of a next-generation AI system that natively understands and generates speech as part of a hardware-software intelligence platform.
You May Be a Good Fit If You - Have hands-on experience building and deploying high-throughput, ultra-low-latency inference engines for large language models or foundational speech models.
- Understand the intricate tradeoffs between latency, throughput, and Time-To-First-Token (or Time-To-First-Audio) in real-time streaming environments.
- Have practical experience with continuous batching, KV cache management (e.g., PagedAttention), and stateful connections for real-time conversational AI.
- Possess a deep understanding of GPU architectures (NVIDIA Ampere/Hopper) and the memory hierarchy, allowing you to identify and eliminate hardware bottlenecks.
- Communicate clearly and collaborate effectively at the intersection of core ML training and backend infrastructure teams.
- Thrive in fast-moving environments and genuinely enjoy the systems engineering challenge of squeezing every last drop of performance out of a GPU cluster.
Strong Candidates May Also Have - Frontier Serving Frameworks: Deep, under-the-hood familiarity with modern LLM serving frameworks such as vLLM, TensorRT-LLM, SGLang, or NVIDIA Triton Inference Server (open-source contributions are a strong plus).
- Real-Time Audio Streaming: Experience handling continuous audio streams over WebSockets or WebRTC, deploying neural audio codecs, and managing chunked audio generation to minimize conversational latency.
- Advanced Inference Techniques: Hands-on implementation of speculative decoding, lookahead decoding, or chunked prefill techniques.
- Model Compression and Quantization: Experience with post-training quantization (PTQ) and deploying models in FP8, INT8, AWQ, or GPTQ without degrading audio naturalness or ASR accuracy.
- Large-Scale Distributed Systems: Experience deploying multi-GPU (Tensor Parallelism) and multi-node inference pipelines and managing autoscaling infrastructure via Kubernetes.
What We Offer - Meaningful early-stage equity and the opportunity to be a foundational member of a core research and engineering lab.
- Highly competitive base salary: $200K to $540K depending on experience, plus performance bonus and equity.
- Top-tier healthcare for employees and dependents, including dental and vision coverage with a generous employer subsidy.
- 401(k) with company matching.
- Unlimited PTO plus 13 paid holidays.
- 12 weeks paid parental leave regardless of gender.
- Hybrid working model with a minimum of 3 days per week in-office.
- Top-of-the-line laptop or workstation, annual team offsites, and a fully stocked office.
