AI Inference / Systems Software ArchitectReports To: VP, Engineering
Position Summary:This is a highly technical role for an individual who deeply understands AI Data Center and Edge AI Inference workloads, model execution, systems software, memory management, and hardware/software co-design.
The successful candidate will work closely with the Chief AI Systems Architect, AI Product Strategist, AI Hardware / Memory Systems Architect, and R&D engineering teams to understand how AI workloads execute on current infrastructure, identify software bottlenecks, and architect software that enables new hardware and memory technologies to deliver measurable customer value.
We are particularly interested in investigating opportunities involving AI workload, software stack, and inference runtime usage of the hardware to deliver the required performance and efficiency.
Key Responsibilities:AI Inference Architecture- Develop a deep understanding of modern AI inference workloads and their system-level behavior
- Analyze the execution characteristics of Large Language Models (LLMs), Generative AI, Agentic AI, Long-context inference, Mixture-of-Experts (MoE), Retrieval-Augmented Generation (RAG), Multimodal AI and Edge inference
- Analyze the differences between prefill and decode workloads
- Characterize the impact of batch size, sequence length, concurrency, model size, attention mechanisms, and memory access patterns
- Identify whether workload bottlenecks are caused by compute, memory bandwidth, memory capacity, latency, interconnect, software overhead, or scheduling
AI Runtime and Systems Software- Architect software infrastructure that allows AI workloads to efficiently utilize heterogeneous memory and compute resources
Areas may include AI inference runtimes, model execution engines, scheduling, resource management, memory management, KV-cache management, data movement, prefetching, caching, model partitioning, load balancing, multi-device execution, distributed inference, accelerator orchestration and telemetry and performance monitoring
- Evaluate and contribute to open-source and commercial AI software stacks such as PyTorch, CUDA, ROCm, Triton, vLLM and other emerging inference runtimes and compiler frameworks
AI Memory Management- Work closely with the AI Hardware / Memory Systems Architect on next-generation memory architectures to make the entire memory hierarchy appear as an efficient, application-aware resource rather than a collection of independent storage tiers
- Explore how software can intelligently manage HBM, DDR, CXL memory, memory pools, disaggregated memory, NVMe, persistent storage, local versus remote memory, model weights, context state and KV cache
- Develop policies and mechanisms for data placement, tiering, migration, caching, prefetching, eviction, compression, decompression, memory sharing, QoS, isolation and resource allocation
Hardware / Software Co-Design- Work closely with the hardware architects to ensure that architectural features and hardware capabilities like CXL memory, memory pooling, smart memory controllers, near-memory compute, PIM, inference engines, AI accelerators, compression engines and specialized cache architectures can be effectively exposed and utilized by the software stack
- Collaborate on AI runtime architecture, memory management, KV-cache management, scheduling, data placement, prefetching, compression, workload orchestration, APIs and software interfaces and hardware telemetry and monitoring
- Ensure that hardware innovations translate into measurable application-level benefits rather than isolated component-level benchmarks
- Develop and lead performance optimization methodologies based on real AI workloads. Measure and optimize tokens/sec, time-to-first-token (TTFT), time-per-output-token (TPOT), latency, throughput, memory bandwidth utilization, memory capacity utilization, accelerator utilization, power efficiency, cost per inference, cost per million tokens and system level TCO
Inference Performance Optimization- Identify opportunities to improve performance through kernel optimization, quantization, sparsity, compression, operator fusion, memory placement, scheduling, batching, speculative decoding, KV-cache optimization, prefill/decode optimization and communication reduction
Software Architecture for CXL and Disaggregated Memory- Help define the software architecture necessary to make CXL-based memory expansion, pooling, and disaggregation practical for AI workloads
- Investigate host memory versus CXL memory placement, shared versus dedicated memory, memory allocation policies, NUMA-aware scheduling, remote-memory latency, bandwidth allocation, memory QoS, memory migration, application-level
- Develop software mechanisms that allow applications to benefit from additional capacity without unacceptable latency or performance penalties
- Develop software architecture for AI context memory systems. Potential capabilities include KV-cache tiering, KV-cache compression, KV-cache quantization, prefix sharing, context reuse, cache eviction, context migration, session-aware caching, distributed context management, GPU-to-remote-memory transfers and context prefetching
versus system-level control and multi-host memory orchestration
AI Context / KV-Cache SoftwareArchitecture Modeling and Prototyping- Develop software prototypes to validate new hardware architectures
- Build workload models and emulation environments before silicon availability
- Create benchmark suites representative of customer workloads
- Evaluate architecture concepts using existing CPUs, GPUs, FPGAs, or emulation platforms
- Quantify software overhead and expected system-level benefits
- Establish performance baselines against existing GPU, CPU, CXL, and inference solutions
Required Qualifications:- Bachelor's or Master's degree in Computer Science, Computer Engineering, Electrical Engineering, or a related technical field; PhD is a plus
- 10+ years of experience in systems software, AI infrastructure, computer architecture, inference systems, compilers, runtimes, or related areas
- Deep understanding of modern AI/ML inference workloads
- Strong systems programming skills, preferably in C/C++, Python, or comparable languages
- Strong understanding of computer architecture, memory systems, operating systems, and data movement
- Experience architecting or optimizing large-scale AI inference systems
- Demonstrated experience translating application/workload requirements into systems architecture
- Strong understanding of performance modeling and benchmarking
- Excellent communication skills and ability to work across hardware and software organizations
Preferred Qualifications:Experience with several of the following:
- LLM inference
- Transformer architecture
- KV cache
- vLLM
- PyTorch
- CUDA
- Triton
- TensorRT
- ROCm
- XLA
- MLIR / LLVM
- FlashAttention
- MoE
- Quantization
- Speculative decoding
- Distributed inference
- GPU systems
- AI accelerators
- HBM
- DDR
- CXL
- NUMA
- Memory pooling / disaggregation
- PIM / near-memory computing
- AI networking
- AI storage
- Edge AI
- Hardware/software co-design
Salary Range: $275,000 
