Principal Software Engineer, Inference

This role has been designed as 'Hybrid' with a requirement that you will work on average 2 days per week from an HPE office.

Job Description:

HPE's Private Cloud AI organization is seeking a Principal Software Engineer to lead the model runtime within HPE AI Essentials, the inference platform used by enterprises to operate large language models on infrastructure they own, including air-gapped and sovereign environments. The principal engineering challenge in this domain is not model deployment but sustained execution efficiency: achieving low tail latency and high GPU utilization on customer-owned hardware of varying generation and configuration. In this role you will define the architecture of that runtime - engine integration, batching, KV cache management, and distributed execution - together with the Kubernetes orchestration layer that supports it. The primary work location is as listed, but could be any other HPE site location in the US; however, remote work options will be considered.
Responsibilities
• Define and own the technical direction of the LLM serving deployment, including engine integration, continuous batching, KV cache management and reuse, and quantized execution
• Partner with inference performance engineering teams, with accountability for time-to-first-token, inter-token latency, throughput per GPU, and P95/P99 tail latency
• Define distributed inferencing strategy, including disaggregated prefill/decode, tensor and pipeline parallelism, KV cache offload across GPU memory, host memory, and RDMA-attached storage
• Evaluate emerging runtimes, quantization schemes, speculative decoding, and mixture-of-experts serving, and determine whether each runtime is adopted, developed in-house, or declined
• Define the orchestration layer supporting the runtime, including model admission, GPU scheduling and partitioning, cache-aware request routing, and autoscaling
• Mentor engineers, lead design and architecture reviews, and present technical direction to business unit and executive audiences
Knowledge and Skills

Required
• Production experience with LLM inference engines such as vLLM, SGLang, TensorRT-LLM, TGI, or NVIDIA NIM, including modification of engine internals
• Comprehensive understanding of inference internals, including continuous batching, paged attention, KV cache reuse and prefix caching, chunked prefill, quantization, and speculative decoding
• Tensor and pipeline parallelism, NCCL collective operations, and the GPU memory hierarchy and interconnect characteristics that govern them
• Expert level proficiency in Kubernetes platform architectures, including operators, custom resources, controllers, and scheduling
• Strong programming proficiency in Go and Python, with the ability to read, debug, and profile C++/CUDA using tools such as Nsight
• Experience with debugging/profiling multi-tier application workloads such as RAG, Agents, etc
• Excellent analytical, debugging, and problem-solving abilities

Preferred
• Upstream contribution to vLLM, SGLang, TensorRT-LLM, LLM-D, LMCache, or KServe
• Disaggregated prefill/decode serving, or KV cache offload and reuse at scale
• RDMA, GPUDirect Storage, InfiniBand, or RoCE
• MIG, fractional GPU allocation, and multi-tenant GPU isolation
• On-premises, air-gapped, or regulated enterprise software delivery
Experience and Education
• Minimum of 12 years of experience in Software Engineering, including +1 years working directly on LLM inference runtimes or production model serving
• Degree in Computer Science or related field

Accessibility

HPE is committed to creating an inclusive and accessible workplace and encourages applications from all qualified individuals, including those with disabilities. If you believe you require accommodation during any stage of the application or interview process, please submit your request by completing our secure form linked here.

Note: This option is reserved for applicants needing assistance/reasonable accommodation related to a disability.

What We Can Offer You:

Health & Wellbeing

We strive to provide our team members and their loved ones with a comprehensive suite of benefits that supports their physical, financial and emotional wellbeing.

Personal & Professional Development

We also invest in your career because the better you are, the better we all are. We have specific programs catered to helping you reach any career goals you have - whether you want to become a knowledge expert in your field or apply your skills to another division.

Unconditional Inclusion

We are unconditionally inclusive in the way we work and celebrate individual uniqueness. We know varied backgrounds are valued and succeed here. We have the flexibility to manage our work and personal needs. We make bold moves, together, and are a force for good.

Let's Stay Connected:



#unitedstates

Job:
Engineering
Job Level:
TCP_05

"The expected salary/wage range for this position is provided below. Actual offer may vary from this range based upon geographic location, work experience, education/training, and/or skill level.
- United States of America: Annual Salary USD 160,000 - 303,000 in Colorado // 152,000 - 349,000 in North Carolina & Texas
The listed salary range reflects base salary. Variable incentives may also be offered."

Information about employee benefits offered in the US can be found at https://myhperewards.com/main/new-hire-enrollment.html

The estimated job application period closure is December 30 2027; this timeline is provided for transparency and internal planning purposes.