JOB SUMMARY
You design, document, and operate the LLM Gateway as a highly available, cost-governed shared service - owning its architecture, onboarding automation, workload and FinOps management, CI/CD, and ongoing reliability so that applications and models can be onboarded and run behind it safely and repeatably.
Key Responsibilities
Architecture & Documentation
Finalize the Gateway architecture and produce the authoritative Technical Document.
Define the HA/DR architecture and its high-availability and disaster-recovery requirements.
Finalize the RACI for the Gateway.
Onboarding Automation
Build automation for user / application onboarding.
Build automation for LLM onboarding.
Workload, Budget & FinOps
Manage LLM workload across Provisioned Throughput (PT) and Pay-As-You-Go (PAYG) usage.
Own Budget / FinOps for the Gateway.
Reliability & Operations
Handle upgrades and debugging.
Build notification / alerting.
Run performance testing and benchmarking.
Delivery & Reporting
Build and maintain CI/CD pipelines (development and maintenance).
Create reports.
Implement customizations.
Required Qualifications
Skill Tied to Python (production-grade)
Onboarding automation, CI/CD tooling, customizations
API gateway / proxy engineering
Architecture and operation of the Gateway
LLM API integration
LLM onboarding; PT/PAYG workload management
HA/DR architecture
HA/DR architecture & requirements
CI/CD & infrastructure as code
CI/CD pipelines (dev & maintenance)
Observability & alerting
Notification/alerting; debugging
FinOps / cost management
Budget/FinOps; PT vs PAYG governance
Performance / load testing
Performance testing & benchmarking
Technical documentation & RACI
Technical Document; RACI finalization
Preferred Qualifications
Experience with LLM gateway tooling (e.g. LiteLLM, API management / cloud-native AI gateways).
Kubernetes / containerized deployment for AI workloads.
Provisioned-throughput / reserved-capacity management with a major model provider.
Experience 8-12 years in software / platform engineering on production systems.
Demonstrated experience operating high-throughput shared services or API gateways.
Hands-on experience integrating and operating LLM APIs in production.
Certifications
None mentioned
