Job Summary
We are seeking a Senior AI Engineer to design, build, deploy, and operate enterprise-grade Large Language Model (LLM) applications that support business processes in regulated financial services environments. The role requires hands-on experience delivering Generative AI solutions from design through production deployment, including ongoing production support. The ideal candidate will have strong expertise in Agentic AI, Generative AI, LLM orchestration, Retrieval-Augmented Generation (RAG), AI evaluation, Python, cloud services, observability, and production-grade AI engineering.
Key Responsibilities
• Design and implement multi-step LLM pipelines, including LLM orchestration, structured output generation, tool/function calling, error handling, retries, and fallback logic.
• Develop and maintain RAG solutions, including indexing strategies, embedding model selection, and retrieval quality optimization.
• Build and maintain LLM evaluation frameworks and golden datasets to measure accuracy, consistency, and regression risk before production releases.
• Establish and enforce quality gates for model, prompt, and configuration changes.
• Implement observability for non-deterministic AI applications, including tracing, structured logging, quality monitoring, and cost monitoring.
• Manage the cost and latency of production AI workloads against defined budgets.
• Collaborate with Product, Business, Risk, Compliance, and Security teams to translate business requirements into technical solutions and support review and approval processes.
• Participate in code reviews, architectural decisions, and technical documentation.
• Provide ongoing production support for deployed AI applications, including incident response, root cause analysis, regression management, and remediation.
• Mentor junior engineers and contribute to engineering standards and best practices.
• Design and operate enterprise-grade AI applications that meet availability, latency, cost management, access control, data handling, and auditability requirements.
• Support security and compliance reviews and manage changes deployed against live production traffic.
Required Qualifications
• Bachelor's degree in Computer Science, Engineering, or a related technical field, or equivalent practical experience.
• Minimum 5 years of professional software engineering experience.
• Minimum 2 years of experience developing LLM or Generative AI applications.
• Demonstrated experience deploying at least one enterprise-grade AI application into production and providing post-launch support. Proof-of-concept-only experience does not meet this requirement.
• Experience delivering AI solutions that serve real users or business processes rather than only internal pilots.
• Hands-on experience with Agentic AI, Generative AI, LLMs, and LangChain.
• Experience with at least one LLM agent or orchestration framework, such as LangGraph, LangChain, LlamaIndex, Semantic Kernel, AutoGen, or Amazon Bedrock Agents.
• Strong experience with LLM evaluation methodologies, including golden dataset curation, metric selection, evaluation of non-deterministic outputs, LLM-as-judge techniques, and established evaluation frameworks or custom evaluation harnesses.
• Strong proficiency in Python 3.11+, including type annotations, automated testing, and asynchronous programming.
• Experience building and operating cloud-based services, including REST APIs, data pipelines, and event-driven workers.
• Experience working with codebases governed by automated quality gates and formal code reviews.
• Experience completing security and compliance reviews in regulated or enterprise environments.
• Post-production ownership involving incident response, regression management, and deployment of changes against live traffic.
• Experience managing non-functional requirements, including availability, latency, cost, access control, data handling, and auditability.
Preferred Qualifications
• Experience with Document AI or Intelligent Document Processing, including OCR, structured field extraction, scanned documents, photographed documents, and handwritten source material.
• Experience designing human-in-the-loop workflows, including confidence scoring, selective automation, escalation, and review processes.
• Experience implementing business rules or policies as versioned and reviewable artifacts.
• Experience with event-driven architecture, particularly Apache Kafka.
• Experience with Kubernetes, Helm, and CI/CD pipeline ownership.
• Experience with Infrastructure as Code (IaC), particularly Terraform.
• Experience with model selection, model benchmarking, fine-tuning, distillation, and cost and quality optimization.
• Experience with financial services, insurance, healthcare, or other regulated industries.
Technical Skills
• AI and Generative AI: LLMs, Agentic AI, LLM orchestration frameworks, RAG, LLM evaluation, and observability tooling.
• Programming and Application Development: Python 3.11+, FastAPI, and Pydantic.
• Data and Databases: PostgreSQL with vector extensions and Aurora PostgreSQL.
• Cloud and AI Platforms: AgentCore and cloud-based AI services.
• Infrastructure and DevOps: Kubernetes, Helm, Terraform, and CI/CD.
• Event-Driven Architecture: Apache Kafka.
• Observability: Datadog and OpenTelemetry.
Notes:
Mandatory Areas Agentic AI; Generative AI; LLMs; LangChain
Must Have Skill 1 Agentic AI / LLM orchestration with hands-on LangChain, LangGraph, LlamaIndex, Semantic Kernel, AutoGen, or Amazon Bedrock Agents
Must Have Skill 2 Generative AI / LLM application development, with minimum 2 years developing LLM/GenAI applications
Must Have Skill 3 RAG including embeddings, indexing strategies, vector retrieval, and retrieval-quality optimization
Must Have Skill 4 Python 3.11+, including type annotations, automated testing, asynchronous programming, APIs/FastAPI
Must Have Skill 5 LLM Evaluation, including golden datasets, metrics, non-deterministic output evaluation, and LLM-as-judge/evaluation frameworks
Must Have Skill 6 Enterprise production AI deployment & post-production support including incident response, regression management, latency/cost management, and live production changes
Must Have Skill 7 AI Observability & Cloud Services including tracing, structured logging, quality/cost monitoring, REST APIs/data pipelines/event-driven services
Must Have Skill 8 Enterprise Security & Compliance, including security/compliance reviews, access control, data handling, and auditability
Domain Experience Regulated industry strongly preferred, particularly Financial Services. Insurance and Healthcare are also acceptable preferred backgrounds.
Experience Minimum 5 years software engineering + minimum 2 years LLM/GenAI application development
Must Have Certifications None specified
Location Remote.
Onsite Requirement No
Number of days onsite Remote
