Senior AI Engineer (Agentic Systems)

Position Overview

Pantheon Data is seeking a Senior AI Engineer (Agentic Systems) to design, build, and operate the agentic framework at the core of our AI platform in AWS GovCloud, serving workloads that process CUI, PII, and PHI. You will own the framework that routes requests to the right agents, composes new agents at runtime from a registry of capabilities, and coordinates their tools, state, and handoffs, so that these systems behave predictably, securely, and auditably under FedRAMP High and DoD IL4/IL5.

This role owns the agent runtime: orchestration, routing, the capability registry, and the contracts that capabilities plug into. Capabilities such as retrieval, document extraction, and application integrations are built by machine learning and software engineers and exposed through the registry; you define how they are described, discovered, composed, and governed. You will partner closely with the Data Engineer, who owns the pipelines and data stores agents draw on; the Cloud Security Engineer, who owns cloud security controls and ATO evidence; and the DevSecOps and Cloud Operations Engineers, who own delivery and infrastructure. This is a systems engineering role, not a prompt-only or prototype role: successful candidates can walk through production agentic systems they have personally built, including the orchestration design, how routing decisions were made and evaluated, how failures were contained, and what they would change.

Responsibilities

• Agent Orchestration: Design and build the orchestration layer that plans, dispatches, and coordinates work across agents, including multi-step and multi-agent workflows, handoffs, parallel execution, cancellation, and human-in-the-loop checkpoints.
• Routing: Build the routing that maps requests and intermediate steps to the right agents, tools, and models, combining deterministic rules, classifiers, and LLM-based routing where each is appropriate; measure routing accuracy and cost, and make routing decisions observable and explainable.
• Capability Registry and Dynamic Agents: Own the capability registry, including schemas for describing capabilities, versioning, discovery, and access policy, and the mechanisms that assemble agents at runtime from registered capabilities, tools, and instructions.
• Tool and Integration Contracts: Define the interfaces capabilities plug into, including tool definitions, Model Context Protocol (MCP) servers, input and output schemas, and error semantics, and help capability owners onboard to them.
• State, Memory, and Reliability: Design agent state, session memory, and context management; build retries, timeouts, idempotency, loop and budget limits, and graceful degradation so agent failures are contained and recoverable.
• Evaluation and Observability: Build evaluation harnesses for agent behavior, including routing accuracy, task completion, tool-call correctness, and regression across model and prompt versions, and trace every agent run end to end for debugging, cost, and latency.
• Agent Security and Auditability: Enforce least-privilege tool access per agent and per request, contain prompt injection across tool and agent boundaries, and record a complete audit trail of agent decisions and actions; work with the Cloud Security Engineer to map these controls to NIST SP 800-53 and support ATO.
• Technical Leadership: Set the architecture and standards for agentic development across the team, review designs and code, and mentor engineers building capabilities on the framework.

Required Skills and Experience

• Bachelor's degree in Computer Science, Engineering, or a related technical field from an accredited university, or equivalent professional experience.
• 8+ years of professional software engineering experience, including 2+ years building production LLM-backed systems.
• Demonstrated experience designing and shipping a production agentic system: multi-step or multi-agent orchestration with tool use, beyond single-prompt chat applications or demos.
• Hands-on experience with LLM tool calling, structured outputs, and at least one agent framework or SDK, such as LangGraph, Strands Agents, the Claude Agent SDK, Semantic Kernel, or a custom framework you built.
• Strong software architecture skills: plugin or registry-based designs, interface and schema design, versioning, and extensibility.
• Experience with distributed and asynchronous systems: queues, event-driven workflows, concurrency, retries, idempotency, and failure handling.
• Experience evaluating nondeterministic systems, including building test sets, automated evaluations, and regression checks for LLM behavior.
• Strong Python engineering skills: readable, maintainable, tested code; debugging; packaging; and API and service design.
• Working knowledge of AI security risks such as prompt injection, excessive tool permissions, and data exfiltration, and of controls that mitigate them.
• Experience with AWS services relevant to this work, such as Amazon Bedrock, Lambda, Step Functions, ECS or EKS, SQS, DynamoDB, and CloudWatch.
• Experience with Git-based workflows, code review, CI/CD, and team-based software delivery.
• Clear written and verbal communication, including explaining agent behavior, tradeoffs, and failure modes to technical and non-technical stakeholders.
• Ability to work effectively remotely in cross-functional teams.
• Ability to meet deadlines and produce quality work.
• Proficient in Microsoft Suite software including Outlook, Word, Excel, SharePoint, and PowerPoint.

Preferred Skills and Experience

• Experience building or operating agentic systems in AWS GovCloud or other environments supporting FedRAMP High or DoD IL4/IL5 workloads.
• Experience implementing or integrating MCP servers and clients, or agent-to-agent protocols, in production.
• Experience with Amazon Bedrock beyond basic inference: Agents, Guardrails, Knowledge Bases, AgentCore, provisioned throughput, and cross-model routing.
• Experience building model routing or cascading strategies that balance quality, latency, and cost across multiple models.
• Experience with agent observability and tracing tools such as OpenTelemetry, Langfuse, LangSmith, or Arize Phoenix.
• Experience integrating retrieval, document processing, or other ML capabilities as tools within an agentic framework.
• Familiarity with federal compliance frameworks and their engineering implications: NIST SP 800-53, CUI handling, audit logging, and least-privilege access in regulated environments.
• Experience with infrastructure as code (Terraform), containerized deployment, and GitLab CI or GitHub Actions in regulated environments.
• Advanced degree in a technical discipline, or demonstrated contribution to team technical direction and mentorship of other engineers.

Clearance Requirements

Applicants selected will be subject to a security investigation and may need to meet eligibility requirements. Candidates must have the ability to obtain and maintain a Secret clearance; an active Secret clearance is preferred.

Work Location: Reston, VA - Remote

• Our company prioritizes the benefits of flexibility and collaboration, whether that happens in person or remotely.
• If the position is remote or hybrid, you may periodically work from a Pantheon Data office location or client site.
• If this position is assigned to a Pantheon Data office location or client site, you'll work with colleagues and clients in person, as needed for specific client requirements.

Interview Requirement: Candidates who are local to the area should be prepared to participate in an in-person interview as part of the selection process. Candidates outside the local area may be considered for a virtual interview.

Compensation

The salary range for this position is $120,000 - $220,000. This is not, however, a guarantee of compensation or salary. Rather, salary will be set based on experience, geographic location and possibly contractual requirements and could fall outside of this range.

Benefits Overview

We are always looking for good people! Pantheon Data is committed to providing its employees with competitive salaries and benefits in order to increase employee satisfaction and productivity. In addition to our benefits, we also offer SmartBenefits through the Washington Metro Area Transportation Authority, where you specify an amount of your pre-tax wages be paid directly to your SmarTrip account. In some cases, tuition assistance may be available for continuing education expenses and certifications related to their position. Additional details may be found at