This is a full-time onsite role based in the location(s) posted and is performed from the designated work location five days per week. No relocation is offered.
The Business Agent Foundry is seeking a Principal AI Agent Engineer to design, build, and scale production-grade AI agents and LLM-powered applications that solve high-value business problems.
This is not a traditional software engineering role with AI layered on top. You will work directly with business partners to turn complex, sometimes ambiguous problems into working AI-enabled solutions, moving quickly from problem definition and prototype to production while maintaining enterprise standards for security, reliability, observability, Responsible AI, and maintainability.
We’re looking for someone who has already shipped AI agents, LLM-powered workflow automation, or applied GenAI solutions into production and can clearly explain how those systems worked, how they were adopted, and the measurable impact they delivered.
What You'll Do:
- Design, build, and deploy production-grade AI agents and agentic workflows that improve customer experience, employee productivity, operational efficiency, quality, or cost.
- Partner directly with business leaders, product teams, and process owners to identify high-value AI opportunities and translate ambiguous requirements into scalable solutions.
- Build conversational, workflow, and task-oriented agents capable of using tools, APIs, enterprise data, retrieval, structured reasoning, and human-in-the-loop approvals.
- Develop agent orchestration using LangGraph, Semantic Kernel, AutoGen, CrewAI, Bedrock Agents, or similar production-capable frameworks.
- Engineer prompts, system instructions, context-management strategies, state, memory, and retrieval flows to improve agent accuracy and reliability.
- Integrate AI solutions with enterprise APIs, databases, knowledge repositories, systems of record, vector stores, and other data sources.
- Implement RAG and advanced retrieval patterns, including enterprise search, semantic retrieval, and context engineering.
- Establish evaluation frameworks using golden datasets, scenario testing, regression testing, human review, and production feedback loops.
- Build observability across prompts, model calls, retrieval, tool execution, latency, errors, and user interactions.
- Design guardrails, escalation paths, human oversight, and secure access patterns for sensitive or higher-risk workflows.
- Apply Responsible AI, privacy, security, and governance principles throughout solution design and deployment.
- Establish reusable engineering patterns, templates, and components that other teams can leverage across the AI platform.
What You'll Bring:
- Demonstrated experience building and shipping AI agents, LLM-powered applications, or GenAI workflow automation into production for real users or business teams.
- The ability to explain the scope and impact of previously deployed AI systems, including architecture, users, workflows, data sources, integrations, adoption, operating model, and measurable business outcomes.
- Deep hands-on experience with Python and backend/full-stack development using technologies such as FastAPI, Flask, or Django.
- Strong understanding of LLM application architecture, including prompt engineering, context engineering, RAG, tool/function calling, session state, structured outputs, and response validation.
- Hands-on experience with agent orchestration frameworks such as LangGraph, Semantic Kernel, AutoGen, CrewAI, or comparable platforms.
- Experience designing APIs, service integrations, authentication, database interactions, and deployment-ready application components.
- Experience connecting AI applications to enterprise data, APIs, knowledge repositories, vector databases, or systems of record.
- Strong software engineering discipline across automated testing, code review, CI/CD, logging, monitoring, observability, and production support.
- The ability to evaluate tradeoffs among AI models based on accuracy, latency, cost, reliability, risk, and business requirements.
- Strong communication skills and the ability to explain complex AI and engineering decisions to both technical and non-technical stakeholders.
What You'll Need:
- Typically 8+ years of software engineering experience, including experience designing and operating complex production systems.
- Significant hands-on experience developing and deploying LLM, GenAI, or agentic AI applications.
- Strong proficiency in Python, APIs, distributed application architecture, and enterprise integration.
- Experience with at least one production-capable agent or AI orchestration framework.
- Experience implementing evaluation, monitoring, observability, and reliability controls for AI-enabled applications.
- Understanding of Responsible AI, data privacy, security, access control, auditability, and AI guardrails.
- Ability to work in an onsite, highly collaborative environment with business and technology partners.
Preferred Qualifications:
- Technical leadership experience establishing architecture, engineering standards, or reusable patterns across multiple teams.
- Experience applying AI within enterprise functions such as customer care, operations, network, sales, finance, service delivery, or employee productivity.
- Experience with advanced RAG, knowledge graphs, enterprise search, ontology-driven retrieval, or semantic data modeling.
- Experience evaluating or serving LLMs at scale, including quality, latency, token/cost, and reliability optimization.
- Front-end development, data visualization, or rapid product prototyping experience.
- Open-source contributions or reusable frameworks related to AI agents, LLM applications, AI evaluation, or GenAI engineering.
Our Principal Software Engineering jobs earn between $155,400.00 - $261,100.00 USD Annual. Not to mention all the other amazing rewards that working at AT&T offers. Individual starting salary within this range may depend on geography, experience, expertise, and education/training.
Joining our team comes with amazing perks and benefits:
- Medical/Dental/Vision coverage
- 401(k) plan
- Tuition reimbursement program
- Paid Time Off and Holidays (based on date of hire, at least 23 days of vacation each year and 9 company-designated holidays)
- Paid Parental Leave
- Paid Caregiver Leave
- Additional sick leave beyond what state and local law require may be available but is unprotected
- Adoption Reimbursement
- Disability Benefits (short term and long term)
- Life and Accidental Death Insurance
- Supplemental benefit programs: critical illness/accident hospital indemnity/group legal
- Employee Assistance Programs (EAP)
- Extensive employee wellness programs
- Employee discounts up to 50% off on eligible AT&T mobility plans and accessories, AT&T internet (and fiber where available) and AT&T phone
Weekly Hours:
40
Time Type:
Regular
Location:
Alpharetta, Georgia, Dallas, Texas, Plano, Texas, USA:GA:Atlanta / 1057 Lenox Park Blvd Ne - Adm:1057 Lenox Park Blvd Ne
Salary Range:
$155,400.00 - $261,100.00