Title: AI Quality Engineer
Location: Hybrid (Vinna, VA) or Remote
Duration: Initial 6 months plus extensions
Work Requirement: USC, GC or Authorized to work in US

Job Description
One of our Financial client is seeking an experienced AI Quality Engineer to establish and execute the quality, validation, certification, and production readiness processes for AI-powered SDLC and enterprise automation solutions.
As *** continues to expand its AI-enabled capabilities, we require an independent validation function responsible for ensuring these solutions are accurate, reliable, secure, explainable, compliant, and production ready for Production deployment.
This is not a traditional manual testing role. It is a specialized engineering position focused on validating AI systems, Retrieval-Augmented Generation (RAG) architectures, agent workflows, orchestration frameworks, observability platforms, security controls, and production readiness requirements. The successful candidate will serve as an independent reviewer and certification authority responsible for evaluating the quality and effectiveness of AI-generated outputs and agent behavior.
The AI Quality Engineer will work closely with AI Engineers, Platform Engineers, Architects, Quality Engineering teams, Security, Risk, and Product Owners to develop repeatable evaluation frameworks, testing methodologies, benchmarking standards, and governance processes for enterprise AI solutions. This individual will play a critical role in ensuring AI solutions meet enterprise expectations for accuracy, transparency, auditability, and operational excellence before production deployment.

In addition to validating AI-powered solutions, this role will serve as the independent quality authority for ***'s Internal Developer Portal (IDP) and Developer Experience (DevEx) platform initiatives. The successful candidate will evaluate and certify new portal capabilities, self-service workflows, automation features, developer onboarding experiences, and platform enhancements to ensure they meet usability, reliability, operational readiness, governance, and business value expectations before release to engineering teams.
Key Responsibilities:
• Define and implement AI certification, validation, and production readiness standards for enterprise AI agents and intelligent automation solutions.
• Develop and execute evaluation frameworks that measure:
* Accuracy
* Relevance
* Groundedness
* Completeness
* Hallucination rates
* Retrieval quality
* Recommendation quality
* User satisfaction
• Design and maintain AI validation datasets, benchmark scenarios, regression test suites, and golden test datasets using platforms such as LangSmith and Azure AI Foundry.
• Validate Retrieval-Augmented Generation (RAG) solutions built using Azure AI Search, Azure AI Foundry, LangChain, LangGraph, and enterprise knowledge repositories.
• Perform architecture reviews and quality assessments for AI solutions deployed on Azure Container Apps, Azure Functions, Azure Databricks, Azure SQL, Cosmos DB, and related cloud-native services.
• Evaluate AI orchestration workflows including prompt execution, agent routing, tool calling, guardrails, retrieval pipelines, human-in-the-loop processes, and MCP integrations.
• Validate security, governance, auditability, and compliance controls including Entra ID integration, RBAC, managed identities, Key Vault integrations, data protection policies, and sensitive data handling requirements.
• Create production readiness and certification checklists covering observability, monitoring, logging, resiliency, supportability, recoverability, and operational readiness.
• Analyze LangSmith traces, agent execution logs, AI evaluation metrics, and telemetry to identify quality concerns and improvement opportunities.
• Partner with development teams to identify and remediate quality, accuracy, security, and performance issues prior to production deployment.
• Create and maintain AI certification reports, scorecards, dashboards, and executive summaries for leadership and governance review boards.
• Drive continuous improvement of AI validation methodologies, testing strategies, and model evaluation frameworks across the organization.
• Establish independent quality gates that AI solutions must satisfy before production approval.
• Lead certification and production readiness reviews for Internal Developer Portal (IDP) capabilities, self-service workflows, engineering automation, and developer experience improvements.
Required Qualifications:
• Bachelor's Degree in Computer Science, Software Engineering, Data Science, Artificial Intelligence, Information Systems, or a related technical field.
• 5+ years of software quality engineering, test architecture, platform engineering, software development, machine learning engineering, or related experience.
• 2+ years of experience working directly with Generative AI, Large Language Models (LLMs), RAG architectures, AI agents, or AI-powered business applications.
• Strong understanding of AI system evaluation methodologies including accuracy testing, hallucination detection, groundedness validation, and AI quality measurement.
• Experience with one or more of the following:
* Azure AI Foundry
* Azure OpenAI
* LangChain
* LangGraph
* LangSmith
* AI agent frameworks
* RAG architectures
• Experience with cloud-native application architectures and Azure services including Container Apps, Functions, Cosmos DB, Azure AI Search, Azure SQL, Key Vault, and Databricks.
• Experience evaluating or supporting developer platforms, internal developer portals, DevOps platforms, CI/CD pipelines, platform engineering initiatives, or self-service engineering capabilities.
• Experience designing automated test strategies, quality frameworks, regression testing, and certification processes.
• Strong understanding of DevSecOps, CI/CD, observability, monitoring, and enterprise software development practices.
• Experience evaluating APIs, microservices, integrations, and distributed systems.
• Strong analytical and troubleshooting skills with the ability to investigate complex technical issues.
• Excellent documentation, communication, and stakeholder engagement skills.
• Ability to operate independently and provide objective, unbiased quality assessments.
Preferred Qualifications:
• Experience building or validating enterprise AI agents and multi-agent systems.
• Experience within Platform Engineering, Developer Experience (DevEx), Site Reliability Engineering (SRE), or DevOps organizations.
• Experience with AI observability, tracing, and evaluation platforms such as LangSmith.
• Experience validating Azure AI Search implementations, vector databases, semantic search, embeddings, and knowledge retrieval solutions.
• Experience implementing AI governance, Responsible AI, model risk management, or AI compliance frameworks.
• Experience in financial services, banking, insurance, healthcare, or other regulated industries.
• Familiarity with Azure DevOps, GitHub, GitHub MCP, Azure DevOps MCP, enterprise developer platforms, and SDLC tooling.
• Experience with performance engineering, chaos testing, resilience testing, and production readiness reviews.
• Knowledge of identity and access management concepts including Entra ID, managed identities, Key Vault integration, and RBAC controls.
• Experience creating quality scorecards, executive dashboards, KPI frameworks, and reporting metrics.
• Experience in Quality Engineering, Test Architecture, Security Engineering, Platform Engineering, or AI Engineering roles.
Ideal Candidate Profile:
A highly technical engineer who understands both modern software engineering and AI systems. This individual can independently assess whether an AI solution is accurate, secure, explainable, compliant, observable, and ready for production. They serve as the organization's independent quality authority for AI-powered solutions and help establish confidence in enterprise AI deployments before release to production.