Familiarity with cloud environments (GCP, AWS, or Azure) and containerization basics (Docker/Kubernetes)
Strong problem-solving skills with a bias toward shipping and iterating
Responsibilities
Design, build, and maintain AI agents within the AgentOS framework
Develop new MCP servers and tools that expose Pinnacle analytics capabilities to AI agents
Implement multi-turn conversational flows and agent orchestration logic for complex queries
Build and maintain the chat UI integration layer connecting AgentOS to the front-end experience
Work with major LLM providers to select and integrate models for specific agent tasks
Develop and own prompt engineering strategies for AI interactions
Build and maintain evaluation frameworks to measure agent accuracy and user satisfaction
Benefits
A chance to build at the frontier of applied AI on a high-visibility platform
Direct impact on how Fortune 500 clients interact with their advertising data
A collaborative, innovative team culture with a strong emphasis on continuous learning
Professional development and conference opportunities
Hybrid work flexibility from our New York City headquarters
Full Job Description
Senior AI Engineer - Pinnacle Role Overview
We are looking for a talented and driven Senior AI Engineer to join the Pinnacle AI (PinAI) team at DoubleVerify. In this role, you will be a core builder of AgentOS - our agentic AI platform that powers conversational, intelligent experiences across the Pinnacle advertising analytics suite.
You will design and implement AI agents, Model Context Protocol (MCP) servers, and LLM-backed tools that allow Fortune 500 clients to interact with their advertising data through natural language. This is a hands-on engineering role for someone who is passionate about applied AI, thrives in a fast-moving environment, and wants to see their work directly impact how the world's largest advertisers make decisions.
Location: New York, NY (Hybrid) What You'll Do Agentic AI Development
Design, build, and maintain AI agents within the AgentOS framework using architectures based on MCP (Model Context Protocol), LangChain, LangGraph, and related tooling
Develop new MCP servers and tools that expose Pinnacle analytics capabilities - including metrics, dimensions, settings, and user data - to AI agents
Implement multi-turn conversational flows, tool-use reasoning chains, and agent orchestration logic that handle complex, real-world advertising analytics queries
Build and maintain the chat UI integration layer, connecting AgentOS to the Pinnacle front-end experience
LLM Engineering & Optimization
Work with major LLM providers (OpenAI, Anthropic, Google, and others) to select, prompt, and integrate models appropriate for specific agent tasks
Develop and own prompt engineering strategies, including system prompts, few-shot examples, and chain-of-thought techniques
Implement Retrieval-Augmented Generation (RAG) pipelines to ground AI responses in Pinnacle's proprietary data and documentation
Continuously optimize model performance through evals, latency tuning, and cost management
Evaluation & Quality
Build and maintain evaluation (evals) frameworks that measure agent accuracy, instruction-following, hallucination rates, and user satisfaction
Define quality benchmarks and run regression evals on every agent change before production deployment
Instrument agents with observability tooling (e.g., LangSmith, Weights & Biases, or custom tracing) to support debugging and continuous improvement
Integration & Platform Engineering
Integrate AgentOS with Pinnacle's existing platform services - including PIAM/Keycloak authentication, Settings APIs, and analytics data layers
Collaborate with backend teams to build and consume APIs that surface data cleanly and securely to AI agents
Participate in architecture reviews and contribute to technical design documents for new AI-powered features
Cross-Functional Collaboration
Partner closely with Product, UX, and Data Science to translate business requirements into agent capabilities and conversational flows
Work with QE teams to define testing strategies for AI features, including behavioral testing and adversarial prompting
Contribute to the team's shared AI engineering standards, best practices, and documentation
Who You Are Required Qualifications
Experience:
5+ years of professional software engineering experience building production-grade systems
2+ years of hands-on experience developing LLM-based applications or AI agent systems
Demonstrated experience shipping AI-powered features end-to-end, from prototype to production
Technical Expertise:
Agentic AI Frameworks: Solid hands-on experience with LangChain, LangGraph, or equivalent agent orchestration frameworks
Model Context Protocol (MCP): Familiarity with MCP or similar modular tool-integration protocols for AI agents (or strong appetite to ramp quickly)
LLM Integration: Experience calling and integrating with LLM APIs (OpenAI, Anthropic Claude, Google Gemini, or similar), including streaming, tool-calling, and structured output patterns
Python: Strong Python proficiency; ability to build clean, testable, production-quality backend services (FastAPI experience a plus)
RAG & Embeddings: Understanding of retrieval-augmented generation patterns, vector stores, and semantic search
Evaluation Mindset: Experience building or using eval frameworks to measure LLM/agent quality and regression-test AI behavior
Cloud & Infrastructure: Familiarity with cloud environments (GCP, AWS, or Azure) and containerization basics (Docker/Kubernetes)
Soft Skills:
Strong problem-solving skills with a bias toward shipping and iterating
Clear communicator - able to explain AI behavior and limitations to both technical peers and non-technical stakeholders
Comfortable operating with ambiguity in a fast-paced, evolving AI landscape
Collaborative team player who contributes actively to shared standards and code quality
Education:
BS/MS in Computer Science, Machine Learning, AI, or a related technical field
Preferred Qualifications
Experience with MCP server development and tool-use patterns in production agentic systems
Familiarity with PIAM, Keycloak, or OAuth 2.0-based authentication patterns
Knowledge of advertising analytics, AdTech platforms, or BI tools (Looker, Snowflake, etc.)
Experience with modern front-end frameworks (React/Angular) for conversational UI features
Contributions to open-source AI projects or published work in AI/ML
Experience with agent evaluation tools such as DeepEval, LangSmith, or Braintrust
Knowledge of responsible AI practices, including bias mitigation and hallucination reduction strategies
What We Offer
A chance to build at the frontier of applied AI on a high-visibility platform used by the world's largest advertisers
Direct impact on how Fortune 500 clients interact with and act on their advertising data
A collaborative, innovative team culture with a strong emphasis on continuous learning and AI adoption
Competitive compensation including base salary, bonus, and equity
Comprehensive benefits package
Professional development and conference opportunities
Hybrid work flexibility from our New York City headquarters
Co-authored with Glean
About DoubleVerify
DoubleVerify is a digital advertising company that provides verification and measurement solutions for brands, agencies, and publishers. The company was founded in 2008 and is headquartered in New York, New York. DoubleVerify's technology helps advertisers ensure that their ads are viewable, brand-safe, and free from fraud. The company also provides data and insights to help advertisers optimize their campaigns. DoubleVerify has offices in over 20 countries and works with some of the world's largest advertisers and publishers.