Job Summary

Pantomath is hiring a Senior Product Manager for AI with direct, hands-on experience building agentic systems to own the strategy and execution of our AI product surface.

This role sits squarely at the intersection of product leadership and applied AI engineering. You will be responsible for defining how Pantomath's agents resolve data incidents: what context they can see, what actions they are trusted to take, how their output is evaluated, and where the boundary sits between what the agent does and what the human decides. That work spans our context layer (structural, behavioral, baseline, risk, and semantic context), our resolution and triage agents, and our MCP-based distribution into the surfaces customers already work in (Teams, Slack, ServiceNow, Jira).

The bar for this role is hands-on. You deeply understand frontier model capabilities and limitations, fluency in the parts that decide whether an agent is trustworthy: what goes into its context and what gets left out, tool and API surfaces the model can actually use correctly, evaluation for non-deterministic systems, and the cost/latency/quality frontier and the unique UX considerations of non-deterministic systems. Enough depth to read a failed trace and name the defect - retrieval, tool definition, context assembly, or prompt.

You will use that experience to set technical direction, write precise requirements, and partner closely with engineering to deliver agents that are reliable enough to earn autonomy in production data environments.
What You'll Do

• Own the AI product strategy and roadmap, making informed, technical decisions about which workflows an agent should own end to end, which it should assist, and which it should stay out of.

• Leverage your applied AI background to evaluate feasibility, failure modes, and long-term maintainability of agentic features.

• Define and defend the agent boundary: what the agent is trusted to do autonomously, what requires human confirmation, and how that boundary widens as trust is earned.

• Translate customer needs, technical constraints, and model capabilities into clear, detailed PRDs that engineers can execute against without ambiguity.

• Partner closely with engineering to design, build, evaluate, and ship agents that meet strict standards for accuracy, latency, cost, observability, and security.

• Build and own the evaluation harness: define golden datasets, quality rubrics, and regression testing so that model and prompt changes ship on evidence rather than anecdote.

• Replicate real-world customer data environments and incident scenarios to validate agent behavior, and ensure production readiness.

• Define success metrics for AI features (resolution accuracy, acceptance rate, time to root cause, deflection, token cost per outcome) and actively monitor, diagnose, and iterate based on data and feedback.

• Work with Go-to-Market teams to support launch readiness, technical enablement, and customer-facing documentation, including how to explain what the agent knows and why it is right.

• Stay current on trends and best practices across foundation models, agent frameworks, MCP and tool-calling standards, retrieval, and evaluation to inform roadmap decisions.

• Advocate for a best-in-class experience for data engineers and analysts, ensuring AI capabilities are transparent, correctable, and trustworthy at scale.

Required Qualifications (Non-Negotiable)
• Hands-on experience building with LLMs in a technical role (e.g., shipping production AI features, building agent workflows, prompt and context engineering, running model evaluations).
• 3+ years of product management experience working on technically complex products such as AI/ML platforms, data platforms, developer tools, or APIs.

Strong, working knowledge of:

• Foundation model capabilities, limitations, and cost/latency trade-offs

• Agent architecture patterns: tool calling, context assembly, retrieval, orchestration, and guardrails

• Evaluation methodology for non-deterministic systems

• Demonstrated judgment about when AI is the right answer and when it is not.

• Proven ability to partner deeply with engineering teams and make informed trade-offs.

• Ability to clearly communicate complex technical concepts to both technical and non-technical stakeholders.

• Bachelor's degree in Computer Science, Data Science, Information Systems, Engineering, or a related technical field (or equivalent practical experience).

Preferred Qualifications

• Experience with MCP, agent frameworks, or embedding AI into third-party surfaces such as Teams, Slack, ServiceNow, or Jira.

• Prior ownership of an AI or agent product within a B2B SaaS or data product.

• Familiarity with data observability, lineage, governance, or metadata platforms.

• Hands-on background in data engineering or data architecture, or experience building AI on top of enterprise data systems.

• Experience in a high-growth SaaS or developer-focused product environment.

Who This Role Is (and Isn't) For

This role is ideal for a PM who:

• Has actually built with models, not just written strategy decks about them

• Is energized by the hard part of agentic products, which is proving they are right often enough to be trusted

• Enjoys being close to the technical details and using that depth to drive product decisions

• This role is not a fit for PMs whose AI experience is limited to evaluating vendors or shipping a chat interface.

Pay Range for this role is 124,000 - 182,000

Department Product Locations San Francisco Bay Area, CA Remote status Hybrid Employment type Full-time