Position Overview:We are looking for an owner for the accuracy, reliability, and cost-efficiency of the AI behind CentrlX, our agentic platform for Manager Research and Investor Relations teams at investment firms.
We have a strong engineering team building the platform: agents, skills, knowledge, automations, connectors, and governance. We are looking for a single owner for output accuracy and efficiency.
You will own the model and prompt layer end to end. You will build the evaluation datasets and harnesses that tell us whether a change actually helped, run structured comparisons across model providers on accuracy, latency, and cost, and drive the resulting changes into the product. You will help create and design agents and skills, and work closely with sales and PS in order to help drive standards for prompting and agents, and then feed that back into the product design. You will do a real share of this work yourself from prompt iteration to writing user facing stories like incorporating interactive feedback. While we have support teams to intake client issues and fix them, you will own AI Quality and be the point person for what is happening and what needs to improve.
This is a product role, not a research role. We want the analytical rigor of a data scientist paired with the judgment of a product manager: someone who can run the experiment, interpret it honestly, and then turn it into a shipped change.
Key ResponsibilitiesEvaluation & AI Quality
- Build and own CentrlX's evaluation foundation from the ground up: golden datasets, grading rubrics, LLM-as-judge pipelines calibrated against human labels, and regression suites that run before prompt or model changes ship.
- Define what "good" means for each core workflow - document digitization and extraction, retrieval and groundedness, Smart Summary, Smart Response, Smart Evaluation, and full multi-step agent runs - and set a measurable quality bar for each.
- Evaluate agent behavior, not just single responses: tool selection, retrieval quality, step sequencing, and whether the finished deliverable holds up to a practitioner's review.
- Turn every real client failure into a permanent eval case, so the same class of error does not come back.
Model Selection & Cost Optimization
- Continuously benchmark models across providers - OpenAI, Anthropic, Google, open-weight, and specialized document models - on accuracy, latency, and cost for each workflow, and make the call on what we run where.
- Own model migrations end to end, including our in-flight move off GPT-4.1 in document digitization, where current alternatives are materially faster, cheaper, and more accurate.
- Track and manage AI spend by workflow, and use routing, model tiering, caching, and context strategy to hold quality while bringing cost down.
- Maintain a working view of the model landscape - releases, pricing changes, deprecations - and turn it into a recommendation with evidence attached, not a newsletter.
Instrumentation & Data
- Define the logging and tracing we need - prompt inputs, retrieved context, prompt text, outputs, tool calls, token counts, latency - and write the stories to get it built.
- Partner with Product and Design to build in-app feedback capture (ratings, corrections, structured reason codes) so labeled data accumulates as a byproduct of normal use instead of a periodic collection project.
- Build and maintain the datasets yourself: pull the data, label it, curate the hard slices, and keep the sets honest with holdouts and rotation.
Ownership & Cross-Functional Partnership
- Serve as the single point of contact for AI quality escalations from Client Success, Sales Engineering, and Professional Services - triage, reproduce, root-cause, and close the loop.
- Write the user stories and acceptance criteria that turn findings into shipped changes, and make the prompt, configuration, and model changes yourself where that is the fastest path.
- Publish a regular quality and cost readout that leadership, engineering, and client-facing teams all treat as the same version of the truth.
- Work with CENTRL's Manager Research, Investor Relations, and diligence practitioners to encode domain judgment into rubrics - in our market, accuracy is defined by industry expertise, not by a generic benchmark.
Minimum Qualifications- Must have work authorization in the USA.
- 3+ years across product management, data science, or applied AI, including at least 2 years working on LLM-based products in production.
- Hands-on Python and SQL. You are comfortable in a notebook pulling data, running batch inference, and computing metrics. This role writes code; it does not only specify it.
- Demonstrated experience building evaluation datasets and harnesses for LLM systems: golden sets, rubric design, LLM-as-judge with human calibration, and regression testing against prompt and model changes.
- Working fluency with at least one eval or LLM observability platform: Braintrust, LangSmith, Langfuse, Arize Phoenix, W&B Weave, Inspect, Promptfoo, or a comparable in-house harness.
- Practical understanding of RAG systems: retrieval quality, groundedness and faithfulness, hallucination detection, and chunking and context strategy.
- Statistical literacy: you can size a comparison, judge significance, and say plainly when a difference is not real.
- Ability to write clear user stories and acceptance criteria and work inside an agile engineering process.
- Strong written communication. This role produces recommendations that executives act on.
Preferred Qualifications- Experience with document AI: OCR and vision-language extraction, table and layout parsing, and structured output from complex PDFs.
- Experience evaluating agentic systems - multi-step trajectories, tool-use correctness, and long-run failure modes.
- Experience in financial services or investment management: due diligence, manager research, investor relations, or DDQ/RFP workflows.
- Experience reducing inference cost at scale through routing, tiering, batching, caching, or context compression.
- Experience designing in-product feedback mechanisms that generate labeled evaluation data.
- Familiarity with agent frameworks and MCP.
- Degree in a quantitative or technical field.
