About the roleLuma Fintech is building a best-in-class LLM-powered document parsing pipeline that extracts structured data from complex financial product term sheets. We are seeking an AI Quality Engineer to own the daily testing, analysis, and iterative improvement of our Claude API-based extraction system. This role sits at the intersection of financial data operations and applied AI, you will be the person who closes the loop between what the model outputs and what the schema demands.
What you'll do- Run daily accuracy evaluations against a defined extraction schema, tracking field-level performance across structured product types (autocallables, CLNs, barrier notes, etc.)
- Design and maintain test cases, regression suites, and gold-standard document sets to benchmark extraction quality over time
- Diagnose extraction failures, distinguishing between prompt logic issues, schema ambiguity, model hallucinations, and edge-case document formats
- Iterate on prompt engineering, system instructions, and context design to improve field-level extraction accuracy
- Work alongside the AI Engineer lead to feed findings into validation logic and rules-based layers that sit on top of LLM output
- Document failure modes with reproducible examples and root-cause hypotheses
- Build and maintain evaluation metrics (precision, recall, field coverage, hallucination rate) and report on accuracy trends
- Flag schema gaps or ambiguities surfaced by real document variance and collaborate with data operations to refine field definitions
- Contribute to RAG improvements by identifying where retrieved context is insufficient or misleading
QualificationsRequired- Hands-on experience working with LLM APIs (Anthropic, OpenAI, or similar) in a production or near-production context
- Strong prompt engineering skills, you understand how instruction design affects model behavior, not just output tone
- Analytical mindset with the ability to systematically isolate variables in model output quality
- Experience designing structured test cases or evaluation frameworks (QA background is a plus)
- Familiarity with JSON schema, structured data output, and data validation patterns
- Ability to read and interpret complex financial or legal documents (term sheets, prospectuses, offering documents), prior financial services exposure strongly preferred
- Strong written communication; you'll be documenting findings for both technical and non-technical stakeholders
Preferred- Experience with RAG pipelines and retrieval evaluation
- Python proficiency for scripting evaluation workflows or parsing outputs
- Background in structured financial products (autocallables, structured notes, credit-linked notes)
- Familiarity with evaluation frameworks or tools (e.g., LangSmith, RAGAS, custom evals)
What Success Looks LikeIn 90 days, you have established a repeatable daily evaluation process, a documented baseline of field-level accuracy across product types, and have driven at least one measurable improvement in extraction quality through prompt iteration.
Why This RoleThis is a high-ownership position on a strategic automation initiative with direct visibility to leadership. You won't be maintaining someone else's test suite, you're building the quality layer of a system that processes real financial data at scale. The role will evolve as the system matures, with opportunity to expand into evaluation infrastructure and model improvement strategy.
The pay range for this role is:
90,000 - 115,000 USD per year (Cincinnati)
