DescriptionProduct Analyst, AI SolutionsThe Product Analyst, AI Solutions is responsible for shaping, optimizing, and maintaining advanced AI-driven features and user experiences within our product suite. This role sits at the intersection of business analysis, AI tuning, and user experience-combining expertise in software requirements gathering, prompt engineering, and AI model evaluation. The Analyst is heavily focused on refining how AI systems behave in real-world scenarios, ensuring outputs are accurate, trustworthy, and closely aligned with client needs.
You partner with AI Engineers, Product Managers, UX/UI Designers, QA, and Subject Matter Experts (SMEs) to extract user requirements, translate those into actionable User Stories and evaluation metrics, and continuously tune AI systems for best-in-class performance and usability.
This role is pivotal in ensuring our AI solutions provide maximum business value, aligning advanced technology with end-user needs while driving continuous improvement and reliability. The Analyst produces high-quality deliverables using systematic processes, with impact across multiple projects, deadlines, and teams.
Principal DutiesAI Tuning & Output Quality- Design, evaluate, and refine prompts, templates, and conversation flows for reliability, cost effectiveness, and user alignment.
- Maintain prompt libraries, track changes, and manage versioning.
- Systematically test AI model outputs for accuracy, consistency, compliance, and tone, applying rerankers, guardrails, and validation layers to ensure business rules are met.
- Collaborate with engineering and QA to promptly integrate feedback and resolve model drift or new scenarios.
Requirements Gathering & Documentation- Proactively gather requirements of moderate scope and complexity from diverse sources (clients, market analysis, SMEs) to drive product development and AI enhancements.
- Document software and system requirements, including user stories, program functions, test cases, and steps required for new features or modifications.
- Ensure all documentation supports future maintainability and growth of AI-enabled features.
Evaluation, Metrics, & Continuous Improvement- Establish systematic evaluation frameworks for AI model outputs, leveraging quantitative and qualitative benchmarks.
- Run experiments, A/B tests, and structured evaluations to compare tuning strategies and track performance metrics (accuracy, coverage, hallucination rates, latency).
- Translate business and user feedback into actionable improvements, owning continuous iteration for released solutions.
Collaboration & UX Alignment- Act as a bridge between technical and non-technical stakeholders, ensuring clear, effective communication and alignment.
- Work closely with product, development, and UX/UI teams to ensure AI-driven features fit seamlessly into business workflows, are intuitive and user-friendly, and meet client needs.
- Help shape the overall user experience for AI interactions, promoting explainability and transparency.
Education, Experience and Special Skills- Bachelor's degree in Computer Science, Management Information Science, Business, or related field OR equivalent work experience.
- 1+ years' experience in software, AI/ML, or related product analysis roles.
- Experience with AI/ML applications (LLMs, NLP, machine learning pipelines).
- Strong understanding of prompt engineering and generative AI interaction design.
- Basic knowledge of data modeling, relational database concepts, and SQL.
- Familiarity with model evaluation techniques and performance metrics.
- Proficiency with Python or JavaScript for prototyping and evaluation.
- Experience with Agile SCRUM development processes.
- Excellent planning, organization, and interpersonal communication skills.
- Ability to work independently and collaboratively across functional groups.
- Experience in ERP systems, enterprise software, or complex business domains.
- Familiarity with RAG (retrieval-augmented generation), embeddings, MLOps, and responsible AI practices.
- Experience with fine-tuning methods (LoRA, QLoRA, PEFT), running structured evaluation series, or human-in-the-loop systems.
