AI Agentic Tester

Prophecy Technologies

$88K — $105K *
Information Technology
Less than 5 years of experience
Job Overview by Ladders

Qualifications

  • 2-4 years of experience in QA/Test Engineering
  • Experience with testing Generative AI, Large Language Models, or AI-based applications
  • Proficient in modern Functional Quality Engineering techniques

Responsibilities

  • Validate AI agent workflows against business requirements
  • Test end-to-end AI use cases, including multi-agent interactions
  • Verify task completion accuracy and output quality
  • Test agent orchestration, tool calling, and workflow execution
  • Evaluate LLM responses for accuracy, relevance, and consistency
  • Test prompt engineering effectiveness and response quality
  • Validate Retrieval-Augmented Generation (RAG) outputs
  • Measure hallucination rates and response reliability
Full Job Description
Role Overview:

The AI Agentic Tester will be responsible for ensuring the quality and functionality of AI-based applications, with a specific focus on Generative AI and Large Language Models (LLMs). This role involves modern functional quality engineering to validate complex AI agent workflows and model outputs.

Key Responsibilities:
  • Validate AI agent workflows against business requirements.
  • Test end-to-end AI use cases, including multi-agent interactions.
  • Verify task completion accuracy and output quality.
  • Test agent orchestration, tool calling, and workflow execution.
  • Evaluate LLM responses for accuracy, relevance, and consistency.
  • Test prompt engineering effectiveness and response quality.
  • Validate Retrieval-Augmented Generation (RAG) outputs.
  • Measure hallucination rates and response reliability.

Required Skills:
  • QA/Test Engineering experience.
  • Experience testing Generative AI, LLM, or AI-based applications.
  • Modern Functional Quality Engineering.

Qualifications:
  • 2-4 Years of experience.

Similar Jobs

More Jobs at Prophecy Technologies

More Information Technology Jobs

Find similar AI Agentic Tester jobs: