AI Operations Engineer

Blytzpay

$70K — $95K *
Information Technology
Less than 5 years of experience
Job Overview by Ladders

Qualifications

  • Deep understanding of LLM memory management and context windows
  • Familiarity with data sanitization and evaluation metrics
  • Exceptional communication and interpersonal skills
  • Bachelor's degree in Computer Science or Data Science
  • Experience with tool interaction and orchestration

Responsibilities

  • Build agentic skills using coding techniques
  • Troubleshoot errors and refine prompts when users encounter issues
  • Manage evaluations with internal employees and translate feedback into improvements
  • Provide feedback on agentic architecture related to memory and context
  • Ensure AI is learning and executing tools correctly

Benefits

  • Opportunity to work on cutting-edge AI technology
  • Collaborative environment with direct impact on AI development
  • Potential for growth and learning in a dynamic field
  • Engagement with cross-functional teams
  • On-site position fosters hands-on collaboration
Full Job Description
Location: Draper, UT (On-site)

Level: Entry to Mid-Level (Perfect for a high-performing recent CS/Data Science grad)

You will be the primary driver of Agent Skill Development. You will sit between our internal users and our core architecture, ensuring the AI is learning, executing tools correctly, and providing accurate value to the company.

or related field

Role

  • Skill Construction: Use agentic coding to build agentic skills (third-party connections and tool-use logic).
  • Troubleshooting: When our internal users are unable to train the agent out of a mistake via feedback, you step in. You'll dig into the code, refine the prompts, and fix the hallucination or logic error.
  • Human-in-the-Loop Management: Work directly with internal employees to manage evals. You'll help them understand how to test the agent and translate their qualitative feedback into quantitative technical improvements.
  • Architectural Strategy: You will work within the existing agentic architecture, providing feedback on how the system handles memory, context, and tool-triggering.

Requirements

  • Agentic AI Expertise: Deep understanding of LLM memory management, context windows, and tool interaction/orchestration.
  • Familiarity with Data Science Principles: Knowledge of data sanitization, evaluation metrics, and data pipelines.
  • Exceptional Communication and Interpersonal Skills: Ability to translate complex technical concepts for non-technical stakeholders and act as a liaison to internal staff.
  • Bachelors Degree in CS/Data Science

Similar Jobs

More Jobs at Blytzpay

More Information Technology Jobs

Find similar AI Operations Engineer jobs: