Caterpillar, Inc

Manager; AI Evaluation Engineering

Caterpillar, Inc$147K — $240K *
Information Technology
5 - 7 years of experience
Job Overview by Ladders

Qualifications

  • 5-7 years of experience in software development and AI evaluations.
  • Proven leadership experience in overseeing AI and software quality teams.
  • Deep knowledge of generative AI architectures and system design.
  • Hands-on experience with modern AI development ecosystems such as Azure and AWS.
  • Expertise in AI observability, monitoring, and testing methodologies.

Responsibilities

  • Lead the evaluation and validation of advanced AI solutions.
  • Provide technical guidance to align team efforts with corporate objectives.
  • Assess team performance, fostering an environment of continuous learning.
  • Ensure the quality and robustness of AI engineering products.
  • Implement best practices in software development for consistent product excellence.

Benefits

  • Medical, dental, and vision benefits
  • Paid time off plan (Vacation, Holidays, Volunteer, etc.)
  • 401(k) savings plans
  • Health Savings Account (HSA)
  • Flexible Spending Accounts (FSAs)
  • Employee Assistance Program
  • Career Development
  • Incentive bonus
  • Tuition Reimbursement
Full Job Description

Career Area:

Technology, Digital and Data

Job Description:

Job Summary:
Join theAI Engineeringteam of Cat Digital and take charge of leading a team dedicated to evaluating and validating our advanced generative AI solutionsincluding intelligent agents, digital assistants, and other innovative capabilities. You will shape the development ofcutting-edgeproducts that redefine how customersand dealersinteract with technology through AI-driven experiences.

What You Will Do:

  • Providing strong technical support and clear direction to ensure the team is aligned with company goals and capable of evaluating advanced AI projects.
  • Overseeing the performance of both individual team members and theteam as a whole, fosteringa culture of learning byidentifyingand addressing training and development needs.
  • Taking ownership of the quality of AI engineering products, making sure all solutions are robust, reliable, and meethigh standards.
  • Establishing and supervising the implementation of engineering best practices tomaintainconsistency and excellence in development processes.


What You Will Have:

  • Products and Services: Knowledge of major products and services and product and service groups; ability to apply knowledge of product and service appropriately to diverse situations.
  • Software Development: Knowledge of software development tools and activities; ability to produce software products or systems in line with product requirements.
  • Software Development Life Cycle: Knowledge of software development life cycle; ability to use a structured methodology for delivering and managing new or enhanced software products to the marketplace.
  • Software Quality Assurance and Testing: Knowledge of software quality assurance and testing; ability to apply appropriate processes, tools, and techniques for assuring a high level of quality in computer software products and systems.

Considerations For Top Candidates:

  • Proven experience leading software quality, test automation, validation, and AI evaluation teams, with the ability to scale processes, tools, metrics, and engineering practices across multiple products and enterprise initiatives.
  • Demonstrated success delivering enterprise-scale software and GenAI solutions across hybrid cloud and embedded/edge environments, leveraging modern software engineering practices including CI/CD, automated testing, incident management, feature flags, and progressive deployment strategies (e.g., blue/green and canary deployments).
  • Deep expertise in GenAI architecture and system design, including LLMs, SLMs, multimodal, speech, and real-time models; prompt engineering; agentic systems; tool use and orchestration; RAG architectures; vector databases, embeddings, and chunking strategies; fine-tuning techniques such as LoRA; and emerging standards including MCP and A2A.
  • Hands-on experience with AI development and evaluation ecosystems, including Azure, AWS, GCP, Azure AI Foundry, SageMaker, Bedrock, Snowflake Cortex, LangChain, LangGraph, Langfuse, Arize, LangSmith, Humanloop, Ragas, DeepEval, Phoenix, or similar technologies.
  • Deep understanding of AI evaluation methodologies and the challenges of non-deterministic systems, including metric design, RAG quality assessment, output reliability, safety evaluation, A/B testing, human-in-the-loop validation, and balancing deterministic and probabilistic testing approaches.
  • Expertise in modern AI observability and monitoring practices, including tracing, telemetry, evaluation data collection, root-cause analysis, and the use of AI technologies to enhance testing through automated test generation, synthetic data creation, scenario expansion, and LLM-assisted evaluation.

Summary Pay Range:

$147,760.00 - $240,110.00

Compensation and benefits offered may vary depending on multiple individualized factors, job level, market location,job-related knowledge, skills, individual performance and experience. Please note that salary is only one component of total compensation at Caterpillar.

Benefits:

Subject to plan eligibility, terms, and guidelines. This is a summary list of benefits.

  • Medical, dental, and vision benefits*

  • Paid time off plan (Vacation, Holidays, Volunteer, etc.)*

  • 401(k) savings plans*

  • Health Savings Account (HSA)*

  • Flexible Spending Accounts (FSAs)*

  • Health Lifestyle Programs*

  • Employee Assistance Program*

  • Voluntary Benefits and Employee Discounts*

  • Career Development*

  • Incentive bonus*

  • Disability benefits

  • Life Insurance

  • Parental leave

  • Adoption benefits

  • Tuition Reimbursement

* These benefits also apply to part-time employees

Visa Sponsorship is not available for this position.

Posting Dates:

September 1, 2026 - September 20, 2026

Any offer of employment is conditioned upon the successful completion of a drug screen.

About Caterpillar, Inc

Progress Rail Services Corporation, a Caterpillar company, is one of the largest integrated and diversified suppliers of railroad and transit system products and services worldwide. Since its acquisition by Caterpillar Inc. in 2006, Progress Rail has used the Caterpillar international footprint to gain quick and widespread notice and approval across the globe. Product technology, reliability and durability, along with legendary aftermarket parts and service support, have been their keys to success. Caterpillar is the world's leading manufacturer of construction and mining equipment, diesel and natural gas engines and industrial gas turbines. The technology and world reach of Caterpillar, combined with the service standard and historical reliability of Progress Rail, ensures their products and services make a growing impact on the world rail industry. With their acquisition of Electro-Motive Diesel (EMD), Progress Rail is the world's largest builder of diesel-electric locomotives for all commercial railroad applications including freight, intercity passenger, commuter, switching, industrial and mining. EMD is also a global provider of diesel power engines for marine propulsion, offshore and land based oil well drilling rigs and stationary power generation. Electro-Motive is the only diesel engine manufacturer of its size to have produced more than 70,000 engines. This success stems from product reliability, serviceability, load acceptance capability, low operating costs and ease of maintenance with standardized components.

Caterpillar, Inc. Careers

Join the robust team at Caterpillar, Inc., a global leader in construction and mining equipment, where innovation drives everything we do. As part of our team, you'll engage in work that improves the world and furthers your career with every project you touch.

Work You’ll Do

At Caterpillar, Inc., you will have the opportunity to contribute to a variety of groundbreaking projects that extend across the globe. Our commitment to leadership in the industry is unwavering, and we achieve it through a culture that fosters diversity and leadership development.

Explore Job Opportunities and Growth

Caterpillar offers a wide range of job opportunities and career paths, providing a platform for professional growth and development. Whether you're interested in engineering, sales, marketing, or management, Caterpillar has a position that will utilize your unique skills and talents. We are continuously hiring innovative minds to propel our mission forward and maintain our status as industry leaders.

Internship Programs

Kickstart your career with a Caterpillar internship. Our internships provide invaluable workplace experience and networking opportunities that often lead to full-time employment at our company. As an intern, you will work alongside experienced professionals and gain insights that will benefit you as you advance in your career.

Benefits and Culture

Choosing a career at Caterpillar means more than just having a job. It means you're part of a team that values diversity and is committed to creating an inclusive environment. We offer competitive benefits to ensure the health and well-being of our employees and their families. These benefits include health insurance, retirement plans, and paid time off, alongside unique perks like professional development and diversity training programs.

Innovation and Leadership

Drive innovation and industry leadership as part of the Caterpillar team. We empower our employees to lead from their first day on the job, encouraging continuous improvement and career advancement. At Caterpillar, your ideas will shape the future of our business and the industries we serve.

Join Our Team

Search open positions that match your skills and interests. We are looking for passionate, curious, and solution-driven team players. Ready to start your journey at Caterpillar, Inc.? Explore our career portal for the latest job opportunities.

Stay Connected

Keep up to date with company news, career tips, and industry insights—all from the people who work here. Subscribe to our careers blog and personalize your subscription to receive updates that align with your career interests.

Prepare for Your Interview

Make a great first impression. Prepare for your interview at Caterpillar by reviewing common questions and practicing your responses. Tailor your resume to highlight the experiences and skills that make you the ideal candidate for the position you are applying for. At Caterpillar, Inc., every role offers the chance to make an impact. Come be a part of our team and drive the future of our global industries. Your career at Caterpillar is not just about what you'll accomplish, but how you'll grow and the innovation you'll lead. Join us and build a career that matters.
Learn more about Caterpillar, Inc
Size
107,700 employees
Market Cap
$125.3 billion
Industry
Net Income
$2.9 billion
Founded
1925
5 Year Trend
+5.8%
Revenue
$41.7 billion
NASDAQ

Similar Jobs

More Jobs at Caterpillar, Inc

More Information Technology Jobs

Find similar Manager; AI Evaluation Engineering jobs: