American International Group

AI Observability Engineer

American International Group • $100K — $130K *
Atlanta, GA 30349In-Person
Information Technology
Less than 5 years of experience
Reposted 4 days ago
Be an Early Applicant
Job Overview by Ladders

Qualifications

  • 3+ years of experience in roles related to AI and machine learning, with a Bachelor's degree
  • Strong knowledge of machine learning models, algorithms, and evaluation metrics
  • Proficient with data pipelines, capable of data cleaning and analysis
  • Familiarity with scaling human oversight in AI systems, especially with LLMs
  • Demonstrated experience in Responsible AI or AI ethics
  • Capable of collaborating effectively across diverse teams
  • Understanding of AI governance policies and practical application in regulated industries
  • Proven problem-solving skills and ability to influence AI initiatives

Responsibilities

  • Monitor machine learning and AI model performance against critical KPIs
  • Detect anomalies and biases in data inputs affecting AI fairness and accuracy
  • Utilize observability tools for generating insights from AI system data
  • Oversee operation of the AI model management platform and enhance observability
  • Review AI solutions to ensure compliance with organizational and ethical frameworks
  • Support AI product teams by conducting reviews and workshops
  • Collaborate with legal and security teams on Responsible AI issues
  • Prepare executive reports and strategic recommendations on Responsible AI

Benefits

  • Comprehensive Total Rewards Program focusing on health and wellbeing
  • Professional development opportunities to support career growth
  • Support for financial security and peace of mind
  • Collaboration-focused work environment emphasizing in-person teamwork
Full Job Description
How you will create impact

The AI Observability Engineer plays a critical role in interaction with our AI product teams to ensure the AI solutions are aligned with the enterprise Responsible AI principles and standards. Additionally, the AI Observability Analyst supports the design and execution of the overall AI Governance program as well as the communication and awareness initiatives associated with Responsible AI principles and methods through the following activities:
  • Model Performance Monitoring: Tracking ML,LLM, Agentic AI key performance indicators (KPIs) like context relevance, accuracy, precision, recall, and F1 scores to ensure model effectiveness
  • Data Quality and Bias Detection: Monitoring data inputs for anomalies, bias, and drift, which can impact the fairness and accuracy of AI systems
  • Utilize observability tools to produce structured, trustworthy insights from detailed data sets on AI system activities.
  • Manage on-going operations of our AI model management platform and pipelines including evolving our KPIs and model observability capabilities
  • Review AI solutions across the business to ensure alignment with AIG's Organizational Policies, AI Governance Framework and Responsible AI principles
  • Provide support to AI product teams, conduct independent reviews, responding to ad-hoc queries, and participating in workshops
  • Collaborate with legal, compliance, security, and other teams to coordinate the review of Responsible AI issues, ensuring a cohesive approach across the organization
  • Create executive-level reports on Responsible AI and provide strategic recommendations to senior leadership
  • Create and update RAI training content for technical and non-technical audiences at all levels of seniority
  • Contribute to the development and improvement of RAI assets: methods, technical tutorials, tools
  • Monitor and remain up to date on emerging global AI regulatory frameworks, standards and policies by participating in workshops, reading publications, networking with the goal of helping to translate into RAI framework updates


What You'll Bring

Required Skills/Experience
  • Minimum 3 years' experience and a Bachelor's degree is required.
  • Machine Learning, Large Language Models knowledge of various models, algorithms, and evaluation metrics is crucial for effective monitoring.
  • Data pipelines and analysis skills with the ability to handle, clean, and transform data for monitoring and analysis.
  • Excited about the problem of scaling human oversight of AI systems and familiar with LLMs development
  • Comfort with ambiguity and curiosity as our team is scaling with strong ownership
  • You bring experience and passion in Responsible AI or AI ethics, with an understanding of AI technologies and frameworks such as Generative AI
  • You have a strong ability to collaborate with cross-functional teams and drive the implementation of Responsible AI practices, ideally within financial services or other regulated industries
  • You have a working knowledge of effective policies and processes around the usage of emerging AI tools and products, bringing best practices from the industry
  • You have a proven track record of solving complex problems, developing mitigation strategies, and influencing others on key AI initiatives
  • You are passionate about staying ahead of emerging global AI regulations and translating these frameworks into practical, actionable updates


Nice to have
  • Prior experience building production LLM, RAG or model inference monitoring pipelines
  • Instrumenting applications using OpenTelemetry distributed tracing or structured logs.
  • Hands on experience with AI Observability tools such as Fiddler, Arize, LangSmith or similar tools.
  • Ability to build dashboards and alerts tied to AI model behavior beyond system uptime.


#LI-PS1

At AIG, we value in-person collaboration as a vital part of our culture, which is why we ask our team members to be primarily in the office. This approach helps us work together effectively and create a supportive, connected environment for our team and clients alike.

Enjoy benefits that take care of what matters

At AIG, our people are our greatest asset. We know how important it is to protect and invest in what's most important to you. That is why we created our Total Rewards Program, a comprehensive benefits package that extends beyond time spent at work to offer benefits focused on your health, wellbeing and financial security-as well as your professional development-to bring peace of mind to you and your family.

Functional Area:

DT - Data

AIG PC Global Services, Inc.

About American International Group

American International Group provides property insurance, life insurance, and financial services.

American International Group Careers

Join the dynamic team at American International Group (AIG), a leading global insurance organization, and propel your career into a future of opportunities and professional growth. At AIG, we are committed to fostering a culture of innovation, leadership, and diversity that is unmatched in the industry.

Work You’ll Do

At American International Group, you will collaborate with some of the brightest minds in the industry, dedicated to solving complex challenges and driving transformative solutions in the world of insurance. Our team is at the forefront of developing innovative insurance products that not only meet the needs of our clients worldwide but also anticipate future trends.

Explore Job Opportunities and Internships

Whether you're a seasoned professional looking for your next challenge or a recent graduate eager to start your career, AIG offers a range of job opportunities and internships across various functions. From underwriting and claims to finance and technology, your next position awaits you at AIG. Our internships provide invaluable hands-on experience, helping you to enhance your skills and make meaningful contributions from day one.

Grow and Develop Your Career

AIG believes in the power of continuous learning and offers extensive training and development programs to help you hone your skills and advance your career. Leadership development, diversity training, and innovation workshops are just a few examples of how we invest in our employees' growth and leadership capabilities.

Benefits and Culture

Joining AIG means more than just having a job; it means being part of a team that values diversity and is committed to creating an inclusive environment where everyone can thrive. We offer competitive benefits that support the health, well-being, and financial security of our employees and their families. At AIG, you will find a supportive network and a culture that applauds innovation and rewards performance.

Networking and Professional Development

We encourage our employees to build professional relationships both within and outside the company through networking events, professional groups, and career fairs. These opportunities not only enhance your career but also expand your professional network, which is vital in today’s dynamic job market.

Join Our Team

Search open positions that match your skills and interests on our Careers page. We are looking for passionate, curious, and solution-driven team players who are ready to take their career to the next level. Explore the exciting and rewarding career opportunities available at American International Group today.

Stay Connected

Keep up to date with the latest industry trends, career tips, and company news by subscribing to our careers blog. Personalize your experience by signing up for job alert emails tailored to your preferences and be the first to know about new openings and exclusive insights from professionals within AIG. At American International Group, your career is just the beginning – it’s a pathway to developing your skills, meeting new challenges, and making a difference in a global company known for its commitment to innovation and excellence. Join us and shape your future with AIG.
Learn more about American International Group
Size
36,600 employees
Market Cap
$46.4 billion
Industry
Finance & Insurance
Net Income
-$5.9 billion
Founded
1919
5 Year Trend
-0.3%
Revenue
$43.7 billion
NASDAQ
AIG
* Ladders Estimates

Similar Jobs

Ad banner background

Get Ready For Your
Next Interview

More Jobs at American International Group

More Information Technology Jobs

Find similar AI Observability Engineer jobs:

NationwideAtlanta, GA