Sr. GenAI Engineer (3-6 month contract)Department: Engineering
Employment Type: Full Time
Location: US
Reporting To: Pete Martin
Compensation: $129.00 - $150.00 / hour
DescriptionCare Lumen | Remote (US Only)
Employment Type:
Contractor (3-6 months, with possibility of extension)Salary Range:
$129 to $150 an hourScopeThe Senior Generative AI Engineer is an individual contributor responsible for testing, evaluating, optimizing, and monitoring enterprise Generative AI systems. The primary focus is improving the quality, reliability, and performance of existing GenAI solutions through systematic testing, evaluation of new model versions, prompt optimization, and application architecture improvements. The role will also help drive evaluation methodology, monitoring capabilities, and AI governance requirements for selected initiatives. Approximately 25% of the role is expected to support AI governance activities, including model documentation, versioning, registry management, experiment tracking, auditability, and related controls.
The role collaborates with engineering, data science, architecture, security, compliance, governance, and business stakeholders. It does not have direct reports or a formal leadership mandate.
What You'll Do- Test and evaluate GenAI applications, AI agents, and LLM-powered workflows for quality, reliability, accuracy, safety, and task effectiveness.
- Develop and execute repeatable GenAI evaluation methodologies, test datasets, benchmark suites, regression tests, and automated evaluation pipelines.
- Use DeepEval and related evaluation approaches to measure GenAI output quality and identify regressions across prompts, workflows, application changes, and model versions.
- Optimize GenAI solution architecture by decomposing complex prompts and workflows into appropriate subtasks, tools, and specialized agents.
- Build and improve multi-agent workflows using orchestration frameworks such as LangGraph and LangChain.
- Optimize prompts, system instructions, context management, agent interactions, and model selection / configurations using evaluation results (including latency and model performance) and production feedback (including user feedback).
- Evaluate new foundation models and model versions to determine quality, behavior, risks, performance, (groundedness, completeness, bias, etc) and suitability for specific use cases.
- Design and implement monitoring capabilities for GenAI systems, including quality metrics, operational telemetry, audit trails, alerts, and performance reporting.
- Help drive AI governance activities for selected initiatives, including model and application documentation, risk assessments, evaluation evidence, model versioning, registry management, experiment tracking, and change traceability.
- Collaborate across technical, governance, security, compliance, and business teams to improve GenAI solutions and support responsible production use.
Competencies- GenAI testing and quality evaluation
- GenAI evaluation methodology and automated evaluation
- Prompt, context, and application architecture optimization
- Multi-agent architecture and orchestration
- Foundation model and model-version testing
- AI governance and model lifecycle management
- GenAI monitoring, observability, and traceability
- Traditional machine learning testing and validation
- Analytical problem-solving and cross-functional collaboration
What's Required- 10+ years of overall professional experience in data science, machine learning, AI engineering, or a related technical field.
- 2+ years of hands-on experience developing, testing, evaluating, or optimizing Generative AI systems.
- Demonstrated experience testing GenAI applications and establishing systematic methods for assessing LLM output quality.
- Hands-on experience with DeepEval, LangGraph, LangChain, Python, and modern AI and machine learning libraries.
- Experience designing or optimizing multi-agent systems, agent orchestration patterns, or complex LLM workflows.
- Strong hands-on experience with Databricks for data, AI, or machine learning workloads, including model versioning, model registries, and experiment tracking.
- Practical experience supporting AI governance, model lifecycle controls, documentation, traceability, and auditability.
- Prior experience on traditional machine learning projects as an ML engineer, data scientist, or similar contributor, with a strong understanding of model testing and validation methods.
- Ability to communicate technical findings, evaluation results, risks, and recommendations to technical and nontechnical stakeholders.
- Ability to maintain at least four hours of overlap with U.S. Eastern Time (EST/EDT) each workday.
- Bachelor's or advanced degree in computer science, data science, engineering, mathematics, statistics, physics, computational science, or another scientific, quantitative, or computational discipline.
What's Preferred- Experience with Databricks Mosaic AI.
- Experience working with PHI / PII data inside highly regulated domain like healthcare.
- Experience testing or optimizing multi-agent GenAI systems in production or production-like environments.
- Experience working with embedding models, semantic retrieval, vector search, or Retrieval-Augmented Generation architectures.
- Experience applying AI governance or Responsible AI requirements in a regulated enterprise environment.
Compensation & BenefitsThe expected total cash compensation for this role is market-driven, with an expected compensation rate of
$129 to $150 per hour. Hourly rate is ultimately determined based on a candidate's skills, expertise, and experience.
Offers are contingent upon successful completion of a background screening. Background screenings will be conducted in compliance with applicable laws and will not be initiated without your consent.
Ready to Apply? 
