OverviewDecisionPoint seeks an AI/ML & MLOps Engineer to lead the architecture, deployment, monitoring, automation, and governance of production machine learning capabilities supporting the USMEPCOM ENLIST program.
This SME-level position will maintain training, validation, and testing datasets; design and operate production ML architectures; automate model deployment; monitor model performance and drift; and control updates to models, datasets, pipelines, and supporting configurations. The engineer will establish reproducible, traceable, and secure processes for moving machine learning capabilities from development through validation and into production.
The AI/ML & MLOps Engineer will work closely with mission owners, data scientists, data engineers, cloud architects, software developers, cybersecurity personnel, testers, and operations teams. The successful candidate must combine advanced machine learning engineering knowledge with production automation, secure cloud architecture, data governance, and operational monitoring expertise.
This position is remote, with Chicago-area candidates preferred.
Note: By applying to this position, you acknowledge and consent to having your resume included in an active competitive government contract bid.
Duties & Responsibilities
The AI/ML & MLOps Engineer will:
- Lead the design and implementation of secure, scalable, and supportable production machine learning architectures.
- Develop, maintain, and govern training, validation, and testing datasets used for model development and evaluation.
- Establish repeatable processes for dataset ingestion, preparation, transformation, labeling, validation, versioning, and approval.
- Ensure datasets are maintained in or transformed into analysis-ready and model-ready formats.
- Implement controls for dataset quality, completeness, integrity, provenance, lineage, security, and authorized use.
- Design and automate end-to-end ML pipelines covering data preparation, training, evaluation, packaging, deployment, monitoring, retraining, and retirement.
- Deploy machine learning models into approved cloud, application, analytical, and operational environments.
- Establish automated model build, test, validation, promotion, deployment, rollback, and recovery procedures.
- Implement model registries and controlled repositories for model artifacts, dataset versions, code, configurations, dependencies, evaluation results, and approvals.
- Develop automated tests for model pipelines, data transformations, interfaces, deployment packages, and production behavior.
- Define model evaluation frameworks, performance thresholds, validation criteria, and release-readiness requirements.
- Evaluate model accuracy, reliability, robustness, performance, scalability, security, and alignment with documented mission requirements.
- Implement production monitoring for model performance, data drift, concept drift, inference quality, latency, availability, errors, and resource utilization.
- Establish thresholds, alerts, escalation paths, and response procedures for model degradation, drift, data-quality issues, and operational failures.
- Investigate model and pipeline anomalies and coordinate corrective actions with data, application, infrastructure, security, and mission teams.
- Control updates to production models, datasets, features, pipelines, configurations, and dependencies through documented review and approval processes.
- Maintain traceability among source datasets, data transformations, model versions, evaluation results, deployment records, and production outcomes.
- Develop rollback and recovery procedures for models, datasets, pipelines, and supporting services.
- Support secure data pipelines used for AI development, model evaluation, deployment, monitoring, and operational decision support.
- Implement technical controls supporting model and data governance, including version control, separation of environments, approval gates, auditability, and change management.
- Maintain audit-ready documentation covering model purpose, training data, evaluation methods, limitations, dependencies, approvals, changes, performance, and known risks.
- Support the evaluation and auditing of model outputs to identify inaccurate, unsupported, or unexpected results.
- Help establish human-review checkpoints and escalation procedures for model outputs that could materially affect mission or operational decisions.
- Support compliance with applicable Government data-handling, cloud-security, cybersecurity, and operational-risk requirements.
- Integrate model development and deployment workflows with continuous integration, continuous delivery, automated testing, infrastructure-as-code, monitoring, and incident-management capabilities.
- Partner with cybersecurity teams to address vulnerabilities, access controls, secrets management, software dependencies, logging, encryption, and authorization requirements.
- Coordinate with data warehouse and data engineering teams on data models, metadata, Extract-Transform-Load processes, data quality, and governed access.
- Provide technical leadership, architecture guidance, code and design reviews, troubleshooting support, and knowledge transfer.
- Prepare architecture diagrams, model lifecycle documentation, deployment procedures, monitoring plans, test results, operating guides, and technical reports.
- Communicate model performance, drift, risks, incidents, corrective actions, and deployment status to technical teams, Government stakeholders, and program leadership.
Qualifications
Required Qualifications
Candidates must meet all the following requirements:
- Active Secret clearance or higher.
- Bachelors degree in computer science, data science, mathematics, engineering, or a related field, or a permitted equivalent.
- Minimum of 12 years of relevant professional experience.
- Experience leading the architecture of production machine learning systems.
- Experience designing and operating production ML or MLOps environments.
- Experience developing and maintaining training, validation, and testing datasets.
- Experience establishing model-evaluation methods, performance measures, validation criteria, and acceptance thresholds.
- Experience deploying machine learning models into production environments.
- Experience automating model build, test, deployment, promotion, rollback, or retraining processes.
- Experience implementing model and data drift monitoring.
- Experience with model governance and data governance.
- Experience controlling model, dataset, pipeline, and configuration updates.
- Experience with cloud-based machine learning platforms or production cloud architectures.
- Experience implementing model, dataset, code, artifact, and configuration versioning.
- Experience building or integrating automated testing, continuous integration, and continuous delivery capabilities for machine learning systems.
- Experience monitoring production model performance, reliability, and operational health.
- Understanding of secure data pipelines, access controls, audit logging, encryption, secrets management, and software supply-chain security.
- Ability to establish traceability across datasets, transformations, model versions, evaluation results, deployments, and operational outcomes.
- Ability to document and communicate model limitations, performance, drift, risks, incidents, and corrective actions.
- CompTIA Security+ CE or another approved IAT Level II certification.
Preferred Qualifications
The following qualifications are preferred but are not required:
- AWS Certified Machine Learning Engineer - Associate or a comparable machine learning platform certification.
- Advanced AWS, Microsoft Azure, Google Cloud, Databricks, or comparable cloud architecture or data engineering certification.
- Experience supporting federal or Department of Defense programs.
- Experience operating machine learning capabilities in regulated, controlled, or mission-critical environments.
- Experience with MLflow, Amazon SageMaker, Azure Machine Learning, Google Vertex AI, Databricks, Kubeflow, or comparable platforms.
- Experience with containerization, Kubernetes, infrastructure as code, and cloud-native deployment patterns.
- Experience implementing feature stores, model registries, metadata repositories, experiment tracking, or data-lineage tools.
- Experience with observability platforms, automated alerting, incident response, and production reliability engineering.
- Familiarity with responsible AI, explainability, output validation, human oversight, bias evaluation, and model-risk management practices.
- Familiarity with DoD Impact Level 4 or comparable Government cloud and data-protection requirements.
- Experience with analytical models, generative AI, retrieval workflows, or agentic AI systems.
- Experience leading technical teams, architecture reviews, or cross-functional ML delivery initiatives.
Skills
- Production machine learning architecture.
- MLOps engineering and lifecycle automation.
- Model deployment and serving.
- Training, validation, and testing dataset management.
- Model evaluation and performance measurement.
- Model and data drift monitoring.
- Model, data, and configuration governance.
- Automated testing and deployment pipelines.
- Cloud ML platforms and secure data pipelines.
- Version control, traceability, and audit documentation.
- Production monitoring and incident troubleshooting.
- Cybersecurity and access-control implementation.
- Technical leadership and architecture communication.
- Cross-functional collaboration with mission, data, software, cloud, security, testing, and operations teams.
