Overview
The AI Data Scientist II in AI, Data Science and Machine Learning will execute defined data science, experimentation, and model evaluation work for systems that interpret and use aerial imagery. Working with AI and data science, engineering, and product partners, this role will develop and apply metrics, test methods, analysis workflows, and evaluation tools that support product objectives. The role will solve technical problems limited in scope, make evidence-based judgments and proposals, and complete task-level deliverables with minimal coaching.
We are a fast-paced, collaborative team driven by continuous improvement. We are looking for a motivated and organized emerging professional who learns new technologies and development lifecycle practices, communicates effectively across team boundaries, and documents work so that others can understand and build on it.
This is a full-time, remote role, with a base salary of $98,000 - $134,000, bonus eligible. Responsibilities
Model and Agentic System Evaluation- Execute benchmarking tasks for machine learning and agentic AI systems using defined measures such as task success, accuracy, robustness, and common failure patterns.
- Apply and help maintain established evaluation methods for computer vision and machine learning models, including detection, segmentation, and classification models.
- Develop and maintain quantitative metrics, test harness components, and curated evaluation datasets for realistic, production-representative scenarios.
- Analyze model outputs, errors, and test results to identify performance issues and propose practical improvements.
- Conduct defined comparisons across model architectures, prompt strategies, tools, and inference configurations, then summarize results and recommendations.
Evaluation and Analysis- Execute defined offline evaluations, including model comparisons, stress tests, and edge-case testing.
- Perform structured error analysis and root-cause investigation across data, model, and system failures, escalating broader issues when appropriate.
- Document code, data, configurations, and evaluation results so that work can be repeated, reviewed, and improved by others.
- Compare model accuracy, reliability, processing time, and resource use, and communicate findings clearly to stakeholders.
Cross-Functional Collaboration and Communication- Work with AI engineers, product managers, and research partners to translate evaluation results into clear, actionable recommendations.
- Provide evidence-based findings and proposals that support model selection, system design, and product objectives.
- Participate in model reviews by preparing analysis, documenting limitations, and raising identified risks.
- Regularly communicate work status, raise issues with potential solutions, and document methods, results, and new concepts for technical and non-technical audiences within and beyond the team.
Tooling and Evaluation Support- Develop and maintain repeatable scripts, notebooks, and test components for evaluating models and AI applications.
- Track agreed performance measures and identify regressions or unexpected results using established team standards.
- Support internal tools for evaluating models, visualizing metrics, and documenting test results.
- Work with engineering partners to incorporate evaluation results into development and testing workflows.
- Learn and apply current practices in agent evaluation, computer vision benchmarking, and machine learning performance measurement.
- Suggest improvements to metrics, benchmarks, and evaluation protocols as system capabilities and product needs evolve.
- Follow team practices for responsible AI, data quality, and appropriate documentation of model limitations.
- Other duties as assigned.
Qualifications
- Bachelor's degree or equivalent practical experience. Coursework, certification, or applied training in data science, machine learning, statistics, computer science, or a related area is preferred.
- Two to four years of relevant experience in data science, machine learning, model evaluation, or a related technical field.
- Hands-on experience evaluating machine learning models using appropriate quantitative metrics and clearly documenting results.
- Working knowledge of computer vision model evaluation, such as segmentation, detection, or classification metrics.
- Exposure to LLM-based, multimodal, or agentic systems and multi-step evaluation workflows.
- Proficiency in Python and commonly used machine learning and data science libraries, such as NumPy, Pandas, or scikit-learn.
- Experience using Git or another version control system to support collaborative technical work.
- Experience working in at least one cloud-based environment, such as AWS or GCP.
Preferred- Experience evaluating image classification, document classification, LLM, or RAG applications.
- Experience creating reusable evaluation scripts, notebooks, dashboards, or quality assurance tools.
- Experience working in a collaborative software or machine learning development lifecycle.
Core CompetenciesThe successful candidate will demonstrate strength in the following competencies as well as foundational competencies which can be found here:
- Drive & Follow Through - Takes initiative and turns ideas into action.
- Adaptability in Uncertainty - Adjusts quickly and performs through change.
- Customer-Centered Mindset - Puts customer needs at the center of decisions.
- Judgment & Problem Solving - Makes sound decisions and solves problems effectively.
- Role-Specific Expertise - Applies strong functional expertise to deliver results.
- Results Accountability - Owns outcomes and delivers on commitments.
- Work Prioritization & Execution - Focuses on priorities and delivers on time.
