Description The role You will own the evaluation of the model from raw data to verdict. You will build the dataset from 14 years of content and analytics (nearly 100,000 published articles on Atlanta Black Star alone, each with Google Analytics history), have AI models score the full archive, validate those scores against trained human raters on a sample, test whether those scores predict real audience behavior, and run live experiments on our properties. Your findings decide what we build and where we invest. We want an honest answer, including if the answer is no. If the model holds up, you will turn the evaluation pipeline into the core of the product: the scoring engine, the generator that proposes signals and plans, and the experiment and learning loop.
This is a remote contract role starting immediately. You will report to Neil Nelson, CEO, who owns the model, and work closely with our editorial team, whose producers serve as raters.
What you will do - Build the evaluation dataset: export story text and metadata from our WordPress sites, pull Google Analytics 4 data by traffic source, and recover older data from other sources where needed.
- Design a balanced, representative sample, including stories that failed, not only the hits.
- Run the scoring: set up blind rating, train and calibrate human raters with the editorial lead, measure how well raters agree, and write and validate the prompts for AI raters against human judgment.
- Test the model: fit weights on one part of the data and validate them on data held back; compare alternative ways of combining scores; control for distribution effects such as platform algorithm changes and traffic source.
- Test early-warning indicators on historical audience data.
- Design, size and analyze live A/B tests on our properties.
- Keep a clean, reproducible record of every dataset, decision, score, prediction and result.
- Report findings plainly to the CEO, against success criteria agreed before the analysis begins.
Position Requirements Required qualifications - 5 or more years in applied machine learning or data science, with at least one project you owned end to end, from raw data to a decision someone acted on.
- Experimentation: designed, sized and analyzed A/B tests or randomized experiments, including power calculations and pitfalls such as peeking and multiple comparisons.
- Statistics for prediction and validation: regression and classification, out-of-sample and time-based validation, and measures of rater agreement such as intraclass correlation or Krippendorff's alpha.
- Data engineering for analysis: fluent SQL; experience with BigQuery or a similar warehouse; Python (pandas, statsmodels or scikit-learn).
- Web analytics: hands-on work with Google Analytics 4, including traffic-source attribution and its limits.
- Language models as measurement tools: prompted large language models through an API to classify or score text at scale (ideally 100,000 documents or more), managed the cost, and checked the output against human judgment.
- Production systems: built and shipped at least one ML or language-model system that other people relied on, including evaluation, monitoring and cost control.
- Judgment and candor: you can explain a result to a non-technical executive in plain language, and you report findings that contradict expectations.
Preferred - Experience in digital media, publishing, or growth analytics at a consumer company.
- Systems trained on human ratings, such as ranking models or reward models, including pairwise-comparison data.
- Causal inference beyond A/B tests: difference-in-differences, uplift modeling, or similar.
- Natural language processing on large text archives.
- Time-series analysis, especially detecting regime changes or early-warning signals.
- Background in behavioral science, psychometrics or survey research, including designing rating scales.
- Experience working with editorial or creative teams.
- A graduate degree in statistics, economics, computer science, psychology or a related quantitative field, or equivalent experience.
Full-Time/Part-Time Full-Time
SalaryPosition Applied ML Scientist, Audience Behavior Platform
Division Atlanta Black Star
Number of Openings 1
Exempt/Non-Exempt Non-Exempt
Hiring Manager(s)Req Number DAT-26-00001
Open Date 11/11/2026
Location DDM GROUP LLC
