Programmatic QA • Testing for LLMs & Agents • Data Quality • Platform Reliability
What You'll Do1. Programmatic QA of Core Features- Extend our scenario test runner - a proprietary harness that captures real production personalization requests and replays them across environments, asserting on expected algorithms and content selection. Grow it into automated regression across every client.
- Write automated tests in Python with pytest across our tiers - unit, integration, HTTP, and end-to-end.
- Build headless Playwright end-to-end tests to verify how personalized content and tracking render on real client pages.
- Harden the pre-deploy quality gate and pre-commit checks that block bad changes automatically.
2. Testing & Standardizing LLMs and Agents- Design evals for non-deterministic AI features - our conversational analytics assistant, AI content builders, and generative SEO - measuring correctness, grounding, and regression across prompt and model versions.
- Test the tool-calling and agentic layers - that function-calling loops pick the right tools and guardrails hold on adversarial input.
- Validate our agent/MCP interface - contract conformance, rate limiting, authorization, and safe failure.
- Help set our standards for shipping AI - catching hallucinations and drift, and benchmarking prompt/model changes before clients see them.
3. Data Quality Engineering- Build automated data-health checks that flag stale rollups, incomplete coverage, and broken aggregations before they hit a client dashboard.
- Validate data pipelines end-to-end - rollups, funnel/rate/financial ingestion, and BigQuery - with drift detection across environments.
- Guard model inputs so the signals our ML depends on stay accurate and complete.
4. Reliability & Performance- Track platform performance - response times, JS load, and page speed - and help keep it fast.
- Stand up quality dashboards - uptime, coverage, data-health, and eval scores.
5. Collaboration & Bug Lifecycle- Work in the codebase alongside engineers to diagnose issues across development, release, and deployment.
- Drive the bug lifecycle - reproduce, capture with a failing test, and verify the fix.
What We're Looking For- 3+ years in QA/SDET or test automation with a code-first approach.
- Strong Python - you write clean test code and can read the app you're testing.
- pytest (preferred) and browser automation (Playwright or Selenium).
- API and contract testing experience.
- A genuine interest in testing AI - comfortable with non-determinism, evals, and prompts.
- Data-savvy - strong SQL, and the instinct to validate pipelines and reconcile data.
- Building automated quality gates into the deploy and release process.
Nice to Have- Testing or evaluating LLM applications - evals, prompt regression, tool-calling agents, or MCP.
- Data or analytics QA - BigQuery or ETL/rollup validation.
- Django, MySQL, or Celery experience.
- Security testing with SAST/DAST tooling.
- Familiarity with machine learning.
- Financial industry, personalization, or CMS/marketing-platform experience.
- Familiarity with AWS.
- SaaS startup experience on a fast-moving, multi-tenant platform.
