About the RoleTake2 AI is hiring a
Forward Deployed Engineer to design, launch, and continuously improve our AI Interviewers for customers. Our AI agents already conduct tens of thousands of structured candidate interviews each month.
This role sits at the intersection of
voice/conversational agents, prompt + flow design, evaluation/scoring rubrics, and production iteration. You'll work directly with customers to understand screening requirements, translate them into structured interviewer behavior, deploy agents into production, and improve performance based on real-world feedback and metrics. You'll also be at the heart of our product motion - we build our roadmap based on our customers and our FDEs sit squarely at the center of that loop.
This is a hands-on, highly analytical role for someone who enjoys turning ambiguous requirements into precise agent behavior, building rigorous evaluation approaches, and shipping improvements quickly in a startup environment.
What You'll DoCustomer Onboarding & Requirements (Customer-Facing)- Lead technical onboarding with customers to understand roles, hiring goals, must-have signals, and constraints.
- Translate customer needs into structured interview flows, role-specific question banks, and scoring rubrics.
- Set clear expectations on what "good" looks like (pass/fail thresholds, evaluation rationale, interviewer tone and style).
Voice Agent Conversation Design (Prompts + Flows)- Design, build, and refine prompts and agent logic that drive interviewer behavior, question sequencing, probing, and candidate experience.
- Ensure interviewer conversations are consistent, role-relevant, and robust to edge cases (evasive candidates, unclear answers, noisy audio, interruptions).
- Implement multi-step structured interview flows with state management and guardrails.
Evaluation & Scoring Systems- Design and maintain AI-based evaluation and scoring aligned to customer rubrics and hiring criteria.
- Improve accuracy, consistency, and explainability of scoring at scale (including calibration across roles/customers).
- Identify bias/fairness risks and contribute to mitigation strategies and compliant evaluation practices.
Deployment, Iteration, and Customer Feedback Loops- Launch new customer interviewers into production and own iteration cycles from early rollout through steady-state performance.
- Use customer feedback + production metrics to prioritize improvements and deliver measurable outcomes.
- Communicate changes clearly to customers and internal stakeholders.
Quality, Reliability, and Scale- Build and own lightweight QA/evaluation pipelines to measure conversation quality, scoring accuracy, and reliability before/after changes.
- Monitor production performance and partner with engineering to balance quality, latency, and cost tradeoffs.
- Contribute to standards and best practices for prompt quality, eval quality, and voice-agent reliability.
In Terms of ExperienceRequired:- Minimum 2 (preferred 4-6) years of experience working in an engineering role
- 1+ (preferred 2+) years working with LLMs, NLP systems, or AI agents in production.
- Strong full-stack software engineering experience
- Demonstrated experience designing and deploying agent workflows (prompts + structured flows) that operate at scale.
- Strong understanding of prompt engineering, agent control, failure modes, and conversational edge cases.
- Experience building or contributing to evaluation/testing/QA frameworks for AI systems.
- Comfort being customer-facing: running technical discovery, translating requirements, and driving onboarding to production.
- Strong analytical mindset (accuracy, consistency, bias, calibration, and edge cases).
Preferred:- 2+ years working as a Forward Deployed Engineer at a B2B AI tech company.
- Familiarity with voice/conversational AI systems, especially real-time or high-volume environments.
- Strong Python skills (APIs, data pipelines, eval harnesses, testing frameworks).
- Hands-on experience with multiple LLMs (GPT, Claude, Gemini, LLaMA/Mistral, fine-tuned models).
- Experience designing multi-step agents with state management and structured outputs.
- Experience operating AI systems in production and iterating based on real-world performance metrics.
- Prior startup experience (high ownership, fast iteration, ambiguity).
- Bachelor's degree in CS/Engineering/Math or related technical field - or equivalent practical experience.
Location & FlexibilityWe're NYC-based and work hybrid (in-office Mon-Thu). We value in-person collaboration but also trust people to manage their time responsibly.
Compensation & GrowthCompetitive salary + meaningful equity. This is a chance to join at a stage where your work meaningfully shapes the product and your career trajectory.
