We’re looking for a versatile Computational Linguist to join our team focused on evaluating and supporting Generative and Agentic AI systems. This role combines linguistic expertise, data analysis, and hands-on experimentation with large language models.  
This role is ideal for someone who can move between qualitative language analysis and quantitative evaluation. You’ll work cross-functionally with Machine Learning Engineers, Analytics team and annotators to design innovative, rigorous, and scalable evaluation processes for LLM-powered workflows.   

Core Responsibilities 
•    Serve as a subject matter expert, engaging with various stakeholders throughout the product lifecycle, and maintain a strong understanding of the Chase Digital Assistant’s model from both customer and technical perspectives. 
•    Manage, monitor, and evaluate and version the Chase Digital Assistant’s intent and entity taxonomy and the model training; Enforce taxonomy versioning practices to ensure traceability and rollback capability 
•    Work closely on the adaptation to LLM-driven workflows, ensuring seamless integration of LLMs with existing conversational AI architectures and event tracking systems. 
•    Implement and evolve metrics and KPIs across Model Correctness, Customer Experience, AI Assurance, and Business Metrics and ensure evaluation is transparent, repeatable, and release-decision-ready 
•    Maintain established metrics and introduce new guardrail metrics for LLM and generative use cases 
•    Manage full artifact suite for LLM models including  descriptions, prompts, evaluation rubrics, LLM-as-judge prompts, guidelines, calibration data, data statistics & reliability measures.  
•    Apply ontology design principles to improve semantic reasoning and data integration aligned to business standards. 
•    Design frameworks to incorporate knowledge graphs within classification and extraction model architectures. 
•    Align knowledge models with RAG pipelines and agent orchestration to enhance AI functionality. 
•    Work with data scientists, software engineers, and business stakeholders to translate requirements into robust solutions. 
•    Identifying optimization opportunities across teams, supporting continuous improvement across model performance, data quality, and feature coverage for improving customer experience. 

Required Qualifications, Capabilities, and Skills 
•    Master's degree in Computational Linguistics, NLP, Linguistics, or related field 
•    2+ experience  in Computational Linguistics or NLP applied to chatbot or conversational AI development 
•    Hands-on experience with Generative AI and Agentic AI frameworks and evaluation (e.g., AutoGen, LangGraph, CrewAI, Sierra) 
•    Linguistic background in discourse & pragmatics 
•    Advanced knowledge of conversational AI product development lifecycle — training, design, conversation analysis 
•    Hands-on experience with LLM integration, prompt engineering, evaluation, and performance monitoring 
•    Proficient in Python, Git, Linux, and Bash scripting 
•    Fluent with NLP/data science libraries: pandas, numpy, scikit-learn, NLTK 
•    Experience with transformer-based models (e.g., BERT, GPT) — fine-tuning and application 
•    Experience with generative AI SDKs and frameworks (e.g., OpenAI, Google, Anthropic, LangChain) 
•    Comfortable with JSONL, CSV, and Jupyter notebook workflows 
•    Experience with ontologies/taxonomies and knowledge graphs 
•    Solid understanding of evaluation methodologies including human-AI comparison and red teaming 
•    Direct experience with financial institutions, financial products, and customer-facing queries 
•    Chase customer service experience highly desirable 
•    Strong written communication for documenting experiments, results, and processes

Preferred Qualifications 
•    Experience with hybrid conversational architectures, generative AI, and LLM-driven flow design 
•    Familiarity with LLM safety, bias, and compliance 
•    Demonstrated success in a highly matrixed organization 
•    Awareness of current GenAI trends and evaluation challenges in subjective NLP tasks 
•    Solve the cold start problem via synthetic data generation for new intents, flows, and low-resource scenarios.