Lead Clinical Data Engineer

Overview

TheLead Clinical Data Engineeris a senior technical specialist responsible for leading the ongoing design and development of the Aligned Data Warehouse, a centralized data repository for biomedical research for Yale School of Medicine in Yale New Haven Health System. In this position, theLead Clinical Data Engineerwill work with manager and other members of the data warehouse team to expand the functionality and integrate new data sources into the ADW and other research databases. The ADW is built on the Microsoft SQL Server technology stack, with data extracted from research and clinical systems, especially the Epic9s data warehouse, and transformed into OHDSI9s Observational Medical Outcomes Partnership (OMOP) Common Data Model and other data marts. The ADW is integrated with Linux and Windows-based systems for natural language processing, Large Language Models and other GPU-based software. This work supports major national and regional research projects and data-sharing initiatives.Responsibilities include:

• Design, build, test, maintain, and control data pipelines and ETL jobs for integrating data into the ADW and other research databases. Such data pipelines are implemented in T-SQL stored procedures, SSIS, and SQL Agent jobs, Microsoft Fabric and make extensive use of metadata-driven dynamic SQL.
• Contribute to DevOps and documentation, developing standards and procedures for database and data pipeline design, operational management, and ongoing maintenance.
• Design, build, test, maintain, and control other processes for data management, including management and loading of flat files in a variety of formats (e.g., csv, tsv, pipe-delimited, XML, JSON).
• Facilitate design sessions, code walkthroughs, peer reviews and produce technical documentation, SOPs and KBAs.
• Performance tune database objects, stored procedures, ETL jobs, and related scripts to optimize both end-user customer queries and data pipeline ETL processing.
• Monitor scheduled ETL jobs and other processes to ensure expected functioning and uptime of data pipelines. Lead and/or coordinate the troubleshooting and remediation of all ETL job failures in a timely manner. On-call support is sometimes required.
• Deliver world-class customer service in all interactions with customers, stakeholders, and other teams. Maintain a customer-focused approach to provide solutions that are science/research-driven.
• Maintain patient privacy and the integrity and security of healthcare data in all databases and systems, including compliance with all applicable laws, regulations, and institutional policies related to its Institutional Review Board (IRB), patient privacy, and IT cybersecurity. These laws include, but are not limited to, the 9Common Rule9 (45 CFR Part 46), HIPAA (45 CFR Part 164), and 42 CFR Part 2.
• Independently investigate and stay abreast of new and emerging technologies that Scientific Computing & Data could leverage to provide new capabilities, boosting efficiencies or quality, and/or lower operating costs
• Lead projects and data stewardship strategies
• Work closely with DBA team and vendors to optimize data pipelines
• Assist with the design, build, test, and performance-tuning of reports and dashboards in SSRS, Microsoft Power BI, or similar business intelligence tools.
• Performs related duties as assigned or requested.

Required Skills and Abilities

1.Extensive experience developing, automating and optimizing data integration workflows with a focus on Microsoft Fabric Data Pipelines supporting large-scale data ingestion, transformation, and integration workloads.

2.Proven ability to design, develop, and optimize scalable data models, semantic layers, and storage architectures that support enterprise reporting, self-service analytics, operational intelligence, and advanced analytics initiatives.

3.Experience developing and maintaining Fabric Notebooks utilizing PySpark, Spark SQL, and related technologies for data transformation, enrichment, validation, and automation.

4.Demonstrated experience designing and implementing Microsoft Fabric Lakehouses, Warehouses, and enterprise data organization strategies that support governance, performance, security, and scalability requirements.

5.Ability to work collaboratively with business stakeholders, analysts, architects, and technical teams to translate complex business requirements into scalable data engineering solutions

Preferred Skills and Abilities

1.In -depth knowledge of associated technology areas that could impact area of responsibility; healthcare technology experience

2.Experienceintegrating electronic health record (EHR), clinical, research, financial or operational healthcare data in cloud based analytics platforms.

3.Certification and experience with Epic9s Clarity & Caboodle databases

4.Experience with the OMOP Common Data Model, PCORnet and other research data models and familiarity with research administrative systems includingOnCore or IRES - IRB

Preferred Education and Experience

Master's degree

Experience in data engineering, analytics engineering, or data integration, including at least five (5) years of designing, developing, and maintaining enterprise-scale healthcare data platforms, ETL processes, and cloud-based data solutions.

Principal Responsibilities

Designs, modifies, develops, writes and implements software programming applications. Supports and/or installs software applications/operating systems. Participates in the testing process through test review and analysis, test witnessing and certification of software. Familiar with a variety of the field's concepts, practices, and procedures. Relies on experience and judgment to plan and accomplish goals. Performs a variety of complicated tasks. May lead and direct the work of others. May report directly to a project lead or manager. A wide degree of creativity and latitude is expected. Required Education and Experience Requires a bachelor's degree in a related area and a minimum of four years of experience in the field or in a related area.

Job Posting Date

10/13/2026

Job Category

Professional

Bargaining Unit

NON

Compensation Grade

Administration & Operations

Compensation Grade Profile

Manager; Program Leader (P6)

Salary Range

$90,000.00 - $165,750.00

Benefits

A general description of the benefits to be offered with this Managerial & Professional position is available at:

Yale Benefits

Time Type

Full time

Duration Type

Staff

Work Model

Remote

Background Check Requirements

All candidates for employment will be subject to pre-employment background screening for this position, which may include motor vehicle, DOT certification, drug testing and credit checks basedon the position description and job requirements. All offers are contingent upon the successful completion of the background check. For additional information on the background check requirements and process visit "Learn about background checks" under the Applicant Support Resources section of Careers on the It's Your Yale website.

Health Requirements

Certain positions have associated health requirements based on specific job responsibilities. These may include vaccinations, tests, or examinations, as required by law, regulation, or university policy.

Posting Disclaimer

Salary offers are determined by a candidate9s qualifications, experience, skills, and education in relation to the position requirements, along with the role9s grade profile and current internal and external market conditions.

The intent of this job description is to provide a representative summary of the essential functions that will be required of the position and should not be construed as a declaration of specific duties and responsibilities of the position. Employees will be assigned specific job-related duties throughtheir hiring department.