Organization OverviewLilly CIA powers the enterprise with governed, AI-ready data products, a converged semantic layer, and the identity and policy infrastructure behind Lilly's digital experiences. Within CIA, the LillyDirect Covered Entity (CE) Pharmacy pod builds and operates the data platform that sits inside the pharmacy's covered-entity trust boundary - handling identified PHI under HIPAA with strict isolation from parent-side systems. This role leads the engineering team that builds and operates that platform.
Position SummaryThe Associate Director of CE Data Engineering owns engineering delivery and standards for the CE Pharmacy data platform on Databricks. This position leads a team of data engineers implementing the medallion pipelines, data contracts, and isolation controls that the CE Data Architect designs - translating architectural intent into a reliable, tested, production engineering practice. The role sets the CE engineering strategy (tooling, automation, quality standards, CI/CD), develops the team, and partners with the CE Data Architect (design) and DataHub Product Owner (product) to ship governed, PHI-compliant data products at scale.
Key Responsibilities- Lead the CE Data Engineering strategy: drive excellence in engineering practices, set standards for pipeline design, testing, and code quality, and ensure the team's work aligns to the CE Data Architect's canonical models and isolation boundary.
- Own the end-to-end engineering delivery lifecycle for CE data products - from requirements and prototyping through production deployment and maintenance - across the team's portfolio of pipelines.
- Evaluate and standardize on Databricks capabilities and integration patterns (Unity Catalog, Delta Lake, Workflows, serverless compute, Lakebase) across the team, balancing performance, cost, and scalability.
- Establish and enforce data quality, testing, and validation methodology (pytest, DLT/Great Expectations) across all CE pipelines; own monitoring and alerting standards so issues are caught proactively, not in production incidents.
- Own the CE CI/CD strategy - GitHub Actions-based, Git-based promotion dev 10 test 10 prod - for data, contract, policy, and agent artifacts, ensuring the team ships safely and repeatably.
- Advocate and enforce standard processes across the CE data asset lifecycle, working closely with the CE Data Architect, DataHub Product Owner, and Policy-as-Code Engineer.
- Partner across governance and platform - HIPAA governance, Privacy, Legal/DLO, and platform owners (EPH) - to keep engineering practice compliant and defensible, advancing architecture or policy blockers rather than working around them.
- Advise collaborators and partner teams on proper use of CE data, and communicate delivery status, risk, and tradeoffs to both technical and non-technical collaborators.
- Lead special projects and multi-functional initiatives for the CE Pharmacy platform as needed.
Basic Qualifications- Bachelor's in Computer Science, Engineering, Information Technology, Information Management, or similar degree.
- 8+ years of experience in data engineering
- 3+ years leading data engineering teams.
- Expertise in SQL, Python, and Databricks / Unity Catalog (Delta Lake, Workflows, notebooks, catalog/schema/grants) at production scale.
- Qualified applicants must be authorized to work in the United States on a full-time basis. Lilly will not provide support for or sponsor work authorization or visas for this role, including but not limited to F-1 CPT, F-1 OPT, F-1 STEM OPT, J-1, H-1B, TN, O-1, E-3, H-1B1, or L-1.
Additional Skills / Preferences- Experience defining and executing data ingestion pipelines for large-scale, enterprise data.
- Experience with regulated or patient/PHI data environments; working knowledge of HIPAA and covered-entity data handling constraints.
- Proficiency with Git-based CI/CD workflows (GitHub Actions or equivalent) for versioned data and pipeline artifacts.
- Solid grasp of data modeling, database develop, and data governance (RBAC/ABAC, row/column-level security).
- Excellent problem-solving skills and a record of translating business and regulatory requirements into engineering standards.
- Good communication and collaboration skills, with the ability to lead through influence across multi-functional and peer leadership teams.
- Consistent record leading and delivering complex multi-functional data engineering projects, ideally in a regulated or healthcare data environment.
- Validated ability to rally resources across engineering, architecture, product, and compliance to hit delivery commitments.
- Familiarity with HIPAA, PHI classification, tokenization/de-identification patterns (Datavant or comparable), and data contract frameworks (ODCS).
- Knowledge of policy-as-code (OPA/Rego) and how engineering standards intersect with automated policy enforcement.
- Knowledge of DevOps practices and tools for CI/CD and infrastructure automation (GitHub Actions, Terraform).
- Experience with Agile/Scrum methodologies and project management tools such as JIRA.
- Experience architecting or supporting AI Skill / agent development and operationalizing it through CI/CD.
- High level of motivation and a strong desire to find creative solutions to challenging situations.
Additional Information- 0-10% travel depending on business need.
Actual compensation will depend on a candidate's education, experience, skills, and geographic location. The anticipated wage for this position is
$132,000 - $193,600
Full-time equivalent employees also will be eligible for a company bonus (depending, in part, on company and individual performance). In addition, Lilly offers a comprehensive benefit program to eligible employees, including eligibility to participate in a company-sponsored 401(k); pension; vacation benefits; eligibility for medical, dental, vision and prescription drug benefits; flexible benefits (e.g., healthcare and/or dependent day care flexible spending accounts); life insurance and death benefits; certain time off and leave of absence benefits; and well-being benefits (e.g., employee assistance program, fitness benefits, and employee clubs and activities).Lilly reserves the right to amend, modify, or terminate its compensation and benefit programs in its sole discretion and Lilly's compensation practices and guidelines will apply regarding the details of any promotion or transfer of Lilly employees.
#WeAreLilly
