Eli Lilly

Sr. Principal Data Engineer - Lakehouse Architecture

Eli Lilly$132K — $244K *
Information Technology
Less than 5 years of experience
Job Overview by Ladders

Qualifications

  • Master's degree in computer science, Engineering, or a related technical field
  • 3+ years of hands-on experience with Lakehouse architectures (Databricks, Snowflake, or similar)
  • 7+ years of overall data engineering experience with large-scale distributed systems
  • Proficiency in Python and SQL for data transformation
  • Expertise in Apache Spark (PySpark or Scala) for data processing
  • Experience with cloud platforms (AWS or Azure) and their data services
  • Strong knowledge of data governance, security, and compliance in regulated industries

Responsibilities

  • Design and implement Lakehouse architecture solutions using Databricks and Snowflake
  • Define and enforce medallion architecture standards across data domains
  • Lead Unity Catalog governance including data schemas and access control
  • Build and maintain real-time and batch data processing systems using Apache Spark and Kafka
  • Architect scalable data pipelines for structured and unstructured data
  • Develop data transformation workflows with tools like DBT and Airflow
  • Collaborate with data scientists and analysts to enhance technical solutions

Benefits

  • Company-sponsored 401(k) and pension plans
  • Vacation and leave benefits
  • Medical, dental, vision, and prescription drug coverage
  • Flexible spending accounts for healthcare and dependent care
  • Life insurance and death benefits
  • Employee assistance programs and fitness benefits
  • Opportunities for participation in employee clubs and activities
Full Job Description
Tech at Lilly is seeking a highly skilled Senior Data Engineer who can implement and optimize large-scale Lakehouse solutions and drive the evolution of our modern data platform while providing technical leadership to a growing team. The ideal candidate will have hands-on experience with modern data engineering technology stack and a proven track record of leading engineering talent in fast-paced environments.

What you will be doing

  • Design and implement comprehensive Lakehouse architecture solutions using technologies like Databricks and Snowflake platforms


  • Define and carry out medallion architecture standards (Bronze / Silver / Gold) across data domains, ensuring data quality, lineage, and discoverability


  • Lead Unity Catalog governance design: schemas, access control policies, and data contracts


  • Build and maintain real-time and batch data processing systems using Apache Spark (PySpark/Scala), Kafka, and Databricks Structured Streaming, and similar technologies


  • Architect scalable data pipelines that handle structured, semi-structured, and unstructured data to deliver AI ready data.


  • Develop data transformation workflows using tools like DBT, Airflow, or Databricks


  • Implement data governance frameworks, including data quality monitoring, lineage tracking, data time travel and security protocols.


  • Build data pipeline testing frameworks: unit tests, data quality assertions (Great Expectations / dbt tests), and schema validation


  • Define and publish data SLAs/SLOs in collaboration with data product owners; own incident response and root-cause analysis for pipeline failures


  • Drive adoption of modern data engineering standard processes including Infrastructure as Code, CI/CD, and automated testing


  • Collaborate with data scientists, analysts, and business collaborators to translate requirements into robust technical solutions


  • Mentor a team of 3-5 data engineers


  • Foster a collaborative team culture focused on continuous learning and innovation


How You Will Succeed

  • Proven ability to mentor junior engineers and facilitate knowledge sharing


  • Strong project management skills with experience leading multi-functional initiatives


  • Coordinate multi-functional projects and ensure effective communication between technical and business teams


  • Demonstrated ability to make architectural decisions and drive technical consensus


  • Embrace a growth mindset and actively seek opportunities to expand your leadership capabilities and technical mastery


What You Should Bring

  • Knowledge in the pharmaceutical or life sciences domain
  • Experience with streaming data technologies (Kafka,Databricks Structured Streaming)


  • Familiarity with data cataloging tools


  • Familiarity with high performance data service framework (Arrow Flight)


  • Expert-level proficiency in Python and SQL for data transformation and pipeline development


  • Strong experience with Apache Spark (PySpark or Scala) for big data processing and analytics


  • Hands-on experience with cloud platforms (AWS or Azure) and their data services, including S3, Glue, Redshift, IAM, and CloudWatch


  • Proficiency with Infrastructure as Code tools (CloudFormation)


  • Experience with containerization (Docker, Kubernetes) and orchestration platforms


  • Knowledge of data modeling techniques for both analytical and operational workloads


  • Hands-on expertise with Databricks: clusters, Delta Lake, Unity Catalog, Workflows, and MLflow integration


  • Understanding of data governance, security, and compliance requirements including GxP, HIPAA, or GDPR data frameworks in regulated industries


  • Experience orchestrating pipelines with Apache Airflow


  • Strong command of dbt for modular, tested, and version-controlled transformations


  • Familiarity with data quality testing frameworks such as Great Expectations or dbt tests for schema validation


  • Experience with data observability: freshness, volume, schema drift, and anomaly detection (Databricks Lakehouse Monitoring or equivalent)


  • Familiarity with open table format interoperability: Delta Sharing or Apache Iceberg


  • Knowledge of DataOps practices and data mesh / data product principles


  • Exposure to ML platform integration: MLflow experiment tracking, feature stores, or model serving


Your Basic Qualifications

  • Master's degree in computer science, Engineering, or related technical field


  • 3+ years of hands-on experience with Lakehouse architectures (Databricks, Snowflake, or similar)


  • 7+ years of overall data engineering experience with large-scale distributed systems, including at least 3 years in a senior or lead capacity


Actual compensation will depend on a candidate's education, experience, skills, and geographic location. The anticipated wage for this position is
$132,000 - $244,200

Full-time equivalent employees also will be eligible for a company bonus (depending, in part, on company and individual performance). In addition, Lilly offers a comprehensive benefit program to eligible employees, including eligibility to participate in a company-sponsored 401(k); pension; vacation benefits; eligibility for medical, dental, vision and prescription drug benefits; flexible benefits (e.g., healthcare and/or dependent day care flexible spending accounts); life insurance and death benefits; certain time off and leave of absence benefits; and well-being benefits (e.g., employee assistance program, fitness benefits, and employee clubs and activities).Lilly reserves the right to amend, modify, or terminate its compensation and benefit programs in its sole discretion and Lilly's compensation practices and guidelines will apply regarding the details of any promotion or transfer of Lilly employees.

#WeAreLilly

About Eli Lilly

ICOS Corporation is a biotechnology company that engages in the discovery, development, and commercialization of therapeutic products. It is engaged in the commercialization of treatments for unmet medical conditions, such as benign prostatic hyperplasia, hypertension, pulmonary arterial hypertension, cancer, and inflammatory diseases. It is the developer of a treatment known as Cialis (tadalafil), a product for the treatment of erectile dysfunction through its joint venture with Eli Lilly and Company in North America and Europe. It is also engaged in contract manufacturing services for third parties. It is in a strategic alliance with Solvay Pharmaceuticals, Inc. ICOS Corporation was established in 1989, based in Bothell, Washington. It is currently operated by Eli Lilly and Company.

Eli Lilly Careers

Joining Eli Lilly offers an unparalleled opportunity to become part of a leading global team dedicated to creating a healthier future. As a company revered for its commitment to innovation and leadership in the pharmaceutical industry, Eli Lilly is where your professional journey can flourish. Work You’ll Do At Eli Lilly, we are passionate about transforming patient care and advancing medical innovation. Our team at Eli Lilly is at the forefront of developing groundbreaking solutions in healthcare. By joining us, you will collaborate with some of the brightest minds in the industry, using cutting-edge technology to make real-world impacts. Lead with Innovation and Leadership Eli Lilly stands out in the marketplace by integrating deep industry expertise with robust research and development efforts. We are looking for professionals who are eager to drive change and lead the way in developing therapeutic breakthroughs. Explore Job Opportunities and Growth Eli Lilly offers a variety of career paths, including full-time positions and internships, across multiple functions such as research, marketing, IT, and sales. Whether you are a seasoned professional or a recent graduate, Eli Lilly provides an environment that promotes career growth and learning opportunities. Our commitment to diversity and leadership training ensures that every employee can achieve their potential. Be Part of Our Team Our team at Eli Lilly is committed to excellence and driven by a mission to improve lives. Employees enjoy a supportive culture that values collaboration, creativity, and diversity. We believe that a diverse workforce fosters innovation and helps us better connect with the communities we serve. Benefits and Culture Eli Lilly is dedicated to supporting our employees, offering competitive benefits, wellness programs, and comprehensive health care. Our culture is built on a foundation of respect, integrity, and quality, making Eli Lilly not just a great place to work, but a community to grow with. Networking and Professional Development Eli Lilly encourages continuous professional development and networking. With access to various training programs and mentorship opportunities, employees can enhance their skills and advance their careers. Our leadership is committed to nurturing talent through effective training and development strategies. Join Our Team Discover the exciting job opportunities at Eli Lilly by exploring open positions that match your skills and interests. We are continuously hiring and looking for individuals who are passionate, innovative, and ready to contribute to our mission of making life better for people around the globe. Stay Connected Keep up to date with the latest at Eli Lilly by following our careers blog. Gain insights from industry leaders and get tips on everything from crafting the perfect resume to preparing for your interview. Eli Lilly is not just a company—it's a place where you can make a difference. Explore the positions available and find out how your talents can help change the world. SEARCH ELI LILLY JOBS Stay ahead in your career with Eli Lilly, where innovation, leadership, and a commitment to diversity and growth lead the way to future advancements.
Learn more about Eli Lilly
Size
35,000 employees
Market Cap
$344.2 billion
Industry
Net Income
$6.1 billion
Founded
1876
5 Year Trend
+5.9%
Revenue
$24.5 billion
NASDAQ

Similar Jobs

More Jobs at Eli Lilly

More Information Technology Jobs

Find similar Sr. Principal Data Engineer - Lakehouse Architecture jobs: