Senior Data Engineer (Databricks)

Insight Global

$110K — $140K *
Healthcare
5 - 7 years of experience
Job Overview by Ladders

Qualifications

  • 7+ years of experience as a Data Engineer in an Azure environment
  • 5+ years of experience building pipelines with Databricks
  • Expert proficiency in Python for scripting and data handling
  • 5+ years of experience using PySpark for ETL processes
  • 2+ years of experience in a healthcare setting
  • 3+ years of experience with diverse data types such as JSON, HL7, XML, CSV

Responsibilities

  • Design, develop, and implement comprehensive data solutions using Azure Databricks
  • Transform legacy SQL workflows into modern Python-based data pipelines
  • Maintain and optimize high-volume production data pipelines
  • Write, test, and enhance PySpark and SQL scripts for better performance
  • Ensure data quality and integrity through robust validation and cleansing
  • Collaborate cross-functionally within an agile team, promoting critical thinking and effective communication

Benefits

  • Medical, dental, and vision insurance starting on the 31st day
  • HSA, FSA, and DCFSA account options available
  • 401k retirement account with employer matching
  • Paid sick leave and other paid time off as per applicable law
Full Job Description
A leading healthcare payor is actively seeking to hire a Senior Azure Data Engineer with deep Databricks expertise to help shape the future of data innovation. This is your chance to join a growing Enterprise Data Office where your work will directly impact patient outcomes, operational efficiency, and data-driven decision-making. Key responsibilities include:
• Design, develop and implement end-to-end data solutions using Azure Databricks.
• Convert legacy SQL workflows into modern Python-based pipelines.
• Maintain and optimize high-volume data pipelines in production.
• Write, test, and tune PySpark and SQL scripts for performance and scalability.
• Ensure data quality and integrity through robust validation and cleansing processes.
• Collaborate cross-functionally, bringing critical thinking and clear communication to a fast-paced, agile team.

PLEASE NOTE: We are looking for someone who values in person collaboration and is willing to come onsite twice a month in Newtown Square, PA 19073.

We are a company committed to creating diverse and inclusive environments where people can bring their full, authentic selves to work every day. We are an equal opportunity/affirmative action employer that believes everyone matters. Qualified candidates will receive consideration for employment regardless of their race, color, ethnicity, religion, sex (including pregnancy), sexual orientation, gender identity and expression, marital status, national origin, ancestry, genetic factors, age, disability, protected veteran status, military or uniformed service member status, or any other status or characteristic protected by applicable laws, regulations, and ordinances. If you need assistance and/or a reasonable accommodation due to a disability during the application or recruiting process, please send a request to [email protected] learn more about how we collect, keep, and process your private information, please review Insight Global's Workforce Privacy Policy: https://insightglobal.com/workforce-privacy-policy/.

Required Skills & Experience

7+ years of experience as a Data Engineer working in a heavy Azure environment

5+ years of experience building pipelines with Databricks

Expertise with Python for scripting, automation, data cleaning and transformation

5+ years using PySpark to implement ETL pipelines

2+ years of recent experience working in a healthcare environment

3+ years of experience working with different data types (JSON, HL7, XML, CSV, etc.)

Benefit packages for this role will start on the 31st day of employment and include medical, dental, and vision insurance, as well as HSA, FSA, and DCFSA account options, and 401k retirement account access with employer matching. Employees in this role are also entitled to paid sick leave and/or other paid time off as provided by applicable law.

Similar Jobs

More Healthcare Jobs

Find similar Senior Data Engineer (Databricks) jobs: