Senior Databricks Data Engineer

Prophecy Technologies

$100K — $140K *
Finance & Insurance
8 - 10 years of experience
Job Overview by Ladders

Qualifications

  • 10+ years of experience in data engineering or related fields.
  • 3-5 years of hands-on experience with Databricks and Spark.
  • Strong proficiency in SQL and data analysis techniques.
  • Experience with ETL processes, data modeling, and performance tuning.
  • Familiarity with Python or Scala for data engineering tasks.
  • Excellent problem-solving and communication skills.

Responsibilities

  • Design, develop, and deploy scalable ETL/ELT data pipelines using Apache Spark, PySpark, and Databricks.
  • Develop and optimize SQL queries for data transformation and analysis.
  • Collaborate with product owners, data architects and analysts to build data models, delta lake structures, and data workflows.
  • Collaborate with data analysts and business teams to deliver actionable insights.
  • Build job orchestration and monitoring solutions.
  • Ensure data quality, performance, and reliability across workflows.
Full Job Description
Role Overview:

We are seeking a hands-on Sr. Databricks Data Engineer to design, develop, and optimize data pipelines and analytics solutions.

Key Responsibilities:
  • Design, develop, and deploy scalable ETL/ELT data pipelines using Apache Spark, PySpark, and Databricks.
  • Develop and optimize SQL queries for data transformation and analysis.
  • Collaborate with product owners, data architects and analysts to build data models, delta lake structures, and data workflows.
  • Collaborate with data analysts and business teams to deliver actionable insights.
  • Build job orchestration and monitoring solutions.
  • Ensure data quality, performance, and reliability across workflows.

Required Skills:
  • 10+ years overall experience in data engineering or related fields.
  • 3-5 years hands-on experience with Databricks and Spark.
  • Strong proficiency in SQL and data analysis techniques.
  • Experience with ETL processes, data modeling, and performance tuning.
  • Familiarity with Python or Scala for data engineering tasks.
  • Excellent problem-solving and communication skills.

Preferred Skills:
  • Knowledge of insurance industry data.

Similar Jobs

More Jobs at Prophecy Technologies

More Finance & Insurance Jobs

Find similar Senior Databricks Data Engineer jobs: