8-10 years of extensive data engineering experience.
Proficiency in Databricks, ETL/ELT, and SQL.
Hands-on development skills in Scala and Spark.
Experience with PySpark and Spark structured streaming.
Strong understanding of Databricks Delta Lake storage and Unity Catalog.
Familiarity with Lakehouse and Medallion architectures.
Expertise in configuring and orchestrating Databricks clusters.
Responsibilities
Implement Databricks solutions using Scala and Spark.
Develop and optimize Spark jobs and data processing workflows.
Perform performance tuning for Scala, PySpark, and SQL code.
Work with Databricks Delta Lake storage and Lakehouse architectures.
Configure and size Databricks clusters and orchestrate jobs.
Handle complex data streaming and transformations using Azure Databricks.
Monitor and debug live Databricks jobs.
Benefits
Opportunity to work with cutting-edge data technologies.
Engagement in complex data engineering projects.
Collaboration with a skilled team of data professionals.
Potential for career growth in a rapidly evolving field.
Full Job Description
Role Overview:
Seeking an experienced Data Engineer with a strong focus on Databricks and Scala to design, develop, and optimize data engineering solutions. The role involves extensive work with ETL/ELT processes, data pipelines, and advanced data transformations within a Lakehouse architecture.
Key Responsibilities:
Implement Databricks solutions using Scala and Spark, including data frames, notebooks, and SQL.
Develop and optimize Spark jobs, data transformations, and data processing workflows.
Perform performance tuning and enhancement for Scala, PySpark, and SQL code.
Work with Databricks Delta Lake storage, Unity Catalog, Autoloader, and Lakehouse/Medallion architectures.
Configure and size Databricks clusters, orchestrate Databricks jobs, and implement Spark structured streaming.
Handle complex and large volume data streaming and transformations using Azure Databricks.
Lead data warehouse and data lakehouse implementations.
Monitor, debug, and resolve issues in live Databricks jobs.
Troubleshoot DevOps pipelines, Azure, and AKS services.
Required Skills:
Extensive Data engineering experience in Databricks, ETL/ELT, and SQL.
Hands-on development proficiency in Scala and Spark.
Experience with PySpark.
Strong understanding of Databricks Delta Lake storage, Unity Catalog, Autoloader.
Knowledge of Lakehouse and Medallion architectures.
Familiarity with Databricks serverless options, Dataset and data frame concepts.
Expertise in Databricks clusters configuration, sizing, and job orchestration.
Experience with Spark structured streaming.
Proficiency in Azure Databricks for complex data streaming and transformations.
Strong hands-on expertise in troubleshooting DevOps pipelines, Azure, and AKS services.
Qualifications:
8-10 years of extensive Data engineering experience.
