JOB SUMMARY
Design, develop, and optimize scalable data engineering solutions using Azure Databricks. Build end-to-end ETL/ELT pipelines and Lakehouse architectures. Implement data governance, security, and performance optimization best practices. Collaborate with cross-functional teams in a consulting environment to deliver enterprise data solutions.

Key Responsibilities
Design and develop ETL/ELT pipelines using Azure Databricks.
Build batch ingestion using Auto Loader and real-time ingestion using Spark Structured Streaming.
Develop and maintain Delta Lake-based data pipelines.
Manage Unity Catalog for data governance, security, and access control.
Create and maintain catalogs, schemas, tables, materialized views, functions, and volumes.
Develop Slowly Changing Dimensions (SCD Type 1 & Type 2) for dimensional data.
Build Change Data Capture (CDC) pipelines for incremental data processing.
Implement Lakehouse Federation and configure foreign catalogs for external data integration.
Optimize data storage using partitioning and Liquid Clustering.
Ensure data quality, integrity, and consistency across data platforms.
Participate in CI/CD implementation and follow DevOps best practices.
Collaborate with business and technical stakeholders to deliver scalable data solutions.

Required Qualifications
7-10 years of experience in Data Engineering.
Hands-on experience with Azure Databricks and Delta Lake.
Strong expertise in ETL/ELT pipeline development.
Experience with Spark Structured Streaming and Auto Loader.
Strong knowledge of Unity Catalog.
Proficiency in SQL and data modeling.
Strong understanding of Data Warehousing concepts.
Experience with Microsoft Azure cloud services.
Excellent analytical and problem-solving skills.