Role Overview:This role involves designing and developing scalable ETL/ELT pipelines using Databricks, including Delta Lake, Auto Loader, and DLT.
Key Responsibilities:- Develop batch, streaming, and CDC ingestion pipelines to enable efficient and near real-time data processing.
- Implement ingestion patterns using Auto Loader with checkpointing and schema evolution for structured and semi-structured data.
- Develop and maintain reusable DBX functions for data transformation, data quality checks, and validation logic.
- Leverage Unity Catalog to establish secure data governance, enabling controlled access, lineage tracking, and management of External Locations and Volumes across workspaces.
- Partner with cross-functional teams to enable secure data sharing using Delta Sharing across internal domains and external partners.
Required Skills:- 15+ years of experience required.
- Digital: Databricks.
- Strong SQL and data modeling skills.
- Experience with cloud platforms and distributed systems.
Qualifications:- Strong SQL and data modeling skills.
- Experience with cloud platforms and distributed systems.
