Job Summary
The Databricks Architect will design, develop, and maintain scalable data engineering solutions using Databricks and Azure Data Factory (ADF). The role will focus on building optimized ETL/ELT pipelines, developing data transformation solutions using Python and SQL, and supporting Credit Risk reporting and analytics. The architect will integrate data from multiple internal and external sources, troubleshoot technical and data quality issues, and collaborate with business users, data engineers, and project stakeholders to deliver reliable enterprise data solutions.
Key Responsibilities
• Design, develop, and maintain scalable data pipelines using Azure Data Factory (ADF).
• Build and optimize ETL/ELT processes to ingest, transform, and load enterprise data into Databricks environments.
• Develop and maintain Databricks notebooks using Python and SQL.
• Create scalable and efficient data processing solutions supporting Credit Risk reporting and analytics.
• Query, analyze, and validate data using Databricks SQL.
• Integrate data from various internal and external sources into enterprise data platforms.
• Troubleshoot pipeline failures, performance bottlenecks, and data quality issues.
• Collaborate with business users, data engineers, and project stakeholders to understand requirements and deliver effective solutions.
• Participate in code reviews and maintain best practices for coding, documentation, and deployment.
• Support production environments and ensure reliable data delivery across platforms.
• Utilize Databricks Apps and related platform capabilities where appropriate.
Required Qualifications
• 5+ years of hands-on Databricks development experience.
• Strong hands-on experience with Azure Data Factory (ADF).
• Proficiency in Python programming.
• Advanced SQL development skills.
• Experience creating and supporting data pipelines and ETL workflows.
• Experience developing and maintaining Databricks notebooks and data transformation processes.
• Strong understanding of data engineering fundamentals and best practices.
• Ability to independently diagnose and resolve technical issues, including pipeline failures, performance bottlenecks, and data quality issues.
• Excellent communication and collaboration skills.
