JOB DESCRIPTION
We are seeking an experienced Databricks Architect to lead the design and implementation of scalable data platforms on Databricks. The role will drive end-to-end architecture, including data ingestion, transformation, optimization, and governance, while enabling advanced analytics and AI/ML use cases. The ideal candidate will have strong expertise in Spark, Delta Lake, cloud platforms (Azure/AWS), and modern data engineering practices, along with the ability to collaborate with business and technology stakeholders to deliver high-impact solutions.
JOB RESPONSIBILITIES
- Lead the design and implementation of scalable, secure, and high-performance data architecture on Databricks
- Define end-to-end data pipelines (ingestion, transformation, serving) using Spark and Delta Lake
- Drive migration and modernization initiatives from legacy platforms to Databricks
- Establish best practices for data engineering, performance optimization, and cost management
- Design and implement data governance, security, and compliance frameworks
- Collaborate with business stakeholders, data scientists, and engineering teams to translate requirements into technical solutions
- Provide technical leadership, mentorship, and guidance to development teams
- Ensure data quality, reconciliation, and reliability across data workflows
- Integrate Databricks with enterprise tools (e.g., MuleSoft, Alteryx, BI/reporting platforms)
- Stay current with Databricks innovations and recommend adoption of new capabilities (e.g., ML, AI, DBSQL, Unity Catalog)
JOB QUALIFICATIONS
- Bachelor’s or Master’s degree in Computer Science, Engineering, or a related field
- 9-12+ years of experience in data engineering, data architecture, or analytics platforms
- Strong hands-on expertise with Databricks, Apache Spark, and Delta Lake
- Experience with cloud platforms such as Azure (preferred), AWS, or GCP
- Proven experience designing and implementing scalable data pipelines and architectures
- Strong knowledge of SQL, Python, and/or Scala
- Experience with data integration tools (e.g., MuleSoft, Alteryx) and modern data ecosystems
- Familiarity with data governance, security frameworks, and compliance best practices
- Experience with performance tuning, optimization, and cost management in Databricks
- Strong problem-solving skills and ability to work in a cross-functional, collaborative environment
- Excellent communication and stakeholder management skills
- Exposure to AI/ML use cases, Databricks SQL, and Unity Catalog is a plus
