Job Summary
The Lead Databricks Architect will drive the strategy, design, and implementation of an enterprise-grade lakehouse platform for the banking and financial services industry. This role requires deep expertise in Databricks architecture, cloud data platforms, data engineering, security, and governance. The architect will build secure, scalable, and high-performance data solutions supporting regulatory reporting, risk management, fraud detection, customer analytics, and real-time transaction processing. The role also involves technical leadership, architecture governance, and collaboration with business and technology stakeholders.
Key Responsibilities
• Design and implement enterprise-grade Databricks Lakehouse architecture using Delta Lake, Unity Catalog, and Databricks SQL, tailored to financial data models.
• Establish architectural patterns for high-throughput batch and real-time streaming data pipelines using Structured Streaming and Auto Loader.
• Integrate data pipelines with core banking systems, payment gateways, and third-party data providers.
• Drive migration strategies from legacy data warehouses and platforms, including Teradata, Oracle, Snowflake, and mainframe environments, to Databricks.
• Implement enterprise data security, access controls, and fine-grained permissions using Databricks Unity Catalog.
• Design data masking, tokenization, and encryption strategies for Personally Identifiable Information (PII) and Sensitive Personal Data (SPD).
• Ensure data lineage, auditing, and metadata management across multi-cloud and hybrid-cloud deployment environments.
• Define best practices for PySpark, Delta Live Tables (DLT), and SQL optimization to improve compute efficiency and reduce total cost of ownership.
• Establish CI/CD, DevOps, and MLOps automation frameworks for code deployment, data quality validation, and model lifecycle management.
• Implement data quality validation practices using tools such as Great Expectations.
• Partner with Data Science, Quantitative Analytics, and Business Intelligence teams to support financial modeling, predictive risk analysis, and customer 360 initiatives.
• Collaborate with business leaders, Chief Risk Officers, and compliance teams to align data platform capabilities with banking business objectives.
• Provide technical leadership, mentorship, and architecture governance to data engineering and analytics teams.
Required Qualifications
• 5+ years of hands-on experience designing and managing large-scale Databricks environments in production.
• Deep knowledge of cloud data architecture and services across AWS, Azure, or GCP.
• Experience with cloud data services such as Azure Data Lake Storage, Azure Data Factory, AWS S3, AWS Glue, GCP Cloud Storage, or BigQuery.
• Strong expertise in Delta Lake, PySpark, Databricks SQL, Structured Streaming, and Delta Live Tables.
• Hands-on experience configuring Databricks Unity Catalog, role-based access control (RBAC), attribute-based access control (ABAC), and data governance frameworks.
• Proficiency with Terraform, Infrastructure as Code, CI/CD pipelines, and Databricks Asset Bundles (DABs).
• Experience with CI/CD tools such as GitHub Actions, Azure DevOps, or Jenkins.
• 3+ years of experience working within a retail bank, investment bank, or financial services institution.
• Familiarity with core banking operations, payments, lending, credit risk, anti-money laundering (AML), or fraud-related data structures.
• Strong understanding of banking data compliance requirements, auditability, and data sovereignty standards.
• Understanding of financial regulatory requirements, including GDPR, SOX, and PCI-DSS.
• Bachelor's or Master's degree in Computer Science, Data Engineering, Information Systems, or a related quantitative field, or equivalent practical experience.
Preferred Qualifications
• Databricks Certified Data Engineer Professional certification.
• Cloud Solutions Architect certification, such as AWS Certified Solutions Architect or Azure Solutions Architect Expert.
