Job Summary:
The Lead Databricks Architect will lead the strategy, architecture, and implementation of an enterprise-scale Databricks Lakehouse platform within the banking and financial services domain. The role requires extensive hands-on experience designing and managing production Databricks environments, along with strong knowledge of banking data, regulatory compliance, security, and enterprise data governance. The architect will build secure, scalable, and high-performance data solutions supporting critical banking functions, including regulatory reporting, risk management, fraud detection, customer analytics, and real-time transaction processing.
Key Responsibilities:
• Design and implement enterprise-grade Databricks Lakehouse architecture using Delta Lake, Unity Catalog, and Databricks SQL.
• Establish architecture patterns for large-scale batch and real-time streaming pipelines using Structured Streaming and Auto Loader.
• Integrate data from core banking systems, payment gateways, and third-party data providers.
• Lead migration initiatives from legacy data platforms such as Teradata, Oracle, Snowflake, and mainframe environments to Databricks.
• Implement enterprise data security, access controls, and fine-grained permissions using Unity Catalog.
• Establish governance frameworks supporting RBAC, ABAC, data lineage, auditing, and metadata management.
• Design data masking, tokenization, and encryption strategies to protect personally identifiable information (PII) and sensitive financial data.
• Ensure compliance with financial industry regulations and standards, including GDPR, SOX, and PCI-DSS.
• Support secure data management across Azure-based hybrid and multi-cloud environments.
• Define best practices for PySpark, Delta Live Tables (DLT), and Databricks SQL optimization.
• Improve platform performance, scalability, compute efficiency, and total cost of ownership (TCO).
• Establish CI/CD, DevOps, and MLOps frameworks for automated deployments and model lifecycle management.
• Implement data quality and validation frameworks using tools such as Great Expectations.
• Collaborate with Data Science, Quantitative Analytics, and Business Intelligence teams to support predictive risk analytics, financial modeling, and Customer 360 initiatives.
• Partner with business leaders, Chief Risk Officers, and compliance teams to align data architecture with banking objectives.
• Provide technical leadership, architecture governance, and mentorship to data engineering and analytics teams.
• Drive architectural best practices and ensure consistent implementation across enterprise data initiatives.
Required Qualifications:
• 5+ years of hands-on experience designing, architecting, and managing production Databricks environments.
• Strong cloud architecture experience across AWS, Azure, or GCP.
• Expertise in Databricks Lakehouse Architecture, Delta Lake, Unity Catalog, PySpark, and Databricks SQL.
• Strong experience with Structured Streaming, Delta Live Tables (DLT), and Auto Loader.
• Strong experience with data governance, security, and access management, including RBAC and ABAC.
• Hands-on experience with Terraform and Infrastructure as Code (IaC).
• Experience implementing CI/CD pipelines using GitHub Actions, Azure DevOps, or Jenkins.
• Familiarity with Databricks Asset Bundles (DABs).
• 3+ years of experience working within retail banking, investment banking, or financial services.
• Strong understanding of banking data models and business processes, including Core Banking, Payments and Transactions, Lending and Credit Risk, Anti-Money Laundering (AML), Fraud Detection, and Regulatory Reporting.
• Knowledge of financial data governance, auditability, regulatory compliance, and data security requirements.
• Bachelor's or Master's degree in Computer Science, Data Engineering, Information Systems, or a related technical field, or equivalent practical experience.
Preferred Qualifications:
• Databricks Certified Data Engineer Professional certification.
• AWS Certified Solutions Architect certification.
• Microsoft Azure Solutions Architect Expert certification.
• Experience leading enterprise data platform initiatives and cloud modernization projects.
• Ability to collaborate effectively with senior business and technology stakeholders.
Candidates must meet both key experience requirements:
• 5+ years of production Databricks architecture experience.
• 3+ years of banking or financial services industry experience.
