Role description
Solution Architect Databricks Lakehouse Migration AI Modernization
Raritan NJ Onsite
We are seeking an experienced Solution Architect to lead the migration and modernization of enterprisescale data lakes from Azure Databricks to AWS Databricks while transforming the platform into an AIready Lakehouse architecture The role involves defining targetstate architecture migration strategy governance frameworks modernization roadmaps and AIGenAI enablement across largescale data ecosystems
The ideal candidate will have deep expertise in Databricks cloud migration modern data architecture and AIdriven data platforms with experience leading complex enterprise transformation programs involving thousands of data assets and multiple downstream applications
Key Responsibilities
Architecture Strategy
Define endtoend migration strategy from Azure Databricks to AWS Databricks
Design scalable Lakehouse architecture using Medallion principles Bronze Silver Gold
Develop Day1 Business Continuity and Day2 Modernization AI Enablement migration roadmaps
Perform application and dependency assessment across upstream and downstream systems
Establish architecture patterns standards and best practices for cloudnative data platforms
Migration Modernization
Lead migration of data pipelines notebooks Delta tables workflows and data products
Modernize legacy ETLELT frameworks using Databricks native capabilities
Optimize performance scalability security and operational efficiency
Define migration approach for CDC batch streaming and realtime workloads
Drive technology rationalization and tool consolidation initiatives
AIReady Data Platform
Build AIready data foundations leveraging Delta Lake and Unity Catalog
Design data architectures supporting ML GenAI RAG feature stores and advanced analytics
Enable governed data access for data scientists and AI applications
Define semantic models metadata strategies lineage and data discovery frameworks
Establish frameworks for AI governance and responsible AI adoption
Data Governance Security
Architect enterprisewide governance using Unity Catalog
Design data lineage auditing access controls masking and compliance standards
Implement data quality observability and monitoring frameworks
Ensure platform adherence to security and regulatory requirements
Stakeholder Management
Collaborate with business data engineering analytics AI security and cloud teams
Lead architecture review boards and executivelevel discussions
Mentor solution architects technical leads and engineering teams
Drive estimation roadmap planning risk assessment and governance reviews
Required Experience
12 years in Data Analytics
5 years as a Solution Architect or Lead Architect
Proven experience in largescale cloud migration programs
Handson experience with Azure Databricks and AWS Databricks
Experience migrating enterprise Data LakesLakehouses across cloud platforms
Strong understanding of Medallion Architecture and Delta Lake
Experience with Modern Data Platforms supporting AIML workloads
Expertise in enterprise integration and largescale data ecosystems
Technical Skills
Databricks Data Engineering
Databricks Workflows
Delta Lake
Delta Sharing
Unity Catalog
Delta Live Tables
Spark PySpark
Databricks SQL
Structured Streaming
Cloud Platforms
Azure Databricks
AWS Databricks
ADLS Gen2
Amazon S3
IAM
VPC
Glue
Lambda
Redshift good to have
Data Integration
Informatica PowerCenter IDMC
Azure Data Factory
Kafka
CDC Frameworks
API Integration
Eventdriven Architectures
AI Advanced Analytics
GenAI Architecture
RAG Frameworks
Vector Databases
Feature Stores
MLflow
Databricks AIML
Azure OpenAI Amazon Bedrock
DevOps Governance
CICD
Git
Terraform
Infrastructure as Code
Data Quality Frameworks
Data Observability
Security Compliance
Preferred Qualifications
Databricks Certified Professional
AWS Solution Architect Certification
Azure Solution Architect Certification
Experience in Life Sciences Supply Chain Manufacturing or Healthcare domains
Experience leading migration programs involving 1000 datasets and multiple business applications
Key Success Metrics
Successful migration with zero business disruption
Reduction in platform complexity and technical debt
Improved performance and cost optimization
Enterprise adoption of AIready data products
Increased governance lineage and compliance coverage
Accelerated delivery of analytics and GenAI use cases
This JD is tailored for the kind of Azure Databricks AWS Databricks modernization and AIready lakehouse transformation programs reflected in your recent DPSCDL migration initiatives"
