10+ years of overall experience in data architecture/data engineering roles
5+ years of architecture experience
Strong hands-on experience with Azure-native services
Extensive experience with Azure Databricks
Proficiency in Python/PySpark and Spark SQL
Responsibilities
Design and implement end-to-end data architecture using Azure-native services and Databricks
Build scalable, high-performance data platforms
Partner with stakeholders to translate business requirements into data solutions
Implement data governance principles using Unity Catalog
Create data ingestion and integration patterns using Medallion Architecture
Develop PySpark notebooks for data transformation
Ensure governance, security, scalability, and compliance across the platform
Benefits
Flexible work arrangements including part-time and remote options
Support for employees returning after health or family-related absences
Commitment to diversity and inclusion in the workplace
Full Job Description
Job Description Overview :
Job Title: Sr. Data Architect - Azure Databricks
Location: REMOTE Hybrid
Key Responsibilities
Design and implement end-to-end data architecture / data ecosystem using Azure-native services and Databricks
Build scalable, high-performance data platforms and robust data frameworks
Partner with stakeholders to translate business requirements into reliable data solutions
Implement data governance principles using Unity Catalog
Create and implement data ingestion and data integration patterns using Medallion Architecture
Apply best practices in data engineering
Develop PySpark notebooks
Create procedures for advanced data transformation
Handle performance tuning in Databricks and Azure SQL DBs
Architect end-to-end data solutions supporting:
Yield improvement
Process control
Predictive maintenance
Operational efficiency
Ensure strict governance, security, scalability, and compliance across the platform
Required Experience
10+ years of overall experience and 5+ years of architecture experience in data architecture / data engineering roles
Strong hands-on experience with major enterprise data platforms
Extensive experience with Azure-native services including:
Azure Data Lake Storage (ADLS)
Azure Data Factory (ADF)
Azure Synapse Analytics
Azure SQL
Strong hands-on experience with Azure Databricks
Required Databricks Expertise
Should have worked with the latest Databricks capabilities including:
Unity Catalog:
Catalog and schema design
Permissions and security
Managed and external tables
Lineage and governance
Delta Live Tables (DLT):
Reliable pipeline development
CDC implementation
Data transformations
Data quality controls
Scaling and performance tuning
Other modern Databricks capabilities and best practices
Data Ingestion & ELT Experience
Experience with data ingestion tools such as:
Fivetran
HVR or equivalent CDC / replication tools
Experience with ELT tools such as:
DBT
Familiarity with:
Real-time / streaming data
Batch processing
Sensor / OT / ET data flows
Requirements
Technical Skills
Strong expertise in data ingestion, transformation, and orchestration frameworks
Hands-on experience building insurance-specific data models
Proficiency in Python / PySpark and Spark SQL
Strong understanding of streaming vs batch processing
Experience with Spark job performance tuning
Ability to optimize:
Data storage formats
Partitioning strategies
Schema design
Experience working with:
Object stores
Data lakes
Cloud compute scaling
Databricks cluster management
Governance, Security & Operations
Deep understanding of:
Data governance
Security and compliance
RBAC and attribute-based access control
Encryption and audit logs
Data privacy requirements
Experience with operational excellence practices:
Monitoring and observability
Reliability and metrics
Backup, failover, and disaster recovery strategies
Preferred Qualifications
Prior experience working on Insurance P&C cloud-based data and analytics projects
Strong analytical and problem-solving skills
Experience defining and enforcing best practices for Databricks workloads and Spark-based pipelines
Solid understanding of distributed data processing concepts
Strong communication and stakeholder management skills
Ability to document architecture decisions, trade-offs, and technical recommendations
Benefits
Diversity Inclusion:Exavalu also promotes flexibility, adapting to the needs of employees, customers, and the business. It might be part-time work, working outside normal 9-5 business hours or working remotely. We also offer a welcome back program to help individuals return to the mainstream after a prolonged absence due to health or family reasons.
