Databricks Data Engineer with a Solution Architect

Prophecy Technologies

$100K — $130K *
Enterprise Technology
8 - 10 years of experience
Job Overview by Ladders

Qualifications

  • 7+ years of experience in enterprise data engineering, with a preference for healthcare data integration.
  • Strong expertise in Databricks, Spark, Delta Lake, and cloud-native architectures.
  • Proficient in Python, SQL, and PySpark, with a focus on data modeling and ETL/ELT design patterns.
  • Experience in implementing Infrastructure as Code (IaC) using Docker and Azure Kubernetes Service (AKS).
  • Knowledge of performance optimization and scalability techniques.

Responsibilities

  • Designs and governs large-scale enterprise data platforms using advanced Databricks and cloud-native architectures.
  • Leads architectural decisions for cloud data and analytics solutions, ensuring scalability and security across clouds such as Azure, AWS, or GCP.
  • Establishes and evolves standards and best practices for data engineering and AI/ML workloads.
  • Recommends innovative technologies and optimization strategies to enhance platform performance and cost-efficiency.
  • Provides technical leadership and support for the adoption of the Databricks Lakehouse Platform across various business segments.

Benefits

  • Opportunities for professional development and continuing education.
  • Access to the latest tools and technologies for data engineering.
  • Flexible work arrangements to promote work-life balance.
  • Supportive team culture fostering innovation and collaboration.
  • Health and wellness programs aimed at maintaining a healthy lifestyle.
Full Job Description
Job Description:As a Databricks Data Engineer with a Solution Architect background, you will lead the design of enterprise-scale data solutions, establish engineering standards and best practices, optimize platform performance and cost, and provide technical leadership for cloud modernization, real-time analytics, AI/ML enablement, and data governance initiatives. This role requires strong expertise in Spark, Delta Lake, Unity Catalog, cloud architecture, Infrastructure as Code (IaC), and modern data platform engineering principles to deliver secure, resilient, and highly scalable solutions. We are seeking a highly skilled and experienced Senior Solution Architect to lead the implementation and adoption of the Databricks Lakehouse Platform as part of our enterprise data strategy. This role will serve as the technical leader and subject matter expert for Databricks, supporting multiple business segments and strategic initiatives across the organization Roles & Responsibilities Responsibilities • Applies advanced expertise in Databricks, Spark, Delta Lake, and cloud-native architectures to design and govern large-scale enterprise data platforms. • Leads architecture decisions for cloud-based data and analytics solutions, ensuring scalability, security, resiliency, and operational excellence across Azure, AWS, or GCP environments. • Establishes and evolves platform standards, best practices, and governance frameworks for data engineering, streaming, AI/ML, and analytics workloads. • Recommends innovative technologies, architectural patterns, and optimization strategies to Role Descriptions: Skills - Databricks (Python and SQLPySpark) - Azure SQL Server (Managed Instance| Azure SQL DB| SQL Server VMs) - Data modeling and ETLELT design patterns - Docker and Azure Kubernetes Service (AKS) for select automation - Performance optimization and scalabilityExperience - 7 years enterprise data engineering - Healthcare data integration experience strongly preferred - Experience with large-scale data harmonization and normalization Essential Skills: Skills - Databricks (Python and SQLPySpark) - Azure SQL Server (Managed Instance| Azure SQL DB| SQL Server VMs) - Data modeling and ETLELT design patterns - Docker and Azure Kubernetes Service (AKS) for select automation - Performance optimization and scalabilityExperience - 7 years enterprise data engineering - Healthcare data integration experience strongly preferred - Experience with large-scale data harmonization and normalization Desirable Skills: Keyword: Skills: Digital : Python~Azure Data Factory Experience Required: 8-10

Similar Jobs

More Jobs at Prophecy Technologies

More Enterprise Technology Jobs

Find similar Databricks Data Engineer with a Solution Architect jobs: