Expert-level skills in Apache Spark, PySpark, and SQL
Experience with Delta Lake and ETL/ELT development
Strong understanding of data governance and security
Experience in Investment Banking, Securities, or Capital Markets
Responsibilities
Design, develop, and maintain scalable data pipelines using Azure Databricks
Build and optimize ETL/ELT workflows with PySpark, SQL, and Delta Lake
Implement and manage Unity Catalog for data governance and security
Collaborate with cross-functional teams to translate business needs into solutions
Optimize Spark jobs and data pipelines for performance and cost efficiency
Ensure data quality and compliance through best practices
Support production deployments and performance tuning
Benefits
Opportunity for contract-to-hire conversion
Work onsite in Pittsburgh, PA
Collaborative Agile environment
Engage with a high-performing data engineering team
Access to professional development opportunities with preferred qualifications
Full Job Description
Data Engineer - Azure Databricks Contract-to-Hire Pittsburgh, PA - Onsite
Job ID J0726-0569
Visa : USC, GC, EAD (No Sponsorship) Position Overview
Seeking an experienced Data Engineer to join a high-performing data engineering team responsible for building modern, cloud-native data platforms on Microsoft Azure.
This role will focus on designing and implementing scalable, secure, and high-performance data solutions using Azure Databricks, Apache Spark, PySpark, SQL, Delta Lake, and Unity Catalog. The position requires strong expertise in enterprise data governance, metadata management, security, and access control within the Investment Banking, Securities, or Capital Markets domain.
The ideal candidate is passionate about cloud data engineering, enjoys solving complex data challenges, and thrives in a collaborative Agile environment.
Required Competencies
Competency Importance Proficiency Level
Apache Spark
Required
Expert
Microsoft Azure
Required
Advanced
Investment Banking and Securities
Required
Advanced
Structured Query Language - SQL
Required
Expert
Unity Catalog
Required
Advanced
Databricks
Required
Expert
Key Responsibilities
Design, develop, and maintain scalable data pipelines using Azure Databricks.
Build and optimize ETL and ELT workflows using PySpark, SQL, Apache Spark, and Delta Lake.
Implement and manage Unity Catalog for enterprise data governance, metadata management, security, and fine-grained access control.
Develop cloud-native data engineering solutions on the Microsoft Azure platform.
Collaborate with architects, business analysts, and cross-functional teams to translate business requirements into technical solutions.
Optimize Spark jobs and data pipelines for performance, scalability, and cost efficiency.
Ensure data quality, reliability, governance, and regulatory compliance through validation, monitoring, and engineering best practices.
Support production deployments, troubleshooting, root-cause analysis, and performance tuning.
Participate in code reviews, technical documentation, and continuous improvement initiatives.
Apply cloud data engineering, security, CI/CD, and DevOps best practices within Azure environments.
Required Qualifications
8+ years of overall data engineering or related technology experience preferred.
Minimum 5 years of hands-on Azure Databricks experience.
Mandatory hands-on experience implementing and managing Unity Catalog.
Expert-level experience with Apache Spark, PySpark, and SQL.
Strong experience developing enterprise-scale data engineering solutions on Microsoft Azure.
Strong experience with Delta Lake, ETL/ELT development, and data pipeline implementation.
Experience with Azure Data Factory and Azure Data Lake Storage Gen2.
Strong knowledge of the Microsoft Azure data ecosystem.
Strong understanding of data governance, metadata management, data security, and access control.
Mandatory experience within Investment Banking, Securities, or Capital Markets.
Experience using Git and implementing CI/CD practices.
Experience working in Agile/Scrum delivery environments.
Excellent analytical, communication, and problem-solving skills.
Must be willing to work onsite in Pittsburgh, PA.
Must be willing and eligible to convert to full-time employment after six months.
Preferred Qualifications
Databricks Certified Data Engineer Associate or Professional certification.
Microsoft Certified Azure Data Engineer Associate, DP-203.
Experience with Infrastructure as Code using Terraform or Bicep.
Experience implementing DevOps practices for Azure Databricks environments.
Familiarity with Microsoft Purview or Azure Purview.
