5-7 years of hands-on experience with data pipeline development in Databricks.
Expertise in Databricks components like Delta Lake and medallion architecture.
Proficiency in Python, PySpark, and SQL for ETL and data manipulation tasks.
Experience with integrating Databricks and various source systems such as ERPs and APIs.
Familiarity with cloud data services, particularly Azure or AWS.
Strong background in monitoring and improving data pipeline performance.
Responsibilities
Build and maintain robust data pipelines in Databricks connecting multiple ERP systems and databases.
Ensure high-quality data ingestion and monitor pipeline performance metrics.
Troubleshoot and resolve data quality issues and pipeline failures.
Collaborate with cross-functional teams to enhance data accessibility and accuracy.
Optimize current ingestion processes as part of data modernization initiatives.
Benefits
Hands-on role in a significant enterprise data modernization effort.
Ownership of critical data connectivity and ingestion processes.
Opportunity to tackle complex engineering challenges across a decentralized network.
Support for improved analytics and AI capabilities within the organization.
Experience in a fast-paced, growth-focused work environment.
Full Job Description
THE TEAM YOU WILL BE JOINING
A leading commercial HVAC services platform with a large North American footprint, headquartered in Charlotte and operating across a highly distributed network of service businesses
A PE-backed growth environment focused on modernizing its data infrastructure and creating greater scalability, visibility, and consistency across the enterprise
A decentralized operating model with multiple locations, systems, ERP platforms, and isolated data environments
A growing data engineering team building and supporting centralized data pipelines and connectivity into the Databricks environment
A collaborative data and AI organization laying the technical foundation for enterprise reporting, analytics, and future AI-enabled business use cases
LOCATION
Charlotte, NC (Uptown) - Hybrid (3 days on-site)
WHAT THEY OFFER YOU
Opportunity to play a hands-on role in a major enterprise data modernization initiative
Direct ownership of data connectivity, ingestion, monitoring, and pipeline management within a growing Databricks environment
Chance to solve complex engineering challenges across a distributed, multi-location organization with numerous source systems
Technical work that directly supports improved reporting, analytics, and AI capabilities across the business
Exposure to a fast-moving, PE-backed organization where scalability, reliability, and practical execution are highly valued
WHY THIS ROLE IS IMPORTANT
Builds and supports reliable data pipelines that ingest information from multiple ERP systems, applications, databases, and operating locations into Databricks
Strengthens connectivity between disconnected source environments and the organization`s centralized data platform
Monitors pipeline performance, identifies failures or data-quality issues, and implements solutions that improve reliability and observability
Helps maintain current ingestion processes while supporting the transition toward a more scalable and standardized target-state architecture
Partners with data architecture, analytics, infrastructure, and business teams to ensure data is accessible, accurate, and available for downstream use
Reduces technical bottlenecks that impact enterprise reporting, analytics, and future AI initiatives
THE BACKGROUND THAT FITS
Strong hands-on experience developing, deploying, and supporting data pipelines within Databricks
Experience with Databricks technologies and concepts such as Delta Lake, notebooks, workflows, jobs, clusters, and medallion architecture
Proficiency with Python, PySpark, and SQL for data ingestion, transformation, and pipeline development
Experience connecting Databricks to a variety of source systems, including relational databases, ERP platforms, APIs, files, and on-premise environments
Understanding of ETL and ELT design patterns, data orchestration, incremental loading, and pipeline dependency management
Experience monitoring production pipelines, troubleshooting failures, and improving data quality, performance, and reliability
Familiarity with cloud-based data environments and services within Azure, AWS, or a comparable platform
Ability to work across data engineering, networking, infrastructure, MSP, and application teams to resolve complex connectivity issues
Experience operating in a distributed, multi-location, or multi-ERP business environment is strongly preferred
Background in HVAC, field services, construction, facilities services, or another decentralized service organization would be a plus
