MissionThis role is focused on building real time streaming data solutions that enable data-driven decisions on live devices and telemetry data.
While the Senior Data Engineer will support batch data processes, the primary focus is on real-time streaming use cases, data services for APIs and microservices, and pipelines that feed both operational applications and ML/AI model development in a medical device IoT environment.
Role Description As a Senior Data Engineer, you will play a key role in driving the design and architecture of data pipelines that consume real-time streaming data from connected medical devices. You will collaborate cross-functionally with business analysts, software engineers, machine learning engineers, and business users to implement technical approaches and infrastructure that support data consumption across the organization.
This is a great opportunity for someone who is passionate about data engineering, thrives in environments where you can take ownership and recommend solutions, and is a self-starter who is open to learning new tools as needed.
Key Responsibilities - Design and build batch and streaming data pipelines on Azure and Databricks, with primary focus on real-time IoT and telemetry use cases within a Medallion architecture.
- Develop ETL/ELT workflows to ingest, transform, and validate large volumes of structured and unstructured data.
- Build and maintain data services, APIs, and microservices for application, analytics, and ML/AI teams.
- Implement real-time streaming solutions using Azure Event Hubs, Azure Stream Analytics, and related Azure integration patterns, with cost-effective throughput, partitioning, and downstream delivery to Databricks.
- Optimize production Databricks pipelines using PySpark, Spark SQL, and Delta Lake, including Spark tuning for performance, reliability, and cost.
- Troubleshoot and resolve complex pipeline issues across Databricks, Azure, and on-premises systems, including root cause analysis and corrective action.
- Partner with data analysts, software engineers, ML engineers, and business stakeholders to translate requirements into technical designs and delivery priorities.
- Apply data quality, validation, and privacy-first practices, and deliver reliable pipelines through software engineering standards, documentation, testing, and CI/CD.
Outcomes - Onboard to the Azure Databricks environment and contribute to troubleshooting, stabilization, and optimization of existing batch and streaming data pipelines.
- Stand up Azure streaming ingestion for telemetry data and deliver production-ready pipelines integrated with Databricks to support API, ML, and downstream analytics consumption within the Medallion architecture.
- Design and deliver data services or consumption patterns that enable business and ML teams to access near-real-time telemetry data reliably, securely, and on a scale.
Qualifications Required Experience:
- Bachelor's degree in Computer Science, Mathematics, Engineering, or a related technical field and 6+ years of professional experience in data engineering, analytics, or warehousing
- OR Master's degree in a related technical field and 3+ years of professional experience in data engineering, analytics, or warehousing
- 5+ years of experience designing, building, and operating big data and real-time streaming pipelines across cloud and on-premises environments
- 5+ years applying DevOps and CI/CD practices to data and analytics workloads
- Production experience building data services, APIs, or microservices for downstream data consumption
Required Technical Skills:
- Strong, hands-on experience designing and building production-grade data pipelines on Databricks
- Demonstrated experience with Spark optimization and tuning in Databricks, including performance analysis, partitioning strategies, caching, shuffle optimization, and cost-aware pipeline design
- Strong experience with Azure cloud services for data engineering and streaming workloads
- Experience building data services, APIs, or microservices for data consumption.
- Data quality, validation, and privacy-aware handling for regulated or sensitive data
Preferred Technical Skills:
- Azure Event Hubs and Azure Stream Analytics for IoT and telemetry ingestion and stream processing
- Additional Azure data and integration services supporting batch and streaming architectures
- Collaboration with ML teams: feature pipelines, training data, or streaming data for model development
Additional Qualifications:
- Candidates must be authorized to work in the United States without current or future need for visa sponsorship.
- Candidates must be able to commute to our Philadelphia, PA office on Tuesdays and Wednesdays.
Benefits - Health Care Plan (Medical, Dental & Vision)
- Paid Time Off (Vacation, Sick Time Off & Holidays)
- Company Paid Short Term Disability and Life Insurance
- Retirement Plan (401k) with Company Match
