About the RoleAs a Lead Data Engineer, you will drive the design and evolution of the company's data platform, building scalable data pipelines and robust Snowflake-based solutions. Working closely with cross-functional teams, you will help deliver reliable, high-quality data that powers business decisions and supports the company's growth.
What You'll Do- Lead the design, development, and optimization of scalable data platforms and pipelines.
- Design, build, and maintain production-grade ETL/ELT workflows for batch and near real-time data processing.
- Drive the migration and modernization of data assets from BigQuery and other analytical platforms into Snowflake.
- Develop robust integrations with enterprise applications, APIs, and external data providers.
- Build and optimize Snowflake data models, schemas, Snowpipe ingestion processes, and query performance.
- Collaborate with business stakeholders, analysts, and engineering teams to translate business requirements into scalable data solutions.
- Implement data quality, validation, monitoring, and observability frameworks.
- Establish and promote best practices for data governance, lineage tracking, metadata management, and security.
- Support both batch and streaming data architectures using technologies such as Kafka, Event Hub, or equivalent.
- Mentor other engineers and contribute to architectural decisions, technical standards, and engineering best practices.
Ensure data platforms meet performance, reliability, scalability, and regulatory requirements.
Required Skills - 7+ years of experience in data engineering within cloud or hybrid environments.
- Strong experience designing, building, and maintaining production-scale ETL/ELT pipelines.
- Advanced SQL skills and proficiency with data transformation techniques.
- Strong Python programming skills for data processing and automation.
- Hands-on experience with Snowflake, including:
- Data modeling
- Snowpipe
- Query optimization
- Schema design
- Experience migrating from BigQuery or similar columnar data warehouse technologies.
- Experience with batch and streaming data processing platforms such as Kafka, Event Hub, or equivalent.
- Experience building enterprise-grade API and source-system integrations.
- Experience implementing data quality and validation frameworks.
- Good understanding of:
- Data governance
- Data lineage
- Access control and security best practices
- Strong problem-solving and communication skills.
- Ability to work effectively in a collaborative, cross-functional environment.
Nice to Have - Experience with Databricks, including:
- Delta Lake
- Unity Catalog
- Spark-based pipelines
- Experience with Spark, Scala, or PySpark.
- Familiarity with Tableau.
- Understanding of HIPAA and HITRUST data handling practices.
- Experience with dbt (data build tool) for transformation layer management.
- Experience working within healthcare, insurance, or regulated industries.
Working Conditions- Full-time, 40h/week
- US-based - US citizenship is required
- Remote
- Contract or B2B arrangement
