Expertise in PySpark for developing data processing pipelines
Strong SQL skills for data manipulation and querying
Experience in building ETL/ELT pipelines from various data sources
Knowledge of data modeling techniques, particularly Star and Snowflake schemas
Familiarity with data validation frameworks and Git for version control
Responsibilities
Design and implement ETL/ELT pipelines using PySpark and SQL
Develop and maintain complex data models in Snowflake
Monitor and optimize Snowflake queries and Spark jobs
Implement automated data validation frameworks
Troubleshoot production pipeline issues and manage version control
Benefits
Opportunities for professional development and growth within the company
Access to advanced tools and technologies
Collaborative team environment
Flexible work arrangements
Comprehensive health and wellness programs
Full Job Description
Role Overview:
This role is for a Data Engineer focused on designing, implementing, and optimizing robust ETL/ELT pipelines and complex data models primarily using PySpark and Snowflake. The engineer will be responsible for ensuring high data quality, integrity, and efficient operations within the data infrastructure.
Key Responsibilities:
Design and implement robust ETL/ELT pipelines using PySpark and SQL to ingest data from diverse sources including APIs, flat files, and relational databases.
Develop and manage complex data models (e.g., Star/Snowflake schemas) and maintain Fact and Dimension tables within Snowflake.
Monitor and tune Snowflake queries and Spark jobs to optimize performance, reduce latency, and manage computational costs.
Implement automated data validation frameworks and testing procedures to ensure a "single source of truth" and high data reliability.
Troubleshoot production pipeline issues and manage version control via Git.