Qualifications

• 5-7 years of experience with data processing and pipeline management.
• Proven experience in data lake architectures and large-scale data processing environments.
• Hands-on with AWS services like S3, EC2, EMR, and Glue.
• Expertise in building ETL pipelines tailored for analytics and reporting.
• Solid understanding of Snowflake data warehousing, including optimization techniques.
• Proficient in using Control-M and Apache Airflow for workflow automation.
• Strong programming skills in PySpark and Apache Spark using Java.

Responsibilities

• Design and develop scalable ETL pipelines for data ingestion into a centralized data lake.
• Automate data processing and aggregation workflows to enhance availability and performance.
• Manage orchestration and scheduling of data pipelines using Control-M and Apache Airflow.
• Create data transformation logic leveraging PySpark and Apache Spark (Java).
• Handle large datasets on AWS infrastructure, ensuring data quality and consistency.
• Optimize existing data pipelines for cost-effectiveness and performance.
• Collaborate with cross-functional teams to gather data requirements and insights.

Benefits

• Comprehensive health benefits including medical, dental, and vision.
• 401(k) retirement plan with company matching contributions.
• Generous paid time off and holidays.
• Professional development opportunities including training and certifications.
• Work in a collaborative environment with a focus on innovation.