Job SummaryThe Data Engineer will design, develop, and maintain scalable data pipelines and data platform solutions. This role requires strong expertise in Python, PySpark, SQL, Snowflake, Data Lakes, Azure Data Factory, and OpenShift. The engineer will work closely with cross-functional teams to ensure high-quality data processing, integration, and availability to support analytics, reporting, and data-driven decision-making.
Key Responsibilities- Design, build, test, and maintain scalable data pipelines and ETL/ELT workflows using Python, PySpark, and SQL.
- Manage and optimize data storage and processing using Snowflake and Data Lakes.
- Integrate data from multiple sources while ensuring data quality, integrity, and security.
- Design and implement automated data workflows using Azure Data Factory or similar orchestration tools.
- Deploy and manage containerized applications and services on OpenShift.
- Collaborate with Data Scientists, Analysts, and business stakeholders to understand data requirements and deliver solutions.
- Monitor data pipelines, troubleshoot issues, and optimize performance.
- Maintain documentation of data architecture, workflows, and best practices.
- Ensure compliance with data governance, security, and regulatory policies.
Required Qualifications- Bachelor's or Master's degree in Computer Science, Data Engineering, or a related field.
- 10+ years of overall experience, with proven experience as a Data Engineer.
- Strong proficiency in Python and PySpark.
- Expertise in SQL and experience working with Snowflake.
- Experience with Azure Data Factory or similar orchestration tools.
- Knowledge of Data Lake architectures and management.
- Experience deploying and managing containerized applications on OpenShift or Kubernetes.
- Familiarity with cloud platforms such as Azure, AWS, or GCP.
- Understanding of data security, governance, and compliance practices.
- Recent experience in the banking domain.
- Strong problem-solving abilities and attention to detail.
- Excellent communication and teamwork skills.
Preferred Qualifications- Experience with big data tools such as Kafka or Hadoop.
- Knowledge of CI/CD pipelines and DevOps best practices.
- Familiarity with additional cloud data services and engineering tools.
