Proficient in Python and PySpark for data processing
Strong SQL skills for database management
Hands-on experience with AWS services, especially AWS Glue and Lambda
Demonstrated ability to build and maintain data pipelines
Responsibilities
Develop and maintain robust data pipelines using Python, PySpark, and SQL
Contribute to the creation of a new data environment from inception
Design and implement scalable data engineering solutions leveraging AWS services
Develop and support efficient data processing workflows with AWS Glue and Lambda
Optimize data pipelines to meet analytics and processing needs
Benefits
Opportunity to work on cutting-edge data engineering projects
Collaborative environment focused on innovation
Access to professional development and training resources
Flexible work arrangements to support work-life balance
Engagement in impactful financial data and analytics initiatives
Full Job Description
Job Summary The Databricks AWS Data Engineer will develop and support a new data environment from the ground up, focusing primarily on data pipelines and cloud-based data engineering solutions. The role requires strong development experience with Python, PySpark, and SQL, along with hands-on AWS expertise, particularly AWS Glue and Lambda. The engineer will contribute to building scalable data solutions that support financial data and analytics initiatives.
Key Responsibilities • Develop and maintain data pipelines using Python, PySpark, and SQL. • Contribute to building a new data environment from the ground up. • Design and implement data engineering solutions using AWS services. • Develop and support data processing workflows using AWS Glue and AWS Lambda. • Build, maintain, and optimize data pipelines to support data processing and analytics requirements.
Required Qualifications • Strong hands-on experience with Python and PySpark. • Strong SQL development skills. • Mandatory experience with AWS services, particularly AWS Glue and AWS Lambda. • Experience developing and maintaining data pipelines. • Ability to participate in technical discussions and whiteboarding exercises.
