Job Summary
The AWS Data Engineer will design, develop, and implement scalable, high-performance data ingestion and transformation pipelines using AWS services and Apache Spark. The role involves integrating complex datasets from multiple sources, developing advanced SQL solutions, maintaining data quality, and building secure and reliable cloud-based data architectures. The engineer will collaborate with cross-functional teams to deliver data solutions supporting business intelligence, analytics, and operational decision-making.
Key Responsibilities
• Design and implement scalable, high-performance data ingestion and transformation pipelines using AWS Glue, Apache Spark, AWS Athena, AWS S3, AWS Lambda, and related AWS services.
• Integrate large-scale and complex datasets from diverse internal and external sources to support business intelligence, analytics, and operational decision-making.
• Ensure data pipelines are secure, repeatable, and optimized for performance and reliability within a cloud-based data architecture.
• Translate business and functional requirements into detailed technical specifications and data engineering solutions.
• Develop and maintain advanced SQL queries, stored procedures, scripts, and automated workflows to support reporting, analytics, and data validation.
• Produce clean, well-documented, and reusable code following software engineering and data processing best practices.
• Collaborate with data scientists, analysts, software engineers, and product owners to deliver integrated data solutions.
• Evaluate and improve data quality by conducting validation tests, implementing data cleansing rules, and resolving data inconsistencies.
Required Qualifications
• Hands-on experience designing and developing data ingestion and transformation pipelines.
• Strong experience with AWS Glue, Apache Spark, AWS Athena, AWS S3, and AWS Lambda.
• Experience integrating large-scale and complex datasets from multiple sources.
• Strong SQL skills, including developing queries, stored procedures, and scripts.
• Experience developing automated workflows for reporting, analytics, and data validation.
• Understanding of data quality validation, data cleansing, and data consistency.
• Ability to develop reusable, well-documented, and maintainable code.
• Experience collaborating with cross-functional technical and business teams.
