Job Summary
We are seeking an AWS Data Engineer with 10+ years of experience designing, developing, and supporting AWS-based data pipelines for enterprise data onboarding, transformation, and consumption. The role involves hands-on development using Python/PySpark and AWS-native data services to build reliable, scalable, and analytics/AI-ready datasets.
Key Responsibilities
• Develop data ingestion and transformation pipelines using AWS Glue, EMR, S3, Python, and PySpark.
• Ingest data from legacy and enterprise source systems into AWS.
• Develop transformations to create standardized, consumption-ready datasets.
• Build and maintain ETL/ELT workflows supporting enterprise data domains.
• Work with Amazon S3 for data storage and processing.
• Develop workloads using AWS Glue and EMR.
• Develop and optimize analytical datasets and workloads involving Amazon Redshift and Athena.
• Support Iceberg-based data structures on S3 where required.
• Work with other AWS data services, such as DynamoDB, when appropriate to the solution.
• Implement data validation, reconciliation, and quality checks within pipelines.
• Identify and resolve data-processing failures and data-quality issues.
• Optimize PySpark/SQL workloads and data pipelines for performance.
• Participate in troubleshooting and production support.
• Develop clean, maintainable, and reusable Python/PySpark code.
• Perform unit and integration testing.
• Participate in peer code reviews.
• Document data flows, mappings, transformations, and technical designs.
• Work with the Data Engineering Lead and client stakeholders to understand source data and transformation requirements.
Required Qualifications
• 10+ years of experience in data engineering, including designing, developing, and supporting AWS-based data pipelines.
• Hands-on experience with Python and PySpark.
• Experience with AWS Glue, EMR, and S3 for data ingestion, transformation, storage, and processing.
• Experience developing and maintaining ETL/ELT workflows.
• Experience with Amazon Redshift and Athena for analytical datasets and workloads.
• Knowledge of Apache Iceberg-based data structures.
• Experience implementing data validation, reconciliation, and data quality checks.
• Experience optimizing PySpark/SQL workloads and troubleshooting data pipelines.
• Knowledge of unit testing, integration testing, and peer code review practices.
• Ability to document technical designs, data flows, mappings, and transformations.
• Strong collaboration skills and ability to work with technical leads and client stakeholders.
