Job Summary
We are seeking a Data Engineer to join a team focused on building and supporting a Python-based, AWS-native data ingestion platform. The platform is responsible for moving and transforming critical financial data from source systems into third-party applications while maintaining compliance, security, and data integrity throughout the data lifecycle. The Data Engineer will develop, maintain, and enhance data pipelines across multiple processing stages while collaborating with engineering and platform teams to deliver scalable, secure, reliable, and high-performing data solutions within a cloud-native AWS environment.
Key Responsibilities
• Design, develop, and maintain data pipeline components using Python and AWS-native services.
• Build and support data ingestion, validation, tokenization, reformatting, and publishing workflows.
• Develop and maintain infrastructure using AWS CDK and Infrastructure as Code practices.
• Support and enhance distributed data processing solutions using AWS Glue, Spark, Lambda, ECS, and Flink.
• Create and execute comprehensive unit, integration, system, and end-to-end testing strategies to ensure platform reliability.
• Participate in system testing and end-to-end validation across multiple environments.
• Monitor, troubleshoot, and remediate production issues across data pipelines and supporting infrastructure.
• Ensure compliance with enterprise data protection standards, including sensitive data handling and governance requirements.
• Address security findings, vulnerability remediation, and dependency management across the platform ecosystem.
• Collaborate with cross-functional teams to deliver scalable, maintainable, and high-performing data solutions.
• Contribute to CI/CD processes and software engineering best practices throughout the development lifecycle.
Required Qualifications
• Experience with Python development in enterprise-scale environments.
• Hands-on experience with AWS services, including Lambda, ECS, Kinesis, S3, DynamoDB, IAM, and AWS CDK.
• Proficiency with Spark and AWS Glue for large-scale data processing.
• Experience with Git, source control management, and CI/CD pipelines.
• Experience with Databricks.
• Experience building and supporting cloud-native data pipelines and data ingestion frameworks.
• Experience developing and maintaining Infrastructure as Code solutions.
• Strong understanding of data validation, transformation, and data quality practices.
• Knowledge of secure software development practices and vulnerability remediation.
• Experience supporting end-to-end testing, system testing, and production deployments.
• Strong problem-solving and troubleshooting skills in distributed systems environments.
Preferred Qualifications
• Experience working with financial services or highly regulated data environments.
• Experience supporting sensitive data handling, tokenization, or data governance initiatives.
• Familiarity with Apache Flink and real-time stream processing architectures.
• Experience building event-driven architectures using AWS services.
• Knowledge of containerized application deployments and orchestration platforms.
• Experience with automated testing frameworks and test-driven development practices.
• AWS Solutions Architect Associate or AWS Developer Associate certification.
• Experience working within Agile software development teams.
