Job Summary
We are seeking a highly motivated Data Engineer (ETL) to design, develop, and maintain scalable data engineering solutions supporting enterprise data aggregation and integration initiatives. This role focuses on building high-performance ETL pipelines, developing Apache Spark applications, processing large-scale datasets, and leveraging cloud-native technologies to deliver reliable and efficient data solutions. The ideal candidate will have strong expertise in Spark, Scala, Java, cloud platforms, and big data technologies while collaborating with cross-functional teams in an Agile environment.
Key Responsibilities
Design, develop, and maintain scalable ETL pipelines for enterprise data integration and movement.
Develop, test, deploy, and support ETL batch jobs using Apache Spark and Amazon EMR.
Build and maintain Spark services for large-scale data processing.
Schedule, monitor, troubleshoot, and optimize Spark batch processing jobs.
Develop Java clients for consuming REST and SOAP APIs.
Develop and maintain Scala-based Spark batch applications.
Process and transform large datasets using Apache Spark.
Store and manage processed data in NoSQL databases such as Cassandra or Yugabyte.
Design and implement cloud-based data warehouse and data lake solutions using Snowflake.
Work with large-scale data storage formats including Parquet and HDF5.
Develop cloud-native data engineering solutions using AWS services.
Collaborate with cross-functional teams to design innovative data solutions that address business requirements.
Participate in code reviews, pair programming, and technical discussions to ensure code quality and best practices.
Support the full Software Development Life Cycle (SDLC), including analysis, design, development, testing, deployment, and maintenance.
Troubleshoot production issues and optimize data processing performance.
Identify opportunities for innovation and continuous improvement in data engineering practices.
Collaborate across multiple Agile teams to deliver scalable and reliable data solutions.
Apply a customer-first mindset when designing and delivering enterprise data solutions.
Required Qualifications
Bachelor's degree in Computer Science, Information Systems, or a related field.
Proven experience in Data Engineering.
10+ years of experience developing Spark services for large-scale data movement.
Experience scheduling, monitoring, debugging, and optimizing ETL Spark batch processes.
Hands-on experience developing Java clients for REST and SOAP API integrations.
Experience developing Scala Spark batch applications.
Experience developing, deploying, and maintaining Apache Spark and Amazon EMR ETL jobs.
Experience processing big data using Apache Spark.
Experience with NoSQL databases such as Cassandra or Yugabyte.
Experience with cloud-based data warehouse and data lake platforms such as Snowflake.
Experience managing large datasets using storage formats such as Parquet and HDF5.
Experience developing cloud-native solutions using AWS services.
Strong understanding of software development lifecycle (SDLC) methodologies.
Strong analytical, troubleshooting, and problem-solving skills.
Experience working in Agile software development environments.
Excellent collaboration and communication skills.
Preferred Qualifications
Experience designing enterprise-scale data integration and aggregation solutions.
Experience supporting external-facing data platforms and APIs.
Experience working with financial services or FinTech data platforms.
Passion for continuous learning and adopting modern data engineering best practices.
