Qualifications

• 5-7 years of experience in data engineering, with deep expertise in Scala and Apache Spark.
• Proficient in AWS services, specifically S3, Glue, and Step Functions.
• Hands-on experience with databases like MongoDB and familiar with handling both structured and unstructured data.
• Working knowledge of Python for scripting and data transformation tasks.
• Experience using version control systems such as Git and project management tools like Azure DevOps or JIRA.
• Familiarity with SQL and behavior-driven development (BDD) frameworks.
• Knowledge of Terraform for managing infrastructure as code is preferred.

Responsibilities

• Refactor existing Scala Spark code to facilitate migration to modern environments.
• Develop and optimize Scala Spark code to enhance performance and maintainability.
• Create thorough unit tests to maintain code quality and stability.
• Work with data formats including CSV and Parquet stored in Amazon S3.
• Integrate and manipulate data from MongoDB effectively.
• Independently establish test environments for validating code changes.
• Collaborate in an Agile team framework for sprint planning and story grooming.

Benefits

• Opportunity to work in a dynamic cloud-native environment.
• Collaborative team culture within an Agile framework.
• Access to modern tools and technologies in big data.
• Potential for professional growth and learning in cutting-edge data engineering practices.