Role description

Job Summary

Seeking a Senior Specialist with 7 to 11 years of experience to design develop and maintain scalable big data solutions using the Big Data Hadoop Ecosystem Spark SparkSQL and Python

Job Description

Utilize the Big Data Hadoop Ecosystem to efficiently process and analyze large datasets Develop and optimize data workflows leveraging Hadoop Spark and SparkSQL for largescale data analytics Develop Python applications and scripts to automate data processing and integration within the Hadoop ecosystem Collaborate with crossfunctional teams to understand data requirements and deliver optimized solutions aligned with business goals Ensure data quality integrity and governance across distributed systems Integrate various data ingestion and transformation tools such as HDFS Hive Sqoop Oozie and Kafka Stay updated with emerging technologies and best practices in Hadoop Spark Python and big data frameworks Optimize system performance and troubleshoot issues related to big data infrastructure to ensure high availability and reliability

Roles and Responsibilities

Design develop and deploy scalable big data solutions using Hadoop and Spark ecosystems Implement and optimize SparkSQL and PySpark jobs to enhance performance and resource utilization Monitor and maintain Hadoop clusters for stability capacity planning and performance tuning Collaborate with data engineers analysts and stakeholders to deliver actionable insights and highquality data solutions Conduct code reviews provide technical guidance and mentor junior team members Participate actively in Agile ceremonies including sprint planning code reviews and release cycles Document technical specifications processes and best practices related to big data projects Ensure security and compliance standards are met within big data environments