Job Summary:
We are seeking a Senior Big Data Engineer with strong experience in designing, developing, and maintaining scalable data engineering solutions. The ideal candidate will have hands-on expertise in Python, Apache Spark, PySpark, Airflow, Kafka, and GCP data services. The role involves building and optimizing data pipelines, processing large datasets, integrating RESTful APIs, and implementing CI/CD practices to ensure reliable and efficient data workflows.
Key Responsibilities:
• Design, develop, and maintain scalable data pipelines using Python, PySpark, Scala, and Apache Spark.
• Develop and manage data workflows and orchestration using Apache Airflow.
• Implement data processing solutions using Spark SQL and Spark Streaming.
• Integrate Apache Kafka for real-time data ingestion and streaming.
• Write, optimize, and maintain complex SQL queries for large-scale data processing.
• Work with relational and NoSQL databases to support data storage and processing requirements.
• Develop and maintain data pipelines using GCP services, including BigQuery and Dataproc.
• Integrate RESTful APIs into data pipelines for data exchange and processing.
• Implement and maintain CI/CD pipelines using GitLab CI.
• Collaborate with cross-functional teams to deliver reliable, scalable, and high-performance data engineering solutions.
Required Qualifications:
• Strong hands-on experience with Python, PySpark, Scala, and Apache Spark.
• Experience with Apache Airflow for workflow orchestration.
• Proficiency in Spark SQL and Spark Streaming.
• Experience working with relational and NoSQL databases.
• Strong knowledge of Apache Kafka and real-time data processing.
• Ability to write and optimize complex SQL queries.
• Experience with Google Cloud Platform (GCP), particularly BigQuery and Dataproc.
• Hands-on experience integrating RESTful APIs into data pipelines.
• Experience with CI/CD implementation using GitLab CI.
Preferred Qualifications:
• Experience with backend development technologies, including Java and Spring Boot.