Job Summary

We are seeking an experienced Lead Streaming Data Engineer / Technical Lead to design, develop, and lead the implementation of large-scale real-time data processing platforms. The ideal candidate will have strong expertise in Apache Spark, AWS Flink, and Apache Kafka Streaming, along with proficiency in Python or Java. The role requires hands-on experience developing Spark and Python data streaming pipelines, as well as technical leadership, architecture decision-making, mentoring, and successful delivery of streaming data solutions.

Key Responsibilities
• Lead the architecture, design, and implementation of real-time data processing pipelines.
• Build scalable and fault-tolerant streaming applications using Apache Spark Structured Streaming, AWS Flink, and Apache Kafka.
• Develop and maintain real-time streaming pipelines using Spark and Python.
• Apply distributed data processing principles to build reliable and scalable streaming solutions.
• Define technical standards, best practices, and coding guidelines for the team.
• Make architecture and technical design decisions for large-scale streaming data solutions.
• Collaborate with business stakeholders, architects, product owners, and engineering teams to understand requirements and deliver effective solutions.
• Optimize streaming applications for performance, reliability, scalability, and cost efficiency.
• Drive solution design reviews, code reviews, and production readiness assessments.
• Mentor and guide development teams across onshore and offshore locations.
• Troubleshoot complex production issues and provide technical leadership during critical incidents.
• Ensure adherence to security, governance, and compliance requirements.
• Support the successful implementation and delivery of enterprise-scale real-time streaming and distributed data processing solutions.

Required Qualifications
• 10+ years of overall professional experience.
• 5+ years of experience working with real-time streaming and distributed data processing solutions.
• Strong hands-on experience with Apache Spark Structured Streaming.
• Strong hands-on experience with AWS Flink.
• Strong hands-on experience with Apache Kafka.
• Experience developing Spark and Python data streaming pipelines.
• Strong proficiency in Python or Java.
• Experience designing and building scalable and fault-tolerant streaming applications.
• Strong understanding of real-time data processing and distributed data processing architectures.
• Experience making architecture and technical design decisions.
• Experience providing technical leadership to engineering teams.
• Strong troubleshooting and problem-solving skills in production environments.
• Experience mentoring and guiding development teams across onshore and offshore locations.
• Understanding of security, governance, and compliance requirements for enterprise data platforms.

Preferred Qualifications
• Experience leading large-scale real-time data processing platform implementations.
• Strong experience optimizing streaming workloads for performance, reliability, scalability, and cost efficiency.
• Experience conducting solution design reviews, code reviews, and production readiness assessments.
• Experience working with cross-functional teams including business stakeholders, architects, product owners, and engineering teams.