Role Overview:We are seeking a highly skilled Data Streaming Engineer with strong expertise in Apache Flink and Apache Spark to design, develop, and optimize large-scale real-time and batch data processing solutions. The ideal candidate will have hands-on experience building high-performance streaming pipelines, event-driven architectures, and enterprise-grade data platforms.
Key Responsibilities:- Design, develop, and maintain scalable real-time data processing pipelines.
- Build streaming applications using Apache Flink and Spark Structured Streaming.
- Develop batch and near real-time ETL workflows.
- Process and analyze large-scale datasets with low latency and high throughput.
- Optimize data pipelines for performance, reliability, and scalability.
- Collaborate with Data Engineers, Architects, and Business Stakeholders to deliver data solutions.
- Implement monitoring, logging, and troubleshooting for data processing applications.
- Ensure data quality, governance, and security standards are followed.
Required Skills:- Apache Flink:
- Strong hands-on experience in Apache Flink DataStream API.
- Expertise in Apache Flink Table API & SQL API.
- Experience designing and implementing: Stateful stream processing, Event-time processing, Windowing operations, Checkpointing and fault tolerance, Watermarks and late event handling.
- Knowledge of Flink deployment and operations in distributed environments.
- Apache Spark:
- Strong proficiency in: Spark Core, Spark SQL, Structured Streaming.
- Experience developing high-volume ETL and data transformation pipelines.
- Performance tuning and optimization of Spark jobs.
- Knowledge of Spark execution architecture and resource management.
Qualifications:
- 8-10 years of experience.
Preferred Skills:
- Experience with Kafka, Event Hubs, or other messaging/streaming platforms.
- Knowledge of distributed systems and microservices architecture.
- Experience with cloud platforms (Azure, AWS, or GCP).
- Familiarity with containerization technologies such as Docker and Kubernetes.
- Experience with CI/CD pipelines and DevOps practices.
- Proficiency in Java, Scala, or Python.
