Overview:Streaming Data Engineer (Kafka & Spark)Location: Alpharetta, GA (willing to travel to client locations)
Employment Type: Full-Time (W2)
Role OverviewWe are seeking a skilled Streaming Data Engineer to build real-time data pipelines using Kafka and Spark. This role focuses on designing low-latency event streaming solutions to enable rapid data processing and analytics.
Key Responsibilities- Develop real-time data pipelines using Kafka for event streaming and Spark Streaming or Structured Streaming for processing.
- Design and manage Kafka topics, producers, and consumers to ensure low-latency data flows.
- Implement scalable streaming architectures to handle high-velocity data with minimal latency.
- Collaborate with data teams to integrate streaming data into analytics and machine learning workflows.
- Optimize Spark jobs for performance and reliability in real-time processing environments.
- Monitor and troubleshoot streaming pipelines to ensure data integrity and system availability.
Required Qualifications- Bachelor's degree in Computer Science, Software Engineering, or a related field (or equivalent experience).
- 3 years of experience as a Data Engineer with a focus on Kafka and Spark Streaming or Structured Streaming.
- Proficiency in building real-time and low-latency data pipelines using event streaming technologies.
- Strong understanding of distributed systems and streaming data architectures.
- Experience with Spark for processing and transforming streaming data in production.
Preferred Qualifications- Familiarity with cloud-based streaming services like AWS Kinesis or Azure Event Hubs.
- Exposure to advanced Kafka configurations for fault tolerance and scalability.
- Knowledge of monitoring tools like Confluent Control Center or Prometheus for streaming pipelines.
Compensation & Benefits- Competitive salary and comprehensive benefits package (healthcare, PTO, 401k).
- Opportunities for professional growth and upskilling in AI and cloud technologies.
Skills:Data Engineer, Kafka, Event Streaming, Spark Streaming, Structured Streaming, Real-Time, Low Latency
