Bachelor's degree in Computer Science or related field, or equivalent experience.
3 years of experience as a Data Engineer focusing on Kafka and Spark Streaming.
Proficiency in building real-time and low-latency data pipelines using event streaming technologies.
Strong understanding of distributed systems and streaming data architectures.
Experience with processing and transforming streaming data using Spark in production.
Responsibilities
Develop real-time data pipelines with Kafka for event streaming and Spark for processing.
Design and manage Kafka topics, producers, and consumers for low-latency data flows.
Implement scalable architectures for high-velocity data processing.
Collaborate with data teams for integration of streaming data into analytics and ML workflows.
Optimize Spark jobs for performance and reliability in real-time environments.
Monitor and troubleshoot streaming pipelines to ensure data integrity.
Ensure system availability for robust streaming solutions.
Benefits
Competitive salary and comprehensive benefits package including healthcare, PTO, and 401k.
Opportunities for professional growth and upskilling in AI and cloud technologies.
Full Job Description
Overview:
Streaming Data Engineer (Kafka & Spark)
Location: Alpharetta, GA (willing to travel to client locations)
Employment Type: Full-Time (W2)
Role Overview
We are seeking a skilled Streaming Data Engineer to build real-time data pipelines using Kafka and Spark. This role focuses on designing low-latency event streaming solutions to enable rapid data processing and analytics.
Key Responsibilities
Develop real-time data pipelines using Kafka for event streaming and Spark Streaming or Structured Streaming for processing.
Design and manage Kafka topics, producers, and consumers to ensure low-latency data flows.
Implement scalable streaming architectures to handle high-velocity data with minimal latency.
Collaborate with data teams to integrate streaming data into analytics and machine learning workflows.
Optimize Spark jobs for performance and reliability in real-time processing environments.
Monitor and troubleshoot streaming pipelines to ensure data integrity and system availability.
Required Qualifications
Bachelor's degree in Computer Science, Software Engineering, or a related field (or equivalent experience).
3 years of experience as a Data Engineer with a focus on Kafka and Spark Streaming or Structured Streaming.
Proficiency in building real-time and low-latency data pipelines using event streaming technologies.
Strong understanding of distributed systems and streaming data architectures.
Experience with Spark for processing and transforming streaming data in production.
Preferred Qualifications
Familiarity with cloud-based streaming services like AWS Kinesis or Azure Event Hubs.
Exposure to advanced Kafka configurations for fault tolerance and scalability.
Knowledge of monitoring tools like Confluent Control Center or Prometheus for streaming pipelines.
Compensation & Benefits
Competitive salary and comprehensive benefits package (healthcare, PTO, 401k).
Opportunities for professional growth and upskilling in AI and cloud technologies.
