Data Engineer

Magnetic   •  

New York, NY

5 - 7 years

Posted 244 days ago

This job is no longer available.

As a Data Engineer, you will work with product management and data science teams to build resilient smart products that make informed decisions in real time based on billions of data records. 

Responsibilities and Duties

  • Build complex batch and real time data pipelines that massages multiple Terabytes of heterogeneous data to fuel our data science team.
  • Build components that uses advance data science and machine learning.
  • Leverage best practices in continuous integration and delivery.
  • Work in cross functional agile teams to continuously experiment, iterate and deliver on new product objectives.
  • Be part of the engineering team of the company and help shape our engineering culture, values, and ways of working.

Qualifications and Skills

  • Proven track record of 4+ years of experience in software development, a part of which was gained in a high-throughput, decision-automation related environment
  • 2+ years of experience in manipulating big data using technologies like Spark, Kafka, Flink, Hadoop and Impala.
  • Experience in developing, deploying or managing data pipelines for supporting data-science-driven decisioning at scale.
  • Experience with AWS or similar cloud provider.
  • Producing high quality code in Python, Java and willing to learn new languages and frameworks.
  • Passionate about testing, and with experience in Agile teams using SCRUM you consider automated build and test to be the norm.
  • An independent thinker who considers the operating context of what he/she is developing
  • Believes that the best data pipelines run unattended for weeks and months on end.
  • Well-versed in (or contributes to) data-centric open source projects.
  • Reads Hacker News, blogs, or stays on top of emerging tools in some other way
  • Proven ability in a high-performance, collaborative environment
  • Experience with large consumer data sets used in performance marketing is a major advantage.