Senior Data Engineer

Less than 5 years experience  • 

Salary depends on experience
Posted on 04/20/18
New York, NY
Less than 5 years experience
Salary depends on experience
Posted on 04/20/18

As a member of this team you are expected to build innovative data pipelines for processing and analyzing our Big Data (250 billion + events per month). More specifically, we are building and migrating our enterprise data systems from a Hadoop/HDFS based system to a cloud based relational data infrastructure. This brings a unique challenge of shifting gears and developing in varied technologies including: MapReduce jobs, MYSQL stored procedures, Snowflake SQL, Pentaho Data Integration, Kinesis and custom solutions in Java and Scala.

As a Senior Data Engineer you will be responsible for:

  • Code fix and maintaining MapReduce (in Java and/or Scala) data transformation jobs
  • Building ETL pipelines which involves developing transformation widgets in Java + UI workflow
  • Building SQL Scripts for processing and analysis of data
  • Data analysis and data quality checks for completeness and accuracy 
  • Team of 4 rotates on-call duties to ensure we meet our SLAs
  • Maintaining reporting systems for the relational database 
  • Help choose technologies that best support optimal solution designs
  • Learn new technologies with the team
  • Helping us use our infrastructure in the most cost efficient way possible


ABOUT YOU

Possessing the following will help you to be successful in this role:

  • 3+ years of experience in data of all sorts (relational, big data, etc)
  • Deep SQL knowledge. You should be very familiar with SQL and understand what makes a database fast.
  • Knowledge of an OOP language and interest in learning Java and/or Scala to write MapReduce jobs and ETL scripts
  • A passion for telling stories with numbers and statistics
  • A keen eye for detecting data defects and anomalies
  • Comfortable juggling multiple technologies and high priority tasks 
  • Nice to have: experience with Distributed columnar databases like Veritca, Greenplum, Redshift, or Snowflake
  • Big Plus: experience with Hadoop / HBase administration
  • You are a self-starter, and you enjoy learning new technologies
Not the right job?
Join Ladders to find it.
With a free Ladders account, you can find the best jobs for you and be found by over 20,0000 recruiters.