Senior Data Engineer

Salary depends on experience
Posted on 10/15/17
San Francisco, CA
Software
Salary depends on experience
Posted on 10/15/17

Responsibilities

  • Selecting, integrating and building any Big Data tools and frameworks required to provide requested capabilities: ex Storm, Hadoop, etc
  • Implementing ETL process
  • Monitoring performance and advising any necessary infrastructure changes
  • Defining data retention policies and capacity planning
  • Work closely with product managers, data analysts, user experience experts, and quality engineers to build new features to empower our business through data

Skills and Qualifications

  • Proficient understanding of distributed computing principles
  • Proficiency with building stream-processing systems, using solutions such as Storm or Spark-Streaming
  • Proficiency with Hadoop, MapReduce, HDFS
  • Good knowledge of Big Data querying tools, such as Pig, Hive, and Impala
  • Experience with integration of data from multiple data sources
  • Experience with NoSQL databases, such as MongoDB, Cassandra, HBase
  • Strong SQL knowledge
  • Knowledge of various ETL techniques and frameworks, such as Flume
  • Experience with various messaging systems, such as Kafka or RabbitMQ
  • BS or MS in Computer Science, Computer Engineering or other related discipline

Nice to Have:

  • Experience with Big Data ML toolkits, such as Mahout, SparkML or H2O
  • Good understanding of Lambda Architecture, along with its advantages and drawbacks
  • Experience with Cloudera/MapR/Hortonworks
  • Management of Hadoop, Storm clusters with all included services
  • Ability to solve any ongoing issues with operating the cluster.
Not the right job?
Join Ladders to find it.
With a free Ladders account, you can find the best jobs for you and be found by over 20,0000 recruiters.