Senior Data Engineer

Talkdesk   •  

San Francisco, CA

Industry: Software

  •  

Not Specified years

Posted 372 days ago

Responsibilities

  • Selecting, integrating and building any Big Data tools and frameworks required to provide requested capabilities: ex Storm, Hadoop, etc
  • Implementing ETL process
  • Monitoring performance and advising any necessary infrastructure changes
  • Defining data retention policies and capacity planning
  • Work closely with product managers, data analysts, user experience experts, and quality engineers to build new features to empower our business through data

Skills and Qualifications

  • Proficient understanding of distributed computing principles
  • Proficiency with building stream-processing systems, using solutions such as Storm or Spark-Streaming
  • Proficiency with Hadoop, MapReduce, HDFS
  • Good knowledge of Big Data querying tools, such as Pig, Hive, and Impala
  • Experience with integration of data from multiple data sources
  • Experience with NoSQL databases, such as MongoDB, Cassandra, HBase
  • Strong SQL knowledge
  • Knowledge of various ETL techniques and frameworks, such as Flume
  • Experience with various messaging systems, such as Kafka or RabbitMQ
  • BS or MS in Computer Science, Computer Engineering or other related discipline

Nice to Have:

  • Experience with Big Data ML toolkits, such as Mahout, SparkML or H2O
  • Good understanding of Lambda Architecture, along with its advantages and drawbacks
  • Experience with Cloudera/MapR/Hortonworks
  • Management of Hadoop, Storm clusters with all included services
  • Ability to solve any ongoing issues with operating the cluster.