Lead Big Data Engineer (B2B) in San Francisco, CA

$100K - $150K(Ladders Estimates)

Glassdoor   •  

San Francisco, CA 94102

Industry: Consumer Technology


8 - 10 years

Posted 53 days ago

We are looking for a talented engineer to join our growing data engineering team. The ideal candidate has significant experience in leading a small group of engineers that build scalable data platforms that enable business intelligence, analytics, data science, and data products. You must have strong, hands-on technical expertise in a variety of technologies and the proven ability to fashion robust, scalable solutions. You should have a passion for continuous improvement quality.

We embrace a wide variety of technologies and work very closely with data scientists and business stakeholders to deliver end to end solutions. If you are interested in a fast paced environment, the latest technologies, and fun data problems come join us!


  • Design and develop big data applications using a variety of different technologies.
  • Develop logical and physical data models for big data platforms.
  • Automate workflows using Apache Airflow.
  • Write data pipelines using Apache Hive, Apache Spark, Apache Kafka.
  • Create solutions on AWS using services such as Kinesis, Lambda, and API Gateway.
  • Provide ongoing maintenance and enhancements to existing systems, and participate in rotational on-call support.
  • Learn our business domain and technology infrastructure quickly and share your knowledge freely and proactively with others in the team.
  • Mentor junior engineers on the team
  • Lead daily standups and design reviews
  • Groom and prioritize backlog using JIRA
  • Act as the point of contact for your assigned business domain

Key Qualifications

  • 7+ years of hands-on experience with developing data warehouse solutions and data products.
  • 4+ years of hands-on experience developing a distributed data processing platform with Hadoop, Hive, Spark, Airflow, Kafka, etc.
  • 2-3 years of hands-on experience in modeling and designing schema for data lakes or for RDBMS platforms.
  • Experience managing a small team of engineers
  • Experience with programming languages: Python, Java, Scala, etc.
  • Experience with scripting languages: Perl, Shell, etc.
  • Practice working with, processing, and managing large data sets (multi TB/PB scale).
  • Exposure to test driven development and automated testing frameworks.
  • Background in Scrum/Agile development methodologies.
  • Capable of delivering on multiple competing priorities with little supervision.
  • Excellent verbal and written communication skills.
  • Bachelor's Degree in computer science or equivalent experience.

Nice To Have

  • Experience building machine learning pipelines or data products.
  • Familiarity with AWS or GCS technologies.
  • Be passionate about or have contributed to open sourced engineering projects in the past.

Valid Through: 2019-10-18