2-4 years of experience in production data pipelines, ideally in high-volume consumer or entertainment sectors
Proficient in Python and SQL, with experience in Apache Airflow or similar tools
Skilled in managing pipelines that process millions of records daily, including handling complex third-party APIs
Strong communicator, able to simplify technical concepts for diverse teams
Bachelor's degree in Computer Science, Data Engineering, or equivalent experience
Adaptable in fast-paced startup environments with shifting priorities
Bonus: Familiarity with music industry data and streaming APIs
Responsibilities
Build and manage Python ingestion pipelines for diverse data sources
Handle extensive datasets for millions of artists, tracks, and playlists
Model and serve data using ClickHouse and Snowflake, transitioning pipelines to S3
Create and maintain data deliveries for enterprise clients like major labels
Ensure data accuracy through monitoring and quality checks
Collaborate with engineers, product teams, and data scientists on new data products
Work in a hybrid model, attending the San Mateo or New York office 3-4 days a week
Benefits
Competitive salary and equity package
Comprehensive health, dental, and vision insurance
Ownership of impactful data pipelines
Access to advanced tools and technologies
Full Job Description
What you'll do
Build and run Python ingestion pipelines that pull data from streaming platforms, social networks, and third-party data vendors, orchestrated in Apache Airflow
Work with data for 10M+ artists, 100M+ tracks, and 15M+ playlists, plus fast-growing datasets for creators, brands, and athletes
Model and serve data in ClickHouse and Snowflake, and help move our pipelines from Snowflake to S3 and ClickHouse
Build and maintain data shares and deliveries for enterprise customers, including major labels and publishers
Keep data fresh and correct: data monitoring, data quality checks, etc.
Work closely with senior engineers, product, and data science on new data products, using AI coding tools as a normal part of your workflow
This position is hybrid, in our San Mateo or New York office 3-4 days a week
Qualifications
2-4 years of experience building production data pipelines, preferably in high-volume consumer or entertainment industries
Strong Python and SQL, with hands-on experience in Apache Airflow (or a similar orchestrator), a columnar warehouse such as ClickHouse or Snowflake, and AWS
Experience building and running pipelines that handle millions of records a day, including messy third-party APIs (rate limits, retries, changing schemas)
Strong communication skills with ability to translate complex technical concepts to cross-functional stakeholders
Bachelor's degree in Computer Science, Data Engineering, or equivalent practical experience
Thrives in dynamic startup environments with rapidly evolving requirements and tight delivery timelines
Bonus: Experience with music industry data, streaming APIs, or social media analytics platforms
What We Offer
Competitive salary and equity package
Comprehensive health, dental, and vision insurance
Real ownership of pipelines that customers use every day
Access to cutting-edge tools and technologies
What We Offer
We believe that great developer experiences lead to better products and happier teams. Our Data Engineering team operates as enablers and force multipliers, working collaboratively to remove friction from the development process. We value automation, measurement, and continuous improvement, always asking "how can we make this better for our engineering team and ensure quality data to form insights?"
