Senior Data Engineer at SpotHero:SpotHero is seeking a Senior Data Engineer to join the Data Engineering squad. This squad interacts with data consumers such as Data Science, Marketing, Engineering, and Business Analysts to provide data platform solutions that meet their day-to-day needs and long term vision.
As a Senior Data Engineer, you'll focus heavily on backend application development with a focus on building reusable infrastructure services for our stakeholders to enable them to model, store, access, process, and analyze SpotHero's data. You'll also design, instantiate, observe and maintain infrastructure services, both AWS-managed and open source solutions. As a Senior Data Engineer, you'll influence the technology choices and patterns established for data-heavy workloads at SpotHero.
What will you do:- Work with our Analytics, Data Science, Marketing and other squads to understand their data storage and processing needs.
- Be a hands-on contributor to the design and implementation of our data platform solutions from the infrastructure layer up to the API.
- Build robust pipelines that make sure data is where it needs to be, when it needs to be there in a manner that scales.
- Build frameworks and tools to help others design and build their own data pipelines in a self-service manner.
- Performance testing and engineering to ensure that our systems always scale to meet our needs.
- Be a key member of the team focused on hands-on contribution to the implementation and operation of our data platform.
What you bring: - We care about your abilities, not how you gained them. You might demonstrate the capabilities below through any combination of relevant professional experience, experience in a research setting, formal education, self-guided learning, open source contributions, or public speaking / writing / teaching experience.
- You are able to design and implement high-quality software in Python.
- Experience using SQL to read and manage data.
- Experience with Airflow, Luigi, Prefect, or other ETL scheduling tools.
- You have experience provisioning and managing infrastructure with infrastructure-as-code tools (we use Terraform, but experience with similar tools like CloudFormation, Pulumi, or SaltStack is totally fine!)
- Hands-on experience using multiple data platforms and tools (e.g. Airflow, Hive, Kafka, Postgres, Redshift, S3, Spark, Trino, Airbyte), and experience deploying, monitoring, and maintaining at least one of them
- Experience designing and implementing software (pipelines, services and client libraries) that is run in Docker containers, automatically tested on a continuous integration (CI) system, and versioned in git. You have experience writing shell scripts, Makefiles, or other configuration to glue together these components.
- Ability to deploy containerized software in Kubernetes, or sufficient experience in similar technologies like Apache Mesos or Amazon ECS.
- Demonstrated experience designing and supporting technology intended to be used by other stakeholders.
- Strong ability to communicate on both business and technology subjects.
Nice to Haves:- Message driven or streaming architectures, such as those with Kafka, Spark, Flink.
- Postgres, MySQL, or other RDBMS experience.
- Redshift, Presto, or other MPP database experience.
- Experience with at least one JVM language (we use Kotlin, but Java or Scala experience works).
- Experience with cloud data pipeline services like dbt Cloud, Airbyte, or Hightouch.
- Experience using the latest AI tooling (Claude, Codex, etc.)
Technology we use:- Our Data Stack is: Our Monolith Database is Postgres and Redis for caching. We also use Redshift as our data warehouse and S3 as our data lake. The data lake is queried using Trino. We use Apache Airflow, Python, dbt, and Apache Spark for ETL. For streaming data, we use Apache Kafka managed by a vendor and use Kafka Connect, Kafka Streams, and Spark Streaming for stream processing. All machine learning work is done in Python, using the PyData ecosystem. Our analysts use Looker for internal-facing business intelligence reports, and Amazon Quicksight for external-facing data visualizations.
Seeking Candidates in: What we are offering:- Career game changer - A truly unique experience to work for a fast-growing company in a role with unlimited growth potential.
- Excellent benefits
- We cover a generous portion of Medical Premiums, 50% of Dental and Vision Premiums, company-sponsored Life Insurance, STD, and AD&D coverage, a 401(k) with match and immediate vesting, and comprehensive leave policies to meet your needs in creating space for life.
- Flexible PTO policy and outstanding work/life balance - We value and support each individual team member.
- UberEats weekly lunch stipend for in-office days
- Udemy License and Personal Learning Budget - We support the professional and personal growth of our people by providing everyone with learning resources and development opportunities.
- Annual parking stipend - Duh. We help people park!
- The opportunity to collaborate with fun, innovative, and passionate people in a casual yet highly productive atmosphere.
- Our commitment to allyship has been a central driver of how we Respect Fellow Drivers. You'll have the opportunity to be part of Employee Resource Groups, access allyship learning resources, and actively contribute to our ongoing effort of making SpotHero inclusive for all.
- Employee programs to grow and support our people such as Discovery Days for Product and Engineering, Gearing up for Aspiring Leaders, and Mentorship Program.
- Wellness program - a workplace that actively supports your physical and mental wellbeing through ongoing events, initiatives, resources, and thoughtful perks and benefits.
Compensation in Illinois:- Depending on your skillset and experience, you can expect your base salary to be between $136,000 - $153,000 as well as a discretionary bonus, and leading total rewards package.
