The Data Engineering team at Poshmark is looking for an experienced software engineer to scale Data lake, ensuring real-time access to quality data for all the stakeholders. The role provides opportunity to showcase in-depth software development skills to build and maintain real-time and batch data pipelines with a focus on scalability and optimizations while collaborating with Data Science, Analytics, and Platform Engineering teams to build tools to access petabyte scale data.
Responsibilities:
- Build highly scalable, available, fault-tolerant data processing systems using AWS technologies, MapReduce, Hive, Kafka, Spark, Flink and other big data technologies to handle batch and real-time data processing within 10s for terabytes of data ingested every day and a petabyte-sized data warehouse.
- Responsible for taking care of some of the most critical data pipelines at Poshmark.
- Identify areas of opportunity from a technical perspective for improvements to the data platform and lead the areas to achieve scalable, easy to use pipelines/infrastructure.
- Participate in architecture discussions, influence product roadmap, build best practices, take ownership and responsibility over new projects.
- Create technical scoping for projects and execute them, taking full ownership, with minimal guidance.
- Work cross-functionally with other engineering teams, product managers, data analysts, data scientists and data engineers to drive projects from conception to release.
6-Month Accomplishments:
- Become familiar with Poshmark's code and infrastructure of data lake house, including realtime and batch processing.
- Take full ownership of at least one major data domain or critical system, ensuring its high availability and scalability.
- Understand exactly what different stakeholders need from the data.
12+ Month Accomplishments:
- Establish yourself as an expert in one of more areas of Poshmark data platform and as a technical lead on the data engineering team.
- Identify technical debt and propose a concrete 6-to-12-month roadmap for systemic improvements and platform modernization.
- Work alongside security and governance partners on access reviews, data classification.
- Collaborate with Product, Data Science, and Engineering teams to seamlessly sequence technical work across various dependencies.
Requirements:
- 4-6+ years of dedicated production experience in Data Engineering or Distributed Systems Engineering.
- Deep production experience building event-driven architectures and real-time processing applications using Apache Kafka and Apache Flink.
- Expert-level Scala, Java,Python and SQL skills paired with hands-on production experience using Apache Spark/Flink for heavy distributed processing workloads.
- Familiarity with modern open table formats like Apache Iceberg or Delta Lake (Parquet/Avro).
- Production experience with a distributed processing engine and a cloud data warehouse or lakehouse platform (e.g., Databricks/Spark, Snowflake, Redshift), utilizing AWS data services (S3, Glue, Lambda, Step Functions)
- Active participant in architecture discussions who is comfortable leading code and technical design reviews, setting engineering standards, and providing technical guidance to junior team members.
- Excellent communication skills, a strong team-player mentality, and an eagerness to evaluate and adopt newer technologies when necessary.