The roleYou'll own the Spark pipelines on Databricks that power Monaco's analytics products, working directly with product teams to turn what they need into reliable, cost-conscious data. This is a hands-on individual contributor role for a mid-level to senior engineer (roughly 2-6 years of experience).
What you'll do- Build, deploy, and operate production Spark data pipelines on Databricks.
- Partner with product teams to turn analytics-product needs into pipeline requirements, including data quality and freshness expectations.
- Set up monitoring and observability so pipeline reliability is measured, not assumed.
- Diagnose performance and cost problems and fix their root causes.
- Recommend pragmatic standards for running pipelines as our data and platform scale.
What you'll bring- Hands-on experience building, deploying, and maintaining production Spark pipelines.
- Deep understanding of how concepts like clustering, shuffles, different join strategies, and batch versus stream processing create tradeoffs in performance, freshness, and cost.
- Experience with reliability, monitoring, and observability, and the ability to explain how you diagnosed a real pipeline failure or bottleneck.
- Strong SQL.
- Basic programming skills in any language (Python is a plus).
- Comfort working as an individual contributor, directly with product teams, and through ambiguity.
Nice to have- Familiarity with the Databricks ecosystem.
- Experience with streaming pipelines and logical replication, such as replicating Postgres into Databricks.
- Experience building RAG pipelines.
- AWS infrastructure experience.
- Platform administration experience: roles, permissions, access controls, and visibility management.
Location- San Francisco. We're an in-person team - 5 days in the office. At this stage, proximity genuinely accelerates product quality and team cohesion.
