Railroad19, Inc is seeking a
Senior Data Engineer with deep, hands-on experience in building modern lakehouse architectures on GCP. This role focuses on designing, developing in Python & Spark, and delivering reusable data-sharing adapters that connect BigQuery-backed data products to Snowflake and Databricks using Iceberg UniForm and Delta Sharing.
Core Responsibilities:- Design and implement the UniForm write layer (Delta + Iceberg dual metadata).
- Build GCS - BigQuery ingestion pipelines for structured operational datasets.
- Develop and implement in Python and Spark.
- Implement Kafka-based CDC patterns for real-time and near-real-time ingestion.
- Develop data lineage, dependency tracking, and modular adapter code.
- Configure Snowflake Horizon external tables for zero-copy reads.
- All data hub tables are to be written once using Delta Lake with Iceberg UniForm enabled... readable by all target consumers without conversion.
- Implement and certify Delta Sharing endpoints for Databricks consumers.
- Build governed access layer components: RBAC, connector registry entries, tenant-scoped authorization.
- Align semantic layer models with LookML and KPI catalog definitions.
- Collaborate with cross-functional teams to deliver end-to-end features.
- Troubleshoot issues across the full stack and contribute to code quality.
Skills/Experience:- 6+ years of proven enterprise-level experience in Python & Spark
- Advanced experience in GCP BigQuery
- Strong working knowledge of Apache Iceberg, Delta Lake, Iceberg UniForm
- Experience with Delta Sharing; Kafka / CDC pipelines
- Specific work experience in Snowflake Horizon Catalog; Databricks Unity Catalog
- Solid experience with Data lake architecture & ingestion pipeline design
- Excellent Communication skills and the ability to work cohesively with multiple teams.
Preferred Experience - Nice to Have- Prior delivery in enterprise SaaS, media, or advertising technology.
- Active daily use of AI-assisted development tools (Claude Code preferred).
$119,998 - $180,000 a year
