The PositionThe Senior Data Platform Engineer is responsible for designing, building, and managing the organization's Databricks environments, with end-to-end ownership of the data platform's infrastructure, security, and performance. This role leads the ingestion of data from internal systems and external third-party providers into the lakehouse, partners with business and analytics stakeholders to build and support Genie dashboards, and manages the Databricks platform (workspaces, clusters, Unity Catalog, jobs) as infrastructure-as-code using Terraform. The ideal candidate has hands-on experience administering Databricks in production, writing Terraform for infrastructure changes, and building data pipelines that hold up at enterprise scale.
Responsibilities
- Design, build, and manage Databricks environments (workspaces, clusters, Unity Catalog, SQL warehouses) supporting enterprise analytics and data science workloads
- Develop and maintain Terraform modules and pipelines to provision, configure, and version-control all Databricks infrastructure and related cloud resources
- Design and build data ingestion pipelines to bring in data from internal systems and external third-party data providers (APIs, SFTP, batch files, streaming sources) into the lakehouse.
- Architect and maintain medallion (bronze/silver/gold) data layers using Delta Lake, ensuring data quality, lineage, and governance via Unity Catalog
- Partner with business and analytics end users to design, build, and support Genie dashboards and enable self-service, natural-language data exploration
- Implement platform security, access controls, and partner with data stewards.
- Work with data engineers, analysts, data scientists, AI engineers, and cloud DevOps engineers to build and maintain CI/CD pipelines, automated tests, and Terraform infrastructure-as-code for the data platform
- Document platform architecture, ingestion pipelines, and operational runbooks
- Assist AI engineers by building and maintaining MCP (Model Context Protocol) connectors and other data/tool integrations that power agentic workflows
Requirements- Bachelor's degree in Computer Science, Data Engineering, Information Systems, or related field, or equivalent practical experience
- 5+ years of experience in data engineering or platform engineering roles, including extensive hands-on experience administering and building on Databricks (clusters, jobs, Unity Catalog, Delta Lake, SQL warehouses)
- Demonstrated experience managing Databricks infrastructure as code using Terraform
- Strong proficiency in Python and SQL for building and optimizing data pipelines
- Experience designing and building data ingestion pipelines from diverse internal and external data sources (APIs, SFTP, third-party vendor feeds, streaming)
- Working knowledge of cloud platforms (AWS, Azure ) and associated data/storage services
- Experience supporting business intelligence or natural-language dashboarding tools (e.g., Databricks Genie )
- Experience implementing data governance, security, and access controls in Databricks (e.g., Unity Catalog permissions, row/column-level security, service principals, secrets management)
- Ability to explain pipeline and dashboard design decisions to non-technical stakeholders, e.g., when scoping Genie dashboard requirements with business users
Preferred- Databricks certifications (e.g., Databricks Certified Data Engineer Professional, Databricks Certified Associate/Professional Platform Administrator)
- Experience with Databricks Genie and building natural-language / self-service analytics experiences for business users
- Experience integrating data from external data providers or vendor APIs at enterprise scale
- Familiarity with CI/CD tooling (e.g., GitHub Actions, Azure DevOps, GitLab CI) for automated deployment of Terraform-managed infrastructure
- Experience with data orchestration tools (e.g., Databricks Workflows, Airflow, dbt)
- Experience building and supporting agentic workloads on Databricks, including managing model/tool access and governance via the Unity Catalog AI Gateway
- Expertise in pharma or pharma-related data (e.g., clinical, R&D, regulatory, or commercial pharma data) and the compliance considerations that come with it
Wisconsin pay range
$155,000-$175,000 USD
California pay range
$175,000-$190,000 USD
Arrowhead provides competitive salaries and an excellent benefit package.
