Full Job Description
Senior Data Platform / Data Product Engineering Lead
Must Have Technical/Functional Skills
Job Description: Senior Data Platform / Data Product Engineering Lead
Role Overview
We are looking for a Senior Data Platform / Data Product Engineering Lead to drive enterprise-scale data product lifecycle enablement across modern data platforms. This role will lead the design, standardization, and adoption of a paved path for data creators, enabling self-service, governed, and scalable data product development.
The role requires deep expertise in Databricks, Airflow (Astronomer), CI/CD automation, data governance, and data marketplace constructs, along with the ability to lead platform transformation initiatives and mentor engineering teams.
Required Skills & Experience
• 10+ years of experience in Data Engineering / Data Platform roles
• Strong hands-on expertise in:
o Databricks (Delta Lake, workflows, DAG)
o Apache Airflow / Astronomer
o Python, SQL, DBT ,AWS
• Proven experience implementing CI/CD frameworks (Harness, GitHub Actions, Azure DevOps)
• Deep understanding of:
o Data governance (catalogs, lineage, contracts, metadata)
o Data quality and masking techniques
o Enterprise data platforms and marketplace ecosystems
• Experience with API-based integrations (e.g., entitlement systems like AccessCentral)
• Monitoring/observability tools (e.g., Datadog)
Job Description: Senior Data Platform / Data Product Engineering Lead
Role Overview
We are looking for a Senior Data Platform / Data Product Engineering Lead to drive enterprise-scale data product lifecycle enablement across modern data platforms. This role will lead the design, standardization, and adoption of a paved path for data creators, enabling self-service, governed, and scalable data product development.
The role requires deep expertise in Databricks, Airflow (Astronomer), CI/CD automation, data governance, and data marketplace constructs, along with the ability to lead platform transformation initiatives and mentor engineering teams.
Core Responsibilities
1. Platform Strategy & Self-Service Enablement
• Define and implement a self-service data platform strategy to reduce onboarding friction.
• Lead automated provisioning of:
o Databricks workspaces (via DevHub)
o Airflow/Astronomer environments
o Access and entitlements (AccessCentral APIs)
• Establish isolated, stable development environments for federated teams.
• Drive platform observability by integrating metrics into tools lik e Datadog.
2. Data Discovery, Access & Governance
• Architect and implement enterprise-wide data discovery and marketplace enablement.
• Drive adoption of:
o Data contracts
o Metadata standards
o Domain-aligned catalogs (Unity Catalog)
• Enable secure access to curated, masked datasets in dev and production environments.
• Implement tagging, access patterns, and entitlement automation.
• Partner with risk/compliance teams to enforce regulatory and governance controls (BFSI-aligned).
3. Data Engineering, Curation & Orchestration
• Lead design of scalable data ingestion, curation, and transformation frameworks.
• Build and standardize modular, reusable frameworks:
o LaunchLake templates
o Airflow DAG libraries
o DBT-based transformation models
• Ensure:
o Data quality and consistency
o Embedded governance and compliance policies
• Enable concurrent development using standardized patterns and environments.
4. CI/CD, Automation & Deployment
• Define and enforce standard CI/CD pipelines across data products:
o Harness (or equivalent)
o Databricks Asset Bundles (DAB)
• Automate:
o DAG deployments (Airflow/Astronomer)
o DBT pipeline releases
• Reduce manual interventions and ensure consistent, repeatable deployments.
• Improve release reliability with feedback loops, notifications, and monitoring.
5. Data Product Publishing & Marketplace Enablement
• Drive publishing of data products to:
o Unity Catalog
o Enterprise Data Marketplace
• Define and enforce:
o Documentation standards
o Data ownership models
o Versioning and contract management
• Enable cross-domain data sharing with embedded governance and access controls.
6. Operations, Observability & Reliability
• Establish a scalable operating model for data product support.
• Implement:
o Monitoring dashboards (Datadog)
o Data quality frameworks
o Usage and performance metrics tracking
• Improve visibility into:
o Pipeline health
o Data lineage
o Access and consumption patterns
• Lead incident management, root cause analysis, and escalation processes.
7. Transformation, Roadmap & Innovation
• Drive execution of platform priorities such as:
o Data contract activation strategy
o Domain catalog integration
o Data masking in development environments
o Data quality frameworks
o DBT adoption and POCs
• Lead maturity uplift from:
o Manual, fragmented workflows standardized, automated paved paths
• Champion continuous improvement and innovation in developer experience.
Salary Range- $100,000-$120,000 a year
