Job Summary This role leads the delivery and day-to-day operation of Chamberlain Group's data engineering, analytics engineering, data quality, and Databricks platform capabilities within the Data and Analytics Foundations group. This hands-on leader manages a blended team of full-time Data Engineers and Analytics Engineers alongside external delivery partners, translating the data foundations roadmap into working pipelines, trusted data products, and a well-governed, cost-efficient lakehouse platform. Success in this role looks like reliable data delivered on predictable timelines, measurable improvement in data quality and platform efficiency, engineers who are growing in their craft, and business stakeholders who trust and increasingly self-serve from the data we publish.
Essential Duties and Responsibilities- Lead the design, delivery, and operation of scalable batch and streaming pipelines, curated data products, Databricks AI/BI dashboards, and natural-language query experiences over governed data, personally engaging in solution design, code review, and hands-on development where it accelerates the team.
- Manage the day-to-day delivery of a blended team of full-time Data Engineers, Analytics Engineers, and external partner resources - setting priorities, sequencing intake, removing blockers, planning capacity, accepting deliverables, and holding partners accountable to agreed quality, throughput, and responsiveness expectations.
- Own the Databricks platform operationally, including Unity Catalog administration, workspace and compute configuration, access provisioning and least-privilege enforcement, secrets management, and cost attribution and consumption optimization.
- Set the standards for data classification, masking, and access control that engineers implement on published data, in partnership with Data Governance and Information Security, and verify adherence through review.
- Own the quality and coherence of the curated and semantic layer the team publishes - Gold-layer models, enterprise metric definitions, certified datasets, and dashboards - holding the Analytics Engineering discipline to agreed standards and arbitrating metric definition disputes that cannot be settled at the working level.
- Establish data quality and observability practices so every pipeline ships with quality rules, monitoring, alerting, lineage, and privacy classification by design, and own production reliability for the data estate, including incident response, root cause analysis, and adherence to agreed service levels.
- Engage, align, motivate, and inspire the team to maximize their impact, raising the technical bar through mentoring, pairing, and code review.
- Drive adoption of and adherence to engineering standards across full-time and partner engineers alike - version control and pull request practices, automated testing, CI/CD, infrastructure and pipeline-as-code, definition of done, and documentation and runbook standards that reduce key-person risk.
- Champion the practical adoption of AI-assisted development tools and agentic workflows across full-time and partner engineers to accelerate development, testing, documentation, and migration; set the expectation that AI-generated code is reviewed and tested to the same standard as human-written code, establish guardrails for data privacy and secrets handling, and sponsor reusable AI assets - shared skills, prompt libraries, and agent configurations - that encode engineering standards and are versioned and adopted team-wide.
- Engage business stakeholders and partner with data architecture, governance, data science, and BI colleagues to translate priorities into scoped deliverables, align on data source integration and modeling standards, and provide the Director with input into the multi-year data foundations roadmap.
- Lead projects that strengthen data foundations directly, reporting delivery status, risks, and dependencies clearly to senior leadership.
- Comply with health and safety guidelines and rules; managers should also ensure compliance across their teams.
- Protect Chamberlain Group's reputation by keeping information confidential.
- Maintain professional and technical knowledge by attending educational workshops, reading professional publications, establishing personal networks, and participating in professional societies.
- Contribute to the team effort by accomplishing related results and participating on projects as needed.
Supervision Exercised- Motivate and lead a high performance team by attracting, developing, engaging and retaining team members
- Drive the performance management and compensation processes by communicating job expectations, monitoring and evaluating performance, providing feedback and facilitating employee development per the company's policies
- Maintain transparent communication by appropriately communicating organization information to team through department meetings, one-on-one meetings, appropriate email, IM and regular interpersonal communications
- Lead and motivate individuals and teams to create a workplace culture that is consistent with the CG mission, vision and values.
Minimum Qualifications Education/Certifications:- Bachelor's Degree in computer science or equivalent relevant work experience.
Experience:- 5+ years of experience in data engineering or analytics engineering, including 2+ years of hands-on delivery on Databricks
- Demonstrated experience leading technical teams, whether as a people manager or as a technical lead - including setting technical direction, reviewing others' work, and developing engineers
- Recent hands-on delivery experience building and operating production data pipelines and data models on the Databricks platform
- Experience working alongside external delivery partners, contractors, or offshore teams, with an understanding of what it takes to get quality, predictable delivery from resources you do not directly employ
- Demonstrated hands-on use of AI development tools such as Claude, GitHub Copilot, or Databricks Genie Code to improve the speed and quality of data engineering or analytics work, and a track record of raising adoption of those tools across a team
- Experience partnering with stakeholders and working with highly technical developers and analysts, with an ability to earn trust
Knowledge, Skills, and Abilities:- Significant hands-on depth in Databricks, including Delta Lake, Unity Catalog, medallion architecture, PySpark and Spark SQL, and Lakeflow Jobs orchestration - sufficient to review others' code, challenge a proposed design, and personally debug a failing production pipeline
- Working knowledge of Databricks platform administration, including Unity Catalog structure and permissions, compute configuration, and consumption cost management
- Advanced SQL and demonstrated data modeling skill, including dimensional modeling and design of curated, consumption-ready data products
- Working knowledge of the analytics consumption layer - governed semantic and metric definitions, dashboards, and natural language or AI agent access to data - sufficient to set standards for it and review the team's work
- Working knowledge of CI/CD and pipeline-as-code practices for data, including Git-based workflows, automated testing, and promotion across environments
- Practical understanding of data quality, observability, and data privacy controls, and how to embed them into delivery rather than bolt them on
- Ability to operate as a player/coach - setting technical direction and reviewing others' work while remaining credible and capable in the code
- Sound judgment about where AI tooling meaningfully accelerates delivery and where it introduces risk, with the discipline to hold AI-assisted work to the same review, testing, and security standards as any other change
- Ability to deal with ambiguity and make quality decisions in a dynamic, fast-paced environment
- Strong presentation, written and verbal communication skills with an ability to communicate effectively to all levels of the organization
- Insistence on high standards with a strong desire to transform and improve and a bias for action
Other:- Ability to travel up to 5%
Preferred QualificationsEducation/Certifications:- Databricks Certified Data Engineer Associate or Professional
- Azure platform certification (e.g., Azure Data Engineer Associate)
Experience:- Experience directly managing the delivery and performance of external partner, systems integrator, or offshore teams, including work allocation, acceptance of deliverables, and holding partners to agreed quality and throughput
- Experience leading both data engineering and analytics engineering disciplines, with clear ownership boundaries between ingestion and the curated and semantic layers
- Experience establishing reusable, version-controlled transformation frameworks with automated testing and documentation built in
- Experience applying agentic AI development tools, such as Claude Code or comparable tools, to data engineering work including legacy code migration, test generation, and large-scale refactoring
- Experience defining team-level standards or enablement for responsible AI tool use in an engineering organization
- Experience with streaming and near real-time ingestion patterns, including Spark Structured Streaming and Auto Loader
- Experience migrating or modernizing legacy ETL onto a lakehouse architecture
- Experience with agile/scrum development methodologies
Knowledge, Skills, and Abilities:- Azure platform expertise, including one or more certifications
- Familiarity with Lakeflow Declarative Pipelines (formerly Delta Live Tables)
- Familiarity with Declarative Automation Bundles (formerly Databricks Asset Bundles) or comparable deployment tooling
- Experience enabling self-service analytics through governed semantic models and curated, well-documented data products
- Familiarity with data catalog, master data management, and data quality tooling
#LI-Hybrid
#LI-JM2
The pay range for this position is $129,700.00 - $226,900.00; base pay offered may vary depending on a number of factors including, but not limited to, the position offered, location, education, training, and/or experience. In addition to base pay, also offered is a comprehensive benefits package and 401k contribution (all benefits are subject to eligibility requirements). This position is eligible for participation in a short-term incentive plan subject to the terms of the applicable plans and policies.
