Position Overview Join us in a high-impact, high-visibility role where you will lead the design and delivery of enterprise-scale Data & AI solutions that power analytics, operational decision-making, and AI innovation across the organization.
We are seeking an experienced Senior Lead Data & AI Engineer with 9-12+ years of experience in Data Engineering and modern cloud-based data platforms, with deep expertise in Databricks, Lakehouse architecture, and AWS. This role will be responsible for building scalable data foundations, reusable engineering frameworks, and AI-ready data products that accelerate business value and enable advanced analytics and Generative AI use cases.
The ideal candidate combines strong hands-on data engineering expertise with solution architecture, technical leadership, and a passion for modernizing data ecosystems through automation, metadata-driven engineering, and cloud-native technologies.
Key Responsibilities Data Engineering Leadership- Lead the architecture, design, and delivery of enterprise-scale data engineering solutions leveraging Databricks and AWS.
- Design and implement scalable, resilient, and reusable data pipelines supporting batch, streaming, and event-driven workloads.
- Drive engineering best practices across data ingestion, transformation, orchestration, testing, monitoring, and deployment.
- Establish reusable frameworks, patterns, and standards to accelerate delivery and improve engineering quality across teams.
Databricks & Lakehouse Engineering- Architect and implement modern Lakehouse solutions utilizing Databricks, Delta Lake, Unity Catalog, and cloud-native services.
- Develop metadata-driven and configuration-based frameworks that standardize ingestion, transformation, quality validation, and deployment processes.
- Design and implement Delta Live Tables (DLT), Lakeflow Pipelines, and Databricks Workflows to automate enterprise data processing.
- Build trusted, governed, and reusable data products that support analytics, reporting, data science, and AI initiatives.
- Optimize platform performance, scalability, reliability, and cost efficiency across large-scale workloads.
Data Platform Modernization- Lead migration and modernization initiatives from legacy platforms to cloud-native data architectures.
- Implement data observability, monitoring, lineage, metadata management, and quality controls throughout the data lifecycle.
- Partner with architecture and platform teams to advance enterprise data platform capabilities and engineering standards.
- Support implementation of CI/CD, Infrastructure-as-Code, automated testing, and DevSecOps practices.
AI & Advanced Analytics Enablement- Build AI-ready data foundations that support machine learning, Generative AI, and intelligent business applications.
- Partner with Data Scientists, AI Engineers, and business teams to operationalize data products and AI solutions.
- Support implementation of vector search, semantic models, retrieval-augmented generation (RAG), and knowledge-driven AI capabilities.
- Contribute to AI orchestration and agent-based solutions where business value can be accelerated through intelligent automation.
- Enable reliable integration between AI applications and enterprise data platforms.
Governance, Security & Data Quality- Implement enterprise-grade governance using Unity Catalog, metadata repositories, and data quality frameworks.
- Ensure compliance with enterprise security, privacy, and regulatory requirements.
- Design automated controls for data validation, observability, lineage, and operational monitoring.
- Partner with governance organizations to improve trust and accessibility of critical data assets.
Stakeholder & Team Leadership- Act as a trusted technical advisor to business, analytics, and technology stakeholders.
- Mentor and provide technical leadership to Data Engineers and platform teams.
- Lead architecture reviews, design workshops, and engineering governance activities.
- Translate complex technical concepts into actionable business outcomes for leadership audiences.
Qualifications- Bachelor's or Master's degree in Computer Science, Engineering, Information Systems, or related discipline.
- 9-12+ years of experience in Data Engineering, Data Platform Engineering, or Analytics Engineering.
- 5+ years of hands-on experience building enterprise solutions on Databricks.
- Strong expertise in Lakehouse architecture, Delta Lake, Unity Catalog, and modern data engineering patterns.
- Proven experience building large-scale data pipelines using Spark, PySpark, SQL, and Python.
- Deep experience with AWS services supporting Data & AI platforms.
- Experience implementing Delta Live Tables, Lakeflow, Databricks Workflows, and metadata-driven engineering approaches.
- Strong understanding of data modeling, data products, data governance, metadata management, and data quality frameworks.
- Experience with CI/CD, Infrastructure-as-Code, automated testing, and cloud-native engineering practices.
- Demonstrated leadership experience mentoring engineers and leading complex technical initiatives.
- Strong communication and stakeholder management skills.
Pay Transparency
Expected Salary Range: 93,000 to 165,000
Role status: existing vacancy
Our compensation ranges are based on role, level, and local market. Individual pay within the range is determined by factors such as job-related skills, experience, and relevant education or training. For part-time roles, pay will be pro-rated based on regularly scheduled hours.
In addition to the range above, Vanguard’s total compensation package may include performance-based incentives, discretionary bonuses, and other perks. We also offer comprehensive benefits such as health insurance, accident and life insurance, and retirement savings plans. Your recruiter will be able to provide additional details on the total compensation offering during the hiring process.
How We Work
Vanguard has implemented a hybrid working model for the majority of our crew members, designed to capture the benefits of enhanced flexibility while enabling in-person learning, collaboration, and connection. We believe our mission-driven and highly collaborative culture is a critical enabler to support long-term client outcomes and enrich the employee experience.
