Job DescriptionThe Tech & AI Department has an opening for a Senior Director, Data Engineering. This position is located in Pleasanton, California.
Central to this mission is democratizing access to data - enabling customer experience innovation, predictive models, and business intelligence by partnering closely with technology teams, data scientists, data analysts, and business users across the enterprise.
This role owns the architecture, the team, and the outcomes. The leader sets technical direction for how data is ingested, modeled, governed, and consumed - and deploys agentic AI to make the engineering organization itself faster and more scalable. This means owning a multi-year roadmap, a technology budget, and a team of 50+ engineers working across three areas: data platform, data analytics, and intelligence platform.
Main responsibilities:- Strategic Leadership: Owns the multi-year data platform roadmap and architectural evolution across lakehouse, Medallion, and streaming infrastructure - driving high-consequence technology decisions with full accountability for ROI, budget, team goals, KPIs, and delivery milestones.
- Agentic AI Across the Engineering Lifecycle: Defines the strategy and operating model for embedding agentic AI - spanning code generation, automated testing, pipeline monitoring, and auto-remediation - across the engineering function to drive measurable gains in velocity and reliability.
- AI-Ready Data Foundations & Data Products: Enforces architectural standards for trusted, governed, and discoverable data - driving data mesh adoption, data contracts, and domain ownership - while overseeing feature stores, vector databases, and RAG pipelines that power production AI.
- Enterprise Intelligence Platform: Delivers a governed conversational and semantic access layer across core retail functions - like merchandising, supply chain, store operations - into a unified, queryable knowledge layer with governance guardrails ensuring trustworthy, business-ready outputs.
- Data Governance, Quality & Observability: Adopts and enforces enterprise governance frameworks, quality standards, and observability practices - ensuring compliance across all data engineering workloads, proactively recommending improvements, and partnering with Audit, Compliance, and Governance leaders on data security and regulatory adherence.
- Engineering Velocity & Reliability: Embeds DataOps and platform engineering best practices - CI/CD for data, SLA/SLO ownership, and cloud cost optimization across GCP and Azure - to build a high-throughput, production-grade engineering organization.
- People Leadership: Leads 50+ engineers through directors and senior managers - owning hiring, performance management, career development, and succession - while building psychological safety and developing managers who grow the next generation of engineering leaders.
- Executive Stakeholder Management: Partners with Data Science, ML Engineering, Analytics, Product, and business domains to align data strategy with enterprise priorities - communicating trade-offs and platform direction to C-suite and VP stakeholders with clarity and conviction.
- Data Analytics & Consumption Layer: Partners with business stakeholders to deeply understand the underlying problem before delivering - translating business intent into governed BI, self-service analytics via PowerBI, and API-driven data products that go beyond the ask and drive measurable outcomes.
We are looking for candidates who possess the following:- Bachelor's degree in computer science, engineering, or a related quantitative discipline - required. Master's degree or MBA - strongly preferred.
- Experience:
- 15 plus years in data engineering, data architecture, or closely related technical disciplines.
- 8 plus years in building and leading data engineering teams
- 5 plus years in senior leadership with direct accountability for large-scale data engineering organizations (25+ engineers) in a complex enterprise environment.
- Proven track record building and scaling production-grade data platforms at Fortune 500 scale.
- Experience owning technology budgets.
- Demonstrated success leading organizations through technology modernization and methodology shifts.
- Technical Depth:
- Expert-level command of data engineering fundamentals - ELT/ETL, data integration, cataloging, wrangling, quality, governance, and lineage - with proven ability to select and implement the right tooling at enterprise scale.
- Expert-level command of cloud data platforms; strong preference for GCP (BigQuery, Dataflow, Pub/Sub, Composer)
- Deep expertise in data lakehouse, Medallion architecture, and large-scale warehouse engineering
- Deep understanding of data mesh principles, semantic layer design, and conversational AI architecture
- Proficient in Airflow, Kafka/Flink, Python, SQL, and Apache Spark
- Strong grasp of data analytics, BI tooling, and self-service reporting platforms (e.g., Power BI, Looker).
- Knowledge of MLOps tooling, feature stores, vector databases, and LLM API integration for production AI/ML systems
- Industry Context
- Experience in retail, CPG, grocery, or a similarly complex, high-velocity transactional industry - strongly preferred.
- Familiarity with retail-specific data domains: supply chain, merchandising, loyalty, store operations, and pharmacy.
- AI Skills:
- Proficiency in leveraging AI tools for daily engineering tasks to enhance productivity, optimize effort, and ensure cost-aware AI assistance.
- Familiarity with Retrieval Augmented Generation (RAG) architectures, including semantic layers for data retrieval and grounding Large Language Model (LLM) responses.
- Conceptual understanding of LLM-based applications (e.g., chatbots, Q&A systems), encompassing prompt engineering, context management, and response generation.
- Exposure to agentic and multi-agent AI systems, including agent roles, tool utilization, memory management, and workflow orchestration.
- Practical experience with LLM application frameworks (e.g., LangChain, LangGraph, or Google GenAI tools) for prototyping and integrating AI-driven solutions.
We also provide a variety of benefits including:- Competitive wages paid weekly
- Access to up to 50% of your earned wages before payday, via our partnership with Stream
- Associate discounts
- Health and financial well-being benefits for eligible associates (Medical, Dental, 401k and more!)
- Time off (vacation, holidays, sick pay). For eligibility requirements please visit myACI Benefits
- Leaders invested in your training, career growth and development
- An inclusive work environment with talented colleagues who reflect the communities we serve
A copy of the full job description can be made available to you.
#LI-MF1
About the TeamPay Transparency:Starting rates will be no less than the local minimum wage and may vary based on criteria such as location, experience, and qualifications.
Candidates with unique qualifications may be considered for compensation above this range. Benefits may include medical, dental, vision, disability and life insurance, sick pay, PTO/Vacation Pay or Flexible Time Off, paid holidays, bereavement pay, and retirement benefits (pension and/or 401k eligibility). [If applicable:] Associates in this position may be eligible for a quarterly bonus.
