The Lead / Principal Data Engineer is the technical owner of Peter Millar's modern data platform, responsible for the architecture, engineering standards, and production operation of our Microsoft Fabric and OneLake environment. This role brings data engineering and architecture in-house owning the medallion (bronze/silver/gold) architecture, CI/CD standards, and the relationship with outsourced engineering partners, while establishing the AI-ready data foundation on which analytics, data science, and generative AI depend.
ESSENTIAL FUNCTIONS:
Platform Architecture & Ownership
- Own the end-to-end architecture of the Microsoft Fabric environment, including Lakehouse and Warehouse design, the medallion (bronze/silver/gold) pattern, and Direct Lake semantic datasets.
- Design and govern OneLake as the single, organization-wide data lake, managing domains, shortcuts, data mirroring, and Delta Lake/Parquet storage standards.
- Own data ingestion and transformation at scale using Fabric Data Factory pipelines, Dataflows Gen2, and Spark notebooks.
- Define and enforce engineering standards: Git-based source control, CI/CD and deployment pipelines, environment promotion, and code review.
Reliability, Performance & Cost
- Establish monitoring, alerting, and observability across pipelines; ensure SLAs for data freshness and availability.
- Optimize Fabric capacity units, and storage layout, for both performance and cost.
- Implement data quality, lineage, and automated testing frameworks across the platform.
AI-Ready Foundation & GenAI Enablement
- Structure governed, well-modeled data in OneLake to serve as grounding for generative and agentic AI use cases.
- Partner with the AI Engineering Manager to integrate Azure AI services and Microsoft Foundry (formerly Azure AI Foundry) into production data workflows, including retrieval (RAG) pipelines over OneLake.
- Implement security and governance controls, row/column-level security and PII handling, required for AI at scale.
Leadership & Vendor Management
- Direct and set the technical agenda for outsourced engineering partners, shifting them from running the platform to extending it on scoped, time-bound projects.
- Serve as technical hiring manager and interviewer for subsequent engineering hires; mentor junior analytics and data engineers.
- Capture and retain institutional knowledge of our architecture and design patterns in-house.
TECHNICAL COMPETENCIES (REQUIRED):
- Microsoft Fabric - deep, hands-on experience with Lakehouse, Warehouse, Data Factory pipelines, Dataflows Gen2, Spark notebooks, and Direct Lake.
- OneLake - demonstrated experience designing and governing OneLake, including domains, shortcuts, mirroring, and Delta Lake/Parquet standards.
- Azure AI - experience integrating Azure AI services / Azure OpenAI into data and retrieval pipelines.
- Microsoft Foundry (Azure AI Foundry) - working knowledge of the model catalog and RAG/agent grounding for GenAI workloads (applicable to this role).
- Azure data platform - Synapse, Azure Data Lake Storage, Azure SQL, Service Bus, and Spark in production.
- Engineering practice - strong CI/CD, Git-based source control, and infrastructure-as-code.
DESIRED EDUCATION AND EXPERIENCE:
- 8+ years in data engineering / data architecture, including 2+ years in a lead or principal capacity.
- Expert SQL and Python; strong Spark (PySpark) experience.
- Proven track record building and operating a Microsoft Fabric / OneLake platform in production (Fabric-specific experience strongly preferred and commands a premium).
- Experience managing or directing outsourced/consulting engineering teams.
- Exposure to Microsoft Purview, data governance, and security/compliance frameworks.
- Bachelor's in Computer Science, Engineering, or related field; Master's or relevant Microsoft certifications (e.g., Fabric Analytics Engineer, Azure Data Engineer Associate) a plus.
