OVERVIEW
Reporting to the VP, Technical Operations, the Senior Data Platform Engineer is a senior, hands-on individual contributor on our Engineering, Data Platform team. This is a full-time, remote role open to candidates based in the United States or Canada, with no direct reports.
As a Senior Data Platform Engineer, you will build, run, and scale that platform: Spark and Delta Lake pipelines from our multi-tenant SQL Server estate, Unity Catalog governance with a shared semantic layer, and the analytics agent on top. You will work in a small platform team, alongside an implementation partner during the initial build, and own the platform's growth from the first customers to the full base.
WHAT YOU'LL DO
1. Lakehouse Engineering & Data Pipelines
- Build and run the lakehouse on Azure Databricks, including change tracking and change data capture (CDC) ingestion from SQL Server through Lakeflow Connect.
- Develop Spark and Delta Lake transformations in Lakeflow Spark Declarative Pipelines and jobs, along with orchestration and serving layers.
- Implement and maintain the medallion and dimensional data model in Delta Lake.
- Tune Spark jobs, Photon, liquid clustering, and storage for performance and cost.
- Measure page and report latency against Lakebase and Databricks SQL, adding a columnar serving store only if the numbers require it.
2. Semantic Layer & Data Governance
- Maintain the governed semantic layer in Unity Catalog (Metric Views) so that dashboards, embedded analytics, applications, and AI agents consume the same certified metrics.
- Enforce per-tenant row and column security at query time.
- Implement StoreForce's retail KPI logic in Metric Views, including fiscal calendars, 53-week comparisons, comp days, and targets.
- Reconcile Metric View results to the existing application's numbers.
3. Platform Scale & Tenant Onboarding
- Scale the platform by configuration: onboard tenants and add KPIs by repeating the established pattern.
- Map each tenant's configurable KPI columns to one governed schema.
- Extend and fix the pattern when it needs a new code path.
4. Data Quality, Reliability & Cost
- Enforce data quality in the pipeline: expectations, data contracts, reconciliation to source, and freshness and volume checks.
- Maintain lineage and Lakehouse Monitoring, treating data incidents like production incidents.
- Keep the platform reliable in production, favoring serverless and managed services over undifferentiated infrastructure.
- Deliver everything as code through CI/CD, with close attention to compute sizing and DBU spend.
5. AI-Powered Analytics
- Build and operate the LLM-powered analytics agent that answers natural-language questions against the semantic layer.
- Use Databricks AI tooling alongside Anthropic Claude and OpenAI models.
- Ensure tenant isolation, runtime guardrails, and an automated evaluation suite that acts as a quality gate.
- Use AI-assisted development to ship quickly and effectively.
6. Collaboration & Platform Ownership
- Work alongside the implementation partner during the initial build and take ownership of the platform at handover.
- Serve as the go-to engineer for data at StoreForce, contributing design decisions backed by evidence.
- Keep engineering standards and runbooks current.
- Support the teams consuming the platform and triage data incidents.
Other duties as assigned by your manager and senior leadership.
QUALIFICATIONS
Competencies Required:
- Technical Depth - Hands-on expert across the Databricks and Spark core who writes production code every day
- Ownership - Takes end-to-end accountability for the platform, from build through production support
- Data Quality Mindset - Treats data correctness and data incidents with the rigor of production software
- Evidence-Based Decisions - Measures before deciding and backs design choices with data
- Learning Agility - Picks up new tools quickly and embraces AI-assisted development
- Collaboration & Communication - Works effectively with partners and consuming teams; explains trade-offs clearly to technical and non-technical audiences
EXPERIENCE / KNOWLEDGE
- 5+ years building data pipelines or backend systems, including at least 2 years running a modern data stack in production
- Hands-on production experience with Databricks: Unity Catalog, Delta Lake, Lakeflow Jobs and Spark Declarative Pipelines (or their predecessors, Workflows and Delta Live Tables), Databricks SQL, and deploying through Databricks Asset Bundles or an equivalent as-code practice
- Strong Apache Spark skills: PySpark and Spark SQL, Structured Streaming, reading query plans, diagnosing shuffle and skew, and tuning jobs and compute for performance and cost
- Strong SQL (window functions, complex joins, query optimization) and strong Python for data work
- Solid data modeling across medallion, dimensional, and incremental patterns, with experience tuning Delta tables (liquid clustering, change data feed, OPTIMIZE and VACUUM strategy)
- Experience implementing data quality and observability (expectations, contracts, reconciliation, lineage, monitoring, alerting)
- Hands-on experience with modern LLM providers (Anthropic Claude and OpenAI), building AI features against their APIs
- Fluent in modern software engineering practices (version control, automated testing, CI/CD) and able to explain everyday trade-offs such as serverless vs. classic compute or streaming vs. batch; infrastructure or DevOps specialization is not required.
PREFERRED QUALIFICATIONS
- Databricks Certified Data Engineer Associate or Professional
- Experience deploying LLMs against structured data in production on Databricks (Genie, the agent framework, Model Serving for external models, MLflow evaluation), particularly evaluation, guardrails, and CI gating
- Experience with multi-tenant SaaS data on Databricks, including per-tenant isolation with Unity Catalog row filters and column masks, and catalog design for many tenants
- Experience ingesting from SQL Server and Azure SQL Managed Instance with Change Tracking or CDC over private networking
- Experience serving curated data to always-on applications from Lakebase or PostgreSQL
- Experience delivering embedded analytics inside a product, not only internal business intelligence
- Experience taking over a platform from a consultancy or another team and running it well afterward.
TECHNICAL ENVIRONMENT
- Ingestion - Lakeflow Connect with SQL Server change tracking
- Storage - Delta Lake on Azure Data Lake Storage Gen2 in StoreForce's own tenancy, in open formats so the data outlives any engine
- Transformation - PySpark, Spark SQL, and Lakeflow Spark Declarative Pipelines
- Governance - Unity Catalog for governance, lineage, and the semantic layer (Metric Views)
- Orchestration & Compute - Lakeflow Jobs, with serverless compute where it fits
- Serving - Databricks SQL for BI and embedded analytics; Lakebase or PostgreSQL as the always-on serving store
- Infrastructure & Security - Everything as code with Databricks Asset Bundles and Terraform; private networking with Entra ID throughout
You don't need experience with every technology: bring hands-on depth across the Databricks and Spark core and be ready to learn the rest. This is a data engineering role, not infrastructure.
KEY PERFORMANCE INDICATORS (KPIs)
- Pipeline success rates, data freshness against agreed SLAs, and availability of serving layers
- Reconciliation of governed KPIs to the existing application's numbers, data quality check pass rates, and number and severity of data incidents
- Time to detect and resolve data incidents, with root causes addressed and runbooks updated
- Dashboard, report, and embedded analytics latency against agreed targets
- Time to onboard a new tenant and to add a new KPI using the standard pattern
- Compute and DBU spend per tenant, and cost trends against budget
- Evaluation suite pass rates, answer accuracy, guardrail effectiveness, and zero cross-tenant data exposure
- Successful handover from the implementation partner, current standards and runbooks, and satisfaction of teams consuming the platform.
- We're remote and offer flexibility in your schedule;
- Employer-paid health benefits including dental, prescription drug coverage, and a Health Spending Account;
- RRSP Matching with First Time Home Buyer support*
- Company paid training and professional development;
- Extended Long Weekends and Religious Observance Days
- Employee Referral Program
- Spotlight Recognition Bonus Program
- Dedicated Employee Assistance Program (EAP)
*Speak with HR for full program details.
