Job DescriptionAs a Sr Data Engineer, you will design, build, and maintain scalable data pipelines and data infrastructure that support analytics, reporting, and data science initiatives. You will collaborate with cross-functional teams to ensure data is accessible, reliable, and secure across the organization, while contributing to the ongoing improvement of data engineering practices.
ResponsibilitiesArchitect, Design, and Deliver Scalable Data Pipelines- Lead the design and implementation of scalable ingestion and transformation frameworks on Azure, enabling efficient processing of structured, semi-structured, and unstructured data across enterprise platforms.
- Build, standardize, and maintain robust ETL/ELT pipelines using Azure Data Factory and Azure Databricks, including reusable patterns, error handling, and automated testing.
- Own complex integrations across on-premises systems, cloud storage, APIs, and streaming platforms, ensuring reliability, scalability, and clear interface contracts.
Lead Databricks Engineering and Platform Optimization- Develop, review, and optimize Databricks notebooks and workflows using PySpark and SQL; establish engineering standards for readability, maintainability, and reuse.
- Implement and govern Delta Lake patterns for efficient storage, versioning, and ACID transactions, including retention, compaction, and schema evolution strategies.
- Leverage and administer Databricks capabilities (Unity Catalog, job orchestration, cluster policies) to balance security, performance, and cost across environments.
Define Data Architecture, Modeling Standards, and Lakehouse Patterns- Design and evolve enterprise data models (star/snowflake and lakehouse-oriented models) to support analytics, reporting, and self-service consumption.
- Partner with data/solution architects to define lakehouse architecture, reference patterns, and design reviews that improve scalability, resilience, and maintainability.
- Lead implementation and optimization of Medallion Architecture (Bronze/Silver/Gold), defining SLAs, data contracts, and layering conventions for scalable, governed processing.
Establish Data Quality, Observability, and Governance Controls- Implement automated data validation, profiling, and cleansing routines; define quality rules, thresholds, and exception workflows aligned to business-critical datasets.
- Ensure adherence to governance policies by implementing lineage, metadata, and cataloging practices; partner with governance stakeholders to close gaps and drive adoption.
Drive Performance Engineering, Monitoring, and Incident Resolution- Monitor and optimize Spark jobs and data pipelines, applying performance and cost tuning (cluster sizing, partitioning, caching, and query optimization).
- Lead troubleshooting and root-cause analysis for latency, failures, and resource constraints; implement preventative fixes and improve runbooks/alerts to reduce recurrence.
Provide Technical Leadership and Stakeholder Partnership- Partner with data scientists, analysts, and business stakeholders to shape data strategy, clarify requirements, and prioritize delivery based on value, risk, and dependencies.
- Translate business needs into durable technical designs (including data contracts and SLAs) and guide implementation to ensure solutions are scalable, maintainable, and supportable.
Engineer Secure-by-Design Data Solutions and Ensure Compliance- Implement and enforce secure data access patterns (RBAC/least privilege), encryption, secrets management, and secure network configurations across the data platform.
- Ensure solutions meet applicable regulatory and internal compliance requirements (e.g., NERC CIP, GDPR, HIPAA where applicable) through controls validation, audit support, and documentation.
Advance Best Practices, Documentation, and Mentorship- Maintain clear documentation of data flows, architecture decisions, and operational procedures; create runbooks and knowledge transfer artifacts to support production operations.
- Promote engineering excellence through code reviews, version control, automated testing, and CI/CD; mentor junior engineers and drive continuous improvement across the team.
QualificationsBachelor's degree in information systems, computer science or related technical field or equivalent work experience. (Typically four years of additional related, progressive work experience would be needed for candidates applying for this position who do not possess a bachelor's degree.)
Eight or more years of experience with advanced knowledge of data architecture, cloud platforms (especially Azure), and enterprise data solutions.
Advanced proficiency with data engineering platforms and tools, particularly Azure Data Factory and Azure Databricks.
Advanced knowledge of core data engineering practices, including data modeling, ETL/ELT pipeline development, and performance tuning for enterprise-scale applications.
Experience across the data technology lifecycle, including solution design, development, optimization, administration, and licensing considerations.
Prior experience in the utility industry, with exposure to relevant data domains and operational environments.
