Job Summary
We are seeking a Lead Engineer to serve as the technical lead for the Internal Analytics platform, the enterprise's central source of truth for data and reporting. This role combines hands-on engineering with technical leadership across Databricks, Azure Data Factory (ADF), SQL, ETL, data and Delta Lake architecture, medallion architecture, DevOps, and emerging data technologies. The Lead Engineer will guide the design, development, and delivery of scalable data solutions that support self-service analytics and enterprise-scale reporting while ensuring performance, accuracy, and reliability.
Key Responsibilities
• Lead the design, development, enhancement, and delivery of enterprise data and software solutions.
• Provide technical leadership across Databricks, SQL, ETL using Azure Data Factory, data engineering, data modeling, and analytics technologies.
• Implement complex initiatives spanning multiple technologies and business functions.
• Participate in requirements gathering, solution design, and architectural discussions.
• Conduct code reviews, enforce development standards, and maintain high code quality.
• Drive CI/CD and Infrastructure as Code (IaC) practices to improve development efficiency and reliability.
• Mentor engineers and encourage continuous learning through pair programming and knowledge-sharing.
• Collaborate across teams to maintain the integrity and performance of the Internal Analytics database.
• Participate in Agile ceremonies and ensure consistent delivery of sprint objectives.
• Design and optimize Databricks solutions, including notebooks, stored procedures, and user-defined functions (UDFs).
• Build and maintain scalable data ingestion, transformation, and curation solutions using modern data pipelines.
• Support data integration from external source systems through API calls and other ingestion mechanisms.
Required Qualifications
• 8+ years of professional data engineering experience, including proven technical leadership.
• Deep understanding of the Azure technology stack, including Databricks, Azure Data Factory (ADF), Key Vault, Azure SQL (ASQL), Azure Data Lake Storage Gen2 (ADLS Gen2), and service principals.
• Strong hands-on experience with Databricks, including design, optimization, notebooks, stored procedures, and UDFs.
• Hands-on experience with ETL tools such as SSIS and Azure Data Factory.
• Experience working with API calls and ingesting data from external source systems.
• Strong experience with SQL, Python, and PySpark.
• Experience building dynamic and scalable data ingestion and curation solutions.
• Experience in data engineering, Microsoft SQL Server, and T-SQL development.
• Strong understanding of data lake and Delta Lake architectures and modern data pipelines.
• Strong understanding of medallion architecture and associated design patterns.
• Strong understanding of data modeling techniques.
• Experience handling large-scale data volumes involving millions of records.
• Proficiency in building and maintaining CI/CD pipelines.
• Experience with Infrastructure as Code technologies such as Terraform or Pulumi.
• Prior experience working in an Agile development environment.
• Strong communication and collaboration skills, including the ability to explain technical concepts to business partners.
Preferred Qualifications
• Exposure to big data tools such as Spark, Hadoop, and Kafka.
• Experience with reporting tools such as Power BI or Tableau.
• Knowledge of .NET development to support cross-team collaboration.
