Job Summary:
The Data Engineer will build, optimize, and manage large-scale data pipelines and solutions that support critical business insights and operations. The role requires strong expertise in T-SQL, hands-on experience with Apache Spark, and the ability to develop data workflows using Python or Scala. The engineer will design and implement scalable data pipelines, data lakes, and warehouse models within Azure cloud environments, ensuring reliable, secure, and high-performance data solutions.
Key Responsibilities:
• Architect and implement data pipelines, data lakes, and warehouse models using Azure Synapse Analytics and Azure Data Lake Storage.
• Develop and optimize complex T-SQL queries, views, stored procedures, and data transformations across dedicated and serverless SQL pools.
• Leverage Apache Spark through Azure Synapse to process and transform large-scale datasets using Scala and/or Python.
• Design and implement ELT/ETL workflows across structured, semi-structured, and unstructured data sources.
• Collaborate with business analysts, data scientists, and stakeholders to deliver reliable and high-performance data solutions.
• Monitor, troubleshoot, and optimize performance across Synapse pipelines, Spark pools, and Azure data services.
• Apply data governance, metadata management, and security practices, including role-based access control (RBAC), encryption, and auditability.
• Work within Agile/Scrum teams and follow modern DevOps and CI/CD practices for data development and deployment.
• Develop and maintain data pipeline architectures, data models, and warehouse solutions.
• Support platform integration, monitoring, governance, and technical documentation.
Required Qualifications:
• 3+ years of hands-on experience in data engineering and ETL development.
• Experience working with Microsoft Azure data services.
• Hands-on experience with at least one of the following: Azure Data Factory, Azure Synapse Analytics, or Microsoft Fabric.
• Strong proficiency in T-SQL, including complex queries, views, stored procedures, and data transformations.
• Hands-on experience with Apache Spark for large-scale data processing.
• Experience developing data workflows using Python and/or Scala.
• Experience designing and implementing ETL/ELT workflows across different data source types.
• Understanding of data lakes, data warehousing, and data modeling concepts.
• Knowledge of data governance, metadata management, and cloud data security practices.
• Experience working in Agile/Scrum environments and using DevOps and CI/CD practices.
Preferred Qualifications:
• Experience with Azure Synapse Analytics dedicated and serverless SQL pools.
• Experience with Azure Data Lake Storage.
• Experience optimizing Spark pools, Synapse pipelines, and Azure data services.
