We are seeking a Data Engineer to design, build, and support modern data pipelines and data platform solutions using Microsoft and cloud-based technologies. This role will be responsible for developing and optimizing scalable data pipelines, supporting the Microsoft Fabric platform, and ensuring high-quality, reliable data is available for reporting, analytics, and business operations.
The ideal candidate has strong experience with SQL, Azure data services, and ETL/ELT development, and is comfortable working across both pipeline development and data platform responsibilities.
ESSENTIAL FUNCTIONS:- Design, build, and maintain ETL/ELT pipelines using Azure Data Factory, Microsoft Fabric Data Factory, and Dataflows.
- Develop and optimize data ingestion workflows across structured and semi-structured data sources.
- Troubleshoot pipeline failures, data issues, and performance bottlenecks to ensure reliable data delivery.
- Support and maintain data platform components within Microsoft Fabric, including Lakehouse, Warehouse, and OneLake.
- Work with medallion architecture (bronze, silver, gold) to structure and manage data transformations.
- Contribute to Spark and notebook-based data processing for large-scale data transformation.
- Monitor data pipelines and platform health, and support alerting and incident response processes.
- Perform data validation, profiling, and cleansing to ensure high-quality and accurate data.
- Work with business stakeholders to gather data requirements and translate them into technical solutions.
- Collaborate with analysts, data scientists, and application teams to support reporting and downstream use cases.
- Write and maintain documentation for data pipelines, transformations, and workflows.
- Participate in code reviews and contribute to CI/CD and DevOps practices using Git/GitHub.
- Follow and help enforce data governance, security, and best practices across the data platform.
COMPETENCIES / EDUCATION / EXPERIENCE:- Bachelor's degree in Computer Science, Information Systems, Data Engineering, or related field (or equivalent experience).
- 3-5 years of hands-on experience in data engineering or a related technical role.
- Strong SQL skills, including query optimization, indexing, and performance tuning.
- Proficiency in Python and/or PowerShell for data transformation and automation.
- Hands-on experience with Azure data services, including Data Factory, Synapse Analytics, Azure SQL, and Data Lake Storage.
- Working experience with Microsoft Fabric, including Lakehouse, Dataflows, and pipelines.
- Solid understanding of ETL/ELT design patterns and data pipeline best practices.
- Experience working with structured and semi-structured data formats (JSON, XML, etc.).
- Familiarity with source control tools such as Git/GitHub and CI/CD practices.
- Strong analytical, problem-solving, and communication skills.
PREFERRED SKILLS:- Experience with Spark (PySpark) for large-scale data processing.
- Familiarity with Delta Lake and Parquet data formats.
- Exposure to Power BI or other data visualization tools.
- Experience with APIs, including development and integration.
- Microsoft Fabric certification (DP-600) or interest in pursuing certification.
