Project description
The project focuses on designing and delivering scalable cloud-based data engineering solutions on Microsoft Azure. The role will support large-scale data integration, analytics, machine learning, AI solutions, and business intelligence through modern data platforms and automated ETL/ELT pipelines.
Responsibilities
Design, develop, and optimize ETL/ELT solutions using Azure Databricks, LakeBase, and Spark.
Build and deploy large-scale data pipelines supporting analytics, machine learning, predictive modeling, reporting, and data products.
Integrate structured and unstructured data from databases, APIs, FTPs, cloud storage, and various file formats.
Develop and optimize complex SQL queries and data models.
Build AI-based solutions for business needs, process automation, and operational efficiency.
Improve data reliability, integrity, performance, and overall platform quality.
Contribute to data architecture, engineering standards, frameworks, and best practices.
Provide technical guidance and mentorship to other data engineers.
Skills
Must have
7+ years of experience in data engineering, data integration, data modeling, data architecture, and ETL/ELT processes.
5+ years of hands-on Python experience and 2+ years of experience with SQL, complex data schemas, and query performance optimization.
At least 2 years of hands-on experience with Redis-backed state management and event-driven processing, as well as building cloud solutions using Azure Databricks with Unity Catalog, Snowflake, Azure Functions, and Service Bus.
Practical experience with Terraform-based DevOps automation and CI/CD tools including GitHub Actions, Git, Artifactory, and Sonar is required.
Mandatory Skills (only names):
Azure Databricks
Apache Spark
Python
SQL
ETL/ELT
Data Engineering
Data Modeling
Redis
Snowflake
Azure Functions
Azure Service Bus
Terraform
GitHub Actions
Git
Artifactory
Sonar
Nice to have
Excellent communication skills
Bachelor's degree in Computer Science, Engineering, Mathematics, or a related discipline.
Strong understanding of Data Lake, Delta Lake, and Data Warehousing architectures, as well as advanced data modeling techniques including slowly changing dimensions, aggregation, partitioning, and indexing.
Exposure to LLM/AI solution delivery is preferred.
Ability to independently troubleshoot and performance-tune large-scale enterprise systems, collaborate effectively across technical and business teams, mentor engineers, and communicate technical concepts to both technical and non-technical audiences.
Other
Languages
English: C1 Advanced
Seniority
Senior
