5-7 years of data engineering experience with 3+ years of hands-on Databricks proficiency in production settings.
Expertise in migrating on-premise data systems to cloud platforms.
Strong skills in Python, PySpark, and SQL; experience with converting SQL dialects to Spark SQL is a bonus.
Knowledge of cloud computing, especially within Azure; integration of cloud functions with data platforms is a plus.
Familiarity with CI/CD practices, data quality frameworks, and tools like Databricks Workflows, Airflow, or dbt.
Responsibilities
Develop and optimize data pipelines using Databricks and cloud technologies.
Lead the migration of on-premises data systems to cloud-based solutions.
Collaborate with teams to ensure data quality and adherence to best practices.
Design and implement CI/CD processes for data applications.
Monitor and troubleshoot data workflows to enhance performance.
Benefits
Opportunities for career development and continuous learning.
Collaborative and innovative work environment.
Access to cutting-edge tools and technologies.
Flexible work arrangements and work-life balance support.
Full Job Description
Job Description
Qualifications & Requirements
Experience 5 to 7 years of data engineering experience, with at least 3+ years of hands-on Databricks experience in production environments.
Migration Expertise Proven experience in migrating on-premises data systems to cloud platforms.
Technical Skills Strong proficiency in Python, PySpark, and SQL. Experience in converting legacy SQL dialects to Spark SQL is a plus.
Cloud Knowledge Familiarity with cloud computing concepts, preferably within Azure ecosystems. Experience integrating cloud functions or runtime services with data platforms is desirable.
Best Practices & Tools Solid understanding of CI/CD, data quality frameworks, and orchestration tools such as Databricks Workflows, Airflow, or dbt.
Preferred Skills
Experience with large-scale data modernization programs
Exposure to financial services or regulated data environments
Strong problem-solving and stakeholder communication skills
Investment banking background would be added advantage.
