Role Description

Role Description:
As a Senior Principle Engineer - Cloud Data Platform (Databricks) at Incedo, you will be responsible for managing and optimizing the Databricks environment, ensuring its performance, scalability, and security. You will work closely with data analysts and data scientists to develop data pipelines and run data science experiments. You will be skilled in big data technologies such as Spark and have experience with cloud computing platforms such as AWS or Azure. You will be responsible for configuring and optimizing the Databricks environment, ensuring that data pipelines are efficient and accurate, and troubleshooting any issues that arise. You will also work with the security team to ensure that the Databricks environment is secure and complies with relevant regulations.

Roles & Responsibilities:
• Designing, developing and deploying cloud-based data platforms using Databricks
• Integrating and processing large amounts of structured and unstructured data from various sources
• Managing and optimizing Spark clusters for efficient data processing
• Implementing and maintaining data pipelines and ETL processes
• Troubleshooting and resolving data platform issues
• Collaborating with data scientists and analysts to deliver insights and solutions
• Foster a collaborative and supportive work environment, promoting open communication and teamwork.
• Demonstrate strong leadership skills, with the ability to inspire and motivate team members to perform at their best.

Technical Skills

Skills Requirements:

• Proficiency in Databricks platform, including Databricks Runtime, Databricks Delta, and Databricks MLflow
• Experience in building scalable and reliable data pipelines using Databricks, Spark, and related big data technologies
• Familiarity with cloud-based infrastructure and deployment, specifically on Azure or AWS
• Strong knowledge of programming languages such as Python, Scala, Java, and SQL

Nice-to-have skills

Qualifications

Qualifications

• 13-15 years of work experience in relevant field
• B.Tech/B.E/M.Tech or MCA degree from a reputed university. Computer science background is preferred