Job Description: Senior Cloud Data EngineerPosition Overview: We are seeking an experienced and innovative Senior Cloud Data Engineer to lead our clients in the design, development, and maintenance of cloud-based data solutions. As a key member of our data engineering team, you will be responsible for designing and developing robust data pipelines, optimizing data workflows, and ensuring the availability and reliability of our client's cloud data platform.
Key Responsibilities: - Design end-to-end data pipelines that efficiently collect, process, transform, and load data from various sources into our cloud environment.
- Collaborate with data architects and other stakeholders to ensure the scalability, reliability, and performance of data pipelines.
- Develop complex data transformation logic to prepare raw data for analysis and reporting purposes.
- Build and optimize Extract, Transform, Load (ETL) processes to ensure data quality and integrity.
- Collaborate with client stakeholders to understand data requirements and deliver actionable insights.
- Lead a team of data engineers, providing technical guidance, mentorship, and fostering a culture of innovation and collaboration.
- Stay current with the latest advancements in cloud technologies and data engineering practices.
Qualifications: - Bachelor's or Master's degree in Computer Science, Engineering, or related field.
- Proven experience (5+ years) as a Data Engineer, with a focus on cloud-based solutions.
- Strong expertise in Microsoft Azure, including hands-on experience with relevant services.
- Data Platforms (Azure Databricks, Azure SQL Database and Azure Synapse Analytics)
- Spark Programming (Python, PySpark and SparkSQL)
- Orchestration and Automation (Azure Data Factory and Logic Apps)
- Storage (Azure Data Storage Gen2)
- Event Processing (Kafka and Azure Event Hubs)
- Proficiency in data modeling, ETL/ELT processes, and data warehousing concepts.
- Knowledge of SQL and NoSQL databases, along with data processing frameworks (e.g., Apache Spark).
- Excellent problem-solving skills and the ability to troubleshoot complex data engineering issues.
- Strong leadership skills, with the ability to guide and mentor junior team members.
- Relevant Microsoft Azure data certifications are a plus.
