Job Title Backend/API DeveloperLocation PCS CATORONTO
Years of Experience 7-10 Years
Job SummaryWe are seeking a highly skilled Backend/API Developer with strong hands-on experience in Azure Databricks and Apache Spark (PySpark/Scala). The ideal candidate will have a solid background in SQL, data transformation techniques, and cloud platforms (Azure/AWS/GCP). You will be responsible for building and optimizing ETL pipelines, ensuring data integrity, and implementing data governance practices.
Responsibilities- Design and develop ETL pipelines using Azure Databricks and Delta Lake, focusing on batch processing (autoloader) and Spark structured streaming.
- Create and manage end-to-end environments, including catalogs, schemas, tables, materialized views, functions, and volumes using Unity Catalog.
- Implement slowly changing dimensions (SCD1 and SCD2) on dimension tables and build change data capture (CDC) pipelines.
- Utilize Lakehouse federation to create foreign catalogs for accessing data from external sources.
- Optimize data processing through effective partitioning and liquid clustering in Databricks.
- Collaborate with cross-functional teams to ensure data governance and security practices are adhered to.
- Participate in CI/CD pipeline development and DevOps practices to enhance deployment efficiency.
Mandatory Skills- Expertise in Azure Databricks
- Expert-level proficiency in SQL
- Regular experience with CI/CD practices
Preferred Skills- Knowledge of data warehousing concepts
- Experience with data modeling and performance tuning in Spark
- Familiarity with data governance and security practices
QualificationsBachelor's degree in Computer Science, Information Technology, or a related field. 10+ years of experience in Data Engineering projects is required.
