5-7 years of experience in ETL development with Informatica, Azure Data Factory, and Databricks.
Proficient in Medallion Architecture and cloud data engineering concepts.
Strong knowledge of programming languages including Python, SQL, and PL/SQL.
Expertise in data pipeline orchestration, deployment, and automation.
Experience with CI/CD processes, data security, and governance practices.
Responsibilities
Design and optimize Azure Data Factory and Databricks pipelines for data integration.
Develop and implement data models using Medallion Architecture principles.
Create data connections for on-premises data sources to support downstream consumers.
Translate Informatica ETL processes to Azure Data Factory and Databricks ELT.
Conduct knowledge transfer sessions and provide documentation to project stakeholders.
Benefits
Opportunity to work with cutting-edge cloud technologies and frameworks.
Involvement in high-impact data projects within the public sector.
Potential for professional growth and obtaining Databricks or Microsoft Fabric certifications.
Collaborative work culture with emphasis on knowledge sharing and team development.
Full Job Description
Job Description Description:
Designing,developing, and optimizing Azure Data Factory and Databricks pipelinesfrom Oracle databases to Lakehouses
Designingand optimizing data models using Medallion Architecture principles
Designing,developing and optimizing data connections from Databricks MedallionArchitecture to on-premises data sources for downstream consumers.
Designingand optimizing relational data models
TranslatingInformatica ETL to Azure Data Factory and Databricks ELT
Knowledge Transfer:
TransferFrom DataOps/Cloud Data Engineer to Designated CSC Resource
When Knowledge Will Be Transferred:
Knowledgetransfer must be completed one week prior to the end of the project or oneweek prior to the consultant leaving the ministry.
What Knowledge Will Be Transferred:
Alldeliverables, including design/supporting/release/training documents mustbe checked into designated version control repositories (for example, gitrepositories or SharePoint). All final documents and working draftsrelated to project requirements or solution design must be stored ondesignated project repositories (for example, SharePoint site, HPQC, TFS,etc.)
Projectmanager and designated ministry staff must be regularly informed inwriting (by email) of where documentation has been stored and must beprovided a minimum of one walk-through of all documentation as part of thefinal knowledge transfer activities.
How Knowledge Will Be Transferred:
Knowledgewill be transferred through 1 on 1 sessions, emails, document updates anddocument review with the team.
Requirements
Experience and Skill Set Requirements:
Must Haves:
Experiencedeveloping ETL processes with Informatica, Azure Data Factory andDatabricks (Python, Lakeflow, and SQL Optimization)
Experiencewith solution development with Databricks in Medallion Architecture.
Experiencewith Cloud data engineer must be familiar and experienced with differentprogramming languages (Python, SQL, t-SQL, PL/SQL, Informatica, ADF, SSIS.Microsoft Fabric) and be able to integrate with many different platformsto create data pipelines, jobs, automate tasks, and write scripts.
Experiencewith Data pipeline and workflow development, orchestration, deployment,and automation and specializing in programming and pipelines to create andmanaging the dataflow, Delta Lake indexing, parallelism and movement ofdata.
Skill Set Requirements:
Technical Experience:
Experiencedeveloping ETL processes with Informatica, Azure Data Factory andDatabricks (Python, Lakeflow, and SQL Optimization)
Experiencewith solution development with Databricks in Medallion Architecture.
Experiencewith Cloud data platforms, data management and data exchange tools andtechnologies
Experiencewith Commercial and opensource data and database development andmanagement and specializing in data storage setting (Delta Lake) andmanaging cloud Data As a Service (DaaS). application Database as a Service(DBaaS), Data Warehouses as a Service (DWaaS), and other storage platforms(both in the cloud and on-premise).
Experiencewith Data pipeline and workflow development, orchestration, deployment,and automation and specializing in programming and pipelines to create andmanaging the dataflow, Delta Lake indexing, parallelism and movement ofdata.
Experiencewith Cloud data engineer must be familiar and experienced with differentprogramming languages (Python, SQL, t-SQL, PL/SQL, Informatica, ADF, SSIS.Microsoft Fabric) and be able to integrate with many different platformsto create data pipelines, jobs, automate tasks, and write scripts.
Experiencewith digital product, data analysis, data exchange, data integration APIs,data provisioning, and data security
Extensiveexperience in designing/developing and implementing data conversation andmigration of VLD (Very large Data) of Online analytical processing (OLAP)and online transaction processing (OLTP) environments to CloudSoftware-as-a-service (SaaS), Platform-as-a-service (PaaS) andInfrastructure-as-a-service (IaaS) environments.
Experiencein design, development and implementation of fact/dimension model, datamapping, data warehouse, data lakehouse for enterprise.
Demonstratedexperience developing star schema multi-dimensional models and documentingdetailed design models.
Experiencemanaging Cloud Data services for project delivery, including storage,repositories, Data Lakehouse, key vault, virtual machine, Azure StorageAccount, SIHR, parquet files etc.
Experiencewith structured, semi-structured, unstructured data collection, ingestion,provisioning and exchange technological development of enterprise datawarehouse and data lake and data lakehouse solutions and operationalsupport.
Experiencewith DataOPS performance monitoring and tuning
Implementingand enforcing data quality checks and principles, including datavalidation, profiling, cleansing, and monitoring across pipelines
Applyingdata governance practices and information architecture standards,including development of conceptual, logical, and physical data models
Integratingand managing Microsoft Entra ID for secure authentication, authorization,and role-based access control across data platforms
Implementingdata anonymization and masking techniques to protect sensitive andregulated data in compliance with privacy requirements
Developingand maintaining data lineage reports to provide end-to-end visibility andtraceability of data movement and transformations
Databricksand/or Microsoft Fabric certifications are considered an asset
Core Skills:
Excellentanalytical, problem-solving and decision-making skills; verbal andwritten communication skills; presentation skills; interpersonal andnegotiation skills
Demonstratesdevelopment and solution design experience with technology, data,databases, statistics, applications, and networking techniques, tools andpractices that enables the candidate to lead the design and development ofbusiness intelligence projects.
Experiencein DataOPS best practices, and Agile project development and deployment
Demonstratesexperience in working with and analyzing large and diverse data sets, datavisualization, KPI development and data profiling to identifyrelationships and themes.
Demonstratesexperience manipulating and analyzing complex, high-volume data fromstructured and unstructured sources.
Demonstratesexperience creating detailed documentation for the tools that aredeveloped, such as business requirements, system requirements andtechnical requirements.
Demonstratesexperience with advanced data visualization tools and techniques,designing high quality interfaces to present information in a meaningfulway.
Troubleshootingof production issues and creating fixes for any unplanned deployments.
Defectinvestigation, resolution and assignment to team members usingdefect/issue.
Demonstratedexperience in coordinating and planning for the modernization of large,complex multi-platform, multitier information technology systems.
Experiencewith Azure DevOps.
Experiencewith AODA/WCAG compliance.
Ateam player with a track record for meeting deadlines
Project Experience and Techniques:
Demonstratesdevelopment and design experience with SDLC processes, Agile and Waterfallmethodologies.
Demonstratesexperience conducting system testing and unit testing, and conducting SIT,SAT and UAT Testing.
Demonstratesexperience communicating regarding day-to-day tasks and issue tracking,reporting and facilitating resolution of issues and risks for all projectactivities.
Demonstratesexperience gathering and consolidating input into changes required tobusiness and/or system requirements, change requests, project artefacts,as well as leading requirement gathering sessions.
General Skills:
Demonstratesexperience problem solving and decision making.
Demonstratesexperience communicating clearly in both written and verbal formats.
Demonstratesexperience working with and leading multiple teams representing variousareas in delivering the project.
Demonstratesability to work collaboratively, with the ability to coordinate multipleprojects with competing priorities and a track record for meeting strictdeadlines.
Demonstratesexperience working collaboratively with other team members and/or groupsand leading a team to ensure optimal solution integration.
Demonstratesexperience leading a team and with providing technical advice andguidance.
Demonstratesexperience working with both the business users and IT development teamsto ensure business requirements are properly reflected in the systemdesign and technical specifications.
Organization Experience:
Previouspublic sector experience in an organization of equivalent size.
