We are looking for a Senior Data Engineer to join our team in Markham, Ontario, Fredericton, New Brunswick or remotely from a location in Canada. This role will be instrumental in building and maintaining the applications and infrastructure that support our data-driven initiatives. You will work closely with data scientists, data engineers, and analysts to develop end-to-end solutions, from robust backend APIs to dynamic reporting interfaces.
Experience in programming and automation is key as is the requirement to have a strong ability to work with databases, distributed computing systems, data pipelining, and Ops.
Key responsibilities include:- Design, develop, and maintain robust backend ETL pipelines utilizing PySpark, Python and SQL.
- Load data into medallion architecture layers from diverse upstream sources-including both streaming and batch.
- Develop and manage APIs to facilitate secure data access and support application functionality.
- Collaborate with cross-functional teams to gather data requirements and deliver technical solutions.
- Participate in all phases of the software development lifecycle, including design, implementation, testing, and deployment.
- Support the Systems Architect in designing infrastructure and data stack components.
- Develop ETL workflows to provide stakeholders with actionable Network Intelligence.
The ideal candidate will possess:- Bachelor's degree in Computer Science, Engineering, or a related discipline.
- At least five years of practical experience in Data Engineering, Data Science, or full-stack development.
- Advanced proficiency in Python for backend software development.
- Extensive experience building pipelines in hybrid on-premises/cloud environments.
- Demonstrated expertise in continuous integration/continuous deployment (CI/CD) practices.
- Strong background in API design and development (e.g., REST APIs).
- Experience with Terraform, and Databricks is considered a strong asset.
- Significant experience working with relational databases, particularly PostgreSQL.
- Familiarity with data engineering principles, ETL processes, and pipeline architectures is desirable.
- Excellent analytical and problem-solving skills, with the ability to work autonomously as well as collaboratively.
- Outstanding communication and teamwork abilities.
Condition of Employment:As a condition of employment and in order to comply with industry related data security standards, this position is subject to the successful completion of a Criminal Background Check. Details will be supplied to applicants as they move through the selection process.
