We9re seeking a Senior Data Engineer to join our Delivery Team. In this role, you9ll design and implement modern data architectures that enable our clients to make data-driven decisions. You9ll lead the strategy, design, and technical direction of scalable data ecosystems across cloud platforms - ensuring integration, performance, and compliance.
As a Senior Data Engineer, you9ll work closely with business stakeholders, data engineers, and analytics teams to design data solutions that align with client goals.
Responsibilities:
- Design and implement end-to-end data architectures, including data lakes, data warehouses, and analytics platforms.
- Define data integration and transformation strategies, ensuring scalability, security, and performance.
- Collaborate with stakeholders to translate business requirements into technical solutions that support analytics, reporting, and AI initiatives.
- Develop data models, ETL/ELT pipelines, and frameworks for structured and unstructured data.
- Provide technical leadership and mentorship to data engineers and developers, promoting best practices in data management and governance.
- Ensure compliance with data governance, security, and privacy standards across platforms.
- Optimize existing data architectures and processes for improved performance and reliability.
- Stay current with industry trends, cloud data services, and emerging technologies such as Databricks, Snowflake, Azure Synapse, and AWS Redshift.
- Act as a trusted advisor to clients, guiding them on architecture decisions and best practices for data modernization.
Required Skills & Experience
- 5+ years of experience in data architecture, data engineering, or analytics solution design.
- Hands-on experience with data lake and warehouse technologies (e.g., Databricks, Snowflake, Redshift, Synapse).
- Deep understanding of data modeling, data integration, and ETL/ELT design.
- Proficiency in SQL and one or more programming languages (Pyspark/Python, Scala), particularly for complex data transformations and optimization within Spark
- Solid understanding of data governance, security, and privacy best practices.
- Proven experience in designing, implementing, and optimizing large-scale ingestion pipelines using Databricks Autoloader
- Deep practical knowledge of building and managing reliable, self-managing ETL/ELT pipelines using Delta Live Tables
- Experience in building high-throughput, low-latency streaming data ingestion solutions using Apache Kafka, Spark Structured Streaming, and Databricks Streaming
- Extensive experience in successfully applying and enforcing the Medallion architecture (Bronze, Silver, Gold layers) within a Databricks environment
- Experience in designing and implementing CI/CD pipelines (using tools like Azure DevOps, GitHub Actions, GitLab CI) specifically tailored for Databricks workflows, notebooks, and cluster configurations, enabling automated deployment and testing
- Experience in planning and executing data migration projects from traditional data warehouses into the Databricks Lakehouse
- Strong working knowledge of at least one major cloud provider (AWS, Azure) regarding data storage, networking, and security concepts relevant to Databricks deployment.
- Proven ability to engage with clients, present technical solutions, and communicate complex ideas clearly.
- Bachelor9s or Master9s degree in Computer Science, Information Systems, Engineering, or a related field.
- Excellent problem-solving, communication, and collaboration skills.
Nice to Have:
- Experience with AI/ML integration and data science workflows.
- Knowledge of data cataloging and metadata management tools.
- Prior consulting or client-facing experience in a technology services firm.
BIG is a high growth Cloud Consulting firm with offices in Edmonton, Calgary, Toronto, Denver, India and the United Kingdom. Our clients are in Canada, UK, India and the US. We are a team of experienced IT professionals who specialize in providing business value to organizations looking at leveraging modern platforms such as Pega, Appian, MuleSoft and Boomi. Our vast experience in the IT industry and our current track record in enterprise software development, allow us to provide a full range of services to our clients. Bits In Glass helps organizations of all sizes to automate their businesses and leverage the power of the web and mobile technologies.
