Req ID: 266814
General PurposeThe Data Engineer will be responsible for designing, developing, and maintaining data pipelines and architectures using Microsoft Azure and Snowflake. This role will focus on optimizing data workflows, ensuring data integrity, and improving performance to support advanced analytics, reporting, and machine learning initiatives. This role is a critical part of CCSWB's Advanced Analytics team, ensuring that high-quality data is readily available for strategic business decisions.
Duties and Responsibilities1. Design, Develop, and Maintain scalable data pipelines to enable Advanced Analytics' initiatives and digital products
- Build and optimize ETL/ELT processes using Azure Data Factory, Databricks, and Snowflake.
- Develop batch and real-time data pipelines to support reporting and AI/ML applications.
- Implement data transformation and cleansing processes to ensure high data quality.
- Automate data workflows to enhance efficiency and reduce manual interventions.
- Monitor pipeline performance and troubleshoot issues to minimize downtime.
2. Oversee and Ensure data quality, integrity, and governance across Advanced Analytics' data ecosystem
- Implement data validation and anomaly detection techniques within pipelines.
- Work with business users and analysts to understand data quality issues and implement solutions.
- Maintain metadata and data lineage documentation for transparency and traceability.
- Collaborate with cross-functional teams to ensure data consistency and reliability.
3. Monitor, Evaluate, and Optimize data storage and processing for performance and cost efficiency
- Design and implement efficient data models to support the Advanced Analytics team' needs.
- Leverage partitioning, indexing, and clustering techniques for optimized query performance.
- Monitor and manage cloud-based storage and compute costs to ensure cost-effectiveness.
- Implement caching and performance tuning strategies for large-scale data processing.
- Analyze workload patterns and recommend infrastructure improvements.
4. Collaborate with data scientists, analytics translators, and business stakeholders to deliver data solutions
- Gather requirements and translate business needs into scalable data engineering solutions.
- Provide support to data scientists for feature engineering and model deployment.
- Partner with business intelligence teams to improve data accessibility for reporting tools.
- Develop reusable data assets and APIs for Analytics.
- Conduct training and knowledge-sharing sessions to promote data-driven culture.
5. Maintain and Enhance security, reliability, and automation of data infrastructure
- Execute the access control policies and role-based permissions in Azure according to Arca Continental's definitions.
- Automate deployment and monitoring processes using CI/CD pipelines and Infrastructure-as-Code (IaC).
- Set up robust logging and alerting mechanisms to proactively detect issues.
- Ensure compliance with internal and external data security regulations.
- Continuously evaluate and implement new tools and best practices for data engineering.
- End-to-end ownership - design, development, deployment, monitoring, production support, and enhancements.
- Production engineering - troubleshooting, root-cause analysis, reliability, and performance.
- Engineering practices - Git, CI/CD, code reviews, testing, and documentation.
- Future readiness - ability to support current solutions while adapting to evolving products, technologies, and architecture.
Qualifications- Bachelor's degree in Computer Science, Data Engineering, Information Systems, or a related field.
- Advanced degree is a plus.
- Strong experience working with cloud platforms, experience working with with Microsoft Azure services, including Azure Data Factory, Azure Databricks, and Azure SQL is a plus.
- Familiarity with Snowflake for data warehousing, including schema design and performance tuning.
- Expertise in SQL and experience with programming languages like Python, Scala, or Java.
- Knowledge of ETL/ELT processes, data modeling, and best practices for data pipeline development.
- Familiarity with CI/CD, Infrastructure-as-Code (Terraform, ARM templates), and DevOps practices.
- Understanding of data governance, security principles, and compliance standards.
- Experience with Apache Spark, Airflow, and API development is a plus.
- Strong SQL skills for database design, querying, and data manipulation.
- Knowledge of scripting languages (e.g., Bash) for automation and data pipeline orchestration.
- Understanding of data serialization formats like JSON, Avro, Parquet, and XML.
- Familiarity with various database systems, including relational databases (e.g., SQL Server, PostgreSQL) and NoSQL databases (e.g., MongoDB, Cassandra).
- Core Data Engineering skills - SQL, Python/PySpark, ETL/ELT, data modeling, and distributed processing.
- Current platform experience - Azure, Azure Data Factory, and Databricks.
- Production engineering - pipeline troubleshooting, performance optimization, data quality, and monitoring.
- DevOps/engineering practices - Git, CI/CD, testing, and deployment.
- Broader/future skills - cloud architecture, Infrastructure as Code, APIs, and other data/orchestration technologies.
- 30% travel projected
