As a Senior Data Engineer, you will be responsible for developing complex data sources and pipelines within our data platform, including Snowflake, Azure, Airflow, and related automation tools.
You will work closely with Data, Architecture, Business Analysts, and Data Stewards to align requirements, specifications, and constraints across solutions. This role also helps identify gaps in resources, technology, or capabilities and partners with the data engineering team to implement effective solutions with a solid focus on customer usability.
Primary Responsibilities:- Integrate data from multiple on-premises and cloud sources and systems, handling ingestion, transformation, and consolidation to create a unified and reliable data foundation for analysis and reporting
- Develop data transformation routines to clean, normalize, and aggregate data, including handling complex data structures and missing or inconsistent data
- Implement data de-identification and data masking in line with company standards
- Monitor data pipelines and data systems to detect and resolve issues promptly
- Develop monitoring tools and automate error-handling mechanisms to ensure data integrity and system reliability
- Use data quality tools such as Great Expectations or Soda to support the accuracy, reliability, and integrity of data throughout its lifecycle
- Create and maintain data pipelines using Airflow and Snowflake as primary tools
- Develop SQL stored procedures to perform complex transformations
- Understand data requirements and design optimal pipelines to fulfill business use cases
- Create logical and physical data models to maintain data integrity
- Support CI/CD pipeline creation and automation using Git and Git Actions
- Tune and optimize data processes
You'll be rewarded and recognized for your performance in an environment that will challenge you and give you clear direction on what it takes to succeed in your role as well as provide development for other roles you may be interested in.
Required Qualifications:- Bachelor's degree in Computer Science or a related field
- Hands-on experience developing data pipelines in Snowflake and writing complex SQL queries
- Proven hands-on experience as a Data Engineer
- Experience with CI/CD pipelines using Git and Git Actions
- Experience building ETL/ELT data pipelines
- Experience with related open-source platforms and languages such as Scala, Python, Java, and Linux
- Experience with relational and non-relational databases
- Experience working on Agile/Scrum projects with high-performing teams
- Knowledge of data modeling techniques, including star schema, dimensional models, and Data Vault
- Knowledge of data warehousing principles, architecture, and implementation
- Knowledge of mesh compliance is desirable but not essential
- Solid knowledge of Python
- In-depth knowledge of Snowflake architecture, features, and best practices
- Good understanding of access control, data masking, and row access policies
- Exposure to DevOps methodology
- Proficiency in SQL, including window functions and advanced features
- Proven solid written and verbal communication skills
- Proven solid analytical and problem-solving skills applied to large data sets
Preferred Qualifications:- Bachelor's degree or higher in Database Management, Information Technology, Computer Science, or a related field
- Experience in Data Engineering
- Experience orchestrating data tasks in Airflow to run on Kubernetes for data ingestion, processing, and cleaning
- Experience with Databricks
- Familiarity with Azure services such as Blob Storage, Functions, Azure Data Factory, Service Principal, Containers, and Key Vault
- Expertise in designing and implementing data pipelines to process high volumes of data
- Motivated self-starter who excels at managing tasks independently and takes ownership
- Proven ability to create Docker images for applications to run on Kubernetes
Pay is based on several factors including but not limited to local labor markets, education, work experience, certifications, etc. In addition to your salary, we offer benefits such as, a comprehensive benefits package, incentive and recognition programs, equity stock purchase and 401k contribution (all benefits are subject to eligibility requirements). No matter where or when you begin a career with us, you'll find a far-reaching choice of benefits and incentives. The salary for this role will range from $91,700 to$163,700 annually based on full-time employment. We comply with all minimum wage laws as applicable.
