Full Job Description
In the role of Data Engineer 2, we'll count on you to:
• Build and maintain batch and streaming ingestion pipelines from source systems into the enterprise data platform.
• Implement robust error handling, retries, and recovery patterns to improve reliability.
• Collaborate with platform teams on connectivity, access, and operational readiness.
• Develop transformation models using a layered structure (raw, conformed, curated) to separate concerns and improve maintainability.
• Use dependency management patterns to ensure correct build order and reproducible transformations.
• Leverage shared code and reusable components to standardize patterns, reduce duplication, and enforce consistent logic.
• Implement incremental processing where appropriate to reduce runtime and warehouse cost.
• Implement data quality tests (e.g., unique, not_null, accepted_values) on critical models and key business entities.
• Configure and monitor source freshness expectations for important upstream datasets.
• Maintain model and column-level documentation; publish documentation and lineage to enable impact analysis and change management.
• Integrate transformations into CI/CD workflows (build, test, docs generation) before promotion to higher environments.
• Participate in code reviews, follow branching standards, and adhere to quality gates and approval processes.
• Support release and rollback practices to maintain stability and reduce risk.
• Tune transformations for performance and efficiency; partner with stakeholders to manage cost impacts.
• Support monitoring and alerting for pipelines and transformations; respond to incidents and perform root cause analysis.
• Document operational procedures and contribute to runbooks/playbooks.
Preferred Qualifications
• Hands-on experience with dbt Cloud or dbt Core in Git-integrated workflows.
• Experience with orchestration tools (e.g., Airflow, Dagster, Prefect) and scheduling patterns.
• Experience implementing CI/CD quality gates for data (tests, linting, docs artifacts).
Required Qualifications
• Bachelor's degree in Computer Science, Information Systems, Data Engineering, or equivalent practical experience.
• Minimum 3 years of experience in data engineering, data integration, or related roles.
• Proven SQL proficiency and experience building transformations and pipelines.
• Familiarity with cloud data platforms and modern ELT patterns.
Qualifications