Senior Data Integration Engineer Responsibilities Pipeline Architecture & Development: Design, build, and optimize scalable, reliable batch and near-real-time ETL/ELT pipelines using Python, PySpark, SQL, and modern cloud integration engines Orchestration & Automation: Develop and manage complex workflow orchestrations (using Apache Airflow or cloud native schedulers) and automate ingestion routines to minimize manual operations Data Modeling & Warehousing: Design and implement modern data warehouse/lakehouse layers (using Snowflake, ClickHouse, Azure Synapse, or Redshift), establishing optimal partitioning, indexing, and Slowly Changing Dimension (SCD Type 2) patterns Data Quality & Testing Integration: Establish rigorous data quality checks and validation frameworks utilizing tools like dbt (data build tool), Soda, or customized PySpark testing suites Collaboration & Design: Work closely with product owners, business analysts, and systems architects to define data requirements, analyze technical constraints, design Source-to-Target Mappings (STTM), and make critical architectural decisions Code Quality & DevOps: Maintain a clean, modular code repository. Lead code reviews, enforce engineering standards, and configure robust CI/CD pipelines (Azure DevOps, GitLab CI, or GitHub Actions) with Docker containers Technical Documentation: Deliver comprehensive, clear technical specs, metadata lineage documentation, architectural diagrams, and data dictionaries Requirements Experience: 5+ years of hands-on experience in data engineering, data warehousing, database design, and end-to-end data integration ETL & Integration Tools: Advanced knowledge of Cloud Integration tools such as Azure Data Factory (ADF), AWS Glue, or GCP Dataflow Orchestration & Real-Time Ingestion: Proficiency in workflow orchestrators like Apache Airflow and exposure to CDC (Change Data Capture) or real-time streaming tools (e.g., Kafka, Debezium) Core Technical Stack: Strong production-level coding skills in SQL (advanced optimization/stored procedures), Python, and PySpark / Apache Spark Analytical Databases & Cloud Warehouses: Experience working with high-performance databases and cloud-native systems (e.g., Snowflake, ClickHouse, PostgreSQL, MS SQL Server, or Azure Synapse) Methodologies: Master-level understanding of data modeling practices (OLAP, OLTP, Star/Snowflake schemas, Delta Lake/Lakehouse patterns, and Data staging processes) DevOps & CI/CD: Hands-on experience with version control (Git) and building automated deployment pipelines (CI/CD) for data products Communication & English: Proven ability to articulate complex technical ideas clearly to both business stakeholders and developers. Fluency in English (Upper-Intermediate level or higher) Nice to have Data Transformation & Quality Tools: Deep knowledge of dbt (data build tool) and schema validation practices Containerization: Experience using Docker or Kubernetes to package and deploy data applications Serverless Engineering: Experience building lightweight, serverless ingestion services (e.g., using AWS Lambda / Azure Functions and RESTful APIs)