Job SummaryWe are looking for a Senior Data Engineer to design, build, and optimize scalable data pipelines and data platforms supporting analytics, reporting, and AIML use cases. The ideal candidate has strong hand son experience with Snowflake on AWS, Python based ETLELT development, and enterprise scheduling orchestration tools like Control M, along with legacy enterprise ETL experience in IBM DataStage. You will collaborate across engineering, analytics, and business teams in an Agile delivery model.
Responsibilities- Design, develop, and maintain batch and near real-time data pipelines using Snowflake, AWS, and Python.
- Build and optimize Snowflake data models (dimensional, data vault, curated marts).
- Develop and maintain ETL/ELT workflows using Python and IBM DataStage; migrate and modernize workloads.
- Implement job scheduling, monitoring, and operational support with Control-M (alerting, retries, SLAs, dependencies).
- Ensure data quality, governance, lineage, and documentation standards across pipelines.
- Perform performance tuning and cost optimization in Snowflake and AWS (query optimization, clustering, warehouse sizing).
- Collaborate with Data Science, AI, BI, Product, and Platform teams to enable data products and curated datasets.
- Participate in Agile ceremonies, contribute to estimation, sprint planning, and execution.
- Troubleshoot production issues, perform root cause analysis, and drive preventative improvements.
Required Technical Skills- Snowflake: Strong expertise in architecture, SQL development, performance tuning, security roles, and data loading/unloading.
- AWS: Hands-on experience with S3, IAM, CloudWatch, Glue, Lambda, EC2, Step Functions, EMR, or Kinesis.
- Python: Strong programming for ETL/ELT frameworks, API ingestion, automation, unit testing, and logging.
- Control-M: Enterprise job scheduling, dependencies, calendars, SLAs, monitoring, and incident handling.
- IBM DataStage: Building and maintaining jobs, complex transformations, and production support.
- SQL: Advanced skills for transformations, optimization, and validation across large datasets.
- CI/CD & Git: Experience with version control and pipeline automation.
- Operational Excellence: Monitoring, alerting, and production support in business-critical environments.
Good to Have- Exposure to AI/ML pipelines or feature datasets.
- Familiarity with data governance tools (catalogue, lineage, data quality frameworks).
- Experience with streaming/event-driven architectures.
Primary Skills: Snowflake, SQL, AWS, Python, DBT & DataStage
Secondary Skills: AWS Glue, Qlik, Control-M & Airflow.
