Job Description
The client is strategically transitioning the Data Warehouse engineering operations onshore and is looking for a driven Data Warehouse Engineer to help lead this initiative.
Your primary focus will be migrating, building, and optimizing ELT pipelines that extract data from third-party APIs and load it into our data lake utilizing managed Delta tables. You will initially work closely with our existing offshore team to complete a comprehensive Knowledge Transfer (KT).
Key Responsibilities
ELT Pipeline Development Design, build, and maintain robust ELT (Extract, Load, Transform) pipelines to migrate data from various third-party API sources into our enterprise data lake.
Databricks Ecosystem Create, manage, and optimize managed Delta tables within a Databricks environment to ensure high data reliability and query performance.
API Integration Securely connect to, authenticate with, and extract data from a wide variety of third-party RESTful APIs.
Data Modeling & Querying Write clean, scalable, and efficient Python and SQL code to transform raw data into analytics-ready datasets.
Knowledge Transfer (KT) Actively participate in onboarding and KT sessions with the current offshore team to seamlessly transition ownership of existing pipelines and architecture.
Documentation Maintain clear technical documentation for data pipelines, API integrations, and Delta table schemas to support future onshore hires.
Required Skills & Qualifications
Experience 4+ years of experience in data engineering, data warehousing, or a similar technical role.
Core Languages Strong, hands-on proficiency in Python and SQL.
Databricks & Data Lakes Practical experience working with Databricks, specifically building and maintaining Delta tables and working within a modern data lake architecture.
API Expertise Proven experience extracting and handling data from third-party APIs (handling pagination, rate limiting, JSON/XML parsing, etc.).
Methodology Solid understanding of ELT methodologies and cloud-based data architecture.
Communication Excellent verbal and written communication skills, with the ability to effectively absorb complex technical information during cross-border knowledge transfers
