Job Description
The client is strategically transitioning the Data Warehouse engineering operations onshore and is looking for a driven Data Warehouse Engineer to help lead this initiative.
Your primary focus will be migrating, building, and optimizing ELT pipelines that extract data from third-party APIs and load it into our data lake utilizing managed Delta tables. You will initially work closely with our existing offshore team to complete a comprehensive Knowledge Transfer (KT).
Key Responsibilities
ELT Pipeline Development: Design, build, and maintain robust ELT (Extract, Load, Transform) pipelines to migrate data from various third-party API sources into our enterprise data lake.
Databricks Ecosystem: Create, manage, and optimize managed Delta tables within a Databricks environment to ensure high data reliability and query performance.
API Integration: Securely connect to, authenticate with, and extract data from a wide variety of third-party RESTful APIs.
Data Modeling & Querying: Write clean, scalable, and efficient Python and SQL code to transform raw data into analytics-ready datasets.
Knowledge Transfer (KT): Actively participate in onboarding and KT sessions with the current offshore team to seamlessly transition ownership of existing pipelines and architecture.
Documentation: Maintain clear technical documentation for data pipelines, API integrations, and Delta table schemas to support future onshore hires.
Required Skills & Qualifications
Experience: 4+ years of experience in data engineering, data warehousing, or a similar technical role.
Core Languages: Strong, hands-on proficiency in Python and SQL.
Databricks & Data Lakes: Practical experience working with Databricks, specifically building and maintaining Delta tables and working within a modern data lake architecture.
API Expertise: Proven experience extracting and handling data from third-party APIs (handling pagination, rate limiting, JSON/XML parsing, etc.).
Methodology: Solid understanding of ELT methodologies and cloud-based data architecture.
Communication: Excellent verbal and written communication skills, with the ability to effectively absorb complex technical information during cross-border knowledge transfers
