JOB SUMMARY
This role involves collaborating with business, IT, Analyst, and Data Science groups to define data requirements. The position requires designing, developing, deploying, and supporting high-performance data pipelines, modeling data platforms, and ensuring data quality. It also includes developing REST APIs for data exposure and mentoring junior data engineers.
Key Responsibilities
• Collect business requirements by working closely with various business, IT, Analyst, and Data Science groups.
• Design, develop, deploy, and support high-performance data pipelines, both inbound and outbound.
• Model the data platform by applying business logic and building objects in the semantic layer.
• Optimize data pipelines for performance, scalability, and reliability.
• Implement CI/CD pipelines for continuous deployment and delivery of data products.
• Ensure the quality of critical data elements, prepare data quality remediation plans, and collaborate to fix quality issues.
• Document the design and support strategy of data pipelines.
• Capture, store, and socialize data lineage and operational metadata.
• Troubleshoot and resolve data engineering issues.
• Develop REST APIs to expose data to other teams.
• Mentor and guide junior data engineers.
Required Qualifications
• 6+ years of experience in data engineering solutions such as data platforms, ingestion, data management, or publication/analytics.
• 2+ years of experience in Google Cloud Platform (GCP) with services like BigQuery, Composer, GCS, DataStream, or Dataflows.
• Expert knowledge of SQL and Python programming.
• Experience working with Airflow as a workflow management tool, including building operators to connect, extract, and ingest data.
• Experience in tuning queries for performance and scalability.
• Experience in real-time data ingestion using GCP Pub/Sub, Kafka, Spark, or similar technologies.
• Proven experience working in an agile environment.
• Proven experience working in incremental execution through successful launches.
Preferred Qualifications
• None specified in the provided text.
Certifications
• None specified in the provided text.
