Role descriptionWhat is in it for you?As a GCP Data Engineer, you will be a part of an Agile team to build healthcare applications and implement new features while adhering to the best coding development standards .
Summary:We are looking for an experienced GCP Data Engineer to design and build an automated solution for data discovery, end-to-end lineage, metadata management, and data quality across multiple GCP projects. The solution will trace data movement from source systems through GCP processing layers, store metadata centrally in BigQuery, and integrate lineage information with Collibra using OpenLineage APIs
Responsibilities: -- Build programmatic upstream and downstream data lineage across multiple GCP projects and BigQuery datasets.
- Discover datasets, tables, views, stored procedures, jobs, and other relevant data assets.
- Trace data movement and intermediate processing through BigQuery, Dataflow, Cloud Data Fusion, stored procedures, Cloud Storage, and related GCP services.
- Use GCP Data Lineage/Dataplex APIs, BigQuery INFORMATION_SCHEMA, job metadata, and other GCP APIs to collect and enrich lineage information.
- Capture available table-level and column-level lineage transformation details, and pipeline execution metadata.
- Extract data quality rules, scans, and results using Dataplex and related metadata services.
- Develop Python/SQL-based services and pipelines to collect, normalize, validate, and store metadata in BigQuery.
- Generate standardized OpenLineage JSON/events and integrate lineage metadata with Collibra.
- Implement secure cross-project access using GCP service accounts, IAM roles, and least-privilege principles.
- Build monitoring, logging, error handling, and audit capabilities for metadata and lineage processing.
- Support AI/agentic workflows for metadata analysis, lineage enrichment, and identification of lineage or data quality gaps.
Experience: -• 7+ Years
Location: -• Remote
Educational Qualifications: -- Engineering Degree - BE/ME/BTech/MTech/BSc/MSc.
• Technical certification in multiple technologies is desirable.
Mandatory skills: -- Strong hands-on experience with Python and SQL
- Strong experience with Google BigQuery, including metadata and INFORMATION_SCHEMA`
- Experience with Dataflow and/or Cloud Data Fusion
- Experience integrating with GCP APIs and REST APIs
- Understanding of data lineage, metadata management, data quality, and data governance
- Experience with GCP IAM, service accounts, and cross-project access
- Experience designing and supporting production-grade data pipelines
Good to have:- Dataplex / GCP Data Lineage API
- Collibra and OpenLineage
- Cloud Run and Cloud Storage
- Terraform / Infrastructure as Code
- Vertex AI / agentic or LLM-based workflows
- SQL parsing and transformation dependency analysis
- Enterprise data governance and metadata platforms
