Optimized Job TitleFull Stack Data Engineer Overview / Summary We are seeking an experienced Full Stack Data Engineer to support the development and maintenance of an enterprise data platform. This role is responsible for designing, building, and optimizing scalable data pipelines and data solutions in a Google Cloud Platform (GCP) environment.
The position focuses on data infrastructure, pipelines, data platforms, data governance, security, performance optimization, and automation. The ideal candidate will have experience with GCP technologies, SQL, Java, Python, cloud-based data platforms, and data engineering solutions.
Key Responsibilities - Collaborate with business and technology stakeholders to understand current and future data requirements.
- Design, build, and maintain reliable, efficient, and scalable data infrastructure for data collection, storage, transformation, and analysis.
- Design, develop, and maintain scalable data pipelines, data models, and applications for efficient and reliable data workflows.
- Design, implement, and maintain data warehouses, data lakes, and data lakehouse solutions for structured and unstructured data.
- Design and develop analytical tools, algorithms, and programs to support data engineering activities.
- Write scripts and automate data platform tasks and processes.
- Design, develop, and maintain scalable data ingestion and curation pipelines from diverse data sources.
- Ensure data is standardized, high quality, and optimized for analytical use.
- Contribute to end-to-end development to support reliable data flow from source to insight.
- Build and manage data platforms using GCP services such as BigQuery, Dataflow, Pub/Sub, and Cloud Functions.
- Implement and manage data governance policies, access controls, and security best practices.
- Support data security through GCP-native security features, data encryption, and data masking techniques.
- Use Astronomer and Terraform for data workflow management and cloud infrastructure provisioning.
- Monitor and improve the performance, scalability, and efficiency of data pipelines and storage solutions.
- Monitor and optimize cost and compute resources for GCP processes.
- Collaborate with data architects, application architects, service owners, and cross-functional teams to define and promote best practices, design patterns, and frameworks for cloud data engineering.
- Automate data platform processes to improve reliability, data quality, operational efficiency, and minimize manual intervention.
- Identify opportunities for improvement and ensure optimal performance of data solutions.
Required Qualifications - Bachelor's degree in Computer Science, Information Technology, Information Systems, Data Analytics, or a related field, or an equivalent combination of education and experience.
- 5-7 years of experience in Data Engineering or Software Engineering.
- At least 2 years of hands-on experience building and deploying cloud-based data platforms.
- Strong proficiency in SQL, Java, and Python.
- Practical experience designing and deploying cloud-based data pipelines using GCP services such as BigQuery, Dataflow, and DataProc.
- Experience with GCP-based data platforms and technologies.
- Understanding of Service-Oriented Architecture (SOA) and microservices and their application within a cloud data platform.
- Experience with relational databases such as PostgreSQL and MySQL.
- Experience with NoSQL databases and columnar databases such as BigQuery.
- Knowledge of data governance frameworks, data encryption, and data masking techniques in cloud environments.
- Familiarity with CI/CD pipelines.
- Familiarity with Infrastructure as Code (IaC) tools such as Terraform and Tekton.
- Experience with automation frameworks.
- Experience with data/analytics dashboards.
- Strong analytical and problem-solving skills, including the ability to troubleshoot complex data platform and microservices issues.
- Experience monitoring and optimizing cost and compute resources for GCP technologies such as BigQuery, Dataflow, Cloud Run, and DataProc.
- Experience with Python, SQL, and DBT/Dataform for building data pipelines.
- Experience with GCP services including BigQuery, Dataflow, Pub/Sub, and Cloud Functions.
- Experience with Astronomer and Terraform for data workflow management and cloud infrastructure provisioning.
- Passion for data, innovation, and continuous learning.
- Master's degree is preferred.
- Hybrid position requiring 4 days per week onsite.
#LI-RM1
