Data Engineer IIIOverview / Summary We are seeking a Data Engineer III to design, build, and maintain scalable data solutions, including data infrastructure, pipelines, platforms, and applications for collecting, storing, processing, and analyzing large volumes of data efficiently and accurately.
This role will focus on developing scalable data ingestion and curation pipelines, building and managing cloud-based data platforms, improving data quality and performance, and collaborating with technical and business stakeholders to support data engineering initiatives.
Key Responsibilities - Collaborate with business and technology stakeholders to understand current and future data requirements.
- Design, build, and maintain reliable, efficient, and scalable data infrastructure for data collection, storage, transformation, and analysis.
- Plan, design, build, and maintain scalable data solutions, including data pipelines, data models, and applications.
- Design, implement, and maintain data platforms, including data warehouses, data lakes, and data lakehouses for structured and unstructured data.
- Design and develop analytical tools, algorithms, and programs to support data engineering activities, including scripting and task automation.
- Develop and maintain scalable data ingestion and curation pipelines from diverse data sources.
- Standardize and optimize data for analytical use while maintaining data quality.
- Leverage Python, SQL, and DBT/Dataform to build robust and efficient data pipelines.
- Utilize GCP services, including BigQuery, Dataflow, Pub/Sub, and Cloud Functions, to build and manage data platforms.
- Implement and manage data governance policies, access controls, and security best practices.
- Support efficient data workflow management and cloud infrastructure provisioning using Infrastructure as Code (IaC) best practices.
- Monitor and improve the performance, scalability, and efficiency of data pipelines and storage solutions.
- Identify opportunities to improve resource utilization and cost-effectiveness.
- Collaborate with data architects, application architects, and cross-functional teams to define and promote best practices, design patterns, and frameworks for cloud data engineering.
- Automate data platform processes to improve reliability, data quality, and operational efficiency.
- Communicate complex technical decisions clearly to technical and non-technical stakeholders.
- Stay current with industry trends and emerging technologies and identify opportunities to improve data platforms and capabilities.
- Develop comprehensive documentation for data engineering processes to support knowledge sharing, collaboration, and long-term system maintainability.
Required Qualifications - 7+ years of experience in Data Engineering.
- Bachelor's degree.
- Experience with Google Cloud Platform (GCP).
- Experience with GCP Cloud Run and BigQuery.
- Experience with Kafka.
- Experience with cloud architecture and cloud computing.
- Experience with software development and application development.
- Experience with SQL.
- Experience with Big Data technologies.
- Experience with Java.
- Experience with Python.
- Experience with application testing.
- Experience with Agile software development.
- Experience with APIs.
- Experience related to Artificial Intelligence and Expert Systems.
#LI-Hybrid #LI-AA1
