OverviewGDC IT Solutions is currently seeking a Data Engineer & Insights Analyst to support the Commonwealth of Pennsylvania's analytics initiatives and the Pennsylvania Longitudinal Data System (PALDS). This position will focus on developing and optimizing enterprise data pipelines, integrating complex datasets, and delivering actionable insights to support data-driven decision-making across state government agencies.
As a Data Engineer & Insights Analyst you will be responsible for designing, developing, and maintaining scalable data pipelines and cloud-based data solutions that transform complex, disconnected datasets into trusted, usable information. This role will support the Pennsylvania Longitudinal Data System (PALDS) by integrating data from multiple sources, improving data quality, and developing analytics products that support education and workforce outcomes, operational improvements, and executive decision-making. The successful candidate will collaborate with data scientists, research analysts, policy analysts, and other stakeholders to build reliable data environments, support advanced analytics, and optimize the Commonwealth's longitudinal data architecture.
Responsibilities- Design, develop, and maintain scalable data pipeline architectures to support enterprise data and analytics initiatives.
- Design and implement cloud-based data solutions using AWS services, including S3, Glue, Redshift, EMR, Athena, Kinesis, and Lambda.
- Build and maintain data orchestration workflows using Apache Airflow, AWS Step Functions, or similar technologies.
- Develop and optimize infrastructure for extracting, transforming, and loading data from multiple sources using SQL and big data technologies.
- Assemble and manage large, complex datasets that meet functional and non-functional business requirements.
- Engineer analytical data environments that support statistical analysis, predictive modeling, machine learning, and evidence-building activities.
- Develop and maintain data pipeline monitoring, alerting, logging, and validation frameworks to ensure data quality and reliability.
- Design and implement data transformation processes, metadata structures, dependency management, and workload management solutions.
- Develop dashboards, reports, analytical models, and decision-support products for executive and non-technical audiences.
- Build analytics tools that provide actionable insights into education and workforce outcomes, operational efficiency, and key business performance metrics.
- Partner with data scientists, research analysts, and policy analysts to support reproducible research and statistical modeling.
- Identify and implement process improvements through automation, infrastructure optimization, and scalable data delivery solutions.
- Perform root-cause analysis to identify data quality issues, resolve technical challenges, and improve internal and external data processes.
- Support data conversions associated with system modernization and replacement initiatives.
- Collaborate with executive, data, design, and support teams to address technical issues and support enterprise data infrastructure needs.
- Ensure compliance with Commonwealth policies, data governance standards, privacy requirements, security controls, and interagency data-sharing agreements.
Minimum Qualifications- Advanced SQL knowledge and experience working with relational databases, query authoring, and various database technologies.
- Experience building, optimizing, and maintaining big data pipelines, data architectures, and complex datasets.
- Advanced experience designing, developing, and refining dashboards, reports, analytical models, and decision-support products for executive and non-technical users.
- Experience working with student-level higher education data, including linking education records with employment and wage data to measure outcomes and generate actionable insights.
- Experience manipulating, processing, and extracting value from large, disconnected datasets.
- Experience building data transformation processes and managing data structures, metadata, dependencies, and workloads.
- Experience performing root-cause analysis on internal and external data to answer business questions and identify opportunities for improvement.
- Strong analytical skills and experience working with unstructured datasets.
- Working knowledge of message queuing, stream processing, and highly scalable big data storage technologies.
- Strong project management, organizational, and problem-solving skills.
- Experience collaborating with cross-functional teams in a dynamic environment.
- Hands-on experience designing and implementing cloud-based data solutions using AWS services such as S3, Glue, Redshift, EMR, Athena, Kinesis, and Lambda.
- Experience with Apache Airflow, AWS Step Functions, or comparable data orchestration technologies.
- Experience developing analytics environments that support predictive modeling, machine learning, advanced statistical analysis, and reproducible research.
- Experience developing data quality controls, pipeline monitoring, automated validation, alerting, and logging frameworks.
- Experience supporting enterprise data modernization, system replacement, and data conversion initiatives.
- Familiarity with data governance, confidential data management, privacy requirements, and interagency data-sharing practices.
