Bachelor's Degree in Computer Science, Software Engineering, Data Science, Engineering, or related field + 2 years of relevant experience; equivalent Master's degree accepted; non-relevant bachelor's degree requires 5+ years of equivalent hands-on experience.
Proficiency in SQL and Python for data engineering and transformation.
Experience with PySpark and Microsoft Fabric for lakehouse transformations.
Strong understanding of data governance and security standards.
Familiarity with cloud data platforms and industrial data environments.
Responsibilities
Partner with analysts and stakeholders to create scalable data solutions.
Deliver analytics-ready datasets for downstream teams.
Enable industrial analytics by providing contextualized operational data.
Design and maintain robust data ingestion pipelines.
Develop Python and SQL transformations for data cleaning and enrichment.
Architect end-to-end workflows for data processing and publication.
Enforce data security and governance standards across platforms.
Benefits
Flexible working arrangement within commuting distance to the office.
Opportunity to work on a strategic, innovative team focused on advanced analytics.
Access to cutting-edge tools and technologies in data engineering.
Collaborative environment with cross-functional teams.
Potential for professional growth and development in a dynamic field.
Full Job Description
Job Description
Position Title:
Data Engineer III
Pay Rate:
$83,000 - $121,000
Multiple factors, including Individual experience, skills and abilities will determine where an employee is ultimately placed in the pay range.
Category/Shift:
Salaried Full-Time
Physical Location:
75 5th Street NW Atlanta, GA 30308 United States
The Job You Will Perform:
The Data Engineer is an important member of the Cost Innovation Team, which is a strategic, innovative, and cross-functional team that is focused on creating value for the company through the advanced analytics of manufacturing data. This position offers a flexible working arrangement provided that the home office is within a typical commuting distance to the A2C office.
Core Responsibilities
Analytics & Data Science Enablement
Partner with analysts, data scientists, engineers, and business stakeholders to translate functional requirements into scalable technical data solutions.
Deliver self-service, analytics-ready datasets that empower downstream teams to consume data without deep knowledge of underlying source systems.
Enable industrial analytics use cases by providing trusted, contextualized time-series, asset, event, and operational data for analytics platforms and applications.
Eliminate redundant data-preparation and contextualization efforts across disparate analytics and application teams.
Help establish standard data definitions, reusable data products, and consistent business context across enterprise and industrial analytics environments.
Support downstream analytics workflows, including advanced analytics, data science, visualization, and industrial process analytics.
Data Ingestion & Integration
Design, build, and maintain robust pipelines to ingest data from relational databases, operational systems, industrial historians, APIs, cloud platforms, and flat files.
Establish reliable and secure connections between source systems, cloud data platforms, and industrial data platforms.
Write optimized SQL queries and extraction logic for high-performance data retrieval.
Implement flexible ingestion patterns including batch, incremental, streaming, and replication-based approaches based on business latency requirements.
Integrate high-volume time-series data and industrial metadata while maintaining relationships between assets, measurements, events, and other operational entities.
Support ingestion and integration patterns used by platforms such as Cognite Data Fusion (CDF), Seeq, and Microsoft Fabric.
Data Storage & Lakehouse Engineering
Design and manage scalable storage structures across raw (bronze), intermediate (silver), and curated (gold) layers.
Leverage Spark-native formats and engines such as Delta Lake to support scalable analytics workloads.
Design curated data structures that support both enterprise analytics and industrial data consumption.
Enforce data retention, archiving, and purging policies to manage data lifecycles.
Monitor and optimize storage, compute costs, and cloud resource utilization.
Data Transformation
Develop Python and SQL-based transformations to clean, standardize, enrich, and unify data.
Develop, tune, and proactively optimize scalable PySpark applications to efficiently process high-volume datasets.
Build modular, reusable transformation components rather than siloed, application-specific logic.
Resolve complex data-quality anomalies including schema drift, missing values, duplicates, timestamp alignment, and inconsistent identifiers.
Transform raw operational and historian data into analytics-ready time series, calculations, conditions, events, and contextualized datasets for downstream consumption.
Enable reusable data transformations that can support platforms such as Seeq, Cognite, Fabric, and custom analytics applications.
Semantic Modeling & Industrial Data Contextualization
Translate complex source schemas into intuitive, business-aligned semantic entities, asset hierarchies, and industrial data models.
Build and maintain relationships between assets, equipment, time series, events, files, and other operational data.
Standardize naming conventions, identifiers, metadata, and relationship mappings across source systems.
Perform data contextualization to connect operational data with relevant equipment, process, and business context.
Support asset-centric and knowledge-graph-style data models commonly used in industrial data platforms such as Cognite.
Pipeline Orchestration & Operations
Architect end-to-end workflows that seamlessly coordinate ingestion, transformation, contextualization, validation, and publication.
Orchestrate data movement between source systems, lakehouse environments, industrial data platforms, and analytics applications.
Implement resilient error handling including retries, automated recovery, checkpointing, and alerting.
Monitor pipeline health, SLA compliance, data freshness, and data drift using centralized observability tools.
Design pipelines capable of supporting both traditional enterprise data and high-frequency industrial time-series workloads.
Data Governance, Security & DevOps
Enforce data security and privacy standards, including row- and column-level security, RBAC (Role-Based Access Control), and masking of sensitive data.
Apply consistent access-control and governance patterns across enterprise data platforms and industrial analytics environments.
Embed automated validation checks as needed to enforce data quality.
Maintain Git-based development workflows using branching, code reviews, and pull requests.
Implement automated CI/CD pipelines for code testing, data validation, configuration management, and repeatable environment deployments.
Support deployment and lifecycle management of data pipelines, transformation logic, semantic models, and analytics configurations across development, test, and production environments.
Maintain comprehensive documentation of data architectures, pipeline schemas, source-to-target mappings, contextual relationships, and lineage maps.
Preferred Experience
Bachelor's Degree in Computer Science, Software Engineering, Data Science, Engineering, or related field + 2 years of relevant experience; equivalent Master's degree accepted; non-relevant bachelor's degree requires 5+ years of equivalent hands-on experience.
Experience working with:
SQL and Python for data engineering and transformation.
PySpark and Microsoft Fabric for lakehouse transformations.
