3+ years in data engineering or related technical role.
Strong proficiency in Microsoft Azure data technologies, especially Azure Data Factory and Data Lake Storage.
Innovative SQL skills, with a focus on complex queries and data transformations.
Experience with both ETL and ELT pipeline development across multiple data sources.
Understanding of data warehousing concepts and dimensional modeling principles.
Familiarity with data validation and reconciliation techniques.
Proficiency in Python for data processing is required.
Responsibilities
Develop and optimize data pipelines for efficient information movement.
Design ETL processes using Azure Data technologies.
Manage the Azure Data Lake environment for reliability and access.
Architect data feeds from diverse internal and external sources.
Troubleshoot data feed failures and processing errors.
Create SQL transformations and maintain standardized data structures for reporting.
Partner with developers to translate business needs into technical data solutions.
Benefits
Opportunity to work with cutting-edge Azure data technologies.
Dynamic work environment with a focus on continuous improvement.
Collaboration with cross-functional teams for rich problem-solving experiences.
Support for professional development and technical training.
Involvement in innovative projects that enhance data architecture.
Full Job Description
Job Description
Key Responsibilities
Develop, maintain, monitor, and optimize data pipelines that move information from operational business systems into the company's enterprise data platform and data lake.
Design and maintain ETL and ELT processes using Azure Data Factory and related Azure data technologies.
Manage and support the company's Azure Data Lake environment, including data organization, storage structures, processing workflows, access, and overall platform reliability.
Architect new data feeds and integrations from enterprise applications, APIs, databases, files, and other internal and external data sources.
Develop processes for ingesting structured and semi-structured data including SQL data, APIs, JSON, CSV, flat files, and other common data formats.
Troubleshoot data feed failures, pipeline errors, processing issues, data discrepancies, and other problems affecting the availability or accuracy of business data.
Monitor and improve data pipeline performance, processing times, query performance, resource utilization, and overall data platform efficiency.
Develop and maintain SQL queries, views, stored procedures, transformations, and reusable data structures used by reporting and business applications.
Develop curated datasets and reporting views that provide Power BI and other analytics tools with consistent, reliable, and understandable business data.
Work with Power BI developers and data analysts to understand reporting requirements and translate business needs into appropriate data structures and models.
Design and maintain data models that support enterprise reporting, analytics, operational reporting, historical analysis, and KPI measurement.
Apply dimensional modeling principles where appropriate, including fact tables, dimension tables, relationships, measures, historical data, and common business entities.
Improve business visibility into data by identifying opportunities to make information easier to access, understand, analyze, and use for decision-making.
Establish and maintain appropriate data quality controls, validation processes, reconciliation procedures, and monitoring to identify missing, inaccurate, duplicated, or inconsistent data.
Develop processes for incremental data loading, change detection, historical data retention, and efficient processing of large data sets.
Establish standards for data naming, definitions, structure, documentation, ownership, lineage, and appropriate use across enterprise reporting.
Support data governance initiatives that create consistent definitions and trusted sources for customers, employees, shipments, financial information, operational metrics, and other key business entities.
Help establish consistent enterprise KPIs and reporting definitions to reduce conflicting calculations and different versions of the same business metric.
Document data sources, pipelines, transformations, dependencies, business rules, data models, and reporting structures.
Maintain visibility into dependencies between source systems, data pipelines, transformations, reporting datasets, and downstream applications.
Participate in the evaluation and implementation of new data technologies, tools, integrations, and architectural improvements.
Work with application developers, infrastructure teams, data analysts, project managers, and business stakeholders to support new projects and data requirements.
Follow appropriate security, access control, data privacy, change management, testing, and development practices when working with enterprise data.
Proactively identify opportunities to improve data reliability, processing performance, automation, scalability, maintainability, and overall data architecture.
Qualifications
3+ years of experience in data engineering, data integration, database development, business intelligence engineering, or a related technical role.
Strong experience with Microsoft Azure data technologies, particularly Azure Data Factory, Azure Data Lake Storage, and DataBricks.
Strong SQL skills with experience developing complex queries, views, stored procedures, transformations, and reporting datasets.
Experience designing, developing, and maintaining ETL and ELT pipelines that integrate multiple enterprise data sources.
Experience integrating data through REST APIs, databases, file transfers, JSON, CSV, flat files, and other common integration methods.
Strong understanding of relational databases, data structures, database relationships, data types, indexing, and query optimization.
Understanding of data warehousing concepts, dimensional modeling, fact and dimension tables, star schemas, and data structures designed for analytics.
Experience preparing and modeling data for Power BI or similar business intelligence and visualization platforms.
Understanding of Power BI data requirements, semantic models, relationships, refresh processes, and reporting performance considerations.
Experience troubleshooting and optimizing data pipelines, SQL queries, transformations, and large data processing workloads.
Knowledge of incremental loading, change data capture concepts, data synchronization, data dependencies, and historical data management.
Experience implementing data validation, reconciliation, monitoring, logging, error handling, and data quality processes.
Understanding of data governance concepts including data ownership, business definitions, lineage, documentation, access controls, quality, and trusted data sources.
Ability to understand business processes and translate business reporting requirements into scalable technical data solutions.
Ability to analyze data discrepancies and determine whether problems originate within source systems, integrations, transformation logic, data models, or reporting layers.
Experience with Python for data processing, automation, integration, or data engineering is required.
Familiarity with Git or other source control and development lifecycle practices.
Familiarity with Azure security, role-based access control, service accounts, credentials, and secure data integration practices.
Experience working with data originating from ERP, CRM, transportation, logistics, financial, warehouse, customer service, or other enterprise business systems is preferred.
Strong analytical, troubleshooting, communication, organization, documentation, and problem-solving skills.
Ability to explain technical data concepts, issues, dependencies, and recommendations clearly to both technical and non-technical stakeholders.
Ability to work independently while collaborating cross-functionally with application development, infrastructure, analytics, project management, and business teams.
Ability to take ownership of critical data processes and proactively identify potential failures, performance issues, data quality concerns, and architectural improvements.
Ability to manage multiple priorities in a fast-paced environment with changing business and reporting needs.
Ability to read, write, and speak English fluently.
Education
Bachelor's degree in Computer Science, Software Engineering, or a related field; or equivalent practical experience
