The Data Engineer works closely with the Engineering and Analytics leads in a small, cross-functional team to develop a comprehensive data platform. This role also leverages data from ever increasing sources to assist in modeling for affinity, personalization, offers, promotions, etc.
ESSENTIAL DUTIES AND RESPONSIBILITIES
The following and other duties may be assigned as necessary:
- Develops new, comprehensive platforms to support Company initiatives.
- Partners with the Business Analytics Team, and internal technology teams to collect and document solution requirements, data access, process automation, and performance objectives. Makes recommendations based upon findings.
- Partners with all necessary technology partners to design, develop, deploy and maintain an enhanced data stack and data processing pipeline.
- Establishes foundation for future implementations and automation requirements for cloud environments.
- Develops and integrates monitoring for the cloud platforms and data pipelines to ensure the highest levels of data quality and availability.
- Maintains strict confidentiality in all departmental and company matters.
- Responsible for ensuring the compliance with all regulatory compliance within area of responsibility and reporting potential issues to Executive Management.
To be successful in this position it will require the following skill set
- Bachelor’s degree (B.A./B.S.) in Computer Science, Engineering, or related field from a four-year college or university; or 5 or more years of software engineering or data automation experience; or equivalent combination of education and experience.
- Must have at least 3 years of experience implementing Data Lake/Hadoop platforms.
- Must be proficient in Microsoft Applications (Word, Excel, Outlook, and Power Point).
- Must have experience developing with one or more programming languages (Java, Python, etc..).
- Must have experience with one or more of Beam, Flume, Kafka, Solr Zookeeper, or similar required.
- Must have experience with one or more SQL on Hadoop technologies (i.e. Hive, Spark).
- Must have knowledge on how to bring structured and unstructured data into a data lake for mining and modelling.
- Must have experience working on data lake implementations specifically AWS Lake Formation (Glue, EMR, S3, Redshift, Athena, etc.)
- Knowledge with CI/CD configuration pipelines into the cloud using Jenkins.
- Experience with tools such as Airflow, Dagster, or DBT highly preferred.
- Must have strong critical thinking skills and technical skills.
- Ability to adapt to changes in prioritization or schedules of items in the work queue as necessary.
- Ability to work in a fast-paced environment, while meeting deadlines and being proactive when working on tasks.
- Must have strong problem solving, organizational, analytical, interpersonal and communication skills (verbal and written).
- Must have excellent project and time management skills with the ability to multi-task and prioritize.
- Ability to effectively present information and respond to questions from groups of managers, clients, customers, and the general public.
- Must have the ability to interact with guests, staff and colleagues and resolve problems and conflicts in a diplomatic and tactful manner.
- Strong client interface and presentation skills highly preferred.