The Lead Data Engineer will support data lake developers, data warehouse architects, data analysts and data scientists on various data initiatives and will ensure optimal data delivery architecture is consistent throughout ongoing projects. He/She must be self-directed and comfortable supporting the data engineering needs of multiple teams, systems and products. He/She will be re-designing the company's data architecture to support our next generation of products and data initiatives.
Create and maintain optimal data pipeline architecture.
Integrate large, complex data sets that meet business requirements.
Identify, design, and implement internal process improvements: automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability.
Evaluate/Review/Implement/Build the infrastructure required for optimal extraction, transformation, and loading of data from wide variety of data sources.
Work with stakeholders at various levels to assist with data-related technical issues and support their data infrastructure needs.
Create data tools for analytics and data scientist team members to help build and optimize our product and services.
Work with data and analytics experts to strive for greater functionality in data ecosystem.
Advanced working SQL knowledge and experience working with relational databases as well as working familiarity with a variety of databases.
Experience building and optimizing big data pipelines, architectures and data sets.
Experience performing root cause analysis on internal & external data and processes and identify opportunities for improvement.
Strong analytic skills related to working with both structured and unstructured datasets.
Build processes supporting data transformation, data structures, metadata, dependency and workload management.
Working knowledge of message queuing, stream processing, and highly scalable 'big data' data stores.
Experience supporting and working with cross-functional teams in a dynamic environment.
Preferred Technical Skills:
Candidate with 10+ years of experience in a Data Engineer role; Graduate degree in Computer Science, Statistics, Informatics, Information Systems or another quantitative field.
Experience with big data tools: Hadoop, Spark, Hive, Impala, Scala etc.,
Experience with relational SQL and NoSQL databases: Redshift, Snowflake, DynamoDB, Neo4J, MongoDB, Cassandra etc.,
Experience with data pipeline and workflow management tools: Informatica, Talend, DataStage etc.,
Experience with cloud services: AWS, GCP etc.,
Experience with stream-processing systems: Kafka, Kinesis etc.,
Experience with scripting languages: Python, Java etc.
Graduate degree in Computer Science, Statistics, Informatics, Information Systems or another quantitative field with 10+ years of experience in a Data Engineer role.