NTT DATA Services currently seeks a Lead Data Engineer (Player/Coach) to join our team in Palo Alto, California (US-CA), United States (US).
Lead Data Engineer will:
- Create and maintain the optimal data pipeline architecture based on platform and application requirements
- Assemble large, complex data sets that meet functional / non-functional business requirements
- Identify, design, and implement process improvements: automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability, etc
- Build the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources
- Work with stakeholders including the Executive, Product, Data and Design teams to assist with data-related technical issues and support their data infrastructure needs.
- Recommend, select and integrate the data tools and frameworks required by the SMART Platform and Applications
- Design and implement data integration strategies and processes using industry standard methods and tools
Incoming Lead Data Engineer should possess:
- 15+ years of overall technology experience, with 7+ years of data management, modeling and architecture design experience
- 4 years of Python
- 5 years' experience with integration of data from multiple data sources
- 2 years' experience using Hadoop, Kafka, HDFS
- 2 years' experience building stream-processing systems, using solutions such as Storm or Spark-Streaming
- 2 years' experience using Spark
- 2 years' experience in a "Lead" role.
Nice to have:
- Experience with NoSQL databases, such as Cassandra, MongoDB, or CouchDB
- Experience with various messaging systems, such as MQTT or RabbitMQ
- Experience working in public cloud environments like AWS, Azure.
- Experience with assisting Data Scientists with conforming and integrating data, data preparation and data cleansing, and then automating that work over time.
- Good understanding of Lambda Architecture, along with its advantages and drawbacks
- Experience with Cloudera/MapR/Hortonworks
- Experience working in large-scale, highly complicated and global environments – Japan experience is a plus
- A strong team-oriented mindset
- Experience with SQL and RDBMS systems for analytic querying
- Experience using various ETL techniques and frameworks, such as Flume and Informatica
- Experience using Big Data querying tools and approaches including Parquet, Hive, and Impala