Role descriptionJob Title: Specialist - Data Engineering
Work LocationIrving TX
Job Description:- Seeking a Specialist with 3 to 5 years of experience in Apache Spark and Scala within the Hadoop Spark Kafka Nifi ecosystem
- Design develop and optimize scalable big data processing solutions using Apache Spark and Scala
- Collaborate with cross functional teams to build data pipelines and integrate with Hadoop Kafka and Nifi
- Analyze large datasets to derive actionable insights and support business decision-making
- Develop robust high-performance applications in a distributed computing environment
- Ensure code quality maintainability and adherence to best practices in Scala programming
- Participate in architecture discussions to enhance data platform capabilities
- Continuously evaluate emerging technologies and tools in the big data ecosystem
- Troubleshoot and resolve complex data processing issues to ensure system reliability
- Document technical designs and implementation details for knowledge sharing
Roles and Responsibilities
- Lead the development and deployment of big data solutions leveraging Apache Spark and Scala
- Collaborate with data engineers architects and stakeholders to gather requirements and deliver solutions
- Optimize Spark jobs for performance and scalability in production environments
- Implement data ingestion workflows using Kafka and Nifi to support real-time and batch processing
- Mentor junior team members and provide technical guidance on best practices
- Ensure compliance with data governance and security policies
- Conduct code reviews and contribute to continuous improvement of development processes
- Monitor system performance and proactively address bottlenecks or failures
- Participate in sprint planning estimation and agile ceremonies to deliver high-quality software
