Role descriptionJob Title: Sr. DATA Engineer
Work LocationIrving TX
Job Description:- Seeking a Senior professional with 3 to 5 years of experience in Python Scala and Spark SQL to design and optimize scalable big data processing solutions within the Databricks Spark SQL Scala Python Java ecosystem
- Develop and maintain scalable data processing pipelines using Apache Spark with Python and Scala
- Utilize Spark SQL to analyze large datasets and extract meaningful insights supporting data driven decision making
- Collaborate with cross functional teams to integrate big data solutions into existing systems
- Optimize Spark jobs for performance and efficiency in distributed computing environments
- Ensure data quality and consistency across various data sources
- Stay updated with the latest advancements in Spark Scala Python and related big data technologies to continuously improve processing capabilities
Roles and Responsibilities
- Design develop and deploy complex data processing workflows leveraging Apache Spark Scala Python and SparkSQL
- Write efficient reusable and maintainable code for Spark applications
- Troubleshoot and resolve performance bottlenecks in Spark jobs and data pipelines
- Collaborate with data engineers' data scientists and stakeholders to translate business requirements into technical solutions
- Conduct code reviews and provide mentorship to junior team members on best practices and coding standards
- Participate in architectural discussions and contribute to the evolution of big data platforms
- Monitor and maintain the health availability and reliability of Spark clusters and data services Karat Interview mandate
