SPARK Data Reconciliation Engineer- NJ
Dallas, TX 75217 In-Person
More than 3 months ago
By clicking Apply, I agree with Ladders'
Job Description
Job Description
Job Title: PySpark Data Reconciliation Engineer
Summary:
We're seeking a skilled PySpark Data Reconciliation Engineer to join our team and drive the development of robust data reconciliation solutions within our financial systems. You will be responsible for designing, implementing, and maintaining PySpark-based applications to perform complex data reconciliations, identify and resolve discrepancies, and automate data matching processes. The ideal candidate possesses strong PySpark development skills, experience with data reconciliation techniques, and the ability to integrate with diverse data sources and rules engines.
Key Responsibilities:
Data Reconciliation Development:
Data Analysis and Matching:
Rules Engine Integration:
Problem Solving and Gap Analysis:
Qualifications and Skills:
Job Title: PySpark Data Reconciliation Engineer
Summary:
We're seeking a skilled PySpark Data Reconciliation Engineer to join our team and drive the development of robust data reconciliation solutions within our financial systems. You will be responsible for designing, implementing, and maintaining PySpark-based applications to perform complex data reconciliations, identify and resolve discrepancies, and automate data matching processes. The ideal candidate possesses strong PySpark development skills, experience with data reconciliation techniques, and the ability to integrate with diverse data sources and rules engines.
Key Responsibilities:
Data Reconciliation Development:
- Design, develop, and test PySpark-based applications to automate data reconciliation processes across various financial data sources, including relational databases, NoSQL databases, batch files, and real-time data streams.
- Implement efficient data transformation, matching algorithms (deterministic and heuristic) using PySpark and relevant big data frameworks.
- Develop robust error handling and exception management mechanisms to ensure data integrity and system resilience within Spark jobs.
Data Analysis and Matching:
- Collaborate with business analysts and data architects to understand data requirements and matching criteria.
- Analyze and interpret data structures, formats, and relationships to implement effective data matching algorithms using PySpark.
- Work with distributed datasets in Spark, ensuring optimal performance for large-scale data reconciliation.
Rules Engine Integration:
- Integrate PySpark applications with rules engines (e.g., Drools) or equivalent to implement and execute complex data matching rules.
- Develop PySpark code to interact with the rules engine, manage rule execution, and handle rule-based decision-making.
Problem Solving and Gap Analysis:
- Collaborate with cross-functional teams to identify and analyze data gaps and inconsistencies between systems.
- Design and develop PySpark-based solutions to address data integration challenges and ensure data quality.
- Contribute to the development of data governance and quality frameworks within the organization.
Qualifications and Skills:
- Bachelor's degree in Computer Science or a related field.
- 5+ years of hands-on experience in big data development, preferably with exposure to data-intensive applications.
- Strong understanding of data reconciliation principles, techniques, and best practices.
- Proficiency in PySpark, Apache Spark, and related big data technologies for data processing and integration.
- Experience with rules engine integration and development
- Strong analytical and problem-solving skills, with the ability to translate business requirements into technical solutions.
- Excellent communication and collaboration skills to work effectively with business analysts, data architects, and other team members.
- Familiarity with data streaming platforms (e.g., Kafka, Kinesis) and big data technologies (e.g., Hadoop, Hive, HBase) is a plus.
About Photon
Learn More About PhotonPhoton Careers
Joining Photon presents an unparalleled opportunity to advance one's career with a leader in digital innovation. Photon is actively seeking talented individuals to join its diverse team of professionals, dedicated to reshaping industries through technology and creativity.
Explore Job Opportunities
Photon offers a variety of job opportunities that cater to a range of skills and interests. Each position at Photon is designed to challenge team members while providing meaningful pathways for professional and personal growth.
Experience the Culture and Benefits
Photon is committed to fostering a workplace culture that promotes diversity and inclusion. The company provides comprehensive benefits designed to support the health, well-being, and financial security of each team member. At Photon, every professional enjoys access to career development programs and diversity training, ensuring they are equipped for leadership roles in an evolving marketplace.
Engage in Professional Growth
Photon believes in the power of innovation and leadership to drive success. The company supports its team members with resources for continuous learning and growth, including advanced training sessions and leadership workshops. Networking within Photon’s industry-leading community allows for unparalleled career advancement and skill enhancement.
Internship Programs
For those beginning their professional journey, Photon offers internship programs that provide a robust foundation in digital solutions. Internships at Photon are characterized by immersive projects and hands-on learning, guided by experts in various fields. These programs are a gateway to full-time employment and a promising career path.
Join the Team
Photon is hiring! Explore open positions that match your skills and interests. Photon looks for passionate, curious, creative, and solution-driven team players. Whether you are starting your career or looking to make a significant impact in your professional journey, Photon has a place for you.
Prepare for Your Interview
Photon values a thorough selection process. Prospective team members are encouraged to prepare by understanding the company’s mission and values. Tailoring your resume to highlight relevant experience and skills will make a significant difference. Engage actively during your interview, demonstrating your knowledge and enthusiasm for the role you are applying for.
Stay Connected
Keep up to date with the latest from Photon by following the company’s career blog. Gain insights from insiders, learn about new job openings, and get tips on preparing your resume and acing your interview.
Career Development at Photon
Photon is dedicated to the professional development of its team members. The company offers various tools and resources, including career coaching and performance feedback, to help individuals reach their full potential. With Photon, career aspirations are nurtured, encouraging every team member to excel and innovate in their respective roles.
Networking and Career Opportunities
Photon encourages its team members to engage in networking within the company and the broader industry. These connections can lead to myriad opportunities, from collaborative projects to mentorship. Networking at Photon is not just about building professional relationships; it's about fostering a community of innovation and mutual growth.
Conclusion
Photon is not just a company; it is a community of driven professionals committed to making a difference. With a focus on innovation, leadership, and diversity, Photon offers a dynamic environment where ambitious individuals can thrive. Explore the possibilities and join Photon’s team of leaders, innovators, and visionaries.
