Qualifications

• 5-7 years of experience in Apache Spark ETL design and development
• Proficient in Unix/Linux command line and shell scripting (Bash, KornShell, Perl)
• Strong SQL skills, including complex joins and querying
• Experience in financial data processing and lifecycle management
• Familiarity with Apache Airflow for managing data workflows and task dependencies
• Strong analytical skills, particularly with SparkSQL for data analysis
• Experience in implementing data quality checks and monitoring.

Responsibilities

• Design and develop scalable ETL data processing pipelines using Apache Spark
• Utilize command line tools and shell scripts for efficient file processing
• Query large datasets with complex SQL joins for data extraction
• Implement Spark-based frameworks for validating financial data throughout projects
• Manage and automate data workflows and task dependencies with Apache Airflow
• Analyze data and ensure readiness for business use with high-quality standards
• Conduct data quality checks to maintain dataset integrity.

Benefits

• Work with cutting-edge technologies in data processing
• Opportunity to innovate in financial data management
• Hands-on experience with high-volume data operations
• Collaborative team environment fostering skill development
• Exposure to project lifecycle management in a financial context.