Job Summary
The Data Engineer will be responsible for designing, developing, and maintaining high-performance, event-driven data pipelines that support large-scale credit card processing systems. The role requires expertise in the Ab Initio ecosystem, real-time streaming technologies, ETL development, and distributed data processing. The ideal candidate will build resilient, low-latency data pipelines that process high-volume transactional data while ensuring scalability, reliability, security, and regulatory compliance.
Key Responsibilities
• Design, develop, and maintain high-performance event-driven data pipelines using the Ab Initio platform.
• Build complex Ab Initio Continuous Flow graphs utilizing stateful joins, sliding time windows, aggregations, and advanced data transformation techniques.
• Develop real-time data ingestion and processing pipelines using Kafka, message queues, and file-based streaming sources.
• Implement resilient Continuous Flow architectures with checkpointing, recovery mechanisms, and exactly-once processing guarantees.
• Utilize Ab Initio components including Reformat, Rollup, Join, Partition, and other transformation components to optimize low-latency data processing.
• Develop ETL pipelines supporting both batch and real-time data ingestion and transformation.
• Monitor, troubleshoot, and optimize production data streams to ensure high availability and 24x7 operational reliability.
• Perform performance tuning and optimization across large-scale data processing environments.
• Implement data validation, data quality, security, governance, and compliance controls across enterprise data platforms.
• Collaborate with data scientists, analysts, application developers, and business stakeholders to deliver high-quality data solutions.
• Develop automation scripts and workflow orchestration to improve operational efficiency.
• Troubleshoot production issues and support ongoing enhancements to enterprise data platforms.
• Participate in code reviews, testing, deployment, and continuous improvement initiatives.
• Support CI/CD implementation for data engineering projects.
Required Qualifications
• Strong understanding of credit card processing systems and payment transaction workflows.
• Extensive experience with the Ab Initio platform, including:
• Graphical Development Environment (GDE)
• Enterprise Meta>Environment (EME)
• Conduct>It
• Continuous Flows
• Experience developing complex Ab Initio Continuous Flow graphs.
• Strong understanding of event-driven architectures and real-time streaming systems.
• Experience with Kafka, message queues, and real-time stateful data processing.
• Strong SQL skills and experience with relational databases such as Oracle and Teradata.
• Experience with Unix/Linux environments and Shell scripting.
• Understanding of data security, data governance, and compliance best practices.
• Experience working with cloud-based big data platforms.
• Experience monitoring, troubleshooting, and optimizing large-scale distributed data environments.
• Strong analytical, troubleshooting, and problem-solving skills.
• Excellent communication and collaboration skills.
Preferred Qualifications
• Experience with Java, Scala, Python, or Kafka application development.
• Experience with containerization technologies such as Docker and Kubernetes.
• Experience implementing CI/CD pipelines for data engineering projects.
• Experience building scalable real-time streaming and big data solutions.
• Knowledge of Apache Flink or similar stream processing technologies.
• Experience with OzoneCH or related enterprise data processing platforms.
