Specialist - Data Engineering

LTM

$90K — $120K *
Information Technology
Less than 5 years of experience
Job Overview by Ladders

Qualifications

  • 5-7 years of experience in Quality Engineering, particularly in automation frameworks.
  • Proficient in Unix/Linux environments with strong shell scripting abilities using Bash, Kornshell, or Perl.
  • Expertise in SQL querying techniques, specifically with complex joins.
  • Experience in designing and developing ETL pipelines using Apache Spark.
  • Knowledge of Data validation frameworks for handling large financial datasets.
  • Experience managing data workflows using Apache Airflow.
  • Hands-on experience with automation framework design for ETL and API testing.

Responsibilities

  • Develop and implement automation frameworks for extensive testing processes.
  • Conduct manual testing to support automation efforts until completion.
  • Design scalable ETL pipelines to process financial data using Apache Spark.
  • Use shell scripting for file processing and command-line tasks.
  • Create and implement comprehensive data validation frameworks.
  • Manage complex data workflows and task dependencies in Apache Airflow.
  • Perform data quality checks and monitoring to ensure accuracy in dataset delivery.

Benefits

  • Opportunity to work on cutting-edge data processing technologies.
  • Engage in meaningful projects that impact financial data integrity.
  • Collaborative team environment focused on quality and testing innovation.
  • Professional growth through exposure to diverse automation frameworks and tools.
Full Job Description
Role description

Mandatory Karat Interview

This role is for Quality Engineering team where 70percent of effort will be for developing automation frameworks for testing Remaining 30percent effort will be on manual testing until its fully automated

Unix SQL and Shell Scripting experience is a must have

Expertise in Designing and developing scalable Apache spark ETL based Data processing pipelines

Strong commandline knowledge in UnixLinux with Shell scripting using Bash Kornshell or Perl and File processing using awk scripts

Expertise in SQL querying and complex joins

Implementing comprehensive Spark based Data validation frameworks transforming large volumes of Financial data within the Project lifecycle

Expertise with complex Data workflows with Apache AirFlow managing task dependencies SLAs etc to ensure timely data delivery and corresponding automated validation controls

Strong Analytical skills and expertise on SparkSQL for Data analysis and validation ensuring the delivery of clean queryready datasets for business consumption

Expertise in Data quality checks and monitoring

Handson with Automation Framework Design for ETL and API

SME in Data Analysis Database testing Messaging queues

Experience with coding standards code reviews source management build processes CICD pipeline

Similar Jobs

More Jobs at LTM

More Information Technology Jobs

Find similar Specialist - Data Engineering jobs: