Summary

The candidate must have a successful track record in handling complex data pipelines for high transaction volume application and reporting/analytics data with minimal supervision. The candidate will be expected to support a variety of structured and semi-structured data in streaming and batch frameworks for both migration from and synchronization with legacy systems. The Senior Data Engineer provides support to the Application database design to aid in eliminating data duplication and enabling selective event-based and schedule-based data transfer to endpoints within the cloud and legacy environment as required. The Senior Data Engineer drives towards programmatic pipeline generation and orchestration to enhance repeatability and rapid deployment, using Databricks, PySpark, and Airflow, out-of-the-box thinking, AWS native capabilities, and CI/CD tools while utilizing established design patterns, specifically the Medallion architecture.

What You'll Be Doing:

• Designs, implements, and operates data management systems using Databricks and PySpark.
• Defines and adheres to best practices for implementing the Medallion architecture diligently across all data layers.
• Orchestrates complex data pipelines and workflows using Apache Airflow.
• Works with product and application owners, data stewards, and business/data analysts to determine, create, and populate required data in an optimal, reliable, and measurable manner.
• Assists the Data Architects and technical leads in designing how data will be stored, accessed, used, integrated, and managed by different data regimes and digital systems.
• Plans, designs, and optimizes data throughput and query performance.
• Participates in the employment of backend database technologies, their configuration and utilization, and the optimization of the full data pipeline infrastructure to support the actual content, volume, ETL, and periodicity of data to match expected responsiveness.
• Utilizes identified COTS, Python, PySpark, and AWS tools and services as needed.
• Creates and executes medium to complex SQL scripts to validate and test data prior to and during the lifecycle of reports and dataset development.

Required Qualifications:

• BA or BS degree in CS, Computer Engineering, Information Technology, or a related field.
• Databricks Certified Data Engineer (Associate or Professional) or equivalent experience.
• 5+ years of hands-on experience with PySpark, Python, and various Python toolkits and libraries for data processing and pipelines.
• Proven experience and deep understanding of the Medallion architecture, with the ability to define and adhere to best practices diligently.
• Hands-on experience orchestrating complex data workflows using Apache Airflow.
• 5+ years of experience and ability to create complex SQL queries and functions.
• 3+ years of work experience on AWS-based datasources such as RDS, Athena, Redshift.
• 2+ years of hands-on experience in creating ETL and data pipeline design patterns for modular and reliable ETL with robust error management and alerting.
• 1+ year of experience in Java and Linux scripting is essential.
• 1+ year of experience working with CI/CD tools including Git for ETL and scripts repository.
• 1+ year of experience using complex data sets and formats such as JSON, AVRO, parquet.
• 1+ year of experience in data profiling and quality management.

Preferred Qualifications:

• Experience working in AWS environments and any AWS certifications is advantageous.
• Experience working with database versioning tools such as Flyway is an advantage.
• Experience in Jenkins scripting is an advantage.

Salary Range

The proposed salary range for this role is $150,000 to $170,000 USD. The salary range provided is a good faith estimate representative of all experience levels. Karsun considers several factors when extending an offer, including but not limited to, the role, function and associated responsibilities, a candidate's work experience, location, education/training, and key skills.

Third Party Resumes: Karsun does not accept unsolicited resumes through or from search firms or staffing agencies. All unsolicited resumes will be considered the property of Karsun and Karsun will not be obligated to pay a placement fee.

Clearance Information

This position requires the eligibility to obtain a security clearance. The Defense Industrial Security Clearance Office (DISCO), an agency of the Department of Defense, handles and adjudicates the security clearance process. More information about Security Clearances can be found on the US Department of State government website: https://www.state.gov/m/ds/clearances/c10978.htm

Location

To be considered for this role, you must reside in one of the following states: CA, CO, DC, FL, GA, IL, MD, NJ, NY, NC, OH, OK, PA, SC, TX, VA, WV.

Applicants must be authorized to work in the U.S. We may consider candidates currently in H-1B status who are eligible for transfer.