Job Title: Python Developer with Unix Cron & AWS
Job Location: Sunnyvale, CA
? Minimum 9+ years of software programming experience
? Strong experience in ETL
? SQL scriptingexperience
? Strong Python Development experience around data management.
? Strong in Unix Cron not just exposure
? Must have the ability to work in a dynamic, fast-paced environment
? Strong communication skills to interact with Agile team members
Data Sources: Cloud based as well as on premise example: SAP CRM, ECC, people soft, PLM, Calliduscloud
? Amazon S3 bucket have folders for specific source files delivered by Fusion middleware.
? Data quality check is not handled by the sources.
? Anaconda Python based programs are in place to clean any abnormalities in the source files and re-write them back in their respective folders this is to avoid load failures.
? Most of the known errors are handled using this mechanism, moving forward you will require to enhance these scripts to include a fix for any new errors.
? Currently mechanism is processing one file at a time, target is to move it to parallel processing.
? Next is a Python program for data pipeline, based on AWS API and snowflake API. This is to write files on snowflake EDW
? There are 3 different layers on the snowflake side Pre=processing, staging and base tables
? UNIX cron is used for scheduling purpose.