Python Developer


Sunnyvale, CA

Industry: Staffing & Recruiting


8 - 10 years

Posted 265 days ago

  by    Krishna Reddy

This job is no longer available.

Job Title:             Python Developer with Unix Cron & AWS

Job Location:     Sunnyvale, CA

Job Description:

?                   Minimum 9+ years of software programming experience

?                   Strong experience in ETL

?                   SQL scriptingexperience

?                   Strong Python Development experience around data management.

?                   Strong in Unix Cron not just exposure

?                   Must have the ability to work in a dynamic, fast-paced environment

?                   Strong communication skills to interact with Agile team members

Data Sources: Cloud based as well as on premise example: SAP CRM, ECC, people soft, PLM, Calliduscloud


High-level architecture:

?                   Amazon S3 bucket have folders for specific source files delivered by Fusion middleware.

?                   Data quality check is not handled by the sources.

?                   Anaconda Python based programs are in place to clean any abnormalities in the source files and re-write them back in their respective folders this is to avoid load failures.

?                   Most of the known errors are handled using this mechanism, moving forward you will require to enhance these scripts to include a fix for any new errors.

?                   Currently mechanism is processing one file at a time, target is to move it to parallel processing.

?                   Next is a Python program for data pipeline, based on AWS API and snowflake API. This is to write files on snowflake EDW

?                   There are 3 different layers on the snowflake side Pre=processing, staging and base tables

?                   UNIX cron is used for scheduling purpose.