5-7 years of experience with big data technologies like Spark, Pyspark, and Python.
Proficient in data mapping, extraction, transformation, and loading (ETL).
Skilled in building analytic reports using CloudWatch and Kibana.
Experience in processing OS and Data Logs into reports and dashboards.
Familiar with Regular Expressions (RegEx) for data manipulation.
Proficient in SQL, MySQL, and PostgresSQL.
Experience with data formats like XML and JSON.
Responsibilities
Support customers in defining new system capabilities and enhancements.
Maintain and extend bulk data pipelines for cloud applications.
Process and convert OS and Data Logs into actionable reports.
Develop and document ETL processes, including data mapping and dictionaries.
Conduct extensive data reviews and quality analysis.
Interface with customers and integration partners to clarify objectives.
Contribute to Agile development processes by defining tasks and scope.
Benefits
Flexible work environment with remote options.
Opportunities for professional development and training.
Access to cutting-edge technology and tools.
Collaborative team culture focused on innovation.
Health and wellness programs.
Full Job Description
Job Description
Support Customer with the requirements for new system capabilities, the extension and maintenance of bulk data pipeline and enhancements for multiple applications in the customers' cloud computing environment.
Required Skills
Demonstrated experience utilizing big data processing tech such as Spark, Pyspark, and Python.
Demonstrated experience data mapping, extraction, transformation and loading.
Demonstrated experience building analytic reports in tools such as CloudWatch and Kibana.
Demonstrated experience processing and converting of OS and Data Logs into reports and metrics Dashboards.
Demonstrated experience with Regular Expressions (RegEx).
Demonstrated experience with SQL, MySQL, PostgresSQL.
Demonstrated experience with Data file type processing XML, JSON.
Demonstrated experience with IDEs and Data Modeling through Notebooks and Visual Studio.
Demonstrated experience ETL'ing data from disparate structured & unstructured data formats into enriched, query-friendly structured data in indexed files.
Demonstrated experience performing extensive data review and data quality analysis.
Demonstrated experience developing ETL design documentation including source and target mapping and data dictionary information.
Demonstrated experience interfacing with customers and integration partners for gaining and clarifying detailed objectives.
Demonstrated experience supporting Agile development by contributing to tasking definition, scope and review.
Desired Skills
Demonstrated experience deploying capabilities on the Databricks unified analytics platform.
Demonstrated experience with Elastic Map Reduce to run big data workloads.
Demonstrated experience in joining multiple complex data sets using Spark.
Demonstrated experience in tuning Spark streaming and batch jobs for cluster utilization and speed.
Demonstrated experience in deploying complex, notebook-based pipelines.
Demonstrated experience in Python data analysis libraries such as Pandas.
Demonstrated experience utilizing Cloud services such as Lambda, SNS/SQS, or EC2.
Demonstrated experience with DevOps tools to include Cloudwatch, Lambda, SQS, Dynamo and RDS.
Demonstrated experience working with Elastic, ElasticSearch/Logstash/Kibana (ELK stack).
