Prepares and processes large data sets (structured and unstructured) through queries, scripts, and ETLs to facilitate accurate reporting and statistical analysis per the requirements of data scientists and analysts.
Design and develop big data solutions using industry standard technologies. Analyze business requirements and develop long-term data warehousing strategies leveraging state of the art big datatechnology. Work with internal and external customers to overcome technical obstacles, answering questions and proposing solutions pro-actively.
Utilizing big data tools, gather, collect and store data. Complete batch processing or real-time processing to provide in a ready-to-query and analyze format. Map and document data into a standardized set of facts and dimensions. Create aggregates for use in reporting and analytics.
Monitor the system, log and troubleshoot errors, build human-tolerant pipelines and address continuous integration. Perform data audits on client data sets to test domain and range values, patterns and build a plan for doing data quality assessments and clean up.
Follow best practices for testing, capacity planning, documentation, monitoring, alerting and incident response. Ensure compliance of acquisition, storage and analysis of data.
Collaborate with team to build and design core technologies. Demonstrate strong implementation aptitude to translate objectives into a scalable solution to meet the needs of the end customer while meeting deadlines.