- Strong/expert Spark (PySpark) Using Jupyter Notebooks, Colab or DataBricks (preferred)
- Hands-on data pipeline development, ingest patterns in Azure
- Orchestration tools, ADF or Airflow SQL Denormalized Data modeling for big data systems
Responsibilities and essential job functions include but are not limited to the following: - Demonstrate deep knowledge of the data engineering domain to build and support non-interactive (batch, distributed) & real-time, highly available data, data pipeline, and technology capabilities
- Build fault-tolerant, self-healing, adaptive, and highly accurate data computational pipelines
- Provide consultation and lead the implementation of complex programs
- Develop and maintain documentation relating to all assigned systems and projects
- Tune queries running over billions of rows of data running in a distributed query engine
- Perform root cause analysis to identify permanent resolutions to software or business process issues
Job Description
Data engineers are responsible for building reliable and scalable data infrastructure that enables organizations to derive meaningful insights, make data-driven decisions, and unlock the value of their data assets.
Job Description - Grade Specific
The involves leading and managing a team of data engineers, overseeing data engineering projects, ensuring technical excellence, and fostering collaboration with stakeholders. They play a critical role in driving the success of data engineering initiatives and ensuring the delivery of reliable and high quality data solutions to support the organizations data driven objectives.
