Job SummaryJoin us as a Lead Data Engineer on the Corporate Data Technology Team. Become an integral part of an agile team dedicated to enhancing, building, and delivering trusted market-leading technology products in a secure, stable, and scalable way. As a core technical contributor, you will be responsible for conducting critical technology solutions across multiple technical areas within various business functions, supporting the firm's business objectives.
Job Responsibilities- Design and implement scalable data frameworks to manage end-to-end data pipelines for workforce data analytics.
- Develop secure, high-quality production code and data pipelines on Databricks, reviewing and debugging processes implemented by others.
- Identify opportunities to eliminate or automate remediation of recurring issues to improve operational stability of software applications and systems.
- Lead evaluation sessions with external vendors, startups, and internal teams to drive outcomes-oriented probing of architectural designs, technical credentials, and applicability for use within existing systems and information architecture.
- Work with business stakeholders to understand requirements and design appropriate solutions, producing architecture and design artifacts for complex applications and implement the right ER Model with right fact and dimension tables.
- Implement robust monitoring and alerting systems to proactively identify and address data ingestion issues, optimizing performance and throughput.
- Implement data quality checks and validation processes to ensure accuracy and reliability of data.
Cloud Functions- Design and implement end-to-end data solutions using Databricks Pipelines and Azure Data Lake Storage (ADLS), Azure Databricks.
- Develop, monitor, and maintain scalable data pipelines to support analytics and data science projects.
- Understand and perform Data Integrity checks and Data Quality checks. Research and utilize Databricks test suites and make them absorbed into data eco system.
- Utilize Azure Databricks to perform complex data transformation, aggregation, and machine learning tasks.
- Assist with designing the architecture and workflow for an Enterprise Data Lakehouse in Databricks who wish to build their own Data Marts
General- Collaborate with cross-functional teams to translate business needs into technical specifications and data models.
- Ensure data quality and integrity by implementing best practices and data governance standards
- Provide ongoing support and enhancements to existing data structures and reporting environments.
- Stay current with industry trends and advancements in Databricks data analytics technologies.
- Leverage AI tools to assist with the creation, maintenance, and updating of comprehensive documentation for all data solutions and processes.
- Implement and manage source control practices using GIT within Azure DevOps to ensure code integrity and facilitate collaboration. Proficient in DAB deployments.
- Participate in the development and maintenance of CI/CD pipelines for automated testing and deployment of BI solutions.
- Each team member is expected to be aware of risk within their functional area. This includes observing all policies, procedures, laws, regulations and risk limits specific to their role. Additionally, they should raise and report known or suspected violations to the appropriate Company authority in a timely fashion.
- Performs other related duties as required.
The information on this description has been designed to indicate the general nature and level of work performed by employees within this classification. It is not designed to contain or be interpreted as a comprehensive inventory of all duties, responsibilities, and qualifications required of employees assigned to this job.
Minimum Education:- Bachelor's degree in computer science, information systems, or a related field or an equivalent combination of education and experience.
Minimum Experience:- Ten years of experience in Information technology including collecting business requirements related to usage of data, performing data mapping and conducting data quality assessments and developing, utilizing and writing new automation.
Required Knowledge, Skills, & Abilities:- Understanding of reporting and/or visualization tools (e.g., Tableau, SSRS, SQL Server)
- Knowledge of a variety of technologies, data models, and insights across all relevant data sources
- Understanding of Governance principles
- Understands concepts like data mining, extraction and analysis as it pertains to a specific bank pillar (e.g., Commercial, Retail, Wealth)
- Analytical and critical thinking skills
- Ability to quickly shift between technology stacks
- Ability to mentor and train team members
- Strong verbal and written communication skills
