Role DescriptionThe Senior QA Testing Specialist- Data Engineer is responsible for ensuring the accuracy, reliability, and integrity of enterprise data platforms built on Databricks. This role combines strong data engineering knowledge with deep quality assurance expertise, focusing on validating complex ETL pipelines, data transformations, and large-scale datasets.
The position plays a critical role in enabling data-driven decision-making by ensuring that analytical and financial data is complete, accurate, and production-ready before release. The role partners closely with Data Engineers, Business Analysts, Product Owners, and Project Managers across globally distributed teams.
Role ObjectivesData Quality & Validation:- Perform end-to-end data validation, including source-to-target mapping, reconciliation, and completeness checks. Validate data transformations across Databricks Medallion Architecture (Bronze, Silver, Gold).
- Ensure data quality standards are met prior to production releases.
Test Strategy & Execution:
- Participate in requirements analysis and solution design reviews.
- Define test scope, scenarios, risk-based coverage, and overall test strategies.
- Design and execute test cases for complex ETL pipelines and Perform regression testing and manage test data conditions.
Data Validation & Automation: - Write and execute advanced SQL queries (CTEs, window functions, complex joins).
- Develop Python validation scripts using PySpark and Pandas to support and enhance data test automation efforts where applicable.
- Monitor Databricks jobs and workflow executions. Investigate pipeline failures, data inconsistencies, and quality issues. Analyze defects both functionally and technically prior to assignment.
Defect Management & Collaboration:- Log, track, and manage defects using JIRA, ensuring high-quality defect documentation.
- Collaborate closely with Data Engineers, Architects, and Business teams to resolve issues.
- Provide clear and timely communication on quality risks, test progress, and release readiness and Escalate issues through defined governance and quality escalation channels.
Qualifications and Skills- 8+ years of experience in Quality Assurance testing or relevant Data Engineering development roles.
- Demonstrated expertise in Advanced SQL, including CTEs, window functions, and complex joins, for large-scale data validation.
- 6+ years of experience in test management strategy; experience in the securities or financial services industry
- 3+ years of experience in designing solutions in process flow, business logic and UI.
- Strong experience within the Azure data ecosystem, with hands-on expertise in Databricks notebooks and cluster management.
- Experience building or supporting data test automation frameworks, with exposure to CI/CD pipelines and Git-based version control.
- Hands-on manual and automated test development and design experience leading medium to large scales or enterprise-wide projects and production implementations.
- Strong command of Python, particularly PySpark and Pandas, for comprehensive data validation and analysis.
- Thorough understanding of ETL processes, data pipelines, and data warehouse architecture, with proven experience in ETL/Data Warehouse testing, including end source-to-target validation.
- Prior experience working directly with business users and finance stakeholders, including exposure to Oracle or similar enterprise financial platforms.
- Strong analytical, critical-thinking, and problem-solving skills.
- Excellent written and verbal communication skills, with the ability to articulate technical concepts clearly.
- Ability to work independently while collaborating effectively across distributed and global teams.
Measures of Success - High level of confidence in production data quality, reflected by reduced defect leakage post-release.
- Strong and consistent test coverage across complex, multi-layer data platforms.
- Effective collaboration with engineering, product, and business teams throughout the delivery lifecycle.
- Proactive identification and mitigation of data quality risks and issues.
- Consistent, on-time delivery of test outcomes without compromising quality standards.
SMBC's employees participate in a Hybrid workforce model that provides employees with an opportunity to work from home, as well as, from an SMBC office. SMBC requires that employees live within a reasonable commuting distance of their office location. Prospective candidates will learn more about their specific hybrid work schedule during their interview process. Hybrid work may not be permitted for certain roles, including, for example, certain FINRA-registered roles for which in-office attendance for the entire workweek is required.
