JOB SUMMARY

We are seeking an experienced Data Quality Engineer to design, implement, and operationalize enterprise-scale data quality frameworks within a modern Databricks Lakehouse architecture. The ideal candidate will possess strong expertise in Databricks, PySpark, Delta Lake, SQL, and data governance, with a proven track record of embedding data quality controls throughout the data lifecycle. This role will be responsible for ensuring data integrity, reliability, compliance, and observability across large-scale cloud-based data platforms while driving quality-by-design principles across the organization.

KEY RESPONSIBILITIES
• Design and implement enterprise-wide data quality frameworks aligned with Lakehouse architecture, including Bronze, Silver, and Gold data layers.
• Define, implement, and enforce data quality rules covering:

- Completeness

- Accuracy

- Consistency

- Timeliness

- Validity
• Develop reusable validation, reconciliation, profiling, and monitoring frameworks within Databricks environments.
• Establish automated data quality checks integrated into ELT and ETL pipelines.
• Embed quality controls directly into Databricks workflows, Spark processing pipelines, and Delta Lake architectures.
• Develop scalable validation processes supporting both batch and real-time data ingestion pipelines.
• Partner with Data Engineers to ensure quality gates are enforced across ingestion, transformation, and consumption layers.
• Optimize data quality processes for performance, scalability, and reliability across large distributed datasets.
• Implement and maintain data observability solutions, including dashboards, alerts, monitoring metrics, and reporting frameworks.
• Monitor data pipelines and proactively identify anomalies, failures, data drift, and data quality degradation.
• Lead root cause analysis (RCA) activities and drive resolution of data quality issues.
• Develop and maintain enterprise data quality scorecards and performance reporting.
• Ensure adherence to enterprise data governance standards, including metadata management, data lineage, traceability, and auditability.
• Collaborate with Data Governance teams to align data definitions, ownership models, and control frameworks.
• Support regulatory compliance requirements through auditable and repeatable data quality processes.
• Define and enforce data quality SLAs, standards, and data contracts across business domains.
• Implement CI/CD practices for data quality rules, monitoring processes, and validation frameworks.
• Automate testing and validation of data transformations, integrations, and pipelines.
• Develop reusable enterprise libraries and frameworks for scalable data quality enforcement.
• Partner with Data Architects, BI teams, Data Engineers, and business stakeholders to drive data quality initiatives.
• Provide technical leadership, mentorship, and best-practice guidance across teams.
• Serve as the subject matter expert (SME) for enterprise data quality strategies and standards.
• Drive continuous improvement and innovation in data quality methodologies, tools, and practices.

REQUIRED QUALIFICATIONS
• Bachelor's degree in Computer Science, Data Engineering, Information Systems, Data Science, or a related field.
• Proven experience in Data Engineering, Data Quality Engineering, or related data management roles.
• Strong hands-on experience with:

- Databricks

- Apache Spark

- PySpark

- Delta Lake
• Extensive experience implementing enterprise data quality frameworks and controls within modern cloud data platforms.
• Advanced SQL development, data validation, and data profiling expertise.
• Experience integrating data quality processes into ELT/ETL pipelines and orchestration frameworks.
• Strong knowledge of data lifecycle management principles and best practices.
• Experience working with large-scale datasets in AWS or Azure cloud environments.
• Strong understanding of:

- Data Governance

- Metadata Management

- Data Lineage

- Data Quality Controls

- Auditability Requirements
• Experience implementing automated validation, reconciliation, and monitoring processes.
• Strong analytical, troubleshooting, and problem-solving skills.
• Ability to translate business requirements into scalable technical solutions.
• Excellent communication and stakeholder management skills.
• Experience working within Agile and DevOps delivery environments.

PREFERRED QUALIFICATIONS
• Experience within Financial Services, Banking, Insurance, or other regulated industries.
• Familiarity with Collibra or other Enterprise Data Governance platforms.
• Experience implementing data observability and monitoring solutions, including:

- Monte Carlo

- Great Expectations

- Deequ

- Similar data quality platforms
• Experience with real-time or streaming data quality validation frameworks.
• Knowledge of regulatory reporting, compliance controls, and data governance programs.
• Experience implementing data contracts and enterprise data standards.
• Exposure to machine learning-driven data quality monitoring solutions.
• Experience supporting enterprise Lakehouse modernization initiatives.

CERTIFICATIONS
• Databricks Certified Data Engineer Associate or Professional (Preferred)
• Databricks Certified Developer for Apache Spark (Preferred)
• AWS Certified Data Engineer - Associate (Preferred)
• Microsoft Certified: Azure Data Engineer Associate (Preferred)
• Collibra Data Governance Certification (Preferred)
• Relevant Data Quality, Data Governance, or Cloud Certifications (Preferred)