Job SummaryWe are seeking a
Senior Data QA Automation Engineer to lead the quality strategy, design, and implementation of automated testing frameworks for our big data platforms. In this senior role, you will own the end-to-end data validation strategy within our
Databricks Lakehouse architecture, ensuring high-quality, reliable, and compliant data across Delta Lakes, ETL pipelines, and enterprise data models. You will work closely with Data Engineering leadership to establish rigorous quality gates and mentor mid-to-junior engineers on data testing best practices.
Key Responsibilities- Strategic Framework Design: Architect, build, and scale automated test frameworks from scratch natively within Databricks using PySpark, Python, and SQL.
- Lakehouse Quality Engineering: Design robust automated assertions for Delta Lake tables, including checking data drift, schema evolution, and historical data validation via time-travel functions.
- Enterprise Pipeline Testing: Code complex automated scenarios to validate large-scale batch and real-time streaming data pipelines (Structured Streaming), ensuring source-to-target integrity.
- Governance Validation: Programmatically verify data lineage, audit logs, and access controls implemented via Databricks Unity Catalog.
- CI/CD & DevOps Ownership: Lead the integration of automated data quality tests into enterprise CI/CD pipelines (e.g., Azure DevOps, GitHub Actions), leveraging Databricks Workflows, APIs, or Airflow.
- Technical Leadership & Mentorship: Act as the subject matter expert for data quality; mentor junior team members, establish QA standards, and advocate for data quality principles across engineering teams.
- Performance Assessment: Design and execute automated performance and scalability tests on Spark jobs, large clusters, and complex query optimizations.
Required Skills and Qualifications- Education: Bachelor's or Master's degree in Computer Science, Data Engineering, or a related quantitative field.
- Experience: 8+ years of experience in data engineering, data QA, or software development engineering in test (SDET), with at least 2+ years of dedicated experience architecting test automation in Databricks.
- Expert PySpark & Python: Mastery of Python and PySpark (DataFrames and SQL APIs) for processing and profiling large datasets.
- Advanced Spark SQL: Deep expertise in writing advanced SQL queries, optimization techniques, and understanding Spark query execution plans.
- Advanced Testing Tooling: Hands-on mastery of big-data validation libraries (e.g., Great Expectations, pytest, Delta Live Tables expectations).
- Cloud Infrastructure: Strong operational knowledge of Databricks deployment on a major cloud provider (AWS, Azure, or GCP).
Preferred Qualifications- Certifications: Databricks Certified Data Engineer Professional or Databricks Certified Machine Learning Professional.
- Streaming Expertise: Experience validating real-time event-streaming architectures (Kafka, Event Hubs, Kinesis).
- Data Ops: Solid understanding of DataOps culture, testing infrastructure as code, and data observability principles.
