Senior ETL & AI/ML Test Automation Engineer
Must Have Technical/Functional Skills
ETL & Data Testing
• Validate end-to-end ETL/ELT pipelines across enterprise data platforms.
• Perform Source-to-Target validation and reconciliation testing.
• Validate data transformations, aggregations, business rules, and lineage.
• Execute data quality checks on large-scale structured and semi-structured datasets.
• Perform schema, metadata, and Delta Lake validations.
• Write complex SQL queries for backend validation and data integrity testing.
• Validate CDC (Change Data Capture) and incremental data loading processes.
• Test batch and streaming data pipelines.
Databricks Testing
• Validate Databricks notebooks, Workflows, Jobs, Delta Live Tables, and Unity Catalog implementations.
• Test Bronze, Silver, and Gold layer data transformations.
• Verify workflow orchestration and dependency handling across Databricks jobs.
• Perform performance validation of Spark and PySpark transformations.
• Validate data quality frameworks implemented in Databricks.
AI/ML Testing
• Validate AI/ML model integrations within enterprise applications.
• Verify model scoring results and prediction outputs.
• Perform data drift, schema drift, and feature validation testing.
• Validate feature engineering pipelines and training datasets.
• Execute model input/output verification against expected business outcomes.
• Validate explainability, traceability, and auditability requirements.
• Test model deployment workflows integrated through MLflow, Databricks ML, Azure ML, or similar platforms.
• Validate GenAI/RAG workflows, prompt-response accuracy, and grounding mechanisms where applicable.
Integration Testing
• Perform end-to-end integration testing across:
o Databricks
o Data warehouses
o APIs
o AI/ML services
o Cloud platforms
o Downstream reporting systems
• Validate data flow between source systems, data lakes, ML models, and consuming applications.
• Perform contract testing and API validation.
Test Automation
• Design and develop automation frameworks for:
o ETL testing
o API testing
o Data validation
o ML model validation
• Build reusable test utilities using Python.
• Integrate automated tests into CI/CD pipelines.
• Develop automated quality gates for deployment validation.
• Generate automated execution reports and test metrics.
required
6+ years of experience in ETL/Data Testing.
3+ years of experience testing Databricks or Spark-based platforms.
Hands-on experience in AI/ML Model Testing or MLOps validation.
Strong coding experience in Python and SQL. Education
Bachelor's degree
Salary Range: $120000 - $130000 a year
#LI-JH1
