Job DescriptionRole: Databricks EngineerExperience4-8 yearsLocationAs per business requirement
Employment TypeFull-time
Job SummaryWe are looking for a skilled
Databricks Engineer with strong expertise in designing, developing, and optimizing modern data engineering solutions on the Databricks Lakehouse Platform. The ideal candidate should have experience building scalable ETL/ELT pipelines, working with large-scale data, and leveraging Apache Spark to deliver high-performance data solutions.
Key Responsibilities- Design, develop, and maintain scalable data pipelines using Databricks.
- Build ETL/ELT workflows for batch and streaming data processing.
- Develop solutions using PySpark, Spark SQL, and Delta Lake.
- Implement Medallion Architecture (Bronze, Silver, Gold) for data transformation.
- Integrate data from various sources including relational databases, APIs, cloud storage, and streaming platforms.
- Optimize Spark jobs for performance, scalability, and cost efficiency.
- Collaborate with Data Architects, Data Scientists, BI developers, and business stakeholders.
- Implement CI/CD pipelines and deployment automation for Databricks workloads.
- Ensure data quality, security, governance, and compliance.
- Monitor, troubleshoot, and optimize production data pipelines.
- Document technical solutions and follow engineering best practices.
Required SkillsCore Technologies- Databricks Lakehouse Platform
- Apache Spark
- PySpark
- Spark SQL
- Delta Lake
- Python
- SQL
Cloud Platforms (one or more)- Microsoft Azure (preferred)
- AWS
- Google Cloud Platform
Azure Technologies (Preferred)- Azure Data Factory (ADF)
- Azure Data Lake Storage (ADLS Gen2)
- Azure Synapse Analytics
- Azure Key Vault
- Azure DevOps
Data Engineering- Data Warehousing
- Data Modeling
- ETL/ELT Development
- Batch Processing
- Streaming (Kafka/Event Hubs)
- Data Lake Architecture
DevOps & Version Control- Git
- Azure DevOps / GitHub
- CI/CD Pipelines
Preferred Qualifications- Experience with Unity Catalog.
- Knowledge of Databricks Workflows and Jobs.
- Hands-on experience with Delta Live Tables (DLT).
- Exposure to MLflow is an added advantage.
- Experience with data governance and security best practices.
- Familiarity with Infrastructure as Code (Terraform) is a plus.
Educational Qualification- Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, or a related field.
Preferred Certifications- Databricks Certified Data Engineer Associate
- Databricks Certified Data Engineer Professional
- Microsoft Certified: Azure Data Engineer Associate (DP-203)
- Azure Fundamentals (AZ-900)
Good to Have- Experience with real-time analytics.
- Knowledge of Lakehouse architecture.
- Experience with Agile/Scrum methodologies.
- Strong analytical and problem-solving skills.
- Excellent communication and stakeholder management abilities.
Mandatory Skills- Databricks
- PySpark
- Spark SQL
- Delta Lake
- Python
- SQL
- Azure/AWS/GCP (at least one cloud platform)
- ETL/ELT Development
- Data Lake Architecture
Nice to Have- Unity Catalog
- Delta Live Tables (DLT)
- MLflow
- Kafka/Event Hubs
- Azure Data Factory
- Terraform
- Azure DevOps/GitHub Actions
