Role Overview:We are seeking an experienced Databricks Consultant to design, develop, and implement scalable data solutions on the Databricks platform. The ideal candidate will have strong expertise in data engineering, cloud platforms, Spark, and modern data architectures, with the ability to work closely with business and technical stakeholders.
Key Responsibilities:- Design and develop data pipelines using Databricks, Apache Spark, and PySpark.
- Build and maintain ETL/ELT processes for large-scale data ingestion and transformation.
- Implement data lake and lakehouse architectures on Databricks.
- Integrate data from multiple sources, including databases, APIs, and cloud storage.
- Optimize Spark jobs and Databricks workloads for performance and cost efficiency.
- Collaborate with business users, architects, and developers to understand requirements and deliver solutions.
- Develop data models and support reporting, analytics, and AI/ML initiatives.
- Implement data quality, governance, security, and monitoring standards.
- Support production deployments, troubleshooting, and performance tuning.
- Prepare technical documentation and provide knowledge transfer to project teams.
Required Skills:- Strong hands-on experience with Databricks.
- Expertise in Apache Spark, PySpark, and SQL.
- Experience with cloud platforms such as Azure, AWS, or GCP.
- Knowledge of Delta Lake, Unity Catalog, and Databricks Workflows.
- Experience with data warehousing and data lake concepts.
- Strong understanding of ETL/ELT frameworks and data modeling.
- Familiarity with CI/CD, Git, and DevOps practices.
- Excellent problem-solving and communication skills.
Qualifications:- Bachelor's degree in Computer Science, Information Technology, Engineering, or related field.
- 5+ years of experience in Data Engineering/Big Data.
- 2+ years of hands-on Databricks implementation experience.
Preferred Skills:- Experience with Azure Data Factory (ADF), Azure Synapse, or AWS Glue.
- Knowledge of Machine Learning and MLOps on Databricks.
- Experience with streaming technologies such as Kafka and Structured Streaming.
- Databricks or Cloud Platform certifications.
