We are seeking a highly skilled
Senior Databricks Data Engineer to design, build, and optimize scalable data platforms and pipelines on cloud-based environments. The ideal candidate will have deep expertise in
Databricks, Apache Spark, Delta Lake, Python/PySpark, SQL, and AWS, with strong experience in enterprise data engineering, performance tuning, and modern data architecture. This role will partner closely with architects, analysts, and business stakeholders to deliver reliable, high-performing data solutions that support analytics, AI, and business intelligence initiatives.
Your role
- Design, develop, and maintain data pipelines using Databricks, Apache Spark, and Delta Lake.
- Build and optimize ETL/ELT processes for ingesting structured and unstructured data from multiple sources.
- Develop and maintain notebooks using PySpark, Scala, or SQL for data transformation and analysis.
- Implement Delta Lake architectures (bronze, silver, gold layers) following medallion architecture best practices.
- Collaborate with data architects, analysts, and business stakeholders to translate requirements into technical solutions.
- Optimize Spark jobs for performance, including cluster configuration, partitioning, and caching strategies.
- Implement data quality checks, validation frameworks, and monitoring for pipeline reliability.
- Integrate Databricks with cloud platforms (AWS) .
- Work with CI/CD pipelines for automated deployment of notebooks and workflows .
- Implement version control and code review practices using Git-based workflows.
Skills and experience
- Bachelor's degree in Computer Science, Information Systems, Engineering, or a related discipline.
- 7+ years of experience in Data Engineering, Data Warehousing, or Big Data technologies.
- 3+ years of hands-on experience with Databricks and Apache Spark.
- Strong expertise in PySpark, SQL, and data transformation frameworks.
- Experience implementing Delta Lake and Medallion Architecture patterns.
- Strong understanding of distributed computing concepts and Spark performance optimization.
- Experience developing and supporting data solutions in AWS Cloud environments.
- Hands-on experience with CI/CD tools and DevOps practices.
- Proficiency with Git-based version control systems.
- Knowledge of data modeling, ETL/ELT design patterns, and modern data lakehouse architectures.
- Strong communication and stakeholder management skills.
The base compensation range for this role in the posted location is: $83,950 - $156,310.
Capgemini provides compensation range information in accordance with applicable national, state, provincial, and local pay transparency laws. The base compensation range listed for this position reflects the minimum and maximum target compensation Capgemini, in good faith, believes it may pay for the role at the time of this posting. This range may be subject to change as permitted by law.
The actual compensation offered to any candidate may fall outside of the posted range and will be determined based on multiple factors legally permitted in the applicable jurisdiction.
These may include, but are not limited to: Geographic location, Education and qualifications, Certifications and licenses, Relevant experience and skills, Seniority and performance, Market and business consideration, Internal pay equity.
It is not typical for candidates to be hired at or near the top of the posted compensation range.
In addition to base salary, this role may be eligible for additional compensation such as variable incentives, bonuses, or commissions, depending on the position and applicable laws.
Capgemini offers a comprehensive, non-negotiable benefits package to all regular, full-time employees. In the U.S. and Canada, available benefits are determined by local policy and eligibility and may include:
- Paid time off based on employee grade (A-F), defined by policy: Vacation: 12-25 days, depending on grade, Company paid holidays, Personal Days, Sick Leave
- Medical, dental, and vision coverage (or provincial healthcare coordination in Canada)
- Retirement savings plans (e.g., 401(k) in the U.S., RRSP in Canada)
- Life and disability insurance
- Employee assistance programs
- Other benefits as provided by local policy and eligibility