We are seeking an experienced Databricks Data Engineer to design, develop, and maintain scalable data pipelines and cloud-based data platforms. The ideal candidate will have strong expertise in Databricks, PySpark, Python, SQL, Delta Lake, and cloud technologies (Azure/AWS), with experience building enterprise-scale data solutions that support analytics, reporting, and machine learning workloads.
Key Responsibilities
- Design, develop, and optimize scalable ETL/ELT pipelines using Databricks and PySpark.
- Build and maintain data ingestion frameworks for batch and real-time data processing.
- Develop data models and implement data transformation logic using SQL and Python.
- Leverage Delta Lake for data management, data quality, and performance optimization.
- Implement data orchestration workflows using tools such as Azure Data Factory, Airflow, or Databricks Workflows.
- Collaborate with data architects, analysts, data scientists, and business stakeholders to understand requirements and deliver solutions.
- Optimize Databricks jobs and Spark applications for performance, scalability, and cost efficiency.
- Implement data governance, security, and compliance best practices.
- Monitor and troubleshoot production data pipelines and provide support for critical issues.
- Participate in code reviews, documentation, and knowledge-sharing activities.
Technical Skills:
Strong/expert Spark (PySpark) Using Jupyter Notebooks, Colab or DataBricks (preferred)
Hands-on data pipeline development, ingest patterns in Azure
Orchestration tools, ADF or Airflow SQL
Denormalized Data modeling for big data systems
Job Description
Data engineers are responsible for building reliable and scalable data infrastructure that enables organizations to derive meaningful insights, make data-driven decisions, and unlock the value of their data assets.
Job Description - Grade Specific
The role involves leading and managing a team of data engineers, defining and executing the data engineering strategy, and ensuring the effective delivery of data solutions. They provide technical expertise, drive innovation, and collaborate with stakeholders to deliver high quality, scalable, and reliable data infrastructure and solutions.
The base compensation range for this role in the posted location is $102,523 to $183,760.
Capgemini offers a comprehensive, non-negotiable benefits package to all regular, full-time employees. In the U.S. and Canada, available benefits are determined by local policy and eligibility and may include:
- Paid time off based on employee grade (A-F), defined by policy: Vacation: 12-25 days, depending on grade, Company paid holidays, Personal Days, Sick Leave
- Medical, dental, and vision coverage (or provincial healthcare coordination in Canada)
- Retirement savings plans (e.g., 401(k) in the U.S., RRSP in Canada)
- Life and disability insurance
- Employee assistance programs
- Other benefits as provided by local policy and eligibility
