Your Opportunity
BDO Digital is seeking an experienced and technically proficient Senior Data Engineer (Databricks Focused) to join our Technology Advisory Services practice. This role blends technical leadership with hands-on data engineering, focusing on delivering scalable, secure, and production-grade data pipelines using Azure and Databricks. The successful candidate will also lead client engagements, aligning data initiatives with business objectives to create real-world impact.
Key Responsibilities
Data Engineering and Platform Development
- Design, build, and optimize robust data pipelines using Databricks, Delta Lake, and Apache Spark.
- Develop scalable ETL processes for structured and unstructured data, ensuring data quality and reliability.
- Implement data integration solutions across cloud and on-premises sources, leveraging Databricks and Azure Data Lake.
- Automate data ingestion, transformation, and validation workflows.
- Monitor, troubleshoot, and enhance data pipeline performance and reliability.
- Ensure compliance, security, and operational excellence across all data workflows.
Client and Team Leadership
- Serve as a primary technical lead for client engagements focused on data engineering and analytics solution delivery.
- Translate business requirements into scalable data architectures and operational plans.
- Collaborate with cross-functional teams to integrate data solutions into production environments.
- Mentor junior team members and promote adoption of best practices in data engineering and cloud analytics.
Data Application
- Convert business requirements and analytics prototypes into robust, scalable data solutions.
- Apply appropriate data modeling and transformation techniques to support analytics and machine learning initiatives.
- Document data pipelines and contribute to internal knowledge sharing.
Qualifications
Required
- Educational Background: A Bachelor’s or master’s degree in Computer Science, Data Engineering, Information Systems, or a closely related discipline. Strong foundation in data structures, algorithms, and distributed systems.
- Professional Experience: A minimum of 5 years of hands-on experience in data engineering, including at least 3 years of direct experience with Databricks, Apache Spark, and cloud data platforms. Proven success in deploying and maintaining large-scale data solutions in production environments.
- In-depth knowledge of the Microsoft Azure ecosystem, with demonstrated experience using services such as Azure Data Lake, Azure Databricks, Azure Synapse, and Azure DevOps.
- Very strong proficiency with Databricks, including hands-on work with Delta Lake, Apache Spark, and data pipeline orchestration. Experience optimizing performance and reliability in production.
- Advanced programming skills in Python and/or Scala, with practical experience using data engineering libraries and frameworks.
- Solid understanding of CI/CD practices, with experience designing and maintaining pipelines using tools like GitHub Actions, Azure DevOps, or Jenkins. Familiarity with infrastructure-as-code tools such as Terraform or ARM templates for automating environment provisioning and deployment.
- Excellent written and verbal communication skills, with the ability to effectively engage with a range of stakeholders, including data scientists, engineers, business partners, and executive leadership. Proven ability to explain technical concepts to non-technical audiences and influence decision-making.
Preferred
- Certifications: Professional certifications such as Databricks Certified Data Engineer or Microsoft Certified, e.g. Azure Data Engineer Associate.
- Experience with data governance, data security, and compliance frameworks.
- Consulting experience and excellent communication and client management skills.
The expected range of compensation for this role is $84,000 to $128,000.
#LI-MM1
