5-7 years of experience in Databricks and Data Architecture
Proficient in designing scalable cloud-native data solutions
Experience with Azure Data Factory and data pipeline optimization
Strong skills in Python for automation and data workflows
Knowledge of Data Lakehouse Architecture and Cloud Solution Architecture
Ability to ensure data quality and performance on Databricks clusters
Experience in documenting data processes and architecture
Responsibilities
Lead the design and development of scalable ETL/ELT pipelines using Databricks and Azure Data Factory
Optimize Spark jobs and Databricks clusters for performance and cost efficiency
Collaborate with cross-functional teams to deliver robust data solutions
Troubleshoot and resolve data pipeline issues for system reliability
Develop and enforce best practices for data security and compliance
Mentor junior team members on Databricks and Data Architecture best practices
Drive continuous improvement initiatives for data engineering processes
Benefits
Comprehensive documentation of data pipelines and processes
Opportunities for mentorship and professional development
Exposure to the latest trends in Databricks and cloud technologies
Collaborative work environment with cross-functional teams
Focus on continuous improvement and innovation in data engineering
Full Job Description
Role description
Job Title: Databricks Architect
Work Location Dallas TX
Job Description:
Seeking a Data Architect Specialist with 5 to 7 years of experience in Databricks and Data Architecture to design and implement scalable cloud native data solutions
Design develop and optimize scalable data pipelines using Databricks within the Azure Data Factory ecosystem
Collaborate with data engineers and architects to build cloud native data solutions leveraging Databricks and Python scripting Implement data transformation cleansing and integration workflows to support business intelligence and analytics needs
Ensure data quality reliability and performance of data processing jobs on Databricks clusters
Utilize Python for automation scripting and enhancing data workflows on Databricks
Work with Azure Data Factory to orchestrate and monitor end-to-end data workflows
Participate in the evaluation and adoption of new tools and technologies to improve data platform capabilities
Maintain comprehensive documentation of data pipelines processes and architecture for knowledge sharing and compliance
Apply Data Architecture principles to design scalable maintainable and efficient data solutions
Possess knowledge of Data Lakehouse Architecture and Cloud Solution Architecture to enhance data platform design and integration
Roles and Responsibilities
Lead the design and development of scalable ETLELT pipelines using Databricks and Azure Data Factory
Optimize Spark jobs and Databricks clusters for maximum performance and cost efficiency
Collaborate with cross functional teams including data scientists analysts and infrastructure teams to deliver robust data solutions
Troubleshoot and resolve data pipeline issues to ensure uninterrupted data flow and system reliability
Develop and enforce best practices for data security governance and compliance within the data platform
Mentor junior team members on Databricks technologies and Data Architecture best practices
Drive continuous improvement initiatives for data engineering processes automation and platform scalability
Stay updated with the latest trends and advancements in Databricks Data Architecture and related cloud technologies
