Databricks

Azure Databricks Lead Primary Skill Azure Databricks PySpark Delta Lake

Databricks • $130K — $160K *
New York, NY 10025In-Person
Information Technology
Less than 5 years of experience
3 weeks ago
Job Overview by Ladders

Qualifications

  • 5+ years of experience with Azure Databricks
  • Proficiency in PySpark and Delta Lake
  • Strong command of SQL
  • Experience in building ETL/ELT pipelines
  • Good communication skills

Responsibilities

  • Lead the design and implementation of Azure Databricks solutions
  • Optimize existing Spark jobs for performance and cost efficiency
  • Collaborate with cross-functional teams to integrate with Azure services
  • Implement governance and security measures using Unity Catalog
  • Conduct code reviews and enforce coding standards across the team

Benefits

  • Onsite work in New York City
  • Opportunity to lead and innovate in Azure technologies
  • Collaborative work environment with cross-functional teams
  • Exposure to advanced data engineering practices
  • Opportunities for professional development and upskilling
Full Job Description
Role description

Databricks Lead Primary Skill Azure Databricks PySpark Delta Lake

Location New York USA Onsite

Must Have Skills

Azure Databricks

Pyspark

Delta Lake

SQL

Secondary Nice To Have Skills NOT MANDATORY

AI engineering GenAI enablement

Soft Skills

Good Communication Skills

3 Qualifying Questions

1Describe an endtoend Azure Databricks solution you led ingestion transformation serving What was the data architecture eg Delta LakeMedallion how did you design ETLELT pipelines and how did you integrate with Azure services ADLS ADF Event Hubs Synapse etc

2Give a concrete example where you improved a slow or expensive Spark job in Databricks How did you diagnose the bottleneck skew shuffle partitioning joins what changes did you implement AQE broadcast joins repartitioning caching file sizing ZORDEROPTIMIZE cluster sizing and what measurable performancecost improvement did you achieve

3How have you implemented governance and security in Databricks Unity Catalog access controls secrets networksecurity model and automated deploymentsmonitoring CICD job orchestration s logging Share your approach to enforcing coding standards and conducting code reviews across the team

About Databricks

Databricks is a unified analytics platform that provides data engineering, collaborative data science, and machine learning capabilities. The company was founded in 2013 by the original creators of Apache Spark, a popular open-source big data processing engine. Databricks provides a cloud-based platform that allows data teams to collaborate and build data pipelines, run machine learning models, and perform advanced analytics. The company has raised over $1 billion in funding and is valued at $38 billion as of November 2021.
Learn more about Databricks
Size
2,000 employees
Industry
Information Technology
Founded
2013
* Ladders Estimates

Similar Jobs

Ad banner background

Get Ready For Your
Next Interview

More Jobs at Databricks

More Information Technology Jobs

Find similar Azure Databricks Lead Primary Skill Azure Databricks PySpark Delta Lake jobs:

NationwideNew York, NY