Databricks Engineer

Apexon

$100K — $130K *
Information Technology
Less than 5 years of experience
Job Overview by Ladders

Qualifications

  • 5-7 years of experience in data engineering or related field.
  • Proven expertise in Databricks Lakehouse, PySpark, and SQL.
  • Hands-on experience implementing Medallion Architecture.
  • Strong performance tuning skills for Databricks workloads.
  • Experience managing Unity Catalog with security configurations.
  • Familiarity with data lineage and secure data sharing practices.
  • Knowledge of CI/CD implementation related to Databricks.

Responsibilities

  • Design and develop scalable data pipelines using Databricks, PySpark, and SQL.
  • Optimize Medallion Architecture for effective data processing.
  • Tune and enhance performance of Databricks workloads and Lakehouse.
  • Manage Unity Catalog for security and data governance.
  • Oversee data lineage, governance, and secure data sharing.
  • Implement CI/CD pipelines and automate testing in Databricks.
  • Collaborate with teams to deliver robust data solutions.

Benefits

  • Comprehensive health insurance including dental and vision.
  • 401K retirement plan.
  • Life insurance along with short and long-term disability coverage.
  • Generous paid vacation and observed holidays.
  • Opportunities for continuous skill development and career advancement.
Full Job Description
Role: Databricks Engineer Location: Remote Position: 1 Employment Type: W2 Only Role Overview: We are seeking an experienced Databricks Engineer with strong expertise in Databricks Lakehouse, PySpark, SQL, Medallion Architecture, and Unity Catalog. The ideal candidate will be responsible for designing, developing, optimizing, and securing scalable data pipelines while implementing best practices for data governance, performance tuning, and CI/CD within the Databricks platform. Key Responsibilities: • Design, develop, and maintain scalable data pipelines using Databricks, PySpark, and SQL. • Implement and optimize Medallion Architecture for efficient data processing. • Perform performance tuning and optimization of Databricks workloads and Lakehouse environments. • Configure and manage Unity Catalog, including row level and column level security. • Manage data lineage, governance, and secure data sharing across enterprise environments. • Implement and maintain CI/CD pipelines and automated testing processes within Databricks. • Collaborate with cross functional teams to deliver reliable and scalable data solutions. • Ensure data quality, security, and compliance across data engineering processes. • Troubleshoot and resolve data pipeline and platform performance issues. • Support ongoing enhancements and operational activities within the Databricks ecosystem. Required Skills: • Strong proficiency in PySpark. • Advanced SQL experience. • Hands on experience with Databricks Lakehouse Platform. • Experience implementing Medallion Architecture. • Performance tuning and optimization within Databricks. • Experience configuring Unity Catalog. • Knowledge of row level and column level security. • Experience managing data lineage and secure data sharing. • Hands on experience with CI/CD implementation and testing processes in Databricks. • Strong analytical and problem solving skills. • Excellent verbal and written communication skills. Preferred Skills: • SAS • SQL • Unix • Linux Expected Deliverables: • Scalable Databricks Data Pipelines. • Optimized Lakehouse Performance. • Medallion Architecture Implementation. • Unity Catalog Security Configuration. • Data Lineage and Governance Solutions. • CI/CD Pipeline Implementation. • Technical Documentation and Operational Support. Education Requirement: • Bachelor's degree required. Our Perks and Benefits: Our benefits and rewards program has been thoughtfully designed to recognize your skills and contributions, elevate your learning experience, and provide care and support for you and your loved ones. We also offer: • Health Insurance with Dental & Vision • 401K Plan • Life Insurance, STD & LTD • Paid Vacations & Holidays • Continuous Skill Based Development and Career Advancement Opportunities

Similar Jobs

More Jobs at Apexon

More Information Technology Jobs

Find similar Databricks Engineer jobs: