Data Engineer

JOB DESCRIPTION

Job Title: Data Engineer

Position Type: Full Time

Work Setting: In Office

Group/Division/Team: Data Engineer

Reports to: Data Engineering Team Lead

FLSA: Salary, Exempt

THE POSITION

You'll be a key member of VaxCare's Product Group, joining our Data Engineering team and reporting to our Data Engineering Lead. We are seeking a motivated and capable Data Engineer to join our team. As a Data Engineer, you will contribute to the design, development, and management of our data processing and analytics infrastructure. The ideal candidate will have hands-on experience working with Spark and Databricks, a solid foundation in data engineering principles, and a desire to grow into a senior technical contributor.

ESSENTIAL RESPONSIBILITIES

• Develop and maintain Delta Lake-based data pipelines using Databricks Workflows, Delta Live Tables (DLT), and Unity Catalog for enterprise data governance
• Build ELT/ETL pipelines using medallion architecture (bronze/silver/gold layers) supporting both batch and streaming workloads with Auto Loader and Structured Streaming
• Implement lakehouse solutions leveraging Delta Lake ACID transactions, Z-ordering, liquid clustering, and partitioning strategies
• Support CI/CD pipelines for data workflows using Git integration and Databricks Asset Bundles
• Contribute to data quality frameworks using Delta Live Tables expectations and custom PySpark validation with automated alerting and SLA monitoring
• Create materialized views and incremental refresh strategies for optimized query performance
• Collaborate with data scientists, ML engineers, and analysts to support feature engineering pipelines and MLOps workflows
• Participate in code reviews and contribute to technical design discussions
• Implement data observability and monitoring using Databricks SQL, Lakeview dashboards, and custom alerting frameworks
• Support cost optimization efforts leveraging Photon engine, serverless compute, and platform best practices
• Troubleshoot and resolve issues related to distributed computing, data skew, and performance bottlenecks with guidance from senior engineers
• Contribute to technical documentation including data contracts, runbooks, and data catalog metadata in Unity Catalog
• Follow DataOps best practices including testing strategies, performance tuning, and data platform engineering principles
• Stay current with lakehouse architecture trends and emerging technologies to continuously improve our data infrastructure

REQUIRED EXPERIENCE & QUALIFICATIONS

Education:

• Bachelor's degree in Computer Science, Data Engineering, Engineering, or related technical field OR equivalent practical experience
• Relevant industry certifications (Databricks Certified Data Engineer Associate, Azure Data certifications) are a plus

Experience:

• 3-5 years of data engineering experience with 1+ years hands-on production experience building data pipelines on Databricks and Apache Spark
• Experience contributing to lakehouse architecture implementations

Technical Skills:

Programming & Languages:

• Strong proficiency in Python (PySpark, pandas) and SQL (complex queries, window functions, CTEs, query optimization)
• Experience with Spark SQL, Delta Lake SQL, and Databricks SQL

Apache Spark Expertise:

• Working knowledge of Apache Spark including:
  • Performance fundamentals (partitioning, broadcast joins, data skew handling, caching strategies)
  • Delta Lake features (ACID transactions, time travel, MERGE operations, CDC, liquid clustering)

Databricks Platform:

• Hands-on experience with Databricks including:
  • Delta Live Tables (DLT) for declarative pipeline development
  • Unity Catalog for data governance, access control, and lineage tracking
  • Databricks Workflows and orchestration
  • Basic understanding of cluster configuration and cost-aware compute selection
  • Databricks SQL and Lakeview dashboards

Data Architecture & Modeling:

• Solid understanding of data modeling techniques:
  • Dimensional modeling (star schema, fact/dimension tables)
  • Medallion architecture (bronze/silver/gold layers)
  • Slowly Changing Dimensions (SCD) implementations
• Strong SQL skills including query optimization and performance tuning
• Familiarity with modern lakehouse patterns and understanding of lakehouse vs. traditional data warehouse trade-offs

DevOps & DataOps:

• Familiarity with DevOps/DataOps practices:
  • Git workflows (branching strategies, pull requests, code reviews)
  • CI/CD pipelines for data workflows (GitHub Actions, Azure DevOps, Jenkins)
  • Testing strategies (unit tests, integration tests, data quality tests)
  • Basic monitoring and observability (logging, alerting)

Collaboration & Growth:

• Strong communication and collaboration skills in cross-functional team environments
• Eagerness to learn and grow into a senior engineering role
• Systematic approach to debugging and problem-solving
• Ability to take feedback constructively and iterate on solutions
• Comfort working alongside senior engineers and asking questions to deepen technical understanding

OTHER RESPONSIBILITIES

• Commute to Office: Must be able to commute to and from the corporate office for each scheduled shift.
• Prolonged sitting or standing: Team members must be able to remain in a stationary position for extended periods, as much of the work involves sitting at a computer or workstation. Some hybrid or office-based roles may also require the ability to stand for portions of a shift.
• Repetitive motions: Significant use of the hands, wrists, and fingers is required for tasks like typing, using a computer mouse, and handling paperwork.
• Treating data engineering as a craft, with strong attention to correctness, performance, and maintainability
• Visual acuity: This job requires the ability to view a computer screen and read documents for extended periods.
• Communication: Team members must be able to clearly express and exchange ideas in English to effectively communicate with customers and co-workers.
• Occasional lifting: Many office-based jobs specify the ability to occasionally lift or move up to 10-20 pounds to handle supplies or equipment.

Physical Requirements/Work Environment:

TASK

Continuous

Intermittent

Seldom

N/A

Regular & Reliable Attendance

X

Standing/Walking

X

Travel/Driving/Operating Vehicle

X

Sitting (desk/computer work)

X

Lifting/Lowering/Carrying (up to 20lbs)

X

Reaching/Bending/Twisting

X

Hand/Wrist Use (typing/mouse/writing)

X

In person attendance for meetings/conferences

X

Operating Mobile Equipment/Machinery

X

By signing below, I acknowledge that I have received this job description.

My signature further acknowledges that I have reviewed this job description and understand that I have the individual responsibility to fulfill all the essential duties and responsibilities listed on a consistent and ongoing basis.

Team Member Name

Team Member Signature Date