Sr Data Engineer, Python + Spark (Data Federation skillset - Data Lakehouse - Eg: Starburst) - New York

Photon$40K — $140K *
Information Technology
5 - 7 years of experience
Job Overview by Ladders

Qualifications

  • 5+ years of experience in Python and Apache Spark tuning.
  • Hands-on experience with Starburst Enterprise, Trino, or Dremio.
  • Proven track record with Delta Lake or Iceberg architectures.
  • Extensive experience with AWS, Azure, or GCP cloud platforms.
  • Expert-level SQL skills for complex analytical queries.
  • Proficiency in data modeling with Star/Snowflake schemas and Medallion Architecture.

Responsibilities

  • Design and implement federated query layers for high-speed analytics across distributed data sources.
  • Build scalable, distributed data processing pipelines using Python and Apache Spark.
  • Manage and optimize table formats like Delta Lake and Apache Iceberg for ACID transactions.
  • Optimize Spark jobs and SQL queries to minimize latency and manage compute costs.
  • Implement governance and security measures to ensure data privacy across platforms.

Benefits

  • Medical, vision, and dental benefits.
  • 401k retirement plan.
  • Variable pay/incentives.
  • Paid time off and paid holidays.
Full Job Description
Job Description

Senior Data Engineer (Data Federation & Lakehouse)

As a Senior Data Engineer, you will be responsible for breaking down data silos. This role focuses on building a unified, high-performance data layer using Data Federation techniques. You won't just move data; you will architect a Data Lakehouse environment where disparate sources feel like a single, cohesive database for our analytics and AI teams.

### Core Responsibilities
  • Data Federation Architecture: Design and implement federated query layers (e.g., Starburst/Trino) to allow high-speed analytics across distributed data sources without unnecessary data movement.
  • ETL/ELT Pipeline Development: Build scalable, distributed data processing pipelines using Python and Apache Spark (PySpark).
  • Lakehouse Implementation: Manage and optimize modern table formats like Delta Lake, Apache Iceberg, or Hudi to bring ACID transactions to our data lake.
  • Performance Tuning: Optimize Spark jobs and SQL queries across the federation layer to minimize latency and manage compute costs.
  • Governance & Security: Implement fine-grained access control and data masking within the federation engine to ensure data privacy across all connected platforms.

### Technical Requirements
  • Python & Spark: 5+ years of experience with Python and deep expertise in Apache Spark tuning (partitioning, shuffling, caching).
  • Data Federation Tools: Hands-on experience with Starburst Enterprise, Trino (Presto), or Dremio.
  • Lakehouse Ecosystem: Proven track record working with Delta Lake or Iceberg architectures.
  • Cloud Platforms: Extensive experience with AWS (EMR, S3, Glue), Azure (Databricks, ADLS), or GCP.
  • SQL Mastery: Expert-level SQL skills for complex analytical queries and query plan analysis.
  • Data Modeling: Proficiency in designing Star/Snowflake schemas and understanding "Medallion Architecture" (Bronze, Silver, Gold layers).

### Preferred "Bonus" Skills
  • Experience with Infrastructure as Code (IaC) like Terraform or Pulumi.
  • Familiarity with dbt (data build tool) for modeling within the federation layer.
  • Knowledge of Kubernetes (K8s) for deploying and scaling Spark/Trino clusters.
  • Background in Data Mesh or Data Fabric methodologies.

    Compensation, Benefits and Duration

    Minimum Compensation: USD 40,000
    Maximum Compensation: USD 140,000
    Compensation is based on actual experience and qualifications of the candidate. The above is a reasonable and a good faith estimate for the role.
    Medical, vision, and dental benefits, 401k retirement plan, variable pay/incentives, paid time off, and paid holidays are available for full time employees.
    This position is not available for independent contractors
    No applications will be considered if received more than 120 days after the date of this post

About Photon

Photon Careers

Joining Photon presents an unparalleled opportunity to advance one's career with a leader in digital innovation. Photon is actively seeking talented individuals to join its diverse team of professionals, dedicated to reshaping industries through technology and creativity.

Explore Job Opportunities

Photon offers a variety of job opportunities that cater to a range of skills and interests. Each position at Photon is designed to challenge team members while providing meaningful pathways for professional and personal growth.

Experience the Culture and Benefits

Photon is committed to fostering a workplace culture that promotes diversity and inclusion. The company provides comprehensive benefits designed to support the health, well-being, and financial security of each team member. At Photon, every professional enjoys access to career development programs and diversity training, ensuring they are equipped for leadership roles in an evolving marketplace.

Engage in Professional Growth

Photon believes in the power of innovation and leadership to drive success. The company supports its team members with resources for continuous learning and growth, including advanced training sessions and leadership workshops. Networking within Photon’s industry-leading community allows for unparalleled career advancement and skill enhancement.

Internship Programs

For those beginning their professional journey, Photon offers internship programs that provide a robust foundation in digital solutions. Internships at Photon are characterized by immersive projects and hands-on learning, guided by experts in various fields. These programs are a gateway to full-time employment and a promising career path.

Join the Team

Photon is hiring! Explore open positions that match your skills and interests. Photon looks for passionate, curious, creative, and solution-driven team players. Whether you are starting your career or looking to make a significant impact in your professional journey, Photon has a place for you.

Prepare for Your Interview

Photon values a thorough selection process. Prospective team members are encouraged to prepare by understanding the company’s mission and values. Tailoring your resume to highlight relevant experience and skills will make a significant difference. Engage actively during your interview, demonstrating your knowledge and enthusiasm for the role you are applying for.

Stay Connected

Keep up to date with the latest from Photon by following the company’s career blog. Gain insights from insiders, learn about new job openings, and get tips on preparing your resume and acing your interview.

Career Development at Photon

Photon is dedicated to the professional development of its team members. The company offers various tools and resources, including career coaching and performance feedback, to help individuals reach their full potential. With Photon, career aspirations are nurtured, encouraging every team member to excel and innovate in their respective roles.

Networking and Career Opportunities

Photon encourages its team members to engage in networking within the company and the broader industry. These connections can lead to myriad opportunities, from collaborative projects to mentorship. Networking at Photon is not just about building professional relationships; it's about fostering a community of innovation and mutual growth.

Conclusion

Photon is not just a company; it is a community of driven professionals committed to making a difference. With a focus on innovation, leadership, and diversity, Photon offers a dynamic environment where ambitious individuals can thrive. Explore the possibilities and join Photon’s team of leaders, innovators, and visionaries.
Learn more about Photon

Similar Jobs

More Jobs at Photon

More Information Technology Jobs

Find similar Sr Data Engineer, Python + Spark (Data Federation skillset - Data Lakehouse - Eg: Starburst) - New York jobs: