Senior Data Engineer

Systems Planning and Analysis, Inc.

$156K — $179K *
Information Technology
8 - 10 years of experience
Job Overview by Ladders

Qualifications

  • Master's degree in Computer Science, Information Systems, Software Engineering, or related field
  • 10+ years of data engineering, software engineering, or related technical experience
  • Extensive hands-on experience with production data pipelines and data products
  • Advanced proficiency in Python and SQL
  • Strong experience with Apache Spark and distributed data processing
  • Experience with Databricks or modern data platforms
  • Ability to communicate complex technical concepts to diverse audiences
  • Possess and maintain an active Secret clearance

Responsibilities

  • Design, develop, and maintain scalable data pipelines using Spark (Python/SQL) in Databricks
  • Lead the integration and transformation of data from multiple federal health systems
  • Establish reusable data engineering patterns and best practices
  • Monitor and optimize production workflows for performance and reliability
  • Define and implement data governance practices for product usability
  • Collaborate with stakeholders to translate data needs into technical solutions
  • Mentor engineers and guide them through technical challenges
  • Identify opportunities for improvement in engineering tools and processes

Benefits

  • Health insurance
  • Flexible spending accounts
  • Health savings accounts
  • Retirement savings plans
  • Life and disability insurance programs
  • Opportunity for paid and unpaid time off
Full Job Description
Overview

Our Financial Management & Business Analysis Portfolio supports the U.S. Army Space and Missile Defense Command (USASMDC); the Office of the Assistant Secretary of the Army (ASA) Financial Management and Comptroller (FM&C); the U.S. Army Financial Management Command (USAFMCOM); the Defense Health Agency (DHA); and many other DoD customers. We ensure the Army FM systems and processes are modernized and integrated to provide a complete range of financial and cost information needed for the Army to conduct business transactions, provide accountability to the public, and to support performance reporting and decision making. We provide effective functional systems support, user technical support, training support, and governance support of the Army's modernized and deployed FM domain ERP systems (GFEBS / GFEBS-SA / GCSS-A (Finance)), ensuring technological capabilities maturation and evolution aligns with Army and FM domain goals and objectives.

We are looking for a Senior Data Engineer to join our team to support the Defense Health Agency Chief Data and Analytics Office in building an enterprise data orchestration layer for the Military Health System. The team will be bringing together information from diverse defense and federal systems to build scalable data pipelines, APIs, and data products that improve how critical health data is integrated, accessed, and used.

Responsibilities

  • Design, develop, and maintain scalable, production-ready data pipelines and data products using Spark (Python/SQL) in a Databricks environment.
  • Lead the integration and transformation of complex data from diverse DoW and federal health systems and sources into reliable, reusable data products.
  • Design scalable approaches for data ingestion, integration, and exchange, including API-based integrations and services.
  • Establish and promote reusable data engineering patterns, standards, and best practices that improve consistency, scalability, and maintainability across data products.
  • Provide technical guidance on data architecture, pipeline design, data modeling, integration approaches, and engineering practices.
  • Monitor, troubleshoot, and optimize production workflows and data pipelines, identifying performance, reliability, and scalability improvements.
  • Define and implement data validation, quality, and governance practices that improve the reliability and usability of data products.
  • Troubleshoot complex technical and data integration challenges, identify root causes, and drive sustainable solutions.
  • Collaborate with engineers, architects, analysts, and customer stakeholders to translate complex data needs into scalable technical solutions.
  • Provide technical guidance and mentorship to other engineers, helping teams navigate complex or unfamiliar technical challenges.
  • Proactively identify opportunities to improve engineering tools, processes, and patterns and help drive their adoption across the team.
  • Take ownership of complex technical areas and help maintain engineering quality, consistency, and cohesion as the platform and portfolio of data products grow.

Qualifications

Required Qualifications:
  • Master's degree in Computer Science, Information Systems, Software Engineering, or a related field
  • 10+ years of data engineering, software engineering, or related technical experience
  • Extensive hands-on experience designing, building, and operating production data pipelines and data products
  • Advanced proficiency with Python and SQL
  • Strong experience with Apache Spark and distributed data processing
  • Experience working with Databricks or similar modern data platforms
  • Experience designing and maintaining ETL/ELT processes for complex, large-scale datasets
  • Experience troubleshooting and optimizing complex production data pipelines for performance, reliability, and scalability
  • Experience working with Git-based development workflows and modern software engineering practices
  • Demonstrated experience providing technical guidance, mentoring engineers, and influencing engineering practices
  • Strong client and stakeholder communication skills, with the ability to translate technical concepts and recommendations for both technical and non-technical audiences
  • Ability to independently navigate ambiguity, identify technical risks, and drive complex engineering challenges toward resolution
  • Experience spotting security, privacy and compliance issues and working with security/compliance/legal stakeholders
  • Possess and maintain an active Secret clearance

Desired Qualifications:
  • AWS cloud experience
  • Experience working with very large datasets, including datasets with billions of records
  • Experience with Palantir Foundry
  • GitLab experience
  • Experience working with Advana or similar DoW data environments
  • Experience working with federal health, financial, or other regulated and sensitive data
  • Experience using AI/ML to accelerate work, including automating routine tasks, accelerating development and debugging

Pay Range Information

At SPA, we strive to deliver a robust total compensation package that will attract and retain top talent. Elements of the compensation package include competitive base pay and variable compensation opportunities. SPA provides eligible employees with an opportunity to enroll in a variety of benefit programs, generally including health insurance, flexible spending accounts, health savings accounts, retirement savings plans, life and disability insurance programs, and a number of programs that provide for both paid and unpaid time away from work. The specific programs and options available to any given employee may vary depending on eligibility factors such as geographic location, date of hire, etc. Please note that the salary information shown below is a general guideline only. Salaries are commensurate with experience and qualifications, as well as market and business considerations. , Pay Transparency Salary range: USD $156,423.94/Yr. - USD $179,191.47/Yr.

Similar Jobs

More Jobs at Systems Planning and Analysis, Inc.

More Information Technology Jobs

Find similar Senior Data Engineer jobs: