3+ years of hands-on Databricks development experience
Production experience with PySpark, Apache Spark, Spark SQL, and Delta Lake
Strong proficiency in Python and SQL; knowledge of REST API data ingestion
Hands-on experience with AWS services like S3, IAM, Lambda, Glue, and Aurora
Experience with Terraform and CI/CD tools like Jenkins or GitLab CI
Familiarity with Agile/Scrum methodologies
Bachelor's degree in Computer Science, IT, Data Science, or related field
Active or previous High-Risk Public Trust clearance
Responsibilities
Develop and optimize scalable Spark applications in AWS Databricks
Design, build, and maintain ETL/ELT pipelines using Medallion Architecture
Administer Databricks workspaces, clusters, and access controls
Manage AWS services and provision infrastructure with Terraform
Optimize queries and troubleshoot large-scale data processing solutions
Integrate various data types from APIs and databases
Implement security and compliance controls in CI/CD pipelines
Collaborate with Agile/Scrum teams and provide technical leadership
Benefits
Opportunity to work on mission-critical federal projects
Engagement in a collaborative Agile/Scrum environment
Potential for professional development and certifications
Exposure to cutting-edge technologies in data engineering
Support for security and compliance in federal environments
Full Job Description
Job Title: Senior Data Engineer
Location: Washinton DC
Summary
The Senior Data Engineer / Databricks Specialist will support a federal project by developing, designing , and optimizing high-performance Spark applications and enterprise data infrastructure in AWS Databricks. The role focuses on scalable ETL/ELT pipelines using Python, PySpark, SQL, and Delta Lake; Medallion Architecture; and secure, reliable data solutions for mission-critical environments.
Responsibilities
Develop and optimize scalable Spark applications in AWS Databricks using Python, PySpark, and SQL.
Design, build, and maintain ETL/ELT pipelines using Medallion Architecture (Bronze, Silver, and Gold layers), ensuring data integrity, completeness, and lineage.
Administer Databricks workspaces, clusters, scheduled jobs, access controls, Unity Catalog, and notebooks.
Manage AWS services including S3, Lambda, IAM, Aurora, and Glue; provision infrastructure with Terraform.
Optimize queries and Spark jobs; troubleshoot and debug large-scale data processing solutions.
Integrate structured, semi-structured, and unstructured data from APIs and databases.
Integrate security and compliance controls into DevSecOps CI/CD pipelines using Jenkins, GitLab CI, or equivalent tools, supporting FISMA/FedRAMP requirements, ATO reassessments, and POA&M resolution.
Implement data quality checks, anomaly detection, validation rules, and automated lineage tracking.
Collaborate with cross-functional Agile/Scrum teams in multi-contractor federal environments; provide technical leadership and production on-call support.
Required Qualifications
At least 3 years of dedicated, hands-on Databricks development experience.
Production experience with PySpark, Apache Spark, Spark SQL, and Delta Lake.
Strong proficiency in Python and SQL and working knowledge of REST API-based data ingestion.
Strong hands-on experience with AWS S3, IAM, Lambda, Glue, and Aurora.
Production experience with Terraform infrastructure as code and CI/CD tools such as Jenkins, GitLab CI, or equivalent.
Experience working in Agile/Scrum delivery environments.
Bachelor's degree in Computer Science, Information Technology, Data Science, or a related engineering discipline.
Active or previous High-Risk Public Trust (Tier 4) clearance.
Preferred Qualifications
Hands-on experience with Unity Catalog, Delta Live Tables (DLT), and MLflow or Model Registry for AI/ML pipelines.
Experience supporting federal government contracts and FISMA/FedRAMP environments.
Preferred Certifications
Databricks Certified Data Engineer Associate
Databricks Certified Data Engineer Professional
Databricks Certified Machine Learning Associate
AWS Certified Solutions Architect - Associate or Professional
