3+ years of hands-on data engineering experience on Databricks
10+ years of overall technical experience
Strong proficiency in Apache Spark (PySpark and Spark SQL)
Experience with Databricks on Azure and/or AWS
Familiarity with Unity Catalog for data governance
Experience integrating structured and unstructured data sources
Knowledge of SQL and cloud-based storage patterns
Secret clearance required
Responsibilities
Design and maintain scalable data pipelines and ETL/ELT workflows on Databricks
Develop and orchestrate workflows using Databricks Workflows/Jobs
Implement data quality checks and metadata management
Collaborate to ensure datasets are discoverable through Unity Catalog
Automate ingestion and transformation processes using Databricks-native patterns
Support API strategy and domain-driven data structures
Troubleshoot pipeline issues and optimize Spark jobs for performance
Benefits
Remote work with onsite requirements at the client site
Opportunity to work with cutting-edge AI and analytics technologies
Collaboration with cross-functional teams
Engagement in a mission-driven environment
Potential for professional development and certifications
Full Job Description
Senior Databricks Data Engineer
Overview
N2IA is seeking a Senior Databricks Data Engineer to build, refine, and maintain the core data pipelines on the Databricks Lakehouse Platform that enable enterprise AI, analytics, and mission systems. This role is essential for making data accessible, reliable, and usable across the agency. You will work with cloud engineers, data architects, and AI developers to ensure data is clean, connected, secure, and optimized for advanced analytics and AI/ML workflows. This role is remote supporting a Quantico, VA customer and onsite work is required at the client site full-time.
Key Responsibilities:
Design, build, and maintain scalable data pipelines and ETL/ELT workflows on Databricks using Delta Lake, Delta Live Tables (DLT), and the medallion (bronze/silver/gold) architecture across cloud and on-prem systems.
Develop and orchestrate workflows using Databricks Workflows/Jobs.
Implement data quality checks, validation, lineage tracking, and metadata management using Unity Catalog and Lakehouse Monitoring / DLT expectations.
Collaborate with other teams to ensure datasets meet requirements and are discoverable through Unity Catalog.
Automate ingestion and transformation processes using Databricks-native patterns such as Auto Loader, Structured Streaming, and the COPY INTO command.
Support API strategy, data modeling, and domain-driven data structures within the Lakehouse.
Troubleshoot pipeline issues and optimize Spark jobs and clusters for performance and cost (e.g., Photon, cluster sizing, Z-ordering, liquid clustering, and workload tuning).
Required Qualifications:
3+ years of hands-on data engineering experience building production pipelines on Databricks.
10+ years of technical professional experience.
Strong proficiency with Apache Spark (PySpark and Spark SQL), Delta Lake, and the Databricks Lakehouse Platform.
Hands-on experience with Databricks on Azure and/or AWS (e.g., Azure Databricks, Azure Data Factory, AWS Glue, or similar orchestration tooling).
Experience with Unity Catalog for data governance, access control, and lineage.
Experience integrating structured and unstructured data sources.
Knowledge of SQL, schema design, data modeling, and cloud-based storage patterns (e.g., Delta tables, cloud object storage).
Secret clearance.
Preferred Qualifications:
Databricks certification (e.g., Databricks Certified Data Engineer Associate/Professional).
Experience supporting AI/ML pipelines, vectorization, or embedding generation, ideally using MLflow, Databricks Mosaic AI, Model Serving, or Vector Search.
Experience with Delta Live Tables and streaming pipelines (Structured Streaming / Auto Loader).
Azure, AWS, or multi-cloud data ecosystem familiarity.
Experience with data governance, access controls, or sensitive data protection (Unity Catalog, row/column-level security).
Exposure to DevOps/CI-CD patterns for data workflows (e.g., Databricks Asset Bundles, Repos, Terraform).