Job Description*AWAITING FINAL AWARD*We are looking for a
Data Engineer to build and scale modern, high-throughput data infrastructure. In this role, you will engineer automated big-data extraction and ingestion pipelines using PySpark and Databricks Unity Catalog, support delivery in Enterprise Platform Data Observability, and solve complex enterprise analytic challenges. You will be building "enterprise capabilities" within the Department of War's authoritative data analytics platform (War Data Platform) for the Chief Digital and AI Officer (DoW CDAO).
Key Responsibilities:- Pipeline Engineering: Develop and operate end-to-end, automated data ingestion pipelines using PySpark and Databricks.
- Data Architecture: Maintain and optimize data structures using Delta Tables to enable seamless data versioning, backups, and rollback safeguards.
- Architectural Integration: Coordinate with Platform and Solution Architects to connect Delta Tables with Vector Databases while strictly enforcing Role-Based Access Controls (RBAC).
The annual base salary range for this role is $110,000-$160,000 (USD) , which does not include discretionary bonus compensation or our comprehensive benefits package. Actual compensation offered to the successful candidate may vary from posted hiring range based upon geographic location, work experience, education, and/or skill level, among other things.
Required Skills- Big Data & Pipelines: Expertise in Python, PySpark, and Databricks for building and operating end-to-end automated data ingestion pipelines.
- Data Architecture & Management: Hands-on experience with Delta Lake Architecture, Delta Tables, and Unity Catalog to manage versioning, backups, and rollbacks.
- Automation & Querying: Strong proficiency in SQL and ETL/ELT automation practices.
- Access & Integration: Familiarity with Role-Based Access Controls (RBAC) and connecting data structures to Vector Databases.
- Clearance: Active Top Secret, SCI Eligible
- Experience:0-5+ years of progressive professional experience in data engineering, software engineering, or a related technical discipline (Levels I and II).
- Education: Bachelor's Degree in Computer Science, Data Science, Information Technology, or a related field.
- Work Location: Arlington County, VA (Hybrid)
Desired Skills- Collaborative spirit to work seamlessly alongside Platform and Solution Architects.
- Strong problem-solving skills with a focus on scalable, high-throughput solutions.
- Passion for delivering innovative, high-impact data reliability and observability tools.
