This position requires an ACTIVE SECRET security clearance. Your application will not be reviewed if you do not have an ACTIVE security clearance at SECRET or above.
We are seeking a highly motivated and intellectually curious Databricks Platform Engineer/Admin to join our team working with a Federal client. The position will be a remote role open to US citizens residing in the United States with a Secret security clearance.
Responsibilities and Duties - Design, build, and maintain batch and streaming pipelines using PySpark, SQL, Databricks Workflows/DLT
- Implement Medallion architecture (bronze/silver/gold) for data curation and quality
- Build ingestion frameworks for structured/semi-structured/unstructured data from files, databases, APIs, and streaming sources (Kafka, Kinesis, Auto Loader)
- Design data models (dimensional or domain-specific) supporting downstream analytics
- Build and consume APIs for data integration with downstream systems
- Optimize Spark jobs for performance and cost (partitioning, caching, cluster sizing)
- Implement data quality checks and pipeline monitoring
- Document data flows and lineage for maintainability
Required Experience, Skills and Qualifications - 5+ years data engineering experience, 2+ years on Databricks
- Strong PySpark and SQL for production-grade transformations
- Experience with both batch and streaming data processing
- Solid grounding in data modeling and data architecture principles
- Python proficiency and Git-based CI/CD workflow experience
- Familiarity with Unity Catalog (catalogs, schemas, grants) from a consumer perspective
- Experience with DLT or Workflows orchestration a plus
Education Requirement - 4 year Bachelor's Degree in Computer Science/Eng or related (highly preferred)
Benefits
- Health Insurance
- 401k match
- Unlimited Vacation
- 11 Federal Holidays
- Remote work
- Training
- Mentoring/Coaching
