Build and operate the ingestion, bronze, silver, and gold pipelines on Databricks, migrating logic and data out of legacy.
Responsibilities- Build and maintain Databricks pipelines (60% of the time)
- Gather and analyze requirements (20% of the time)
- Monitor and troubleshoot data jobs (10% of the time)
- Document processes and workflows (10% of the time)
- Perform additional assigned tasks
Requirements- US Citizenship
- Secret Clearance
- Excellent interpersonal skills and ability in overseeing/prioritizing work requests
- Excellent written and oral communication skills
- Strong data engineering, ETL/ELT development, or data warehousing
- Strong PySpark and/or Spark SQL skills; solid SQL fundamentals
- Solid understanding of the Medallion Architecture (Bronze/Silver/Gold layering)
- Proficiency with Delta Lake (ACID transactions, schema evolution, time travel, MERGE/Upsert patterns)
- Experience designing both batch and incremental/streaming ingestion pipelines
- Familiarity with the Agile process
Desired Skills- Databricks Certification/experience/familiarity preferred
- Knowledge of and experience with the DoD
- Familiar with Customer and/or DoD IT architectures
- Federal experience
- Experience in post go-live production support and break/fix
- Focus on customer service and responsiveness, with sound customer handling skills
- Experience working within AWS Gov Cloud
- Experience working migrating code from non-accredited to accredited environments within AWS
