Description: The Mid-Level Data Engineer will support data migration, pipeline engineering, and modernization efforts for enterprise data lakehouse architectures. This role involves converting legacy Informatica artifacts into clean Python/PySpark code, migrating database schemas, and building automated data reconciliation pipelines. Working closely with senior engineering leadership and database managers, the ideal candidate will enforce high standards of data quality, data validation, and version control in a secure federal environment.
Mid-Level Data Engineer Job Duties:- Execute daily data migration operations including data profiling, schema mapping, pipeline conversion, and automated reconciliation for Low and Medium complexity Informatica artifacts.
- Convert Informatica mappings into well-documented Python/PySpark code, ensuring all business logic and data quality controls are preserved.
- Migrate legacy Hive tables to Delta Lake format on S3 using Databricks ingestion tools.
- Build and execute automated data reconciliation scripts to validate migration accuracy and establish Change Data Capture (CDC) pipelines for ongoing synchronization.
- Commit all converted code into Azure DevOps with clear documentation and inline comments while maintaining Unity Catalog configurations.
- Perform daily data profiling and side-by-side validation within legacy enclave environments.
- Support Power BI and ESRI integration testing and validation.
- Participate actively in peer code reviews, daily Agile ceremonies, and collaborative data validation sessions.
- Contribute to Data Quality Assessment Reports and support training and knowledge transfer activities.
- Performs other duties as assigned.
Security Clearance Requirements:- Public Trust / Tier 4 Eligible: No clearance required to apply; must be a U.S. citizen willing to undergo a background check to obtain a Public Trust / Tier 4 clearance.
Experience Requirements:- Minimum experience required varies by degree level: PhD with 0 years; Master's degree with 3 years; Bachelor's degree with 5 years; or High School Diploma with 9 years of relevant experience.
- Proficiency in Python for data transformation and pipeline development, as well as SQL for query development and schema analysis.
- Experience with ETL/ELT processes, data migration methodologies, and cloud data platforms (AWS, Azure, or GCP).
- Familiarity with version control systems (Azure DevOps, Git) and data quality concepts including profiling, cleansing, and reconciliation.
Education Requirements: Bachelor's degree in Computer Science, Data Engineering, Information Technology, or a related field is preferred (or equivalent combination of education and experience).
Desired Skills / Qualifications:- Experience with Databricks (notebooks, jobs, workspace navigation), PySpark, Apache Spark, and Delta Lake or Apache Iceberg table formats.
- Proven track record converting visual ETL tools (Informatica, Talend, SSIS) to code-based pipelines.
- Experience with Hive, HiveQL, or Hadoop ecosystem components.
- Familiarity with federal IT environments, security requirements, and CI/CD pipelines for data engineering workflows.
Location: Washington, DC
Status: Full time
Schedule: Day shift, Monday-Friday
Physical Requirements: Must be able to remain in a stationary position for long durations of time. Also, must be able to continuously operate a computer and other office productivity machinery.
Travel Required: No
This job description is subject to change at any time.
