In this role you will own the full lifecycle of ETL pipeline development - designing, building, testing, and deploying serverless, event-driven workflows that ingest and transform 30+ priority datasets (e.g., R1 annual reports, waybill, financial and cost-of-capital data) into a Medallion (Bronze-Silver-Gold) architecture. You will build the supporting data models, schemas, and validation frameworks, capture lineage, and produce the technical documentation that keeps every pipeline reliable, auditable, and scalable.
The ideal candidate brings deep AWS-native data-engineering expertise, a data- and quality-first mindset, and strong collaboration skills. You must be adaptive to changing priorities, comfortable in an Agile environment, and effective at partnering with database, governance, and analytics stakeholders to sequence and deliver datasets under SLA.
What You'll Be DoingAs the Data Engineer (ETL Developer) at Redan, you will:
- Design, build, test, and deploy ETL/ELT pipelines for priority datasets using AWS-native services (Glue, Lambda, Step Functions, Redshift, RDS PostgreSQL, S3) or approved AWS-hosted third-party tools
- Build serverless, event-driven data workflows within a Medallion (Bronze-Silver-Gold) Lakehouse architecture, with built-in error handling, retries, and notifications
- Develop supporting data models, schemas, validation frameworks, and database structures; enhance data models to support expanded analytics and reporting
- Implement scalable data validation and quality checks - schema enforcement, error handling, and lineage capture (e.g., Great Expectations)
- Produce complete technical documentation - ERDs, data dictionaries, pipeline specifications, lineage diagrams, and deployment packages
- Collaborate with the client to prioritize datasets, define sequencing, and identify phased deployment options
- Support thorough testing across development, staging, and production environments; ensure pipelines support automated retries, idempotent reprocessing, and scale to 2x baseline data volume
- Apply infrastructure-as-code (Terraform) for consistent, repeatable deployments, and integrate pipelines into CI/CD
- Ensure pipelines meet performance standards - 6397% monthly successful run rate, 6398% data-validation pass rate, and data freshness within defined SLA windows
- Support ATO maintenance, FISMA compliance, and protection of PII and CUI in coordination with the agency IT and security teams
Profile of SuccessRequired:- Minimum 5 years of data-engineering experience designing and maintaining AWS-based data pipelines (7+ preferred)
- Proven hands-on experience with AWS data services - Glue, Lambda, Step Functions, and S3 - building serverless, event-driven ETL/ELT workflows
- Experience implementing Medallion/Lakehouse designs with tiered refinement layers
- Experience administering or building on Amazon Redshift, RDS (PostgreSQL or SQL Server), or comparable managed data stores
- Proficiency in Python and SQL, plus PySpark, Pandas, or similar data-processing frameworks
- Experience with data-validation tools (e.g., Great Expectations), data catalogs (AWS Glue, Lake Formation), CI/CD, and infrastructure-as-code (Terraform)
- Experience designing relational and dimensional schemas, indexes, and partition strategies, with validation, error handling, and lineage capture
- Experience delivering under SLAs with documented quality controls and formal acceptance in a federal or regulated environment
- Familiarity with Agile/Scrum methodologies and ceremonies
- Excellent written and oral communication; strong troubleshooting and attention to detail
Preferred:
- AWS certification (e.g., Data Engineer - Associate, Developer - Associate, or Solutions Architect - Associate/Professional)
- Experience with public open-data platforms (CKAN, Huwise) and BI/visualization tools
- Experience supporting optional AI/RAG data initiatives (document ingestion, retrieval) aligned to federal AI guidance (OMB M-26-04)
- Familiarity with federal data-management expectations - auditability, traceability, metadata, and quality frameworks
- Prior federal experience; ability to ramp quickly on a regulatory-domain context
Conditions of Employment
- U.S. Citizenship is required
- Must obtain and maintain a favorably adjudicated Minimum Background Investigation (MBI, Moderate Risk) prior to accessing client systems, and maintain that clearance level throughout the period of performance
- Fully remote; work performed within the continental United States; no travel is required or anticipated
