Senior Data Engineer with extensive experience designing, building, and maintaining data platforms and pipelines for large-scale, enterprise applications in support of the Department of Homeland Security. Tools necessary for excelling in this role include SQL (PostgreSQL or MySQL preferred), Elasticsearch/OpenSearch, data modeling and transformation frameworks, and DevSecOps practices in the Cloud.
This role will focus heavily on modernizing legacy data architectures and re-architecting the application's data layer, shifting from Elasticsearch to PostgreSQL or MariaDB as the system of record for transactional data. You will play a critical role in evolving scoring models, intelligence modifiers, and capability gap logic while ensuring alignment and consistency between the new relational database and Elasticsearch.
Key Responsibilities:- Design and architect robust, scalable data models, schemas, indexes, and stored procedures in a relational database to support transactional workloads and complex business logic.
- Lead the migration of the system of record from Elasticsearch to a relational database, including data modeling, transformation, validation, and cutover strategies.
- Develop and maintain data pipelines and integration processes to keep the database and Elasticsearch aligned for search and analytics use cases.
- Implement and evolve scoring models, intelligence modifiers, and capability gap logic in the data layer to support mission-critical decision support.
- Optimize database performance, including indexing strategies, query tuning, and partitioning to support high-volume, low-latency operations.
- Define and enforce data quality, data validation, and data governance standards across legacy and modernized components.
- Collaborate with application engineers to integrate APIs and services with the underlying data models and stored procedures, ensuring seamless functionality and consistent data semantics.
- Provide technical guidance and solutions for legacy data system modernization while minimizing downtime and operational risk during migration.
- Identify and remediate data-related technical debt in older systems while transitioning to modern data engineering patterns, tools, and frameworks.
- Integrate and operate CI/CD pipelines used for building, testing, and deploying data services and schema changes in cloud environments.
- Ability to pass a Public Trust investigation.
- Must have US citizenship.
- Bachelor's degree in a related field.
- Minimum 15 years general work experience.
- 8+ years of experience as a Data Engineer or in a closely related role, with strong experience in legacy data system modernization.
- 5+ years of hands-on experience with relational databases including from-scratch schema design, index creation, stored procedures SQL development, performance tuning, and transaction management.
- 3+ years of experience working with Elasticsearch or OpenSearch, including index design, mappings, and data ingestion strategies.
- Demonstrated experience designing and implementing data models and logic to support scoring models, rules engines, or similar decision-support systems.
- 5+ years of experience supporting applications in production, troubleshooting complex data issues in legacy environments, and proposing effective remediation strategies.
- Strong understanding of data modeling for OLTP/transactional systems, including normalization/denormalization tradeoffs and referential integrity.
- Solid understanding of Git, version control workflows, and collaborative development practices.
