Position OverviewThe Senior Data Platform Engineer serves as a hands-on technical lead for the Federal Client Datahub, an event-driven data platform built on the Confluent Platform for Apache Kafka and deployed on Kubernetes. The role owns the full portfolio of Java microservices - Kafka consumers, producers, and Kafka Streams applications, as well as non-Kafka services such as REST API web services - along with the deployment and ongoing maintenance of the supporting Confluent Platform and an Elasticsearch cluster. Beyond the data platform, this position serves as solution architect, technical subject matter expert, and senior technical decision maker for the wider FLAG system, and acts as a technical point of contact for DOL stakeholders and integrating federal partner teams.
Key Responsibilities- Kafka Streams application development - Design, build, and maintain stream processing applications including stateful topologies, foreign-key joins, and state stores.
- Microservice and integration development - Build and maintain Spring Boot services, Kafka consumer/producer applications, and REST API web services integrating with Oracle, MySQL, MongoDB, Elasticsearch, managed search services, and downstream partner APIs.
- Kubernetes workload engineering - Own the deployment topology for all platform workloads: StatefulSets with persistent state, liveness/readiness/startup probes, pod anti-affinity, PodDisruptionBudgets, graceful shutdown, and resilience to node lifecycle and consolidation events.
- Production operations and incident response - Serve as senior escalation for production issues; perform root-cause analysis across the stack (JVM, Kafka client, DNS/networking, database, container runtime) and produce written RCAs with corrective actions.
- Confluent Platform and Elasticsearch administration - Deploy, configure, upgrade, and tune the Confluent Platform components (brokers, Connect, Schema Registry, REST Proxy) and the Elasticsearch cluster; manage schema compatibility, connector configuration, consumer group offsets, index lifecycle and versioned index migrations, topic sizing, and storage expansion.
- Database and query engineering - Own data access and query performance across Oracle, MySQL, MongoDB, Elasticsearch, and Azure Search indexes - including PL/SQL and reporting views, document modeling and aggregation pipelines, index mappings and query DSL, and search index schema design; diagnose performance regressions and environment-specific data issues.
- Solution architecture and technical leadership - Provide architectural direction and technical decision-making for the wider FLAG system, including integration points with the Appian application and the public-facing web applications.
- Stakeholder engagement - Act as a technical point of contact for DOL government stakeholders and integrating federal partner teams; translate technical constraints into clear options and tradeoffs for non-engineering decision makers.
- Platform modernization - Lead framework and runtime upgrades across the codebase (JDK, Spring Boot, serialization libraries, client SDKs), including compatibility analysis, remediation, and phased rollout through environments.
- Security compliance and vulnerability remediation - Execute recurring container image scan cycles; remediate CVEs via Dockerfile changes and dependency overrides, and author false-positive and risk-acceptance justifications that meet federal POA&M documentation standards.
- Observability and monitoring - Build dashboards, log-based metrics, and alerting for service health and availability; instrument applications for meaningful operational signals.
- Technical communication and mentorship - Write design documents, migration plans, and recommendation memos; mentor other developers on the platform and its operational practices.
Technical Skills and CapabilitiesCore Languages and Frameworks- Java (required)
- Spring Boot, Spring Framework, Spring Data
- Maven multi-module builds
- SQL / PL-SQL
Streaming and Messaging- Apache Kafka (brokers, topics, partitions, consumer groups, offsets, KRaft)
- Kafka Streams (stateful processing, joins, state stores, application reset, rebalancing/threading model)
- Confluent Platform (Schema Registry, Kafka Connect, REST Proxy)
- Apache Avro and schema evolution/compatibility
- Kafka Connect connectors (JDBC sink/source, Elasticsearch sink)
Cloud and Container Platform- Kubernetes (Deployments, StatefulSets, PVCs, probes, PDBs, anti-affinity, HPA, ConfigMaps/Secrets)
- Docker and container image hardening
- Helm, kubectl
- AWS (EKS, EC2, S3, IAM, CloudWatch, Lambda, EventBridge, ECR)
- Azure exposure (desirable)
Data Stores and Search- Oracle Database (query tuning, stored procedures, views, execution plans)
- MySQL
- Elasticsearch (index design, aliases, versioned index migration, Java client, PIT pagination, cluster deployment and upgrades)
- MongoDB (CosmosDB Mongo API)
- Managed search services such as Azure AI Search
Operations, Security, and Tooling- Observability: CloudWatch Logs Insights, log-based metrics, dashboards, alerting
- Container vulnerability scanning and remediation
- Git, CI/CD pipelines, Linux development environments
- REST API design and OpenAPI/Swagger documentation
Does this opportunity sound like a fit for you? If so, join our talent community and click to apply now!!#NAI #DICE
