OverviewDecisionPoint seeks a Performance & Resilience Test Engineer to evaluate the scalability, stability, responsiveness, failover behavior, and recovery capabilities of mission-critical software supporting the USMEPCOM ENLIST program. This role develops regression, API, integration, load, stress, endurance, and resilience tests; creates realistic workload models; and produces reproducible evidence of system behavior under normal, peak, degraded, and recovery conditions.
The Performance & Resilience Test Engineer works with architects, software developers, API and integration engineers, infrastructure and cloud teams, database specialists, cybersecurity personnel, CI/CD engineers, operations teams, and Government stakeholders. The position identifies performance bottlenecks, validates failover and recovery procedures, and provides evidence-based recommendations supporting release readiness, capacity planning, and operational resilience.
This position is remote, with Chicago-area candidates preferred.
Note: By applying to this position, you acknowledge and consent to having your resume included in an active competitive government contract bid.
Duties & ResponsibilitiesThe Performance & Resilience Test Engineer will:
- Develop and maintain performance, load, stress, spike, endurance, volume, scalability, failover, and recovery tests.
- Build automated regression, API, and integration tests supporting repeatable performance and resilience validation.
- Analyze mission workflows, usage patterns, transaction volumes, concurrency, data characteristics, interfaces, and operational objectives.
- Develop workload models that represent normal operations, peak demand, surge conditions, degraded environments, and recovery scenarios.
- Define test objectives, performance baselines, service thresholds, acceptance criteria, and measurable expected results.
- Develop test plans, scenarios, scripts, data, configurations, and execution schedules.
- Prepare representative test data for functional, high-volume, boundary, exception, failover, and recovery scenarios.
- Maintain repeatable processes for test-data creation, setup, refresh, reset, reconciliation, and cleanup.
- Execute tests in development, integration, test, staging, performance, or other Government-approved environments.
- Automate test execution through approved CI/CD pipelines where appropriate.
- Produce reproducible test results, logs, reports, metrics, graphs, and objective evidence.
- Measure and analyze response time, throughput, latency, concurrency, error rates, resource utilization, saturation, queue depth, and transaction completion.
- Establish performance baselines and compare results across builds, releases, configurations, environments, and infrastructure changes.
- Identify application, database, interface, network, cloud, infrastructure, configuration, or external-service bottlenecks.
- Correlate test results with application logs, infrastructure metrics, traces, events, and monitoring data.
- Work with technical teams to isolate root causes and recommend appropriate corrective actions.
- Validate failover behavior for applications, services, interfaces, databases, infrastructure components, and dependent capabilities.
- Test approved recovery procedures and confirm that services return to an acceptable operational state.
- Validate recovery sequencing, data consistency, transaction handling, dependency restoration, and post-recovery functionality.
- Evaluate behavior during component failures, network interruptions, service degradation, capacity constraints, and dependency outages.
- Verify that systems fail safely and provide appropriate errors, alerts, retries, timeouts, and recovery behavior.
- Support disaster-recovery, continuity, restoration, and operational-readiness exercises where applicable.
- Document performance defects, resilience gaps, capacity risks, recovery failures, and release impacts.
- Support defect triage, root-cause analysis, correction verification, and targeted regression testing.
- Coordinate with architects and engineering teams to evaluate scalability, capacity, extensibility, and recovery approaches.
- Coordinate with CI/CD engineers to integrate repeatable performance checks, thresholds, reports, and quality gates.
- Coordinate with operations teams to incorporate production observations and operational feedback into future testing.
- Support Government demonstrations, user acceptance testing, release testing, and operational validation.
- Provide performance and resilience results, residual risks, and release-readiness recommendations to technical and program stakeholders.
- Maintain test source code, scripts, configurations, data, and documentation in approved version-control repositories.
- Participate in sprint planning, backlog refinement, reviews, demonstrations, retrospectives, and release-readiness activities.
- Contribute to continuous improvement of performance engineering, workload modeling, observability, failover testing, and recovery validation.
QualificationsRequired QualificationsCandidates must possess all the following:
- An active Secret clearance or higher.
- A bachelor's degree in computer science, information technology, engineering, or a related field, or an equivalent permitted by the applicable labor category.
- At least six years of relevant professional experience.
- Experience designing and executing load and performance tests.
- Experience developing realistic workload models based on user behavior, transaction volumes, concurrency, or operational demand.
- Experience analyzing performance results and identifying application, integration, database, infrastructure, or network bottlenecks.
- Experience planning or executing failover tests.
- Experience validating system, application, service, or data recovery.
- Ability to build or maintain regression, API, integration, and automated test coverage.
- Ability to prepare and maintain representative test data.
- Ability to produce repeatable automated execution and reproducible test results.
- Ability to define measurable performance baselines, thresholds, acceptance criteria, and expected results.
- Ability to correlate test results with logs, metrics, traces, events, or other observability data.
- Ability to document performance defects, resilience gaps, capacity risks, recovery failures, and release impacts.
- CompTIA Security+ CE or another Government-approved certification satisfying the required IAT Level II qualification and applicable current DoD 8140 work-role requirements, when required for the assigned duties.
- Strong analytical, automation, troubleshooting, technical documentation, and communication skills.
- Ability to collaborate effectively with architects, developers, integration engineers, database specialists, infrastructure teams, cybersecurity personnel, CI/CD engineers, operations teams, and Government stakeholders.
Preferred Qualifications The following qualifications are preferred:
- A recognized performance-testing, software-testing, cloud, or site-reliability certification.
- Experience supporting federal or Department of Defense software-development, modernization, testing, or continuity programs.
- Experience testing mission-critical, regulated, security-sensitive, high-volume, or high-availability systems.
- Experience with performance-testing tools such as LoadRunner, JMeter, Gatling, k6, NeoLoad, Locust, or comparable platforms.
- Experience developing test automation using Java, C#, Python, JavaScript, TypeScript, or another supported programming language.
- Experience testing APIs, web services, event-driven systems, message brokers, databases, batch interfaces, or cloud-native services.
- Experience with observability and monitoring tools such as Splunk, Dynatrace, AppDynamics, New Relic, CloudWatch, Azure Monitor, Grafana, Prometheus, or comparable platforms.
- Experience with AWS, Azure, containers, Kubernetes, serverless services, or hybrid cloud environments.
- Experience with CI/CD platforms such as Jenkins, GitHub Actions, GitLab CI/CD, Azure DevOps, AWS CodePipeline, or comparable solutions.
- Knowledge of site reliability engineering concepts, including service-level indicators, service-level objectives, error budgets, saturation, and capacity planning.
- Experience with disaster-recovery testing, continuity exercises, recovery-time objectives, or recovery-point objectives.
- Experience with resilience testing, controlled fault injection, or approved chaos-engineering practices.
- Experience using Jira, Azure DevOps, TestRail, Zephyr, Xray, Confluence, or comparable testing and collaboration tools.
- Familiarity with Agile, DevSecOps, secure software-development, configuration-management, and release-management practices.
