5+ years in SRE, DevOps, or cloud infrastructure engineering.
2+ years building production monitoring with observability platforms like Datadog or New Relic.
Hands-on experience with AWS ECS/Fargate, RDS, OpenSearch, ALB, and IAM.
Proficient in CloudWatch metrics, alarms, and dashboards.
Skilled in AWS CloudFormation and/or CDK.
Experience with SLOs and designing alerts based on user-facing symptoms.
Scripting skills in Python or similar language.
Responsibilities
Set up and maintain monitoring across various platforms including APM and error tracking.
Establish CloudWatch metrics, alarms, and dashboards for AWS services.
Enforce a consistent tagging scheme across AWS resources and monitors.
Deploy and configure monitoring agents in Fargate task definitions.
Connect AWS accounts to monitoring tools using least-privilege IAM roles.
Write and maintain infrastructure code in AWS CloudFormation/CDK.
Tune alerts to ensure they lead to actionable insights and reduce false alarms.
Benefits
Hybrid work model (1 day in office, 4 days remote).
Opportunity to work with cutting-edge technologies in cloud infrastructure.
Engagement in incident response and post-incident reviews for continuous improvement.
Collaboration with application teams to enhance service monitoring.
Potential for professional growth in a regulated environment.
Full Job Description
Overview We're looking for a Senior DevOps Engineer to own and carry out a layered monitoring and alerting strategy for containerized applications on AWS Fargate. Your goal is to catch real problems that affect users, while keeping alert noise near zero. Every alert must lead to action.
Responsibilities
Set up and maintain monitoring across synthetic tests, real-user monitoring (RUM), application performance monitoring (APM) and tracing, error tracking, and service-level objectives (SLOs).
Set up and maintain CloudWatch metrics, alarms, and dashboards for ECS/Fargate, RDS, OpenSearch, and Application Load Balancers (ALB).
Enforce a consistent tagging scheme across AWS resources, monitors, and SLOs, and keep data collection limited to what's needed.
Deploy and configure monitoring agents (including sidecar containers) in Fargate task definitions. This includes health checks for RDS (PostgreSQL/MySQL), OpenSearch, and self-managed databases on EC2/ECS.
Connect AWS accounts to monitoring tools using least-privilege, read-only IAM roles.
Write and maintain the infrastructure code for this work in AWS CloudFormation/CDK.
Keep alerting clean. Tune, demote, or remove any alert that doesn't lead to action, and use persistence and multi-location thresholds to prevent false alarms.
Build application dashboards and create standard alert message templates.
Work with application teams to add tracing to their services and to define SLOs.
Take part in incident response and post-incident reviews during business hours, using metrics, traces, and logs to find root causes.
Required qualifications
5+ years in SRE, DevOps, or cloud infrastructure engineering.
At least 2 years building production monitoring with a full observability platform, such as Datadog, New Relic, Dynatrace, Grafana, or Elastic Observability.
Hands-on experience with AWS ECS/Fargate, RDS, OpenSearch, ALB, and IAM.
Hands-on experience with CloudWatch metrics, alarms, and dashboards.
Skilled with AWS CloudFormation and/or CDK.
Practical experience with SLOs, error budgets, and designing alerts based on user-facing symptoms rather than underlying causes.
Working knowledge of PostgreSQL and/or MySQL, plus OpenSearch/Elasticsearch, from an operations standpoint.
Scripting skills in Python or a similar language.
Able to obtain and keep a Public Trust determination.
Possession of a bachelor's degree from an accredited college/university according to the Council for Higher Education Accreditation (CHEA) in related field.
Preferred qualifications
Experience with Datadog (synthetic tests, RUM, APM, monitors, SLOs).
Experience with Sumo Logic: writing queries, building dashboards, and setting up alerts.
Experience with CloudWatch Logs, including Logs Insights queries and metric filters.
Experience with OpenTelemetry or adding tracing to services in more than one language (Node, Python, Java, .NET, Go).
Experience in federal or other regulated environments.
AWS certification (SysOps Administrator, DevOps Engineer, or Solutions Architect).
Salary: $120,000-$160,000 based on experience.
Work Location: Hybrid (1 day in office, 4 days remote per week) in Rockville, MD 20850