Job SummaryWe are seeking an experienced Senior Production Support Analyst to support a multi-year observability and monitoring modernization initiative. This role requires deep expertise in the Grafana ecosystem, enterprise monitoring solutions, and cloud-native observability practices. The successful candidate will help assess the current monitoring landscape, implement and integrate observability solutions, standardize monitoring practices, and support the modernization of enterprise monitoring capabilities.
Key Responsibilities- Assess existing monitoring and observability tools to identify functional overlap, coverage gaps, consolidation opportunities, and provide recommendations.
- Design, implement, and support the Grafana Stack, including Grafana, Loki, Tempo, Mimir/Prometheus, and Alloy.
- Develop monitoring architectures covering metrics, logs, traces, high availability, capacity planning, and retention strategies.
- Integrate Grafana with enterprise monitoring and incident management platforms such as Dynatrace, Splunk, xMatters, and identity/SSO providers.
- Develop and standardize dashboards, alerting frameworks, SLIs, SLOs, and on-call notification processes.
- Support application and infrastructure instrumentation using OpenTelemetry, exporters, and monitoring agents.
- Assist with migration of dashboards, alerts, and monitoring workloads from legacy platforms to Grafana.
- Automate deployment, configuration, and maintenance activities using Infrastructure-as-Code and CI/CD tools.
- Deploy and manage observability solutions within Kubernetes, Red Hat OpenShift, and Linux environments.
- Implement security best practices, including RBAC, secrets management, audit logging, and data retention policies.
- Create runbooks, architecture documentation, training materials, and facilitate knowledge transfer to internal support teams.
- Collaborate with application, infrastructure, and operations teams to ensure monitoring standards and operational excellence.
Required Qualifications- Bachelor's Degree in Computer Science, Information Systems, or a related field.
- Equivalent combination of education and professional experience may be considered.
- Minimum 7 years of relevant IT experience, including 3+ years of hands-on production experience with the Grafana Stack.
- Proven experience implementing and administering enterprise-scale observability and monitoring solutions.
- Strong expertise with Grafana, Loki, Tempo, Prometheus, and/or Mimir.
- Experience with PromQL, LogQL, and Grafana alerting capabilities.
- Hands-on experience with OpenTelemetry instrumentation and collectors.
- Experience deploying observability solutions on Kubernetes and Red Hat OpenShift environments.
- Proficiency in automation and scripting using tools such as Bash and Python.
- Experience with Infrastructure-as-Code tools including Terraform and Ansible.
- Strong understanding of CI/CD pipelines and Git-based development workflows.
- Experience with enterprise monitoring tools such as Dynatrace, Splunk, or similar platforms.
- Linux system administration experience, preferably with RHEL.
- Knowledge of networking, storage, security principles, and telemetry data pipelines.
- Excellent analytical, troubleshooting, written, and verbal communication skills.
Preferred Qualifications- Experience integrating monitoring platforms with identity and access management solutions such as Okta, OIDC, or SAML.
- Experience leading observability platform migrations and tool rationalization initiatives.
- Familiarity with large-scale enterprise monitoring and cloud-native environments.
- Experience developing observability standards, best practices, and governance models.
Certifications- Grafana Certified Professional (Preferred)
- Red Hat OpenShift Certification (Preferred)
- Kubernetes Certifications such as CKA or CKAD (Preferred)
- Terraform Associate Certification (Preferred)
- Relevant Cloud Platform Certifications (AWS, Azure, or GCP) (Preferred)
