Splunk Observability Engineer

System One Holdings, LLC

$110K — $130K *
Information Technology
Less than 5 years of experience
Job Overview by Ladders

Qualifications

  • 5+ years of hands-on experience as a Splunk administrator in large production environments
  • Advanced proficiency in Search Processing Language (SPL)
  • Strong experience with AWS services like EC2, ALB, and CloudWatch
  • Practical knowledge of OpenTelemetry and its integration with Splunk
  • Proficient in Python for scripting and automation tasks
  • Familiarity with GitLab, GitHub, Terraform, and CI/CD pipelines
  • Excellent troubleshooting abilities and effective communication skills

Responsibilities

  • Support and upgrade Splunk Enterprise on AWS infrastructure
  • Manage the complete data lifecycle from ingestion to recovery
  • Write and optimize complex SPL queries for data analysis
  • Design dashboards for monitoring system performance and incident response
  • Troubleshoot and resolve ingestion failures and outages
  • Restore data continuity after service interruptions
  • Minimize data noise and optimize storage costs

Benefits

  • Opportunity to work with cutting-edge observability platforms
  • Work in a collaborative environment with a focus on team troubleshooting
  • Gain hands-on experience with AWS and distributed systems
  • Contributing to open-source initiatives through automation and integrations
  • Flexible on-call rotations and crisis management opportunities
Full Job Description
Splunk Observability Engineer :

Hands on role supporting large Splunk and observability platforms in an AWS production setting.

  • Run, upgrade and support Splunk Enterprise on AWS (EC2)
  • Manage the full data lifecycle: ingestion, indexing, search, dashboards, alerts, retention and recovery
  • Write and tune complex SPL queries
  • Build dashboards for monitoring and incident triage
  • Fix ingestion failures, missing data, volume spikes and outages
  • Restore and replay data after interruptions
  • Reduce noise, storage use and cost
  • Connect Splunk with CloudWatch and OpenTelemetry
  • Build and maintain OpenTelemetry agent and collector packages
  • Automate tasks with Python
  • Handle patching and security fixes
  • Lead troubleshooting calls across teams
  • Take part in on call rotation


Requirements

  • Strong Splunk admin experience in large production setups (forwarders, indexers, search heads)
  • Advanced SPL skills
  • Strong AWS experience: EC2, ALB, CloudWatch
  • Hands on OpenTelemetry experience, including Splunk integration
  • Python for automation
  • GitLab, GitHub, Terraform Enterprise, CI/CD, JSON
  • Strong troubleshooting and clear communication skills


For immediate consideration, please connect with me on LinkedIn at https://www.linkedin.com/in/dpotapenko and then email me your resume, work authorization status, current location, availability, and compensation expectations directly to [email protected] - make sure to include the exact job title and job location in your email message.

#M1

#LI-DP1

Ref: #404-IT Pittsburgh

Similar Jobs

More Jobs at System One Holdings, LLC

More Information Technology Jobs

Find similar Splunk Observability Engineer jobs: