Splunk at Splunk is also a strong partner with the Products and Engineering teams, where we’ve contributed to everything from Splunk-built addons (e.g. Zoom, Gmail, Falcon Data Replicator), to the health checks and alerts that ship with Splunk’s Distributed Management Console (DMC), to the feature roadmap for Data Stream Processor (DSP). If you’re looking for a team where you can make a big impact, this is it!
As a Principal Splunk Engineer, you will be responsible for all aspects of Splunk solutions that related to IT operations and security use cases. The innovations built by our team will be widely shared and you should be eager to drive these efforts and share your knowledge with the Splunk community.Responsibilities:
- Collaborate with the team to develop cutting-edge solutions for corporate data collection to Splunk Cloud. You will have the opportunity to work on projects that involve the latest technologies, such as serverless data collection using AWS Lambda and the development of microservices running on Kubernetes.
- Seize opportunities to automate Splunk administration and deployment workflows using your knowledge of DevOps tools (Ansible, Terraform, GitLab CI/CD pipelines), Python, REST APIs, and Splunk's product portfolio.
- Develop, test, and deploy a variety of Splunk configurations, Splunk technology add-ons, modular inputs, external lookups, and custom search commands using Python and Splunk SDKs.
- Implement automated testing, continuous integration, and continuous deployment to streamline how we operate Splunk and data collection services internally.
- Instrument infrastructure monitoring, application performance monitoring (APM), and tracing using Splunk Infrastructure Monitoring (SignalFx) combined with Ansible roles and Terraform providers for Splunk’s corporate IT services.
- Implement security best practices related to our team’s Linux-based infrastructure, Python code, and Docker services.
- 12+ years working with distributed systems at scale in IT (with some experience in service operations preferred)
- 5+ years of software development experience in Python (expert level)
- 5+ years of DevOps experience (Ansible, GitLab CI/CD, Terraform, Docker, Kubernetes)
- 5+ years performing Splunk administration, architecture, and development
- Strong knowledge of IT Security (encryption, firewalls, secure systems design, vulnerability management, secure software design, credential management, proxies, WAFs, etc.)
- Deep understanding of IT fundamentals, including Linux and Microsoft operating systems, networking, security, cloud (AWS), and database technologies (MySQL, Postgres)
- Excellent communication skills, both verbal and written; able to explain complex technical topics to both internal and external customers with ease
- A solid technical understanding of Splunk products
- Fully proficient in git and version control systems (GitLab, BitBucket, GitHub)
- Passionate about IT, technology, and following the latest industry trends
Preferred knowledge and experience: These are a huge plus.
- Experience with Docker, Kubernetes, and AWS is highly preferred
- Experience building CI/CD pipelines
- Experience administering Linux systems
- Experience integrating systems with Hashicorp Vault
- Experience with load balancing technologies
- Experience developing content in Splunk ITSI a plus but not required
- Certifications in Splunk, AWS, and/or Docker
- Bachelor’s degree or higher in Computer Science, Information Technology, Computer Information Systems, Information Security, or a closely related field
What We Offer You:
- An opportunity to innovate cutting-edge solutions using Splunk software for Splunk itself. Plus, the ability to share those innovations with our customers! How cool is that?
- A rare opportunity to join one of the best Splunk engineering teams in the world with a stable, collaborative, and supportive work environment that encourages work-life balance.
- A constant stream of new things for you to learn. We're always expanding into new areas, bringing in open source projects and contributing back, and exploring new technologies.
- A set of extraordinarily hardworking, innovative, open, fun and dedicated peers, all the way from engineering and QA to product management and customer support.
- Growth and mentorship. We believe in growing engineers through ownership and leadership opportunities. We also believe mentors help both sides of the equation.