Job SummarySHI's IDCS organization specializes in the factory integration and deployment of enterprise data center infrastructure, including servers, networking equipment, storage, and complete rack-level solutions. Our in-house automation platform deploys, configures, and validates racks across multiple customers, vendors, and cold rooms in parallel.
We are seeking a Software Engineer to build and maintain the applications that sit on top of that platform, including the operator web application, pipeline and orchestration engine, inventory and addressing services, and CLI tooling. This role will turn deployment and QC workflows that engineers run manually into reliable, observable software that integration technicians and engineers can run independently.
This role is well suited for an engineer who enjoys building real products for internal operators and is comfortable working alongside hardware, networks, and a factory floor. The Software Engineer will collaborate with systems engineers, network engineers, integration and QC teams, and OEM technologies, while also mentoring junior engineers.
Role DescriptionApplication and Platform Development- Design, build, and maintain the operator-facing web application, including rack views, pipelines, dashboards, work orders, and approvals.
- Develop backend services and REST APIs in Python using FastAPI or similar technologies to expose inventory, device actions, and run status.
- Build and extend the pipeline/DAG engine, including stage ordering, retries, parallel fan-out across devices, run history, and resume-from-failure behavior.
- Build real-time features such as live run output, device status, and rack health using SSE or WebSockets.
- Develop and maintain CLI tools and a command router that dispatches actions by customer, device type, and OEM.
- Create reusable libraries and frameworks rather than one-off scripts, with clear module boundaries and per-page front-end separation.
Data and Inventory- Own the schema and data access layer for the inventory database, including racks, devices, addressing, credentials, and run results.
- Maintain the inventory as the single source of truth for addressing and device identity, ensuring consistency across the tools that read and write to it.
- Build reporting that produces QC coversheets, command logs, and audit evidence for each run.
Device and Vendor Integration- Integrate with vendor and infrastructure APIs, including Redfish, DHCP server APIs, and device CLIs over SSH.
- Support heterogeneous hardware, including Dell, HPE, Lenovo, and Supermicro servers; Cisco, Arista, and Juniper switches; PDUs; and console servers through a common adapter model.
- Build validation, timeout, and error-handling logic to ensure hung or unreachable devices cannot stall the platform.
Quality, Security, and Operations- Build automated checks that scan racks and servers for health issues and open work orders.
- Implement role-based access, audit logging, and secure credential handling, ensuring plaintext secrets are not stored in code, logs, or API responses.
- Add approval gates to ensure device-changing commands do not run without explicit operator approval.
- Write unit, integration, and end-to-end tests, using Git branching and code review to safely deploy to production.
- Deploy and support services on RHEL using gunicorn/uvicorn behind nginx, systemd, and log analysis.
- Write documentation, runbooks, and SOPs.
- Mentor junior engineers and interns.
Behaviors and Competencies- Communication: Can effectively communicate complex ideas and information, and can adapt communication style to the audience.
- Listening: Can recognize underlying messages and non-verbal cues in communication, responding appropriately to both explicit and implicit information.
- Presenting: Can prepare and deliver presentations, addressing key points and responding to questions with clarity.
- Adaptability: Can adjust to changes in the work environment, manage multiple tasks, and effectively handle uncertainty.
- Problem-Solving: Can identify problems, propose solutions, and take action to resolve them without explicit instructions.
- Initiative: Can identify opportunities for improvement, propose solutions, and take action without explicit instructions.
- Prioritization: Can identify critical tasks, arrange them based on importance, and execute without explicit instructions.
- Time Management: Can generally use time effectively and is working towards improving task prioritization and deadline management.
- Technical Expertise: Can apply technical knowledge and skills effectively in most situations, with occasional guidance.
- Teamwork: Can work effectively in a team, contributing ideas and effort, and respecting the contributions of others.
Skill Level Requirements- Minimum of 5 years of professional software engineering experience.
- 3+ years of hands-on Python development building production applications, services, or tooling, rather than primarily scripting.
- Experience designing, building, and supporting REST APIs and web applications in production.
- Experience designing and maintaining relational database schemas and the associated data access layer.
- Experience deploying and supporting services on Linux in production.
- Experience integrating with external systems through APIs and device interfaces, including REST, SSH/CLI, or similar technologies.
- Experience writing automated tests and shipping through Git-based code review.
Software Development- Strong Python skills, including typed, tested, production-quality code.
- REST API design using FastAPI, Flask, Django, or similar technologies.
- JavaScript, HTML, and CSS, with experience developing modern front ends without heavy framework dependence.
- SQL and relational databases, including MariaDB/MySQL.
- JSON/YAML.
- Git, including branching, pull requests, and code review.
- Asynchronous and concurrent programming, including threads, async I/O, and parallel fan-out.
Linux and Services- Linux administration, with RHEL preferred.
- Bash/shell scripting.
- SSH, systemd services, and package management.
- Web server and reverse proxy fundamentals, including nginx and gunicorn/uvicorn.
- Log analysis and troubleshooting.
Networking- TCP/IP, VLANs, and subnetting.
- DHCP/DNS.
- Network troubleshooting.
Infrastructure- Understanding of enterprise server hardware, BIOS/UEFI, and firmware.
- Out-of-band management technologies, including BMC, IPMI, and Redfish.
- PXE/network boot concepts.
Engineering Practices- Secure coding and secrets handling.
- Experience mentoring junior engineers.
Preferred Requirements- Experience with Ansible, including playbooks, roles, and inventories.
- Experience with Paramiko, Netmiko, or similar network device automation libraries.
- Experience with Kea DHCP, including control agent APIs, reservations, and leases, as well as iPXE.
- Experience with Kubernetes and/or Docker.
- Experience with Prometheus and Grafana.
- Experience with real-time visualization technologies such as Canvas, WebGL, or three.js.
- Experience integrating AI/LLM tooling with human approval controls.
- Experience working with Dell, HPE, Lenovo, Supermicro, Cisco, Arista, Juniper, Raritan, Vertiv, or Avocent/OpenGear devices.
- Manufacturing or factory-integration experi
- ence.
Other Requirements- Ability to work effectively alongside hardware, networking, integration, and QC teams in a factory integration environment.
- Ability to create reliable, reusable software and frameworks rather than one-off scripts.
- Ability to work across heterogeneous server, networking, power, and console-server technologies.
- Ability to develop and support secure, observable production applications and services.
- Ability to create clear technical documentation, runbooks, and SOPs.
- Demonstrated ability to mentor junior engineers and interns.
The estimated annual pay range for this position is $140,000 - $180,000 which includes a base salary. The compensation for this position is dependent on job-related knowledge, skills, experience, and market location and, therefore, will vary from individual to individual. Benefits may include, but are not limited to, medical, vision, dental, 401K, and flexible spending.
