Principal Network Developer - Regional Network Reliability

Job Description

Principal Network Developer - Regional Network Reliability

Location: Nashville, TN

Network Design, Development, and Validation:

• NW engineering background - experience in designing and configuring CLOS NW for cloud services
• Lead firmware upgrades, configuration refreshes, feature rollouts, and controlled network changes in partnership with operations and deployment teams.
• Deep experience designing large-scale leaf-spine and multi-fabric data center networks aka CLOS architecture.
• Hands-on expertise with Ethernet switching, BGP, EVPN/VXLAN, IP addressing, routing scale, ECMP, QoS, and network segmentation.
• Strong knowledge of management, OOB, console-server, and production data-center network architectures.
• Experience with network operating systems and hardware platforms such as Arista EOS, Cisco Nexus/NX-OS, NVIDIA Spectrum/Cumulus, or comparable platforms.
• Experience designing or operating AI/HPC/GPU cluster networking, including RDMA/RoCE, lossless Ethernet, and high-bandwidth fabrics.
• Experience leading firmware lifecycle management, configuration standardization, vulnerability remediation, and network change governance.
• Participates in solution architecture discussions, providing technical guidance on network requirements, tradeoffs, and design decisions.
• Identifies and assesses complex risks in network design and recommends mitigation strategies before deployment.
• Collaborates with vendors and internal stakeholders to align on hardware, firmware, software, and cloud network code requirements.
• Designs playbooks for resolving common and uncommon network issues involving moderately complex systems.
• Analyzes network workflows to identify inefficiencies and proposes scalable solutions to improve performance.

Automation and Scripting:

• Experience with network automation using Python, Ansible, Terraform, APIs, Git, and CI/CD.
• Experience with capacity planning, hardware-resource/TCAM analysis, and large-scale routing-table design.
• Can partner with NW automation teams to design and develop NW modeling and configuration rendering solutions, to be able to review configuration diffs and mitigate configuration issues on NW switches.
• Designs and builds advanced modules within automation frameworks to support testing, operations, and service reliability across multiple environments.
• Automates high-impact network tasks for production and lab environments, and optimizes repetitive, cross-team workflows to improve operational efficiency.
• Develops and maintains advanced dashboards, telemetry tools, and alerting systems to enable proactive monitoring and faster issue resolution.
• Writes, enhances, and documents reusable scripts that streamline routine network operations across teams and product domains.
• Modifies and extends infrastructure pipelines and configuration management tools to meet evolving requirements using existing playbooks and custom logic.
• Independently configures and extends tools to support new products or services, ensuring alignment with platform standards and integration best practices.

Testing and Quality Assurance:

• Designs and develops advanced test strategies and reusable test cases that validate complex network systems and enhance long-term network integrity and reliability, including testing network features in lab environment and rolling it out in production.
• Leads team-level test practices and cross-functional collaboration, helping define robust testing protocols and ensuring consistency in execution across environments.
• Implements and refines post-incident validation methods, integrating break-fix outcomes and lessons learned into continuous test improvements.
• Reviews and approves L1 and L2 network changes, and presents high-impact changes to cross-team change management boards, ensuring risks are understood and mitigated.
• Partners with cross-functional teams to drive pre-production validation efforts, ensuring systems and environments meet compliance and deployment-readiness criteria.
• Prepares and delivers audit documentation and test evidence to support internal Governance, Risk, and Compliance (GRC) processes, maintaining alignment with security and policy requirements.

Monitoring and Reliability:

• Works with others (e.g. monitoring teams) to customize dashboards, telemetry pipelines, and alerting systems, defining service level objectives (SLOs) based thresholds and metrics to monitor network health.
• Partners with monitoring and Site Reliability Engineering (SRE) teams to refine alerting systems and enhance early detection of network anomalies and degradation.
• Leads incident response during support rotations, driving root cause analysis and coordinating cross-team resolution for escalated issues.
• Builds and improves internal tools that enable frontline support teams to efficiently respond to network failures and recurring operational issues.

Cross-Team Collaboration and Leadership:

• Drives test and deployment milestones in collaboration with project/program managers, adapting plans to address risks and ensure cross-team alignment.
• Mentors junior engineers and acts as a technical SME, providing hands-on guidance in resolving complex issues.
• Leads customer engagements on technical escalations and delivers clear, actionable root cause analysis (RCA) documentation.
• Influences team-level roadmap and architecture decisions, contributing solutions that align with broader engineering goals.
• Coordinates with vendors and internal teams to resolve standards misalignment, ensuring integration meets technical and operational expectations.

Performance and Capacity Management:

• Leads analysis of network performance metrics (e.g., latency, throughput, packet loss) to identify systemic inefficiencies and drive scalable improvements.
• Forecasts infrastructure needs using performance and capacity data, ensuring systems are prepared for anticipated traffic and service growth.

Planning & Execution:

• Manages and coordinates moderately complex tasks, monitoring timelines and deliverables to ensure timely completion and adherence to requirements for a moderately-sized project or initiative. Efficiently delegates, monitors, and prioritizes work across multiple projects, providing technical oversight and adjusting plans to address shifts in resources or timelines.

Collaboration & Partnership:

• Collaborates across the organization to align on expectations and achieve shared objectives. Leverages understanding of business leaders, stakeholders, and/or customers to ensure proposed solutions meet their needs. Supports inclusivity by actively seeking and listening to diverse perspectives, ensuring others feel heard and respected.

Problem Solving:

• Identifies and addresses moderately complex issues by analyzing a wide range of data and/or information to identify solutions in accordance with standard practices. Proactively escalates unresolved or critical issues with a thorough assessment and suggests potential solutions. Reviews, contributes to, and documents problem-solving strategies.

Continuous Learning:

• Pursues learning opportunities to expand knowledge and skills and/or tools in new areas and stays abreast of the latest industry trends and best practices. Proactively seeks and leverages ongoing feedback and training to improve skills. Coaches and mentors junior team members, fostering continuous learning and knowledge sharing within and across teams.

Continuous Improvement:

• Develops ideas, recommends updates, and/or collaborates on the implementation of process improvements to increase the efficiency and effectiveness of processes, protocols, and workflows across teams, and evaluates the impact on key stakeholders. Solicits feedback from others on ideas for alternative approaches and methods for continued improvement.

Performance and Development:

• Contributes to the talent development pipeline by participating in candidate interviews, assessing candidates, and providing hiring recommendations.

Qualifications:

• Deep experience designing large-scale leaf-spine and multi-fabric data center networks.
• Hands-on expertise with Ethernet switching, BGP, EVPN/VXLAN, IP addressing, routing scale, ECMP, QoS, and network segmentation.
• Strong knowledge of management, OOB, console-server, and production data-center network architectures.
• Experience with network operating systems and hardware platforms such as Arista EOS, Cisco Nexus/NX-OS, NVIDIA Spectrum/Cumulus, or comparable platforms.
• Experience designing or operating AI/HPC/GPU cluster networking, including RDMA/RoCE, lossless Ethernet, and high-bandwidth fabrics.
• Experience with network telemetry and observability: gNMI, streaming telemetry, SNMP, syslog, inventory/CMDB integration, and monitoring systems.
• Experience leading firmware lifecycle management, configuration standardization, vulnerability remediation, and network change governance.
• Strong incident-management and root-cause-analysis skills.
• Ability to communicate technical architecture and tradeoffs to engineering, operations, leadership, and customers.

Responsibilities

Key Responsibilities
Network Design, Development, and Validation:
-Designs and develops advanced network fabric and systems to support large-scale enterprise, data center, or cloud-based networking environments.
-Validates production networks and develops scaling strategies for varied use cases.
-Participates in the network solution and architecture design/validation process, providing expertise and guidance.
-Identifies and assesses complex risks in network design before deployment, recommending mitigations.
-Collaborates with vendors and internal groups (e.g., quality assurance) to ensure hardware, firmware, software, and cloud network code.
-Designs playbooks for investigations to resolve common issues related to network areas.
-Analyzes network workflows and proposes optimizations to enhance performance and efficiency.
Automation and Scripting:
-Designs and builds moderately complex modules within moderately complex automation frameworks for testing and operations to enhance efficiency.
-Automates critical network tasks, including production and lab environments and optimizes repetitive support functions to improve processes.
-Builds and customizes moderately complex dashboards, telemetry systems, and alerting mechanisms to proactively monitor network health.
-Writes and enhances advanced scripts to automate routine tasks across teams and business units.
-Uses and adjusts infrastructure pipelines, configuration items, using established playbooks and resources.
-Independently configures tools to meet requirements of new products or services.
Testing and Quality Assurance:
-Develops standalone, moderately complex test cases and test strategies to ensure and enhance network integrity.
-Serves as technical lead for test practices and strategies, providing guidance to the team and collaborating with other internal stakeholders in the development of robust testing protocols.
-Implements new approaches to improve break-fix validation and post-event root cause analysis (RCA), ensuring issues are resolved and lessons are integrated within the team.
-Conducts L1 network change reviews and creates and presents network changes to change management review boards.
-Collaborates with stakeholders to validate systems and environments through pre-production testing, ensuring compliance with internal standards prior to deployment.
-Provides audit documentation and evidence as part of Governance, Risk, and Compliance (GRC) to ensure alignment with governance and security requirements.
Monitoring and Reliability:
-Works with others (e.g. monitoring teams) to maintain advanced dashboards, telemetry systems, and alerting mechanisms to monitor network health, including defining metrics, trends, and thresholds to ensure service level objectives (SLOs).
-Uses tools from the monitoring team to monitor alerting systems and provides feedback to enhance the detection and diagnosis of network anomalies.
-Participates in operational support rotations, providing guidance in incident response and root cause investigations, and collaborating with stakeholders across team to resolve escalated issues.
-Develops and enhances tools and solutions for frontline support teams to drive efficiencies in responding to network failure conditions.
Cross-Team Collaboration and Leadership:
-Works alongside project program managers and uses templated project plan to ensure achievement of critical test and deployment milestones and project alignment.
-Mentors junior engineers and provides subject matter expert (SME) guidance on technical issues.
-Engages with customers to clarify technical issues, and contributes to the delivery of root cause analysis (RCA) documentation.
-Contributes to roadmap and software architecture discussions, ensuring ideas are shared within team and implemented where necessary.
-Collaborates with vendors and internal teams to ensure adherence to networking standards, identifying areas of misalignment and identifying resolutions.
Performance and Capacity Management:
-Analyzes network performance and service metrics (e.g., latency, throughput, packet loss) to identify network efficiency issues and takes next steps to scale network.
-Uses capacity information through security and performance metrics to ensure the infrastructure can support anticipated growth.

Core Responsibilities
Planning & Execution:
-Independently manages work, monitoring timelines and deliverables to ensure projects or initiatives stay on track and meet requirements. Proactively prioritizes work and adapts to resource or timeline shifts, suggesting adjustments to maintain project efficiency.