Providing Out-of-Band Connectivity to Mission-Critical IT Resources

Network Management Best Practices

A collage of concepts related to network management best practices for resilience and security.

Network management involves administering, controlling, and monitoring an organization’s network. For most companies, the top priority for network teams is ensuring the continuous availability of critical business services, even during disruptive events like natural disasters, ransomware attacks, and infrastructure failures. Network resilience is the ability to continue operating (if in a degraded state) and delivering digital services in the face of adversity. This guide discusses the network management best practices for improving and supporting network resilience.  

Network management best practices

Network Management Best Practices for Resilience

Isolated Management Infrastructure (IMI)

  • Moves management interfaces off the production network to protect them from cybercriminals

  • Out-of-band (OOB) management ensures continuous remote access to IMI even when production infrastructure is offline

  • Isolated Recovery Environments (IREs) allow teams to restore infrastructure and services without risking reinfection

Network Automation

  • Reduces the risk of failures or security breaches by eliminating human error in configuration changes

  • Simplifies fleet management tasks like connectivity checks, device location monitoring, and software patching

  • Application-aware routing, intelligent load balancing, and automatic failover ensure optimal performance and availability

Network Security

  • Zero trust security protects valuable data and resources from attackers already on the network

  • SASE and SSE extend enterprise security policies and tools to remote users, applications, and devices

  • AIOps provides enhanced security monitoring, threat detection, and remediation capabilities

Isolated Management Infrastructure (IMI)

Major ransomware attacks and breaches happen so frequently that cybersecurity professionals must now operate as if the network has already been compromised. This high-threat atmosphere led to the rise of the Zero Trust Security methodology discussed below. It’s also why a recent CISA Binding Directive outlines the best practice of isolating your management interfaces to a designated management network.

Moving all control functions for network infrastructure off the production LAN reduces the risk of cybercriminals accessing your management interfaces and “crown jewel” assets. This practice is known as isolated management infrastructure (IMI), and it separates the management plane from the data plane using designated network infrastructure. Doing so prevents attackers on the production network from finding and accessing the interfaces used to control servers, firewalls, routers, and other critical infrastructure devices. Thanks to management network segmentation and zero-trust security controls, hacking an IMI is almost impossible.

A diagram showing a multi-layered isolated management infrastructure.

The best practice is to use out-of-band (OOB) serial consoles (a.k.a. console servers or terminal servers) to help construct the IMI. An OOB management solution uses dedicated network interfaces (such as 4G/5G cellular LTE or fiber) to provide an Internet connection for remote management access that doesn’t rely upon the primary production network at all. The benefit of using an OOB console server for the IMI is that teams have continuous access to monitor, manage, troubleshoot, and recover remote infrastructure when the production network is unavailable. Additionally, routing management ports to terminate on OOB terminal servers deployed top-of-rack creates multiple layers of management isolation to protect critical assets from criminals on the network.

A diagram showing the components of an isolated recovery environment.

Another network management best practice aided by IMI and OOB serial consoles is an isolated recovery environment (IRE). An IRE is built with designated infrastructure that is easily and quickly deployable, including an OOB control plane (such as a serial console), redundant storage & compute, and security and recovery tools. This gives teams a safe environment to recover from ransomware attacks without worrying about reinfection. Ideally, the IMI will use devices that consolidate network functions to enable easy deployments and scaling of IRE and OOB, but those devices should have robust features that can host the apps, tools, and services required to rebuild systems and restore data.

Network automation

Modern networks are large, complex, and ever-expanding, with user expectations growing more demanding every day. Even the best network administrator sometimes makes mistakes, either through negligence or because they have an overwhelming amount of work to do. Maybe they copy and paste the wrong security setting for a particular firewall appliance in a rush to deploy a new site on time; perhaps they miss a critical device health alert because they’re responding to a separate incident. These human errors, while understandable, can have devastating consequences on network resilience by causing security breaches, equipment failure, and service outages.

Network automation removes human error from the equation, ensuring network management tasks are carried out perfectly every time. Automation streamlines the most tedious network and fleet management tasks so teams can improve efficiency without allowing anything to fall through the cracks. Automation tools also respond to changing network conditions faster than human administrators to optimize the performance and availability of critical systems and services.

Network Automation Examples

Infrastructure as Code (IaC) abstracts infrastructure configurations from the underlying hardware so they can be written and deployed as repeatable, automatable scripts.

Zero Touch Provisioning automatically downloads and installs new network device configurations with little to no human interaction to streamline remote deployments.

Software-Defined Wide Area Networking (SD-WAN) decouples WAN control functions from the underlying hardware to enable features like application-aware routing, intelligent load balancing, and automatic failover to improve performance and availability.

Automatic Patch Management ensures software vulnerabilities are closed before being exploited by cybercriminals while providing automatic recovery and rollback in case of issues.

Network security

As discussed above, network breaches occur so frequently that it’s now a security best practice to assume attackers are already on the network. This is part of the zero trust security methodology, which follows the principle of “never trust, always verify” regarding all the users, devices, and applications that access the network. Zero trust security uses strong authentication methods (e.g., 2FA or one-time passwords), hardware roots of trust, and network micro-segmentation. These methods prevent attackers from moving around the network and accessing valuable resources (such as management interfaces).

Another security-related network management best practice is to extend zero-trust controls and policies to the network’s edges, such as to work-from-home devices, branch offices, and other remote business sites. This is achieved using edge-centric security solutions such as Security Service Edge (SSE) and Secure Access Service Edge (SASE). These technologies route remote, web-destined network traffic through a whole stack of cloud-based security solutions. This allows organizations to apply consistent security to edge traffic without creating bottlenecks at a centralized firewall or deploying additional security appliances at each site.

A diagram illustrating a basic SASE network security architecture.

Another emerging network management best practice, especially for complex, automated infrastructures, is using artificial intelligence (AI) and machine learning to aid security and recovery. For example, AIOps solutions analyze data pulled from various sources on the network, including monitoring platforms, security appliances, and system event logs. AIOps is excellent at detecting anomalies, extrapolating potential consequences, and positing solutions. It can find novel and zero-day threats on the network, spot the signs of an imminent device failure, and perform root-cause analysis (RCA) to discover the source of problems. AIOps enhances the management, automation, and security practices on this list to improve the overall efficiency and resilience of enterprise networks.

Network management FAQs

1. How do I ensure interoperability amongst network management solutions?

Managing a modern network requires many different solutions, often from many different vendors. All these solutions must work together to prevent the management plane from getting too complex and ensure there are no coverage gaps. One option is to stick within one vendor’s ecosystem, but you may miss out on beneficial features or pay for functionality you don’t need. The best approach is to use a vendor-neutral (a.k.a. vendor-agnostic) network management platform to unify all your tools. To learn more, read The Benefits of Vendor Agnostic Platforms in Network Management.

2. What’s the difference between network automation and orchestration?

Network automation and network orchestration are two concepts that are often referenced together, leading to some confusion about the difference between them. Network automation focuses on individual tasks and processes, such as deploying a single software update. Network orchestration involves coordinating and managing multiple tasks and processes, or even entire workflows, such as configuring and deploying all the software on a server. To learn more, read IT Automation vs Orchestration: What’s the Difference?

3. Is network resilience the same as redundancy and backups?

Redundancy and backups are both critical to business continuity, but they do not equate to network resilience. Backups are copies of data, configurations, and code that are used to restore failed (or compromised) production systems. Redundancy duplicates services, applications, and systems so the primary versions can be “failed over” in case of failure or attack. Resilience is an organization’s overall ability to recover or adapt when major disruptions occur. To learn more, read Network Resilience: What is a Resilience System?

Resilient network management with Nodegrid

These network management best practices represent the industry-leading solutions for addressing the most common resilience challenges facing organizations. The network resilience experts at ZPE Systems can help you implement these practices with Gen 3 out-of-band management solutions and a vendor-neutral network management platform that supports automation. ZPE’s Nodegrid platform is the perfect ransomware recovery multi-tool, providing an isolated control plane as well as access to all the tools and software needed to restore critical operations.

Network management best practices for ransomware recovery and resilience

Learn more about using Nodegrid to improve ransomware resilience by downloading our white paper, 3 Steps to Ransomware Recovery.

Download Whitepaper

ZPE Systems offers various solutions to help you implement your enterprise network management strategy.

Including data center infrastructure management, critical remote infrastructure management, and a secure uCPE gateway for distributed branch & edge networks. To learn more, contact us online. 

Contact Us

Network Resilience: What is a Resilience System?

A digital web of interconnected network resilience concepts being selected by a business person in a suit.

Network resilience means being able to withstand or recover from adversity, service degradation, and complete outages with minimal business disruption. The longer business-critical services are down, or systems are breached, the greater the risk of significant financial, reputational, and legal consequences. A resilience system is a set of technologies that enable an organization to continue operating while teams work to repair failures and recover from cyberattacks. But what exactly is a resilience system, and what does it look like? This guide to network resilience defines resilience systems, provides example use cases, compares them to related technologies like backups and redundant systems, and describes the key components required to build them.

What is a resilience system?

A resilience system provides all the infrastructure, tools, and services necessary to continue operating, if in a degraded state, during major incidents. It also includes everything needed to recover data, rebuild systems, perform security testing, and continue delivering core business functionality. A resilience system is typically isolated from the production network, preventing cybercriminals from finding and compromising it and ensuring teams have continuous access even if the primary network goes down.

Resilience system use cases

Some examples of the challenges that resilience systems help overcome include:

1. Ransomware recovery

In a ransomware attack, cybercriminals infect systems with malware that spreads throughout the network and encrypts any data it encounters. Modern ransomware now uses packaged attacks that move at machine speed, instantly incapacitating entire networks. Organizations completely lose access to critical systems and data until they pay a ransom, often in untraceable cryptocurrency. Ransomware is an exceptionally tenacious form of malware and tends to reinfect backup data and rebuilt systems, significantly hampering recovery efforts and increasing the duration and cost of the attack. The best practice for resilience systems is to isolate them on an out-of-band (OOB) network, inaccessible to hackers who have breached the production in-band network. Doing so creates a safe, isolated recovery environment (IRE) where teams can restore critical data and systems without the risk of reinfection. The resilience system includes all the tools and hardware needed to restore critical business services and infrastructure. An IRE significantly accelerates ransomware recovery and minimizes downtime, so businesses can avoid paying ransoms and reduce the overall cost of attacks.

2. Network outages

Enterprise network architectures and supply chains are highly complex, with lots of moving parts that rely on external vendors to maintain availability. Just one of those vendors dropping the ball could take the entire organization offline, severely impacting network resilience. For example, in 2023, an expired cryptographic certificate caused Cisco’s Viptela SD-WAN appliances to fail on reboot, completely taking down affected networks until the issue was resolved. With a resilience system, Viptela customers could have potentially avoided this downtime by failing over to alternative network resources. For example, a resilience system with integrated cellular failover allows branches to continue connecting to and delivering critical business services while also providing a lifeline for remote teams to access and recover failed systems. A resilience system also provides observability and automatic notifications so teams are instantly alerted to issues like certificate expirations and can respond quickly to recover critical services.

3. Shift to remote work

Incidents like ransomware attacks and equipment failures happen frequently enough that companies can create detailed plans and proactively implement solutions to minimize their impact, but not all adverse events are so predictable. When the COVID-19 pandemic struck, the massive shift to remote work strained the network resources of most organizations. Instead of maintaining a limited number of branch offices, teams suddenly had to treat every employee as a new branch, leading to performance degradation and outages as they scrambled to reinforce the business’s remote capabilities. A resilience system gives teams the tools and resources they need to provision additional infrastructure, manage networking logic, deploy new security solutions, and more, even while the primary network is offline or under a heavy load. A resilience system is the key to quickly adjusting network performance and security to adapt to sudden changes like a transition to fully remote operations.

Do backups and redundancy equate to network resilience?

The short answer is no; backups and redundancy do not equate to network resilience, though they do contribute to making systems more resilient.

  • Backups are copies of data, configurations, and application code used to do a hot or cold restore when a production system fails. The underlying infrastructure must remain operational for teams to access and use backups, and unless additional resilience measures are taken, it’s easy for backups to become infected or compromised, severely hampering recovery efforts.
  • Redundancy involves duplicating critical systems, services, and applications as a failsafe in case the primaries go down. Organizations can “fail over” to the redundancies to continue critical business operations during outages. However, redundant systems are just as susceptible to failures and infections without additional resilience measures like out-of-band management and isolated management infrastructure.

Backups and redundancy are part of network resilience but alone are not enough to ensure business continuity. Resilience systems focus on maintaining the architecture of the production network while adding the ability to recover or adapt to adversity. The next section discusses all the tools and technologies that make up network resilience systems.

What does a resilience system look like?

There are four key components that go into a resilience system.

Key Components of a Resilience System

Alternative Networking

Full-stack routing and switching, Wi-Fi, VoIP, virtualization, software-defined network overlays for SDN & SD-WAN

Alternative Compute

Full-stack compute, containers, virtual machines, and any other resources needed to run applications and deliver services

Storage & Storage Recovery

Enough storage to recover systems and applications as well as support content delivery

Automation

Tools like zero-touch provisioning (ZTP) to facilitate speedy recovery while minimizing human error

Alternative networking and compute resources ensure the organization can failover in the event of a network failure or continue delivering services when production servers are unavailable. Teams also need enough storage to restore backup data, build new systems, and support the content delivery network (CDN). Automation solutions like zero-touch provisioning (ZTP), configuration management, and security validation tools accelerate the recovery process while mitigating the risk of human error. Combined, these components enable teams to reduce the frequency, severity, and duration of outages, improving overall network resilience.

Network resilience with ZPE Systems

A resilient network will continue delivering critical business services in the face of any challenge, whether from cybercriminals, supply chain issues, global events, or even plain human error. A resilience system is isolated from the production network to ensure security and availability, and it consists of all the tools and technologies needed to troubleshoot, recover, and deliver your most crucial data, applications, and infrastructure. The Nodegrid platform from ZPE Systems is the perfect foundation for a resilience system. Nodegrid is a vendor-neutral, out-of-band management solution capable of running your choice of third-party software. Nodegrid allows you to build a highly customizable IRE containing all the tools needed to safely recover from ransomware. You can even use Nodegrid to deliver services while the primary network or systems are down, making it your all-in-one network resilience multi-tool.

Want to ensure network resilience by accelerating ransomware recovery?

Minimize the business impact of ransomware with the help of our whitepaper, 3 Steps to Ransomware Recovery. Learn how to follow Gartner’s best practices to build an Isolated Recovery Environment

Download Whitepaper

Out-of-Band Management: What It Is and Why You Need It

Thumbnail – What is out-of-band management

This scenario is every IT professional’s worst nightmare: it’s the middle of the night, a remote site on the other side of the country has gone offline, and nobody knows why. A single minute of downtime can cost anywhere from several hundred dollars to tens of thousands of dollars, and the nearest tech is a six-hour plane ride away. Consider 2024’s CrowdStrike outage and the devastation caused for banks, airports, and many other organizations.

A bar chart showing the average hourly cost of downtime by industry.
Data Source: SolarWinds

Out-of-band management offers the solution: a way for teams to access critical remote infrastructure during outages and breaches without “out-of-chair” expenses. Out-of-band management allows organizations to recover remote infrastructure faster, reducing the duration and expense of downtime.

This guide to out-of-band management answers critical questions about what this technology is, why you need it, and how to choose the right solution.

What is out-of-band management?

Out-of-band management (OOBM) involves controlling network infrastructure and workflows on an out-of-band network. An out-of-band network is an entirely separate network that runs parallel with your production (or in-band) network but doesn’t rely on any of the same infrastructure or services. OOBM allows teams to administer network infrastructure remotely on a dedicated connection, such as secondary Fiber or cellular LTE, that will remain available even if the in-band network goes down from an equipment failure, ISP outage, or ransomware attack.

A diagram showing how out-of-band management works.

The biggest reason to use out-of-band management is to ensure continuous, uninterrupted access to critical remote infrastructure even when the primary network is down. OOBM allows teams to recover from outages and cyberattacks faster and more cost-efficiently because they can access, troubleshoot, and restore systems without rolling trucks or hiring on-site services.

Out-of-band management provides a lifeline for teams to access critical remote infrastructure when the production network is offline. It allows them to immediately begin troubleshooting and repairing the issue to restore services ASAP. With OOBM, companies save money on recovery expenses, and minimize the duration and business impact of downtime.

What is an OOBM serial console?

Front and back views of the Nodegrid out-of-band management serial console.

Some organizations use OOBM jump boxes (or jump servers) that are connected to both the in-band and out-of-band networks, allowing administrators to “jump” from one network to the other for management. Examples of low-cost jump boxes include the Intel NUC and the Raspberry Pi. However, OOBM jump boxes are security risks because they do not effectively isolate the management infrastructure, plus they require an entire duplicate infrastructure of devices and services to create the out-of-band network. The best practice for security, resilience, and efficiency is to deploy an all-in-one, out-of-band management solution.

An out-of-band management solution uses hardware devices known as serial consoles, which connect to infrastructure devices via their management port (usually RS232 Serial, Ethernet, or USB). Serial consoles are known by lots of other names, including terminal servers, console servers, console server switches, serial routers, and serial switches.

The serial console has dedicated network interfaces to provide an Internet connection for remote management access, often fiber or 4G/5G cellular LTE, so they don’t connect to or rely upon the primary production network at all. This gives teams the ability to continuously monitor and administer critical remote infrastructure even during an ISP or WAN outage that would make a jump box inaccessible.

 Administrators remotely access an OOBM serial console via this dedicated link and, from there, can view and manage all connected infrastructure from a single, convenient software platform. This software is typically deployed on-premises and runs as a VM (virtual machine)  either on the serial console itself or on a separate machine, but there are some cloud-based OOBM network management software tools.

Out-of-band management software varies from provider to provider, with most offering second-generation (or Gen 2) solutions that provide some built-in automation capabilities but do not support vendor-neutral integrations with third-party tools. Newer, third-generation (or Gen 3) solutions use an open, x86 Linux-based operating system to allow easy integrations with other vendors’ software for automation, orchestration, security, monitoring, and more.

The benefits of out-of-band management

Out-of-band management can help you:

  • Improve network performance: Performing resource-intensive management, automation, and orchestration workflows on the out-of-band network reduces the strain on the production network for better speed and reliability.
  • Accelerate ransomware recovery: The OOBM network can be used to create an isolated recovery environment (IRE) where teams can safely rebuild and recover from ransomware attacks without the risk of reinfection, reducing the duration and expense of ransomware-related outages.
  • Streamline repairs and rebuilds: OOBM provides the ability to deploy the tools and applications needed to isolate, cleanse, rebuild, and restore services that have been affected by failures and ransomware.

The security and resilience benefits of out-of-band management are discussed further below.

How does out-of-band management improve security and resilience?

Network breaches and ransomware attacks occur so frequently that most businesses know it’s no longer a question of “if,” but “when” they’ll be hit. Once cybercriminals compromise a device or account and can move around the network, it’s only a matter of time before they find the management interfaces and take complete control over critical infrastructure.

OOBM and management infrastructure isolation

Serial consoles create an out-of-band network by directly connecting to the management port of infrastructure devices and moving all control functions off of the production LAN. This isolates the management plane from the data plane, which is part of a cybersecurity best practice known as isolated management infrastructure (IMI). An IMI further segments the management network and routes management ports to terminate on top-of-rack, OOBM serial switches, creating multiple layers of isolated management. The isolated management plane is always remotely accessible to engineers via the OOBM connection, but it remains hidden from any cybercriminals who may breach the production network.

Multi Layered OOB IMI – ZPE Systems

 

OOBM and ransomware recovery

Out-of-band management also improves security and resilience by aiding in ransomware recovery. According to a Sophos survey, 70% of companies hit by ransomware take longer than two weeks to recover, due in no small part to the pervasive nature of the malware used and how frequently rebuilt systems and recovered data get reinfected. Today’s ransomware attacks are now pre-packaged and move at machine speed – meaning instantly – across infrastructure, bringing entire businesses down before they’ve even realized they’re under attack. The longer the business is offline, the more revenue (and customer trust) is lost, causing recovery costs to skyrocket.

An IMI using out-of-band management gives teams an isolated recovery environment (IRE) where they can recover data and rebuild systems without the risk of reinfection. The IRE allows organizations to get services back online faster to reduce the financial and reputational consequences of ransomware attacks.

A diagram showing the components of an isolated recovery environment.

Resilience is defined as the ability to continuously operate and deliver services, if in a degraded fashion, even while undergoing major failures and breaches. Out-of-band management improves resilience by ensuring that teams have continuous access to critical remote infrastructure no matter what’s going wrong with the production environment. OOBM serial consoles also isolate the management infrastructure to protect it from attackers on the primary network and provide a safe environment for teams to recover from ransomware.

Why choose Nodegrid for out-of-band management?

Many network teams think of out-of-band as being a huge expense and time sink. Setting up  proper infrastructure for OOBM and IMI typically requires 6 or more boxes at each business site for routing, switching, firewall, storage, cellular access, and a jump box. The Nodegrid platform from ZPE Systems reduces the cost and headache of out-of-band management by combining all these functions and more into a single box. Teams can easily drop a Nodegrid box in each site at a fraction of the cost of deploying a traditional OOBM network.

A diagram showing ZPE’s multi-function capabilities for IMI in branch and edge sites.

The first Gen 3 OOBM solution

Nodegrid is the first and only Gen 3 out-of-band management solution. Nodegrid OOBM devices use the x86 Linux-based NodegridOS, which is capable of running VMs and Docker containers to host your choice of third-party applications for automation, orchestration, security, SD-WAN, and more. Nodegrid’s ability to host other vendors’ software ensures that teams have access to all the tools they need to troubleshoot and recover infrastructure from within the IMI environment, making it the perfect network resilience multi-tool.

Nodegrid OOBM software is available as an on-premises solution or a highly scalable cloud-based app, and both support easy integrations with tools for monitoring, automated configuration management, and more. This enables teams to consolidate and streamline their workflows, maximizing efficiency while reducing the risk of human error.

Nodegrid’s other key features include:

  • Built-in 5G/4G LTE and Wi-Fi options for OOB and network failover
  • OOB support over IPMI, ILO, DRAC, CIMC, vSerial, and KVM
  • Robust hardware security like BIOS protection, UEFI Secure Boot, and an encrypted solid-state disk
  • SAML 2.0 and two-factor authentication (2FA)
  • Support for legacy and mixed-vendor infrastructure without expensive adapters

ZPE Systems offers a wide range of out-of-band management devices to fit any deployment size and use case, including the 96-port Nodegrid Serial Console Plus (NSCP) for large and hyperscale data centers, and the Nodegrid Gate SR, which combines branch gateway routing and OOB serial console functionality for remote business sites like retail stores and manufacturing plants.

Nodegrid OOB serial console comparison


Guest OS
Docker Apps
Wi-Fi
Cellular (Dual-SIM)
Serial Ports
Data Sheet
Nodegrid Serial Console S Series
1
1-2
No
1
16, 32 or 48
Nodegrid Serial Console Plus (NSCP)
1
1-2
Yes
1
16, 32, 48 or 96

Nodegrid OOB network edge router comparison


Guest OS
Docker Apps
Wi-Fi
Cellular (Dual-SIM)
Serial Ports
Data Sheet
Nodegrid Link SR
1
1-2
Yes
1
1
Nodegrid Bold SR
1
1-2
Yes
1-2
8
Nodegrid Hive SR
1-2
1-3
Yes
1-2
8
Nodegrid Gate SR
1-3
1-4
Yes
1-2
8
Nodegrid Net SR
1-6
1-4
Yes
1-4
16-80
Nodegrid Mini SR
1
1-2
Yes
1
Via USB

Get scalable network resilience with the only Gen 3 out-of-band management solution

Only Nodegrid OOBM delivers network control, security, automation, and resilience with a completely vendor-neutral platform. To see Nodegrid out-of-band management in action, request a free demo.

Request a Demo

Why network resilience requires Isolated Management Infrastructure

Isolated Management Infrastructure

Summary

Business relies on network resilience. But network management now requires more than a ‘break/fix’ approach. Decentralized and growing architectures are spread across cloud, premises, and large IoT footprints. For shrinking IT teams, this presents resilience problems, as many are unaware of the best practice to help them care for their vast digital estate, maintain complex and delicate architecture, and recover quickly in case of attack. This gap means the network is a source of anxiety.

Big Tech solved these problems 10+ years ago, by doubling down on their management architectures. This best practice, which is now recommended by CISA, involves fully separating management networks from production networks into what’s called Isolated Management Infrastructure. IMI goes far beyond serial console and out-of-band access, providing the management and service delivery capabilities teams need to reduce on-site upkeep, stabilize delicate architecture, and accelerate ransomware recovery.

Big Tech’s network resilience secret lies in ZPE Systems’ Nodegrid hardware and software. Nodegrid is the only network resilience platform, delivering the Gen 3 capabilities that are required to build IMI. Now, organizations in every industry benefit from the best practices that have been trusted to run the public cloud for over a decade. Download the solution guide to get your step-by-step walkthrough on building IMI for network resilience.

Network Resilience Doesn’t Mean What it Did 20 Years Ago

Network resilience requirements have changed

Enterprise networks are like air. When they’re running smoothly, it’s easy to take them for granted, as business users and customers are able to go about their normal activities. But when customer service reps are suddenly cut off from their ticketing system, or family movie night turns into a game of “Is it my router, or the network?”, everyone notices. This is why network resilience is critical.

But, what exactly does resilience mean today? Let’s find out by looking at some recent real-world examples, the history of network architectures, and why network resilience doesn’t mean what it did 20 years ago.

Why does network resilience matter?

There’s no shortage of real-world examples showing why network resilience matters. The takeaway is that network resilience is directly tied to business, which means that it impacts revenue, costs, and risks. Here is a brief list of resilience-related incidents that occurred in 2023 alone:

  • FAA (Federal Aviation Administration) – An overworked contractor unintentionally deleted files, which delayed flights nationwide for an entire day.
  • Southwest Airlines – A firewall configuration change caused 16,000 flight cancellations and cost the company about $1 billion.
  • MOVEit FTP exploit – Thousands of global organizations fell victim to a MOVEit vulnerability, which allowed attackers to steal personal data for millions.
  • MGM Resorts – A human exploit and lack of recovery systems let an attack persist for weeks, causing millions in losses per day.
  • Ragnar Locker attacks – Several large organizations were locked out of IT systems for days, which slowed or halted customer operations worldwide.

What does network resilience mean?

Based on the examples above, it might seem that network resilience could mean different things. It might mean having backups of golden configs that you could easily restore in case of a mistake. It might mean beefing up your security and/or replacing outdated systems. It might mean having recovery processes in place.

So, which is it?

The answer is, it’s all of these and more.

Donald Firesmith (Carnegie Mellon) defines resilience this way: “A system is resilient if it continues to carry out its mission in the face of adversity (i.e., if it provides required capabilities despite excessive stresses that can cause disruptions).”

Network resilience means having a network that continues to serve its essential functions despite adversity. Adversity can stem from human error, system outages, cyberattacks, and even natural disasters that threaten to degrade or completely halt normal network operations. Achieving network resilience requires the ability to quickly address issues ranging from device failures and misconfigurations, to full-blown ISP outages and ransomware attacks.

The problem is, this is now much more difficult than it used to be.

How did network resilience become so complicated?

Twenty years ago, IT teams managed a centralized architecture. The data center was able to serve end-users and customers with the minimal services they needed. Being “constantly connected” wasn’t a concern for most people. For the business, achieving resilience was as simple as going on-site or remoting-in via serial console to fix issues at the data center.

Network architecture showing simplicity of data center connected via MPLS to branch office

Then in the mid-2000s, the advent of the cloud changed everything. Infrastructure, data, and computing became decentralized into a distributed mix of on-prem and cloud solutions. Users could connect from anywhere, and on-demand services allowed people to be plugged in around-the-clock. Services for work, school, and entertainment could be delivered anytime, no matter where users were.

Network architecture showing complexity of data center, CDN, remote user, branch office, all connected via many paths

Behind the scenes, this explosion of architecture created three problems for achieving network resilience, which a simple serial could no longer fix:

Too Much Work

Infrastructure, data, and computing are widely distributed. Systems inevitably break and require work, but teams don’t have the staff to keep up.

Too Much Complexity

Pairing cloud and box-based stacks creates complex networks. Teams leave systems outdated, because they don’t want to break this delicate architecture.

Too Much Risk

Unpatched, outdated systems are prime targets for packaged attacks that move at machine speed. Defense requires recovery tools that teams don’t have.

Enabling businesses to be resilient in the modern age requires an approach that’s different than simply deploying a serial console for remote troubleshooting. Gen 1 and 2 serial consoles, which have dominated the market for 20 years, were designed to solve basic issues by offering limited remote access and some automation. The problem is, these still leave teams lacking the confidence to answer questions like:

  • “How can we guarantee access to fix stuff that breaks, without rolling trucks?”
  • “Can we automate change management, without fear of breaking the network?”
  • “Attacks are inevitable — How do we stop hackers from cutting off our access?”

Hyperscalers, Internet Service Providers, Big Tech, and even the military have a resilience model that they’ve proven over the last decade. Their approach involves fully isolating command and control from data and user environments. This allows them to not only gain low-level remote access to maintain and fix systems, but also to “defend the hill” and maintain control if systems are compromised or destroyed.

This approach uses something called Isolated Management Infrastructure (IMI).

Isolated Management Infrastructure is the best practice for network resilience

Isolated Management Infrastructure is the practice of creating a management network that is completely separate from the production network. Most IT teams are familiar with out-of-band management as this network; IMI, however, provides many capabilities that can’t be hosted on a traditional serial console or OOB network. And with increasing vulnerabilities, CISA issued a binding directive specifically calling for organizations to implement IMI.

Isolated Management Infrastructure using Gen 3 serial consoles, like ZPE Systems’ Nodegrid devices, provides more than simple remote access and automation. Similar to a proper out-of-band network, IMI is completely isolated from production assets. This means there are no dependencies on production devices or connections, and management interfaces are not exposed to the internet or production gear. In the event of an outage or attack, teams retain management access, and this is just the beginning of the benefits of having IMI.

A network architecture diagram showing Isolated Management Infrastructure next to production infrastructure

IMI includes more than nine functions that are required for teams to fully service their production assets. These include:

  • Low-level access to all management interfaces, including serial, Ethernet, USB, IPMI, and others, to guarantee remote access to the entire environment
  • Open, edge-native automation to ensure services can continue operating in the event of outages or change errors
  • Computing, storage, and jumpbox capabilities that can natively host the apps and tools to deploy an IRE, to ensure fast, effective recovery from attacks

Get the guide to build IMI

ZPE Systems has worked alongside Big Tech to fulfill their requirements for IMI. In doing so, we created the Network Automation blueprint as a technical guide to help any organization build their own Isolated Management Infrastructure. Download the blueprint now to get started.