What happens when your primary network fails and your backup isn’t there to save you?

That was the question on September 21, when a communications failure disrupted flights across the Northeast. A telecom circuit serving an FAA facility in Philadelphia failed, and when the system attempted to use its backup, officials discovered that the backup fiber connection had also been cut during construction in New Jersey. The resulting outage led to ground stops and significant delays at major airports including Newark, Philadelphia, JFK, and LaGuardia.

This is the perfect reminder that redundancy doesn’t equal resilience.

Truly resilient architecture needs independent paths that can survive failures and give engineers a way to reach critical infrastructure when the primary network goes down. That’s where out-of-band (OOB) management becomes part of the failover strategy.

But before getting into how OOB could have helped, let’s clarify what it does – and what it doesn’t do.

Production Failover Protects Operations. OOB Failover Protects Recovery.

Let’s get one thing out of the way: out-of-band management failover wouldn’t have prevented the FAA incident. It would not have carried the radar data or controller-to-pilot communications, because these require their own separately engineered, certified, and capable (in terms of performance) production failover architecture.

OOB failover solves a different problem:
How do engineers reach the infrastructure when the network they normally use is down?

Why This Matters

During a production failure, the immediate priority is to restore connectivity. But there’s an entire recovery process that needs to happen. Engineers need to be able to access the routers, switches, firewalls, etc. to determine what failed and what they need to do to restore service.

This is the value of OOB. It doesn’t replace or take over for the production network. It gives engineers a separate path to the equipment that can help diagnose the failure, reconfigure the network, activate any available backup path, and prepare the infrastructure to resume service as soon as connectivity is restored.

Otherwise, coming back online after connectivity is restored means waiting hours for someone to physically reach the equipment, which leads to:

• Longer recovery times
• Additional truck rolls
• More operational disruptions
• Dependence on local personnel
• Reduced visibility during an outage

In the case of the FAA, out-of-band wouldn’t have kept the network online, but it would have given engineers a way to work on the infrastructure while the production paths were unavailable. This makes all the difference for a fast recovery, and it’s why redundancy alone doesn’t provide real resilience.

Do You Have Two Connections, Or Two Paths?

There’s an adage in preparedness: two is one, one is none. But in networking, this basic redundancy doesn’t necessarily provide a buffer against outages.

If the primary and backup connections share the same carrier, physical route, or infrastructure, one incident (like a fiber cut) affects both. That’s exactly what happened in the FAA incident and left engineers unable to reach the infrastructure. It’s important to understand the difference between redundancy and resilience:

• Redundancy asks: “Do we have a backup connection?”
• Resilience asks: “If there’s a failure, can we still reach the infrastructure without going on-site?”

Imagine the FAA had set up a different backup, like 5G, that was immune to the fiber cut. They could have still remotely reached their infrastructure to accelerate recovery. The key to making out-of-band truly “out-of-band” is to have its own dedicated links that are completely isolated, both logically and physically, from the production network.

But what does it look like?

Building a Resilient Recovery Path

Out-of-band can use separate ISP providers (if available), MPLS, or link types that don’t rely on traditional terrestrial infrastructure, like 5G/LTE or satellite. Building this resilient out-of-band management path involves deploying serial console servers that connect to the production infrastructure, and using cellular modems, satellite connectivity, etc. that are dedicated to giving remote access to these serial console servers.

This gives engineers the ability to completely bypass the production network and still remotely access all of their production infrastructure. From there, they can:

• Diagnose failed infrastructure
• Access device consoles
• Change routing
• Activate backup circuits
• Roll back configurations
• Restart equipment
• Coordinate recovery across distributed sites

Think of the primary WAN as a single point of failure, and out-of-band is the safety net. When the WAN fails and operations falls apart, out-of-band catches all the pieces and makes it easier (and faster) to put everything back together.

Once your OOB path is set up and fully isolated, there’s one more crucial step to take…

Test The OOB Path

Putting the OOB infrastructure in place is a great start, but you need to make sure everything works so you don’t have a false sense of resilience. There’s nothing like trying to use your safety net in a real scenario, only to find out your SIM or APN settings aren’t configured properly.

Here are some tips:

• Verify that your SIM cards are properly provisioned and activated, and ensure your APN settings are correct.
• Make sure your system can detect loss of the primary connection and can automatically fail over to the backup link.
• Test your complete recovery workflow, from failover, to troubleshooting and failback.

The Takeaway: Protect Your Recovery Path

The FAA outage showed us that a backup is only useful if it survives the failure you’re trying to protect against. But even if your backup is available, you still need a dedicated management path. This is crucial to a fast recovery because it allows you to maintain secure remote access and get your infrastructure ready for when connectivity is restored.

This is exactly why organizations are deploying the next generation of out-of-band – something called Isolated Management Infrastructure – using ZPE Systems’ Nodegrid. Nodegrid combines out-of-band, multiple failover links, centralized management, and remote recovery into a streamlined architecture.

One Nodegrid device provides serial, Ethernet, and USB access to routers, switches, firewalls, servers, and the full stack of critical equipment. And when it comes to failover, Nodegrid supports 5G (via built-in modem), satellite, MPLS, and other link types without adding additional devices.

The result is an independent path that keeps critical infrastructure reachable, gives engineers the access they need to diagnose and recover problems remotely, and reduces dependence on on-site intervention.

Get in Touch For a Failover Assessment

Our engineers can help you identify where your current failover strategy may leave critical infrastructure unreachable, and where an independent management path can improve recovery.

Get in touch for a failover assessment and learn how Nodegrid can help you maintain secure access when your primary or backup network goes down.

Modern IT infrastructure depends on the wide area network (WAN) to connect everything together. Data centers, branch offices, cloud environments, edge locations, and remote sites all rely on WAN connectivity to exchange data, run applications, and maintain centralized operations.

But the WAN also provides the connectivity required to operate the infrastructure itself (think device troubleshooting, config updates, etc.). This creates a problem: when the WAN fails, organizations lose their production services and their ability to manage the infrastructure responsible for restoring them. It’s like walking a tightrope without having a safety net below.

If your remote management capabilities depend on your production network, the WAN can easily become a single point of failure that’ll leave you battling long outages.

What is the WAN?

Unlike a local area network (LAN), which connects devices within a location, the wide area network is the connectivity layer that links geographically distributed networks and infrastructure across cities, regions, or countries. This includes everything from data centers and corporate HQs, to industrial sites, edge kiosks, and remote infrastructure.

Common WAN link types include MPLS, dedicated circuits, broadband, SD-WAN, and Internet connections.

The WAN is the connective tissue between distributed infrastructure. But its importance goes beyond just moving application traffic. That’s because it touches all three network planes.

The WAN Touches Every Network Plane

Modern networks have three primary planes that each serve a different purpose. We broke these down in a previous article, but here are the basics:

1. Production/data plane: Carries the traffic (app workloads, voice, video, etc.)
2. Control plane: Determines how traffic moves (think BGP, OSPF, VXLAN, EVPN)
3. Management plane: How engineers access, troubleshoot, configure, and monitor infrastructure

The management plane is especially important when it comes to the WAN and the production network. That’s because this plane is vulnerable in traditional network architectures.

How The WAN Becomes a Single Point of Failure

When everything is working fine, management access is like air: you don’t even need to think about it. You can SSH into a router, change a firewall config, or restart a hanging server. It’s very straightforward because of one thing: the WAN is what connects you to the infrastructure.

But this is a problem. This means that you really don’t have a dedicated management plane, since management access relies on the production network. When the WAN fails, the production network is what you need to fix — but you no longer have access to it.

Here’s what happens when the WAN fails:

• Data plane disruption: Applications and users lose connectivity.
• Control plane disruption: Routing and network operations are jeopardized.
• Management plane disruption: Engineers lose remote access to the devices they need to troubleshoot.

When you can’t reach the equipment you need to fix, a production outage turns into an entirely different issue.

What Actually Goes Wrong When the WAN Fails?

ISP and Carrier Outages

Sometimes, the service provider itself has an outage and there’s not much you can do about it. Recent data from Cisco ThousandEyes shows just how often these events occur. During the week of August 24-30, 2026, ThousandEyes observed 297 global ISP outage events, 203 of which were in the United States alone.

Outages instantly become a remote-management problem in these cases when the WAN is the only path back to a site that’s gone offline. You’re left with two options: Wait for your provider to restore the connection, or send an engineer to the affected site(s). Both can easily take hours, sometimes days, before normal operations are restored.

Fiber Cuts and Physical Disruptions

What happens when a construction crew accidentally sends a backhoe bucket or an auger straight through an underground fiber run? If you rely on the WAN for management access, physical disruptions like these could have you waiting for days before they’re fixed.

Cloudflare tracked more than 180 Internet disruptions during 2025, including those caused by cable cuts, power outages, and extreme weather. Some were brief, but others lasted for days. These failures are especially difficult (and frustrating) because you could be hundreds of miles away, and your devices could be perfectly healthy, but the path you use to reach them is physically broken.

Routing and Configuration Errors

Increasing network complexity directly contributes to configuration and change-management problems. Uptime Institute’s 2025 analysis found that 23% of impactful outages were attributed to IT and networking issues, like routing errors, incorrect configs, and firmware problems.

The network equipment is exactly what you need to fix, but a configuration error has eliminated the path you need to access it.

Third-Party Network Failures

There’s an entire ecosystem of ISPs, carriers, cloud providers, colocations, and other third party networks. If any part of this ecosystem fails, it can affect your ability to reach remote infrastructure.

Uptime Institute reports that 39% of those surveyed experienced an outage caused by a third-party networking issue. Essentially, you can lose access to your own infrastructure because of a failure you don’t control.

That’s the common thread with all these outage scenarios. When a WAN failure takes down your production network, your management access goes down with it.

How To Stay Operational During WAN Failures

Build an Isolated Management Infrastructure

Adding redundancy seems like the logical solution. But having true operational resilience means going beyond having a second WAN circuit. You need to be able to confidently answer this question:

Can you still reach the infrastructure if the primary network fails?

If the answer is no, you still have a shared-fate problem: your management access depends on the same infrastructure you’re trying to recover.

An Isolated Management Infrastructure (IMI) solves this by creating a dedicated management layer that is completely independent of the production network. IMI does not rely on the WAN, routers, switches, or other infrastructure that carry production traffic. Instead, IMI provides a separate path that’s purpose-built for maintaining and recovering production infrastructure. Think of it as a safety net that keeps business from crashing down if there’s a primary network outage.

What Does IMI Look Like?

Isolated Management Infrastructure is made up of several components:

1. A separate management network that is logically and physically isolated from production infrastructure. This is created by deploying out-of-band serial consoles.
2. Independent connectivity using 5G, satellite, and/or other link types that don’t rely on terrestrial infrastructure. These provide access to your OOB serial consoles, and thus your production equipment.
3. Direct access to the variety of critical IT, including via serial, Ethernet, USB, KVM, and power interfaces. This gives you management access to your entire equipment stack.

This completely changes the recovery process. There’s no more waiting for the WAN to come back or dispatching an engineer to the site. Engineers can use the IMI to pinpoint affected equipment, make config changes, and/or completely rebuild systems.

Take Mercado Libre for example. This Latin America e-commerce giant experienced an unexpected outage at one of their distribution hubs. But because they had installed ZPE Systems’ Nodegrid as their IMI, they were able to keep operating as if nothing happened. “The solution paid for itself with just this one outage,” they said. Read the full Mercado Libre case study here.

Get in Touch For a Demo of AI Resilience

Our engineers will walk you through the best practices and show you Nodegrid’s capabilities first-hand. See how easy it is to point, click, and manage your distributed AI fleet. Fill out the form to get started.

Designing AI infrastructure that engineers can recover when things break

AI infrastructure is expanding fast across enterprise networks. GPU clusters are powering model training, while edge inference systems process data in real time across industries like finance, manufacturing, retail, and telecommunications. Most AI environments are working well from the perspective of the computing infrastructure.

But with many deployments scaling beyond centralized data centers and into distributed edge environments, network engineers are facing a familiar problem: How do you maintain control when failures happen?

This is an important question to address because AI environments amplify the operational impact of failures. Large GPU clusters, distributed edge locations, and latency-sensitive workloads all depend on reliable network connectivity. When something breaks, recovery becomes just as important as redundancy.

AI Infrastructure Places Different Demands on the Network

Traditional enterprise applications are generally tolerant of brief interruptions. A user might reconnect to an application or retry a transaction without significant impact. But AI workloads are much less forgiving.

Large language model (LLM) training, distributed inference pipelines, and real-time analytics depend on continuous communication between compute, storage, and networking resources. Interruptions have a ripple effect because modern AI environments rely on:

• GPU clusters connected by high-bandwidth Ethernet or InfiniBand fabrics
• East-west traffic that exceeds traditional north-south application flows
• Distributed storage systems supplying training data
• Kubernetes orchestration platforms scheduling GPU workloads
• Edge inference nodes continuously exchanging telemetry with centralized systems

For example, a failed top-of-rack switch, an incorrect routing policy, or a misconfigured spine switch can isolate entire GPU pools from storage or orchestration services. Even if the compute nodes remain operational, workloads stall while engineers work to restore connectivity.

For network engineers, resilience is now a job of ensuring the infrastructure remains observable, accessible, and recoverable especially when failures occur.

Understanding the Different Network Planes

Before we go further, it helps to distinguish between three separate but closely related parts of the network, which will help understand where failures happen and their effects:

Production (or data) plane: carries application traffic, including AI training data, inference requests, storage traffic, and user communications. For example, when you submit a prompt to an AI application, the production plane is what carries your prompt, the model’s responses, and supporting data.

Control plane: runs the protocols that determine how traffic moves through the network, including BGP, OSPF, EVPN, VXLAN, spanning tree, and other routing and switching functions. Think of this like the network’s navigation system. The control plane decides the best path for traffic to take, but it doesn’t carry the traffic itself.

Management plane: provides the interfaces engineers use to monitor, configure, and recover infrastructure. This includes SSH, HTTPS, APIs, SNMP, serial console access, intelligent PDUs, and out-of-band management. This is the path that engineers use to log into devices, push config changes, and recover systems during an outage.

Image: The three network planes: The data plane, which carries the workload; the control plane, which decides where the workload goes; and the management plane, which gives engineers access to manage and recover equipment.

Where AI Infrastructure Failures Actually Happen

Control Plane Failures That Disrupt The Management Plane

Not every outage begins with a hardware failure. In fact, a lot of incidents originate in the control plane with:

• BGP policy mistakes
• OSPF adjacency failures
• EVPN/VXLAN configuration errors
• VLAN or VRF misconfigurations
• ACL changes that unintentionally block management traffic

These issues may leave routers, switches, and servers fully operational but unreachable over the production network. If management traffic shares the same infrastructure as application traffic, engineers lose SSH, API, and monitoring access right when they need it most.

This is why it’s important to completely separate the management network from the production network (more on this below).

Automation Mistakes That Create a Big Blast Radius

Infrastructure-as-Code (IaC) and network automation are great for improving consistency, but they also increase the blast radius of configuration errors. A single Ansible playbook, Terraform deployment, or automation workflow can unintentionally affect hundreds of devices simultaneously.

When that happens, engineers need a recovery path that does not depend on the production network remaining operational. This is one of the reasons out-of-band management continues to be standard practice in hyperscale and service provider environments.

GPU Clusters That Are Left Idle

AI infrastructure is built around the most expensive compute resources in the data center. Experts project global AI investments to exceed $1 trillion in 2026, and it’s easy to see why. A single GPU server can cost hundreds of thousands of dollars, and production AI clusters often consist of dozens or hundreds of these systems working together.

Model training requires GPUs to operate as a coordinated cluster. If a network issue isolates part of the cluster or prevents nodes from communicating with storage or orchestration platforms, the training job stalls or fails altogether. Even though the GPUs are powered on, they’re no longer doing productive work.

In inference environments, the impact is different but just as significant. An unreachable edge AI node that stops processing camera feeds, sensor data, or real-time transactions reduces application performance or forces workloads to fail over to other locations.

Idle hardware is just the beginning of the costs. Organizations also lose productive compute time, delay model development, miss service-level objectives (SLOs), and increase operational overhead while engineers work to restore connectivity.

No infrastructure is immune to failures, so eliminating every outage isn’t realistic. The goal instead is to minimize Mean Time to Recovery (MTTR) by giving engineers immediate access to diagnose and recover systems, so expensive GPU resources spend more time running workloads and less time waiting for someone to restore the network.

Addressing a Common Question About AI

One common response to conversations about AI resilience is: “Our AI infrastructure is already working fine. Why do we need to change anything?”

It’s a valid question. After all, if it ain’t broke…

But this underscores something we’ve been talking about for years. It’s not about redesigning networks that already work just fine. It’s about building the ultimate safety net that’ll save you from the growing operational risks of scaling your infrastructure.

Managing a GPU cluster inside a data center is completely different than managing a fleet of AI systems spread across hundreds of remote sites. Engineers need to retain visibility and control when the inevitable routing mistake, WAN outage, or misconfig happens. Resilience needs to be built into the architecture.

Resilience Should Be Built Into AI Architecture

The networking industry has spent decades designing highly available production networks through redundant links, resilient routing protocols, and fault-tolerant hardware. AI infrastructure deserves the same level of attention, but redundancy alone isn’t enough.

Traditional high-availability designs focus on strengthening the production and control planes. Equally important is strengthening the management plane and being able to confidently answer this question: “When an engineer loses connectivity to a remote GPU cluster at 2am, what path do they have to recover it?”

If the answer depends on the production network coming back first, the architecture still has a single point of failure. This is why it’s key to fully separate management from production. This isolation lets engineers diagnose, control, and recover AI infrastructure exactly when it matters most.

So, “Why do we need to change anything?” It’s not really about changing the architecture. It’s more about adding an alternate management path that lets you recover systems even if the production network is down.

That’s where out-of-band and IMI come in.

IMI: The Evolution of Out-of-Band

Out-of-band management is usually thought of as a console server attached to a few routers, meant to give engineers backup access in case the main network goes down. But modern out-of-band — what’s called Isolated Management Infrastructure (IMI) — has much more operational capability, including the ability to fully rebuild systems.

An IMI includes:

• Serial console access to routers, switches, firewalls, storage arrays, and GPU servers
• Independent Ethernet management interfaces
• 5G/LTE or satellite connectivity for WAN independence
• Remote power management through intelligent PDUs
• Secure jump-host capabilities with centralized authentication
• Automated recovery workflows triggered by monitoring platforms

Image: Isolated Management Infrastructure provides the operational capabilities to fully rebuild and restore networking systems, even if the network is offline.

This creates a management plane that remains operational regardless of whether the production network is available. So instead of having to rebuild connectivity before troubleshooting can begin, engineers can connect via 5G/LTE or satellite to immediately investigate logs, inspect device health, restore configurations, or power-cycle failed systems.

ZPE is Essential to Resilient Architecture

ZPE’s solutions are an essential component of resilient network architecture. They’re used by many organizations as an independent operational layer alongside the production network. At each AI or edge site, one ZPE Nodegrid device can aggregate management access for routers, spine and leaf switches, firewalls, GPU servers, storage, PDUs, and hypervisor/compute platforms.

Image: ZPE’s solutions consolidate multiple functions into a single device and aggregate management access to a variety of AI production infrastructure, with the ability to connect via RS-232, Ethernet, USB, and OCP interfaces.

This IMI gives engineers secure access during routing failures, WAN outages, failed software upgrades, config errors, and many other outage scenarios. Nodegrid devices can also host virtualized functions (like routing and firewalls) to help further consolidate the overall tech stack. This gives organizations a consistent operational model no matter how many AI sites they have or where their workloads are deployed.

Get in Touch For a Demo of AI Resilience

Our engineers will walk you through the best practices and show you Nodegrid’s capabilities first-hand. See how easy it is to point, click, and manage your distributed AI fleet. Fill out the form to get started.