3 Real Lessons in Network Resilience

By Ahmed Algam

Over the past few months, I’ve seen many real-world examples of this. These incidents drove home a hard truth about architecting for network resilience:

Out-of-Band (OOB) access isn’t optional. It’s essential.

Here are three short but very real stories that made this point crystal clear.

1. The Power Outage That Didn’t Stop Us

Our Fremont office went dark. Completely dark. There was a power outage and our provider failed to give us a heads-up, so it took us by surprise.

No power meant routers, ESXi hosts, Proxmox servers, backup systems, and even Wi-Fi were knocked offline. It was a total blackout.

But we weren’t scrambling. We had architected a true out-of-band path using LTE. Even with the production network down, we still had a way in.

From miles away, we diagnosed the problem, rebooted critical infrastructure, and got things running again before most people even noticed.

Lesson: Your recovery plan is only as good as your last mile. If your failover path isn’t truly independent, it’s not a plan – it’s wishful thinking.

2. The Engineer Who Locked Himself Out

A partner’s network went down during a routine change. Not uncommon. What was uncommon? The fact that they had no access to fix it.

All their management traffic – SSH, APIs, everything – was routed through the same production network that had just failed. When that network died, so did their ability to reach any routers or switches. The team was flying blind.

We got the call, helped them recover, and discussed IMI best practices afterward.

Lesson: Never mix management and user traffic. You need a control plane that exists outside your data plane, especially when uptime is mission-critical.

3. “That’s So Obvious Now…” – The Failover Fail

A customer had the right idea: install a 4G modem as a failover path. This is common, and it’s a great way to gain access in case the main path goes down.

But the modem was physically wired into their primary Cisco router.

When that router failed (power surge), so did the modem. To make things worse, their monitoring agent was running in-band. So when the network collapsed, their monitoring did, too. No visibility, no access, no control.

We pointed out this problem. Then we suggested running the agent on dedicated OOB gear instead. Their response?

“That’s so obvious now…but I didn’t even think about it.”

Lesson: Monitoring doesn’t help if it goes down with everything else. Build it into your OOB infrastructure. Make it resilient, not just present.

What I Want You To Take Away From These Stories

Resilience isn’t just about having backup tools or extra hardware. It’s about designing for failure.

It’s about building your architecture so that even if the core goes dark, you still have eyes and hands on the network.

Out-of-Band isn’t a Luxury. It’s your Lifeline. Make sure to Architect it like one.

Here Are Resources to Help Build Your OOB Lifeline

• Rollback Gone Wrong: How Out-of-Band Saved Our Engineering Backbone
• After The Firewall Fails: How Gen 3 Out-of-Band Cuts the Ransomware Killchain
• Out-of-Band Deployment Guide
• The CrowdStrike Outage: How to Recover Fast and Avoid the Next Outage

Get Hands-On Help From Our Engineers

My colleagues have years of experience architecting these resilience practices. Please use the form to send us a message and get help with your specific use case.

Every IT person knows that network updates are routine. Sometimes they can work perfectly, and other times, the update messes things up and you have to roll back to the last good configuration.

But what do you do when the rollback goes wrong?

Here’s my first-hand experience with this exact scenario.

We recently implemented what was intended to be a routine update for our engineering network. The change timeframe, internal signoff, test coverage, and rollback strategy were all set up. Every step was even pre-documented. The setup was textbook.

But if you’ve been in IT for more than five minutes, you know that upgrades don’t always fail in the first stage.

The initial steps had gone smoothly and we had a sense of confidence. But midway through, this one failed. It damaged a core routing service and stopped our ability to reach our remote sites.

No big deal. We had a step-by-step rollback plan that we validated in the lab. We even walked through it with a dry run.

Then things took an unexpected turn.

Our rollback failed!

Why? In the background, one dependent service was automatically upgraded. This silently triggered a chain reaction. We found ourselves dealing with Entra ID login loops, DHCP failures, and version mismatches across multiple services. Our internal DNS collapsed. With DNS gone, so was access to our identity provider, our management tools, and even our door badge system.

The access control system was no longer available to us. It was one of those nights. The rollback was supposed to save us, but what could save us from the rollback?

Luckily, we had out-of-band management

We were saved by Out-of-Band (OOB) management through our ZPE architecture.

We used secure OOB serial and cellular failover. That gave us direct control of the devices, even when the core network was down and identity services were unreachable. We stayed operational.

Image: The out-of-band management path is a dedicated access network. It acts as a safety net for instances when a rollback fails and recovery processes must take place.

Fortunately, we had already segmented the engineering network from the business network. That isolation meant the failure didn’t spread. We could take our time rebuilding the broken pieces without impacting customer operations or internal productivity tools.

All services were back up and running within a few hours.

What did we learn?

I’m posting this because a lot of IT teams, particularly those in growth-stage businesses, neglect early architecture segmentation or OOB access. It is considered a “Phase 2” assignment. However, it’s the only way out should things go wrong, which they will.

Here’s what we learned:

• The quality of your assumptions determines how well your rollback strategy works. A rollback that depends on “nothing else changes” is fragile by design.
• DNS, Identity (like Entra ID), and VPN are interdependent. They form a delicate triangle, and when one goes, the others often follow.
• Out-of-Band is a fundamental design need, not just a catastrophe recovery tool. If you’re managing remote or critical infrastructure, there is no substitute for direct, independent access.
• Documentation is important. Access is more important. All the runbooks in the world won’t help if you can’t reach the system that runs them.

Prepare for failure. Walk through your worst-case scenario. Don’t count on luck to save you.

“No one ever tells you this part…”

My name is Ahmed Algam. I am a Network & Systems Administrator for ZPE Systems – A Brand of Legrand, with over 20 years of experience in network administration, system infrastructure, Microsoft ERP solutions, and enterprise IT management. I have a B.S. in Computer Science and will soon complete my Master’s of Information and Data Science.

In the early days of my IT career, I learned how to build systems from scratch, configure networks, and apply patches.

Like many, I was trained to focus on the obvious goals: keep things running, keep everything secure, and automate what I can.

But what no one taught me? What to do when everything goes dark – literally.

That’s exactly what happened recently.

ZPE’s Fremont branch lost power unexpectedly and without notice from our provider.

One by one, our services went down

• ESXi Hosts
• Backup Servers
• VPN Tunnels
• Core Routers and Switches

Here is the part that I wish I knew 20 years ago…

You won’t be rescued by dashboards, spreadsheets, or documentation when IT goes dark. What WILL save you is system design, specifically out-of-band management.

And for which I am lucky that design did save us.

Without Out-of-band (OOB), I would have had to spend the whole night at the office manually rebooting, configuring, and troubleshooting everything. It’s a nightmare for IT admins because you might get the call while you’re attending your kids’ sporting events, attending college courses, or spending quality time with your family. IT emergencies can really intrude on your life outside of work. It’s just part of the job.

But I was so grateful to have OOB because it gave me a separate path dedicated to recovery, which was just what I needed. I was able to instantly remote-into my infrastructure without leaving home.

Image: Isolated Management Infrastructure uses out-of-band management (OOBM) serial consoles to access production devices when they are offline.

Within minutes, I was able to:

• Remotely connect through our OOB console
• Restart critical infrastructure
• Monitor recovery independently of the production path

I didn’t have to head for the office or change the plans I had with my family. With our OOB system in place, I knew that I could fix the problem, have services restored before sunrise, and still get a good night’s sleep.

This wasn’t luck

It was the result of:

• Planning for the worst-case scenario, not just the routine
• Having OOB in all essential areas
• Testing access methods instead of assuming they’ll just work
• Separating management traffic from production flows
• Staying calm with an architecture designed to withstand chaos

Even highly-skilled IT teams come to a full standstill during disruptions

It has nothing to do with a lack of talent or skill. The reason is their inability to access the malfunctioning systems.

So here’s my advice to every IT professional:

• Now is the time to prepare for the worst
• Make an OOB network
• Separate management paths from production (and test access!)

Because when the lights go out, that’s when real IT begins.