Every IT person knows that network updates are routine. Sometimes they can work perfectly, and other times, the update messes things up and you have to roll back to the last good configuration.

But what do you do when the rollback goes wrong?

Here’s my first-hand experience with this exact scenario.

We recently implemented what was intended to be a routine update for our engineering network. The change timeframe, internal signoff, test coverage, and rollback strategy were all set up. Every step was even pre-documented. The setup was textbook.

But if you’ve been in IT for more than five minutes, you know that upgrades don’t always fail in the first stage.

The initial steps had gone smoothly and we had a sense of confidence. But midway through, this one failed. It damaged a core routing service and stopped our ability to reach our remote sites.

No big deal. We had a step-by-step rollback plan that we validated in the lab. We even walked through it with a dry run.

Then things took an unexpected turn.

Our rollback failed!

Why? In the background, one dependent service was automatically upgraded. This silently triggered a chain reaction. We found ourselves dealing with Entra ID login loops, DHCP failures, and version mismatches across multiple services. Our internal DNS collapsed. With DNS gone, so was access to our identity provider, our management tools, and even our door badge system.

The access control system was no longer available to us. It was one of those nights. The rollback was supposed to save us, but what could save us from the rollback?

Luckily, we had out-of-band management

We were saved by Out-of-Band (OOB) management through our ZPE architecture.

We used secure OOB serial and cellular failover. That gave us direct control of the devices, even when the core network was down and identity services were unreachable. We stayed operational.

Image: The out-of-band management path is a dedicated access network. It acts as a safety net for instances when a rollback fails and recovery processes must take place.

Fortunately, we had already segmented the engineering network from the business network. That isolation meant the failure didn’t spread. We could take our time rebuilding the broken pieces without impacting customer operations or internal productivity tools.

All services were back up and running within a few hours.

What did we learn?

I’m posting this because a lot of IT teams, particularly those in growth-stage businesses, neglect early architecture segmentation or OOB access. It is considered a “Phase 2” assignment. However, it’s the only way out should things go wrong, which they will.

Here’s what we learned:

• The quality of your assumptions determines how well your rollback strategy works. A rollback that depends on “nothing else changes” is fragile by design.
• DNS, Identity (like Entra ID), and VPN are interdependent. They form a delicate triangle, and when one goes, the others often follow.
• Out-of-Band is a fundamental design need, not just a catastrophe recovery tool. If you’re managing remote or critical infrastructure, there is no substitute for direct, independent access.
• Documentation is important. Access is more important. All the runbooks in the world won’t help if you can’t reach the system that runs them.

Prepare for failure. Walk through your worst-case scenario. Don’t count on luck to save you.

“No one ever tells you this part…”

My name is Ahmed Algam. I am a Network & Systems Administrator for ZPE Systems – A Brand of Legrand, with over 20 years of experience in network administration, system infrastructure, Microsoft ERP solutions, and enterprise IT management. I have a B.S. in Computer Science and will soon complete my Master’s of Information and Data Science.

In the early days of my IT career, I learned how to build systems from scratch, configure networks, and apply patches.

Like many, I was trained to focus on the obvious goals: keep things running, keep everything secure, and automate what I can.

But what no one taught me? What to do when everything goes dark – literally.

That’s exactly what happened recently.

ZPE’s Fremont branch lost power unexpectedly and without notice from our provider.

One by one, our services went down

• ESXi Hosts
• Backup Servers
• VPN Tunnels
• Core Routers and Switches

Here is the part that I wish I knew 20 years ago…

You won’t be rescued by dashboards, spreadsheets, or documentation when IT goes dark. What WILL save you is system design, specifically out-of-band management.

And for which I am lucky that design did save us.

Without Out-of-band (OOB), I would have had to spend the whole night at the office manually rebooting, configuring, and troubleshooting everything. It’s a nightmare for IT admins because you might get the call while you’re attending your kids’ sporting events, attending college courses, or spending quality time with your family. IT emergencies can really intrude on your life outside of work. It’s just part of the job.

But I was so grateful to have OOB because it gave me a separate path dedicated to recovery, which was just what I needed. I was able to instantly remote-into my infrastructure without leaving home.

Image: Isolated Management Infrastructure uses out-of-band management (OOBM) serial consoles to access production devices when they are offline.

Within minutes, I was able to:

• Remotely connect through our OOB console
• Restart critical infrastructure
• Monitor recovery independently of the production path

I didn’t have to head for the office or change the plans I had with my family. With our OOB system in place, I knew that I could fix the problem, have services restored before sunrise, and still get a good night’s sleep.

This wasn’t luck

It was the result of:

• Planning for the worst-case scenario, not just the routine
• Having OOB in all essential areas
• Testing access methods instead of assuming they’ll just work
• Separating management traffic from production flows
• Staying calm with an architecture designed to withstand chaos

Even highly-skilled IT teams come to a full standstill during disruptions

It has nothing to do with a lack of talent or skill. The reason is their inability to access the malfunctioning systems.

So here’s my advice to every IT professional:

• Now is the time to prepare for the worst
• Make an OOB network
• Separate management paths from production (and test access!)

Because when the lights go out, that’s when real IT begins.