Network engineers need powerful tools to keep digital services online and customers happy. This is especially true during economic downturn, when organizations must freeze hiring and put more strain on existing staff. Revenue relies on network availability, and with experts predicting a recession this winter, significant operational challenges are inevitable for most organizations.

The burden of overcoming these challenges falls on network engineers. Success means maintaining reliable services and reaping any professional benefits (salary increases, promotions, etc.). Failure, on the other hand, means the very realistic possibility of major business losses and job cuts, including yours.

In order to make sure you don’t fall into the latter scenario, here are five must-have tools and techniques to help network engineers overcome these challenges.

Tool 1. OOBI-LAN™

Out-of-band (OOB) management is an essential part of a network engineer’s toolkit. At the conceptual level, out-of-band is meant to provide management access to production equipment, even if the production equipment is offline.

One major problem is that many organizations invest a lot of time and money into their production infrastructure, but not into any dedicated OOB infrastructure. In other words, they deploy OOB solutions that rely in part on their production equipment, such as OOB VLANs connected to in-band switches. All it takes is a mistake, misconfiguration, or attack to bring down the production and management networks, leaving network engineers to rebuild the entire system from scratch while their services remain offline to customers. This is simply not acceptable in a slow economy, where the business’ resources and revenue are already too thin.

From the pandemic lockdowns, organizations have learned that they need a way to more quickly recover their network locations. According to the Uptime Institute’s 2022 Outage Analysis, outages lasting longer than 24 hours increased to nearly 30% in 2021. This has led many to build dedicated OOB infrastructure for the LAN (OOBI-LAN). They deploy a serial console locally to establish connectivity to the management ports of their sensitive equipment. Network engineers must use this serial console to access their production infrastructure. This serial console minimizes the attack surface since it’s the only device connected to the Internet, and allows network engineers to restore services even if production equipment is down.

Tool 2. OOBI-WAN™

A critical tool for network engineers is out-of-band that enables remote WAN management. But typically, organizations employ a WAN management strategy that also relies on their production infrastructure, such as for creating VPN tunnels for management traffic. If a VPN tunnel becomes broken or the production gear fails, network engineers are suddenly left without remote access to their equipment.

Aside from a lack of availability, traditional OOB access comes with real security risks. Exposing LTE modems to the Internet, leveraging untrusted third-party VPN services, using OOB hardware that’s old and unpatched, and worse — exposing the management port of devices to public Internet. All of these are attack surfaces, any of which can give access to your infrastructure and be used as the pivot point to get to the rest of the infrastructure.

Image: Management access depends on production equipment to establish VPN tunnels. 

On top of their OOBI-LAN, organizations have built dedicated OOB infrastructure for the WAN (OOBI-WAN – there’s a Star Wars reference somewhere in there) for added resilience against these scenarios.

Image: OOBI-WAN and OOBI-LAN create a fully separate out-of-band infrastructure that can be used to completely rebuild production infrastructure. 

OOBI-WAN uses MPLS, IPsec, or SD-WAN links to create an overlay network dedicated specifically to management traffic. This gives network engineers private access to their infrastructure for management and troubleshooting, essentially creating a completely separate OOB network that does not rely on any part of the production network. OOBI-WAN lets network engineers use their WAN connection to remotely access their OOBI-LAN and fully rebuild their distributed networks, regardless of the state of their production infrastructure.  

A key part of OOBI-WAN is the inherent security that is built at all layers. To build secure OOBI-WAN, the best practice is to use OOBI-SDWAN™ which automates the building of VPN tunnels between all the nodes that need to be managed. OOBI-SDWAN provides the expected auto-VPN feature which means VPN encryption keys remain secure, as they don’t need to be copied/pasted/typed into multiple third-party devices. OOBI-SDWAN also ensures that an SLA is provided on the OOBI network along with observability dashboards of connectivity and the access state of the network. The combination of OOBI-SDWAN with a zero trust security framework is the best way to gain reliability in a way that reduces your risk.

Tool 3. Fully independent automation infrastructure

Another tool that network engineers are becoming familiar with is automation. Network automation codifies repetitive tasks to reduce workloads for configuration management, compliance, and troubleshooting. During a slow economy, being able to scale an IT team’s efforts is especially valuable to business operations and end customers.

There is one major concern, however: having automation that runs loose and begins destroying the network, much like a bull in a China shop. Network engineers typically must learn new automation tools and programming languages, which requires trial and error. And because there is a lack of a best practice reference architecture, teams don’t know any better than to automate directly on the production network. This causes anxiety, as one mistake could bring down the network, cause catastrophic losses, and leave network engineers without an efficient way to recover.

Image: The orange section describes dedicated automation infrastructure used for safely implementing automation.

In recent years, teams have been deploying automation on dedicated infrastructure like their OOB network. This automation infrastructure sits between the production infrastructure and the orchestration infrastructure, and serves as a safe way to build an automation pipeline. Open, Linux-based appliances like the Nodegrid Net SR combine a variety of functions and can host automation tools, like those for observability and analytics, version control, and source of truth. This independent automation infrastructure allows network engineers to ensure the integrity of configuration changes, software updates, and remediation protocols in an out-of-band manner, rather than testing directly on the production network. They can scale their capabilities, and in case of errors, roll back to a golden configuration that keeps services online.

Tool 4. Remote access to local jump box

Network engineers have another tool at their disposal: the jump box (a.k.a. jump server, jump host). A jump box hosts tools for maintaining operations, and these include file servers, image storage, configuration management tools, and troubleshooting commands. The jump box is a valuable asset for normal operations and for restoring services, such as when a device fails and needs its image rebuilt.

The issue with jump boxes is that they are typically a separate device that requires power, cooling, rack space, and maintenance. Some jump boxes also require on-site technicians to physically connect to the equipment needing repair.

Many organizations have adapted by upgrading their OOB infrastructure with appliances that can run full virtual machines (VMs). These can run all the tools mentioned above as well as with Docker containers, while consolidating power consumption, cooling resources, and rack space. The OOB appliance can double as a jump box. Combined with OOBI-LAN and OOBI-WAN, network engineers get remote access to re-image a device, diagnose DNS/routing issues, and perform any other necessary tasks. Key point is that discrete jump boxes – Like the Intel NUC — to be converted to virtual jump boxes running on a secure OOB platform like the Nodegrid Service routers.

Tool 5. Smart hands

A final way that network engineers get help through a slow economy is by outsourcing to so-called ‘smart hands.’ Employing smart hands means involving a third-party expert who can take on some of the IT workload. It’s a viable strategy, especially for teams feeling crushed by corporate belt tightening and the resulting mountain of tasks.

Companies who take this approach must be aware that the skills of smart hands varies greatly, as does the cost. This means it’s essential to strike a balance between which tasks to outsource, and which tasks to keep in house. For example, many organizations use smart hands for simple jobs such as replacing hardware and installing equipment at new sites. For more specialized jobs that require deeper knowledge of the environment, such as fixing a misconfigured IP address or route, teams use in-house personnel. This balance helps organizations get the support they need to keep operations running.

Get a cheat sheet to implement these tools fast

Some companies thrive during economic downturn, because they’ve intelligently placed these tools within their network architecture. Over the past decade, we’ve worked with these companies — including the largest tech giants — to describe in painstaking detail how they set up their infrastructure. We just released all 40+ pages of this validated reference architecture, complete with implementation diagrams and examples.

It’s called the network automation blueprint and it combines all of these tools. Network engineers can confidently answer questions like:

• How do we meet SLAs with a smaller workforce?
• How can we keep sites operating without physical access to equipment?
• How can we perform weekly updates/patching without breaking things?

The blueprint is your cheat sheet to implementing a more resilient network, and fast. Click the button below to download your copy now.