Providing Out-of-Band Connectivity to Mission-Critical IT Resources

Breaking Down The 2023 Ragnar Locker Cyberattacks

Breaking Down the 2023 Ragnar Locker Cyberattacks

This article was written by James Cabe, CISSP, a 30-year cybersecurity expert who’s helped major companies including Microsoft and Fortinet.

Throughout 2023, several organizations were successfully hit by Ragnar Locker cyberattacks. The affected victims spanned the globe and were forced to shut down much of their critical operations, while the attackers demanded tens of millions of dollars in ransom payments. Despite the group being taken down by law enforcement in October, organizations are re-evaluating their defensive measures — and more importantly, their recovery strategies — to combat these attacks.

If you read my previous articles about the ongoing MOVEit breach and the ransomware that hit MGM, you probably know that isolation is key. It helps you fight through attacks by cutting the kill chain, so that you can restore services quickly without reinfection.

Who Carries Out Ragnar Locker Cyberattacks?

Recent Ragnar Locker cyberattacks were carried out by the Dark Angels Team cybercriminal group. Dark Angels Team’s modus operandi is to breach a company’s defenses, spread laterally, and steal data that can be used to extort the target company. The approach they take involves gaining access to the Windows domain controller, where they deploy ransomware. They encrypt devices using Windows and ESXi encryptors, which gives organizations little recourse aside from taking their critical systems offline in order to stop the spread.

Dark Angels banner

How Do Ragnar Locker Cyberattacks Start?

Ragnar Locker breaches, like all ransomware attacks, require a kill chain that must first be initiated. MITRE ATT&CK defines this as the ‘initial,’ and in these attacks, the initial comes from social engineering. Email stuffing is often the tactic of choice, whereby the attacker sends an email that appears to have a trail of replies or forwards (see the example below). Email trails like this trick spam filters and land directly in the target’s inbox. When an employee clicks a malicious link inside the email, the attack kicks off.

An email showing an example of email stuffing.

Image: Email stuffing is used by marketers and threat actors alike to bypass spam filters.

How Do Companies Discover Ragnar Locker Cyberattacks?

After the Ragnar Locker cyberattack kicks off, the bad link uses Java to load the locker ransomware, then a series of batch scripts installs a payload consisting of virtual box emulation software. This emulation software takes over and encrypts the host, and displays the ransomware message (see image below).

A Ragnar Locker ransomware message shown in a notes file.

Image: A Ragnar Locker ransomware message showing on encrypted devices.

How Do Ragnar Locker Cyberattacks Spread?

The attack spreads by gaining access to Windows domain controllers and then attacking the management interfaces of the VMware ESXi machines. Most organizations don’t properly segment or isolate these management interfaces. This makes them especially vulnerable even to older Babuk ransomware source code that is an ESXi encryptor. Basically, the attackers only need to gain access to the management network, and then they can attack the production network.

From Intel471: “VMware’s ESXi is called a ‘bare metal’ hypervisor because the underlying hardware on which it is installed doesn’t need an operating system. ESXi allows the hardware to be utilized for multiple virtual machines (VMs), which saves on hardware costs. ESXi is a fruitful target for attackers since it may be connected to several VMs and the storage for them. Security experts warn ransomware actors have built specific binaries to target these systems. Groups joining this trend include HelloKitty, Black Basta, Cheerscrypt and GwisinLocker.”

They continue, “Over the last few years, several vulnerabilities have been identified in ESXi, including CVE-2021-21974. The vulnerability is a heap overflow vulnerability within Open Service Location Protocol (OpenSLP), which is a network discovery tool. The vulnerability is remotely exploitable over port 427, and has a Common Vulnerability Scoring System Version 3.0 (CVSSv3) base score of 8.8. It’s suspected that it may be the vulnerability exploited in this attack. VMware said that “significantly out-of-date products” were targeted with vulnerabilities that had been addressed. It affects ESXi versions 7.0 before ESXi70U1c-17325551, 6.7 before ESXi670-202102401-SG and 6.5 before ESXi650-202102101-SG. Due to other vulnerabilities in OpenSLP, VMware disabled OpenSLP starting in 2021 in ESXi versions 7.0 U2c and ESXi 8.0, which is the current version.”

Ultimately, these attacks exploit a combination of a lack of management plane isolation to the VMware management interfaces, specifically on port 427 (OpenSLP), and a lack of patching and updating. Organizations also typically lack a backup authentication mechanism for the control plane, as well as Privileged Access Management, which are both good fallback options.

How Can Companies Stop Ragnar Locker Cyberattacks?

Ragnar Locker ransomware and other attacks are successful because companies don’t employ proper management plane isolation. Attackers can gain access to VMware management interfaces, and then they essentially have the keys to the kingdom. That’s it. No amount of defense can save you.

If you recall CISA’s binding operational directive, they call for an isolated management infrastructure. This is what we refer to as IMI. Rather than serving as a defense, like we think of traditional cybersecurity products, the IMI is an architecture that allows you to fight back. It’s your quick-reaction force, your cavalry, your secret weapon that ensures you always have a counterattack ready to deploy.

IMI is infrastructure that is dedicated — and most importantly, fully isolated from production assets — to ensuring operations can recover quickly from breaches and outages. Here’s a graphical breakdown:

Isolated Management Infrastructure diagram

The IMI includes all of the tools you need for rerouting traffic, decommissioning affected gear, wiping/re-imaging devices, and restoring infrastructure. You can also incorporate automation to speed the process along and make recovery something that happens in minutes or hours at the most. Aside from being completely isolated from production assets, the IMI itself is also segmented and employs zero trust practices. This means that you and only you have access to your secret weapon for cutting the ransomware kill chain.

How Do You Use Isolated Management Infrastructure?

An IMI can host an IRE (Isolated Recovery Environment), which is used to cut off all user data and remote access (except for OOB) to an entire infected site. A properly implemented recovery environment should automate most of these activities to speed up the recovery. One of the first considerations is the requirement for a secondary organization in your IAM that is not attached to normal operations. This is what is known as a set of “Break the Glass” accounts. These are known in military circles but have made it into formal practice as part of a strong playbook for ransomware. Once you do this, you can instantiate selected Zero Trust remote access to the site using credentials that are not in the scope of the attack, and then bring up a communications channel for a virtual war room using software like Rocket Chat, Jitsi, Slack, or other standalone communications tools that are installable on the IRE environment. 

Avoiding normal authentication methods or IAM and normal communication channels is required for the integrity of the recovery and strengthens the recovery playbook. During this time, no email may be used that is associated directly with the organization. Ideally, email should never touch an account that is associated with it either.

The next step is to create a new set of clean side networks that do not directly connect to the main backbone or put it behind another firewall for triage good/bad. Using a sniffer software running on the IRE, the recovery team can then run a passive scan or an active scanner against all machines continuing to try to send email to Exchange/M365. You can give access to people that are deemed good (not sending traffic) but lock off (with an EDR) the ability to open Outlook for a while, while keeping them on the web email. From there, continue working through to find all the sending drivers to see if they have a good backup. If not, back up the infected drive for offline data retrieval for later. Then re-image while scanning the UEFI BIOS during boot (if needed, run an IPMI scan). If the site has a list of assets that are considered crown jewels, prioritize these.

Once you have a segmented “clean side” established with all the network services required to operate the site (DNS, IAM, DHCP), then Internet access can be restored to this site on a limited basis; which means only out-bound communications, nothing in-bound. Restorative operations can continue apace. making sure that the infected side assets are captured in backup for later forensics following chain-of-custody if damages exceeding insurance limits are found to be the case. This is decided in the war room.

Download the Isolated Management Infrastructure Blueprint

Now is the time to lay the groundwork for your IMI so you can fight back against ransomware. Download the Network Automation Blueprint, which gives you a step-by-step guide to building your Isolated Management Infrastructure.

Get in touch with me!

True security can only be achieved through resilience, and that’s my mission. If you want help shoring up your defenses, building an IMI, and implementing a Resilience System, get in touch with me. Here are links to my social media accounts:

Nodegrid OS and ZPE Cloud achieve industry’s highest security with Synopsys

Synopsys and ZPE validation

How do you address security across the software development life cycle?

“Security is the cornerstone of ZPE’s infrastructure management solutions,” says Koroush Saraf, Vice President of Product Management and Marketing at ZPE Systems. “Our automation platform touches every aspect of our customers’ critical infrastructure, from networking and firewall gear, to servers, smart PDUs, and everything else in their production network. The ZPE portfolio is architected with the strongest security and implemented with the same level of scrutiny.”

Given the critical nature of enterprise networking, security is paramount to ZPE’s customers.

“The average time taken to apply patches and fix vulnerabilities can be more than 205 days,” says Saraf. “This is due to many reasons: limited resources and time, concerns that something may break, or in some cases, admins don’t even know that a critical patch is available. That’s why ZPE takes on the responsibility for customers. They’re assured that the systems running their infrastructure are running the latest, most secure software. And if a patch fails, our built-in undo button reverts to a safe configuration before any damage can be done.”

Saraf adds, “Like with all modern organizations, ZPE uses a complex mix of proprietary, open source, and third-party software obtained through a variety of sources from the software supply chain. Think third-party libraries, packaged software from ISVs, IoT and embedded firmware, and especially open source components. In fact, studies show that over three-quarters of the code in any given application is likely to be open source.”

“Most third parties won’t provide the source code behind their software,” notes Saraf. “But the question remains whether that supplier is as security-conscious as ZPE. Again, we found the solution with Synopsys, which gives us insight into any third-party software we include without requiring access to the source code.”

The solution: Comprehensive security testing with Synopsys AST

Different security solutions focus on different aspects of vulnerability detection and risk mitigation. By layering multiple solutions such as static analysis, dynamic analysis, and software composition analysis, ZPE covers a wide range of potential vulnerabilities, ensuring that code quality and security issues are identified at various stages during the software development life cycle and across different types of code.

Table showing ZPE Systems' security in layers

Coverity® provides the speed, ease of use, accuracy, industry standards compliance, and scalability to develop high-quality, secure applications. Coverity identifies critical quality defects and security vulnerabilities as code is written, early in ZPE’s development process when they are easiest to fix. Coverity seamlessly integrates automated security testing into CI/CD pipelines, supports existing development tools and workflows, and can be deployed either on-premises or in the cloud.

WhiteHat™ Dynamic is a software-as-a-service dynamic application security testing solution that allows businesses to quickly deploy a scalable web security program. No matter how many websites or how often they change, WhiteHat Dynamic can scale to meet any demand. It provides security and development teams with fast, accurate, and continuous vulnerability assessments of applications in QA and production, applying the same techniques hackers use to find weaknesses This enables ZPE to streamline the remediation process, prioritize vulnerabilities based on severity and threat, and focus on remediation and its overall security posture.

Black Duck® helps ZPE identify supply chain security and license risks even when it doesn’t have access to the underlying software’s code. This is a critical security tool for the modern software supply chain. Black Duck Binary Analysis can scan virtually any software, including desktop and mobile applications, third-party libraries, packaged software, and embedded system firmware. It quickly generates a complete Software Bill of Materials (SBOM), which tracks third-party and open source components, and identifies known security vulnerabilities, associated licenses, and code quality risks.

The result: A notable reduction of CVEs

“One of the outcomes from taking a comprehensive, layered approach to security testing has been a notable reduction in CVEs on the systems we deploy,” says Saraf.

“I think a lot of industry players don’t give enough attention to patching CVEs. They wait until after a security incident, or until a customer specifically asks. Unfortunately, it’s normal to see unpatched, outdated software running on critical infrastructure. The Equifax breach of 2017 is just one example that exposed the personal data of millions. It’s a particular problem with IoT and embedded devices—many of those systems get installed and forgotten. But it’s another attack surface, especially if you use the equipment for critical infrastructure automation.”

“ZPE’s goal is to reduce the attack surface of our systems to as close to zero as possible, either by making sure that software vulnerabilities are identified and addressed, and that our software is running the most secure and up-to-date versions. It’s an ongoing process— what is vulnerability-free today won’t necessarily be so tomorrow—which is why ZPE always stays security-conscious. I think the company’s commitment to security has positioned ZPE as a trusted partner for enterprises seeking secure automation solutions for their critical infrastructure needs.

 

Download the document for details about Synopsys and ZPE Systems

How to Fight the latest Ransomware Attacks

Nodegrid plus Synopsys is the most secure platform for Isolated Management Infrastructure (IMI). This architecture is recommended by the FBI and CISA, and allows you to fight back when ransomware strikes. Check out our latest IMI articles from cybersecurity veterans James Cabe and Koroush Saraf, who have helped companies including Fortinet, Microsoft, and Palo Alto Networks.

OOB Network Management Software Tools

ZPE – Comparison-2(1)

Network management software tools enable administrators to provision, monitor, and maintain networks and network infrastructure without manually touching each individual device or service. We’ve previously covered the best network management software for both cloud-based and hardware-based networks. This post dives deeper into the hardware-based network management tools deployed in on-premises or private cloud environments via serial consoles (a.k.a. console servers, serial console servers, serial console routers, or serial switches).

These network management software tools provide out-of-band (OOB) management access to connected network infrastructure in data centers and remote sites. Serial consoles directly interface with network systems and devices, creating an isolated management network that doesn’t depend on the production LAN, WAN, or ISP. OOB management ensures continuous remote access even during major outages, so teams can troubleshoot and recover branch offices, edge computing sites, and other remote environments without costly truck rolls or managed service support. OOB console servers also unify management of all connected infrastructure, giving administrators a single platform to monitor, control, and automate the entire distributed network architecture.

Looking for a new out-of-band network management device? Read our guide:

Comparing the Best Out-of-Band Management Devices

We compare offerings from the four best OOB network management software tool providers and discuss the features, advantages, and disadvantages of each to help network teams make the best choice for their environment.

The best OOB network management software tools

Platform

Key Features

ZPE Systems Nodegrid Manager/ZPE Cloud

  • 5G/4G LTE, Wi-Fi, POTS, or fiber OOB and failover support

  • On-premises or cloud-based software options

  • Robust hardware and software security including 2FA and SAML 2.0

  • x86 CPU and Linux-based Nodegrid OS support Guest OS and Docker containers

  • Vendor-neutral software integrates with third-party tools like Ansible, Chef, Python, and Ruby

  • Extends ZTP and other automation to legacy and mixed-vendor infrastructure

  • OOB support over IPMI, ILO, DRAC, CIMC, vSerial, and KVM

  • Device auto-discovery via network scan and custom probes

  • Power management integrated with serial session

  • Supports full range of environmental monitoring sensors

Vertiv Avocent DSView

  • 4G LTE OOB and failover support

  • HTML5 KVM and Serial viewers

  • Telnet/PuTTY serial interfaces

  • Session log, report, and archive

  • Data Center Zone definitions

  • ZTP, SOAP API, Python, Perl, PDU, and UPS automation

  • Schedule and on-demand firmware management

  • Web secure 2048 SSL certificate

  • Two-factor authentication (2FA)

PerleVIEW

  • 4G LTE, Wi-Fi, POTS, or fiber OOB and failover support

  • Health monitoring and event handling

  • Integrates with SNMP NMS

  • Automated device discovery

  • Device CLI scripting

  • Configuration backup, firmware, and change management

  • Single sign-on for in-band connections

  • Device PING tool

  • Audit trail log

Opengear Lighthouse

  • 4G LTE or fiber for OOB and failover

  •  x86 processor can run Guest OSes and automation

  • Supports over 100 power vendors’ equipment

  •  Automatic port discovery

  • Lighthouse playbooks for streamlined automation

  • Integrated SAML and AAA authentication

  • Opengear NetOps modules support Bash, Docker, Pearl, Python, and Ruby

  • SSH direct to consoles

  • Keystroke logging 

Disclaimer: This comparison was written by a 3rd party in collaboration with ZPE Systems using data gathered from publicly available data sheets and admin guides, as of 10/13/2023. Please email us if you have corrections or edits, or want to review additional attributes: Matrix@zpesystems.com

ZPE Systems Nodegrid Manager/ZPE Cloud

ZPE Systems offers two network management software tools based on their Nodegrid line of serial consoles and integrated edge routers. Nodegrid Manager is on-premises software, and ZPE Cloud is a cloud-based tool, but both provide out-of-band management access to on-premises infrastructure in data centers, branches, private clouds, and other remote sites. Nodegrid hardware offers a variety of connectivity options for OOB, failover, and WAN, including Wi-Fi and 5G. Nodegrid also supports a full range of environmental monitoring sensors for greater control over conditions in remote deployments.

Nodegrid solutions are protected by robust hardware and software security features, including UEFI Secure Boot, an embedded firewall with selectable cryptographic protocols, and 2FA and SAML 2.0 authentication. The x86 CPU architecture and Linux-based OS support Guest OSes and Docker containers, so Nodegrid boxes can directly host third-party software for security, automation, orchestration. Both Nodegrid Manager and ZPE Cloud are also completely vendor-neutral, supporting third-party automation scripts and tools including RedHat Ansible, Chef, Python, Ruby, and more. Nodegrid can even extend ZTP and other automation to legacy infrastructure that otherwise wouldn’t support it.

Nodegrid’s open platform essentially makes it a customizable network management multi-tool that’s capable of consolidating many different software solutions and network services. The primary limitation to this vendor-neutrality is that some other providers may require the purchase of additional licenses to run their tools on the Nodegrid platform.

For a real-world example of Nodegrid’s ability to streamline network management, read our case study, Vapor IO: Re-architecting the Internet.
.

Pros

Cons

  • 5G/4G LTE, Wi-Fi, POTS, or fiber OOB and failover support

  • On-premises or cloud-based software options

  • Robust hardware and software security including 2FA and SAML 2.0

  • x86 CPU and Linux-based OS support Guest OS and Docker containers

  • Vendor-neutral software integrates with third-party tools like Ansible, Chef, Python, and Ruby

  • Extends ZTP and other automation to legacy and mixed-vendor infrastructure 

  • Other vendors may require additional licenses to run VNFs and other tools on Nodegrid

Vertiv Avocent DSView

The DSViewTM network management software works with Vertiv Avocent out-of-band serial consoles, the ACS 800 and ACS 8000, which use 4G LTE cellular for out-of-band management and failover. Like the other options on this list, DSView consolidates the management of connected network infrastructure in a single web-based platform. In addition to serial port control, this software also provides keyboard/video/mouse (KVM) and MIB-based (Management Information Base) controls. It also supports environmental monitoring via sensors, but ACS8000 serial console servers only come with one sensor port – and the ACS800 doesn’t have one at all.

DSView management software includes automation support for Zero Touch Provisioning (ZTP), SOAP API, Python, and Perl scripts, and automated PDU (power distribution unit) and UPS (uninterruptible power supply) management. It also provides console event logging and notifications, such as “dying gasp” alarms when systems unexpectedly lose power. However, the software is not extensible with third-party automation or orchestration integrations, and the ACS800 and ACS8000 serial console hardware solutions run on an ARM CPU architecture that can’t support VMs or Docker.
.

Pros

Cons

  • 4G LTE OOB and failover support
  • Environmental monitoring sensor support
  • ZTP, SOAP API, Python, Perl, PDU, and UPS automation
  • Serial, KVM, and MIB-based controls
  • FIPS 410-2 cryptography and 2FA security
  • No support for third-party automation or orchestration
  • Doesn’t run VMs or Docker containers
  • Serial console lacks an embedded firewall

PerleVIEW

PerleVIEW network management software tools are based on Perle’s IOLAN SCG & SCR OOB console servers. The platform includes automated device discovery, automatic event handling, device scripting, ZTP, and collection of device statistics and health statuses. Perle focuses on meeting the FCAPS network management framework created by the International Organization for Standardization (ISO). FCAPS standards for Fault management, Configuration, Administration, Performance management, and Security management, and PerleVIEW’s features target each of these areas. For example, fault management is streamlined through automated event handling with customizable alerts and SNMP probes.

However, the software does not support any third-party integrations, limiting its automation and security capabilities. Aside from automated device discovery and event handling, PerleVIEW doesn’t offer any automation for end devices. It also doesn’t support two-factor authentication (2FA), and only supports single sign-on (SSO) for in-band connections.
.

Pros

Cons

  • 4G LTE, Wi-Fi, POTS, or fiber OOB and failover support
  • Automated device discovery, event handling, health and device log collection, and device scripting
  • Integrates with SNMP NMS (network management station)
  • Single sign-on for in-band connections only
  • Follows FCAPS standards
  • No support for third-party automation or orchestration
  • Doesn’t run VMs or Docker containers
  • Lacks support for environmental monitoring sensors

Opengear Lighthouse

Lighthouse is a network management platform that works with Opengear console servers, such as the OM2200 Operations Manager and the CM8100. The base version of Lighthouse provides out-of-band management access, integrated SAML and AAA authentication, and integrations with third-party notification and alert systems. Lighthouse supports over 100 power vendors’ equipment to streamline PDU and UPS management, and the x86 processor can run Guest OSes and automation.

With additional licenses, Lighthouse software is extensible with Opengear NetOps modules that provide greater automation capabilities with support for Bash, Docker, Pearl, Python, and Ruby. The upgraded Automation edition also includes ZTP for end devices and RESTful APIs. Beyond that, however, automation and orchestration abilities are limited because Lighthouse isn’t vendor-neutral, and Opengear has a limited ecosystem of integrations.
.

Pros

Cons

  • 4G LTE or fiber for OOB and failover
  • Integrated SAML and AAA authentication
  • Supports over 100 power vendors’ equipment
  • x86 processor can run Guest OSes and automation
  • Opengear NetOps modules and ZTP available for automation
  • Only upgraded Automation edition supports ZTP for end devices
  • NetOps modules require additional licenses
  • Limited integration ecosystem

Key takeaways

All four options on this list provide out-of-band (OOB) access to remote network infrastructure, giving teams a lifeline in case of production failures or outages. Vertiv Avocent’s DSView software provides some automation capabilities and environmental monitoring, but doesn’t support third-party automation, VMs, or Docker containers. PerleVIEW focuses on meeting FCAPS network management standards but has very limited automation and security capabilities. Opengear Lighthouse is extensible with NetOps modules and other automation features, but they require additional licenses or fees, and you’re limited to Opengear’s ecosystem of integrations.

ZPE Systems offers Nodegrid Manager as an on-premises application or ZPE Cloud as a cloud-based tool. Both options use vendor-neutral hardware and software to create an open platform you can customize with your favorite third-party apps and integrations. ZPE’s OOB network management software tools enable end-to-end automation over a highly secure and reliable out-of-band control plane for a more resilient network infrastructure.

 

Deploy the best OOB network management software tools with ZPE Systems

Reach out to ZPE to learn how to build a resilient network with Nodegrid OOB network management software tools. Contact Us

Living Spaces Furniture: Scaling to 50 sites with only 3 network staff

Collapsing the stack and centralizing management helps Living Spaces accelerate scaling across the U.S.

Blake Johnson – Living Spaces Furniture Network Architect

“We’ve quadrupled business, but Nodegrid is actually shrinking our workload, especially as we implement new automation. It’s a gamechanger for network folks. Period.” — Blake Johnson, Network Architect, Living Spaces Furniture

Living Spaces is a prominent furniture retailer in the United States. Their store locations include large showrooms, where customers can view furnishings for indoor and outdoor spaces, and plenty of warehouse space for storing on-hand inventory. These locations must serve customers with responsive shopping experiences, which depend on the network infrastructure.

Increasing demand helped Living Spaces grow out of its home state of California, into states including Arizona, Colorado, Oklahoma, Texas, and others. Their out-of-band infrastructure was crucial to spinning up new locations and maintaining operations. But they faced a significant problem: this infrastructure was incredibly complex and costly, requiring many dedicated cellular and out-of-band devices at each location. See why their three-person network team needed a solution that could:

  • Reduce costs and eliminate the need for $300,000 per year in SIM contracts
  • Reduce workloads and risks, by centralizing management and minimizing entry points
  • Accelerate deployments by allowing automation

ISP Network Architecture

An engineer installs fiber optic patch cables at a customer site that’s part of an ISP network architecture.
Internet service providers (ISPs) are the backbone of modern society, responsible for connecting businesses, services, and people to the Internet and to each other. ISP networks are vast, distributed, and complex, making them challenging to manage effectively. However, failing to do so has major consequences. For example, in July of 2022, Rogers Communications in Canada suffered a network system failure after a maintenance update, causing an outage that lasted more than 15 hours and took down emergency services and other critical infrastructure.

An ISP network architecture must be designed for resilience to prevent major incidents from occurring that affect consumers, communities, and the provider’s reputation. But significant challenges stand in the way, including a reliance on legacy infrastructure, and an inability to troubleshoot and recover failed gear remotely. This post discusses why these challenges exist and what ISPs can do to overcome them.

ISP network architecture challenges

Many ISP networks lack resilience because providers are failing to adapt to a rapidly changing landscape. With networks growing larger and more complex every day, new technologies like AI (artificial intelligence) and software-defined networking are needed to manage infrastructure efficiently and deliver innovative services. Additionally, providers get stuck in a break-fix cycle that leaves teams struggling to maintain service level agreements or focus on innovation. Let’s look at the causes of these challenges and discuss how to build more resilient ISP network architectures.

Legacy infrastructure creates technical debt and hampers growth

The challenge:

The solution:

Reliance on legacy systems creates technical debt and prevents ISPs from implementing new technologies

Vendor-neutral platforms like Gen 3 serial consoles extend automation, software-defined networking, and other advanced technologies to legacy infrastructure until it can be replaced.

Internet service providers often have a network architecture that’s a mix of new and legacy infrastructure. However, engineers with the experience to support older solutions are no longer working in the field, either because they’ve been promoted to leadership positions or retired. When legacy hardware fails, inexperienced engineers need time to overcome this skills gap, and ISPs may even need to bring in consultants. This increases the cost of failures, creating what’s known as “technical debt” – when a solution is more expensive to support than the value it brings to the organization.

In addition, ISPs can improve network resilience and provide better service to customers, by adopting new technologies like AI, 5G, software-defined networking (SDN), and Network as a Service (NaaS). But legacy hardware hampers the ability to adopt these technologies. For example, NaaS abstracts the need for MPLS circuits and customer-premises gear, making architectures more cost-effective and improving the customer experience. NaaS brings SDN concepts like programmable networking and API-based operations to WAN & LAN services, hybrid cloud, Private Network Interconnect, and internet exchange points. It optimizes resource allocation by considering network and computing resources as a unified whole and attempts to automate as much as possible. The trouble is, ISPs struggle to implement NaaS and other beneficial new technologies because their legacy hardware simply can’t support it.

Solution: Legacy modernization with a vendor-neutral platform

The ideal solution is to replace legacy infrastructure with modern hardware and software that supports the latest technologies. But for many ISPs, an overhaul like this is too costly and intensive. The next-best option is to bridge the gap with a vendor-neutral network modernization platform that extends automation, AI, and 5G connectivity to otherwise unsupported systems.

For example, serial consoles (also known as terminal servers, console servers, and serial console switches) provide remote management access to network infrastructure. The newest generation of these devices, known as Gen 3, are vendor-neutral by design so that they can control third-party and legacy hardware. Through a combination of built-in features and integrations, Gen 3 serial consoles can use technology like zero-touch provisioning (ZTP), AIOps, and automated configuration management to control connected hardware that otherwise wouldn’t support it. Some solutions, such as the Nodegrid platform from ZPE Systems, can even directly host SDN and NaaS software from other vendors, so ISPs can start implementing network improvements right away while they gradually replace their outdated infrastructure.

Physical infrastructure is difficult to manage and troubleshoot remotely

The challenge:

The solution:

ISP network admins can’t respond to changing environmental conditions or recover failed hardware remotely

Environmental monitoring connected to an out-of-band (OOB) management solution ensures continuous remote access on a dedicated, isolated network that enables fast and cost-effective recovery.

ISP network architectures involve a great deal of physical infrastructure, which is often deployed in remote edge sites and customer premises. Even with software- or service-based network solutions, hardware is needed to host that software, and the physical environment for that hardware is often less than ideal. Drastic weather changes, power outages, and other unexpected scenarios can happen without notice and rapidly bring down an ISP network. These events often cut off remote management access as well, making troubleshooting and recovery difficult, time-consuming, and expensive. In fact, supporting this physical infrastructure often consumes so much time and effort that it prevents ISPs from focusing on delivering better services and software to their customers.

Solution: Out-of-band management with environmental monitoring

The first part of the solution involves monitoring the environment that houses remote, physical infrastructure. An environmental monitoring system uses sensors to detect changes in airflow, temperature, humidity, and other conditions that affect the operation of network hardware. These sensors give ISPs a virtual presence in edge deployments and customer sites so they can quickly respond to changing conditions before systems overheat or circuitry corrodes.

The second part involves providing management teams with reliable remote access to physical infrastructure that won’t go down if there’s a production network outage. Out-of-band (OOB) management solutions use serial consoles with dedicated network interfaces used just for management access. This creates a parallel, out-of-band network that’s completely isolated from production network services and infrastructure. Additionally, many serial consoles use cellular connectivity via 4G or 5G to OOB access, providing a wireless lifeline to connect, troubleshoot, and restore remote infrastructure. OOB management allows ISPs to troubleshoot and recover failed hardware remotely, even during total network outages, so they can get services back up and running faster and less expensively.

The environmental monitoring system should run on the OOB network so remote admins can continue to monitor conditions while they recover failed hardware. The out-of-band management solution also needs to be vendor-neutral so ISPs can deploy third-party automation, AI, and NaaS on the OOB network. For example, Nodegrid Gen 3 serial consoles provide OOB, environmental monitoring, and a vendor-neutral platform to host third-party software at the edge. Nodegrid even enables fully automated responses to changing environmental conditions in those edge environments before admins are aware of a problem.

To learn more about building a resilient, automated network infrastructure with Nodegrid, download the Network Automation Blueprint.

Download Now

ISP network architecture resilience with Nodegrid

ISP network architectures must be resilient, meaning service providers must find a way to bridge the gap between legacy and modern systems while ensuring continuous remote access to manage, troubleshoot, and recover hardware at the edge. The Nodegrid ISP network infrastructure solution  from ZPE Systems is a vendor-neutral, Gen 3 platform that delivers legacy modernization, environmental monitoring, out-of-band management, and much more.

Nodegrid delivers ISP network architecture resilience in a single platform

Request a free demo to see Nodegrid ISP network architecture solutions in action.

Watch a Demo

Edge Management and Orchestration

shutterstock_2264235201(1)

Organizations prioritizing digital transformation by adopting IoT (Internet of Things) technologies generate and process an unprecedented amount of data. Traditionally, the systems used to process that data live in a centralized data center or the cloud. However, IoT devices are often deployed around the edges of the enterprise in remote sites like retail stores, manufacturing plants, and oil rigs. Transferring so much data back and forth creates a lot of latency and uses valuable bandwidth. Edge computing solves this problem by moving processing units closer to the sources that generate the data.

IBM estimates there are over 15 billion edge devices already in use. While edge computing has rapidly become a vital component of digital transformation, many organizations focus on individual use cases and lack a cohesive edge computing strategy. According to a recent Gartner report, the result is what’s known as “edge sprawl”: many individual edge computing solutions deployed all over the enterprise without any centralized control or visibility. Organizations with disjointed edge computing deployments are less efficient and more likely to hit roadblocks that stifle digital transformation.

The report provides guidance on building an edge computing strategy to combat sprawl, and the foundation of that strategy is edge management and orchestration (EMO). Below, this post summarizes the key findings from the Gartner report and discusses some of the biggest edge computing challenges before explaining how to solve them with a centralized EMO platform.

Key findings from the Gartner report

Many organizations already use edge computing technology for specific projects and use cases – they have an individual problem to solve, so they deploy an individual solution. Since the stakeholders in these projects usually aren’t architects, they aren’t building their own edge computing machines or writing software for them. Typically, these customers buy pre-assembled solutions or as-a-service offerings that meet their specific needs.

However, a piecemeal approach to edge computing projects leaves organizations with disjointed technologies and processes, contributing to edge sprawl and shadow IT. Teams can’t efficiently manage or secure all the edge computing projects occurring in the enterprise without centralized control and visibility. Gartner urges I&O (infrastructure & operations) leaders to take a more proactive approach by developing a comprehensive edge computing strategy encompassing all use cases and addressing the most common challenges.

Edge computing challenges

Gartner identifies six major edge computing challenges to focus on when developing an edge computing strategy:

Gartner’s 6 edge computing challenges to overcome

Enabling extensibility so edge computing solutions are adaptable to the changing needs of the business.

Extracting value from edge data with business analytics, AIOps, and machine learning training.

Governing edge data to meet storage constraints without losing valuable data in the process.

Supporting edge-native applications using specialized containers and clustering without increasing the technical debt.

Securing the edge when computing nodes are highly distributed in environments without data center security mechanisms.

Edge management and orchestration that supports business resilience requirements and improves operational efficiency.

Let’s discuss these challenges and their solutions in greater depth.

  • Enabling extensibility – Many organizations deploy purpose-built edge computing solutions for their specific use case and can’t adapt when workloads change or grow.  The goal is to attempt to predict future workloads based on planned initiatives and create an edge computing strategy that leaves room for that growth. However, no one can really predict the future, so the strategy should account for unknowns by utilizing common, vendor-neutral technologies that allow for expansion and integration.
  • Extracting value from edge data – The generation of so much IoT and sensor data gives organizations the opportunity to extract additional value in the form of business insights, predictive analysis, and machine learning training. Quickly extracting that value is challenging when most data analysis and AI applications still live in the cloud. To effectively harness edge data, organizations should look for ways to deploy artificial intelligence training and data analytics solutions alongside edge computing units.
  • Governing edge data – Edge computing deployments often have more significant data storage constraints than central data centers, so quickly distinguishing between valuable data and destroyable junk is critical to edge ROIs. With so much data being generated, it’s often challenging to make this determination on the fly, so it’s important to address data governance during the planning process. There are automated data governance solutions that can help, but these must be carefully configured and managed to avoid data loss.
  • Supporting edge-native applications – Edge applications aren’t just data center apps lifted and shifted to the edge; they’re designed for edge computing from the bottom up. Like cloud-native software, edge apps often use containers, but clustering and cluster management are different beasts outside the cloud data center. The goal is to deploy platforms that support edge-native applications without increasing the technical debt, which means they should use familiar container management technologies (like Docker) and interoperate with existing systems (like OT applications and VMs).
  • Securing the edge – Edge deployments are highly distributed in locations that may lack many physical security features in a traditional data center, such as guarded entries and biometric locks, which adds risk and increases the attack surface. Organizations must protect edge computing nodes with a multi-layered defense that includes hardware security (such as TPM), frequent patches, zero-trust policies, strong authentication (e.g., RADIUS and 2FA), and network micro-segmentation.
  • Edge management and orchestration – Moving computing out of the climate-controlled data center creates environmental and power challenges that are difficult to mitigate without an on-site technical staff to monitor and respond. When equipment failure, configuration errors, or breaches take down the network, remote teams struggle to meet resilience requirements to keep business operations running 24/7. The sheer number and distribution area of edge computing units make them challenging to manage efficiently, increasing the likelihood of mistakes, issues, or threat indicators slipping between the cracks. Addressing this challenge requires centralized edge management and orchestration (EMO) with environmental monitoring and out-of-band (OOB) connectivity.

    A centralized EMO platform gives administrators a single-pane-of-glass view of all edge deployments and the supporting infrastructure, streamlining management workflows and serving as the control panel for automation, security, data governance, cluster management, and more. The EMO must integrate with the technologies used to automate edge management workflows, such as zero-touch provisioning (ZTP) and configuration management (e.g., Ansible or Chef), to help improve efficiency while reducing the risk of human error. Integrating environmental sensors will help remote technicians monitor heat, humidity, airflow, and other conditions affecting critical edge equipment’s performance and lifespan. Finally, remote teams need OOB access to edge infrastructure and computing nodes, so the EMO should use out-of-band serial console technology that provides a dedicated network path that doesn’t rely on production resources.

Gartner recommends focusing your edge computing strategy on overcoming the most significant risks, challenges, and roadblocks. An edge management and orchestration (EMO) platform is the backbone of a comprehensive edge computing strategy because it serves as the hub for all the processes, workflows, and solutions used to solve those problems.

Edge management and orchestration (EMO) with Nodegrid

Nodegrid is a vendor-neutral edge management and orchestration (EMO) platform from ZPE Systems. Nodegrid uses Gen 3 out-of-band technology that provides 24/7 remote management access to edge deployments while freely interoperating with third-party applications for automation, security, container management, and more. Nodegrid environmental sensors give teams a complete view of temperature, humidity, airflow, and other factors from anywhere in the world and provide robust logging to support data-driven analytics.

The open, Linux-based Nodegrid OS supports direct hosting of containers and edge-native applications, reducing the hardware overhead at each edge deployment. You can also run your ML training, AIOps, data governance, or data analytics applications from the same box to extract more value from your edge data without contributing to sprawl.

In addition to hardware security features like TPM and geofencing, Nodegrid supports strong authentication like 2FA, integrates with leading zero-trust providers like Okta and PING, and can run third-party next-generation firewall (NGFW) software to streamline deployments further.

The Nodegrid platform brings all the components of your edge computing strategy under one management umbrella and rolls it up with additional core networking and infrastructure management features. Nodegrid consolidates edge deployments and streamlines edge management and orchestration, providing a foundation for a Gartner-approved edge computing strategy.

Want to learn more about how Nodegrid can help you overcome your biggest edge computing challenges?

Contact ZPE Systems for a free demo of the Nodegrid edge management and orchestration platform.

Contact Us