Cisco admins are scrambling to patch a critical flash memory overflow vulnerability in over 200 Cisco Systems IOS XE-based models of wireless access points (APs), caused by a recent flawed software update.If the issue is not corrected quickly, the AP’s memory will become so flooded that new software updates will be blocked and the AP rendered insecure, or possibly even bricked.The problematic library update causes a specific log file in the flash memory of affected access points to grow by about 5MB a day. Over time, Cisco said in an advisory this week, this could consume “a big portion” of the available memory space.“The longer an AP runs the affected software, the higher the probability that a software download will fail due to insufficient space,” the advisory says.Analyst Rob Enderle of the Enderle Group said that ‘buggy logs’ are a common trope in networking. But, he added, “this particular case is dangerous because it targets the physical limitations of flash memory on hardware that is notoriously difficult to access once it becomes bricked or enters a boot loop. In the world of networking, this is a high-impact, medium-rarity event.”He explained, “what makes this unique is the Catch-22 it creates. To fix the bug, you must upgrade the software. However, the bug itself prevents the device from having enough space to download the fix. If an admin waits too long, the device may require manual, physical intervention or become permanently stuck in a boot loop.”Johannes Ullrich, dean of research at the SANS Institute, called this particular problem uncommon, although he acknowledged flash memory space in IoT devices like access points is limited and may fill up from time to time.“But,” he added, “there is a bigger issue: A competent [vendor] vulnerability management program must always include verification that the patch was indeed applied as expected. There are many reasons why a patch may not be applied correctly, and this is just one way a patch may fail to apply.”Kellman Meghu, CTO of incident response firm DeepCove Cybersecurity, said overflowing a fixed device’s memory due to a bug “would have me rather annoyed with this vendor. This is very rare in my experience, and something that was an issue way back when storage costs were a factor. I would expect my vendor to be able to clean and manage storage for fixed devices. If this device is supported, this would be an RMA [return merchandise authorization] or fix issue, and expectation [for vendor action] would be right away/proactive.”[Related content: Cisco Webex SSO flaw]Affected are access points running IOS XE versions 17.12.4, 17.12.5, 17.12.6, and 17.12.6a. These include Cisco Catalyst 9130AX series APs, as well as 9130AX models with a Stadium Antenna, Catalyst 91361, 91621, 9163E, 91641, 9166D1, and IW9167 series APs, and Wi-Fi 6 Outdoor APs,There are two ways for admins to solve the problem: Download a Cisco tool called WLANPoller, which automates execution of a fix across multiple APs, or manually use the show boot command on each device to look into the boot partition and see if it has enough space for an upgrade. Greater detail on the necessary action is in the Cisco advisory.Cisco says a mandatory precheck of an AP’s status should be run as close to the scheduled maintenance window as possible. But because the affected log file grows daily, Enderle said, “you sure don’t want to wait until [AP] failure.” Manual fixing will probably take 5-10 minutes of active work per AP, he cautioned, plus another 15-20 minutes soak time to make sure the fix takes if the AP does have room for the upgrade. But if the AP has space problems, the time per device could jump to around 20-45 minutes.And if the AP has failed, then it would take one to two hours to fix, he added, and would need physical access to the device.Using WLANPoller will make the process faster, he added.Enderle said that if an admin finds an AP whose flash memory is already too full to upgrade, a reboot sometimes clears temporary buffers or allows a small window for a manual transfer. However, with this specific log bug, a reboot may not be enough if the file is persistent. Admins should contact Cisco for the emergency cleanup script before attempting a mass push, he said.Ultimately, Enderle said, the pushing of a flawed update is a supply chain integrity issue. CSOs should ask their teams, ‘Do we have monitoring in place for hardware health metrics (CPU, RAM, Flash), or only for ‘Up/Down’ status?’ An AP that is Up but has 0MB of free flash memory is a liability, he said.CSOs should look at this vulnerability as a Critical Availability Risk, he added. “While it isn’t a data breach, the potential for a site-wide Wi-Fi outage (due to failed automated updates or boot loops) can halt business operations,” he noted, adding that CSOs should also enforce a policy where even “minor library updates” are still tested in a lab environment for seven to 14 days. “This 5MB/day log growth would likely have been caught in a lab before hitting a production fleet of 5,000 APs,” Enderle said.This article originally appeared on NetworkWorld.