https://krebsonsecurity.com/2023/06/barracuda-urges-replacing-not-patching-its-email-security-gateways/ Advertisement [5] Advertisement [6] Krebs on Security Skip to content * Home * About the Author * Advertising/Speaking Barracuda Urges Replacing -- Not Patching -- Its Email Security Gateways June 8, 2023 25 Comments It's not often that a zero-day vulnerability causes a network security vendor to urge customers to physically remove and decommission an entire line of affected hardware -- as opposed to just applying software updates. But experts say that is exactly what transpired this week with Barracuda Networks, as the company struggled to combat a sprawling malware threat which appears to have undermined its email security appliances in such a fundamental way that they can no longer be safely updated with software fixes. [barracuda] The Barracuda Email Security Gateway (ESG) 900 appliance. Campbell, Calif. based Barracuda said it hired incident response firm Mandiant on May 18 after receiving reports about unusual traffic originating from its Email Security Gateway (ESG) devices, which are designed to sit at the edge of an organization's network and scan all incoming and outgoing email for malware. On May 19, Barracuda identified that the malicious traffic was taking advantage of a previously unknown vulnerability in its ESG appliances, and on May 20 the company pushed a patch for the flaw to all affected appliances (CVE-2023-2868). In its security advisory, Barracuda said the vulnerability existed in the Barracuda software component responsible for screening attachments for malware. More alarmingly, the company said it appears attackers first started exploiting the flaw in October 2022. But on June 6, Barracuda suddenly began urging its ESG customers to wholesale rip out and replace -- not patch -- affected appliances. "Impacted ESG appliances must be immediately replaced regardless of patch version level," the company's advisory warned. "Barracuda's recommendation at this time is full replacement of the impacted ESG." In a statement, Barracuda said it will be providing the replacement product to impacted customers at no cost, and that not all ESG appliances were compromised. "No other Barracuda product, including our SaaS email solutions, were impacted by this vulnerability," the company said. "If an ESG appliance is displaying a notification in the User Interface, the ESG appliance had indicators of compromise. If no notification is displayed, we have no reason to believe that the appliance has been compromised at this time." Nevertheless, the statement says that "out of an abundance of caution and in furtherance of our containment strategy, we recommend impacted customers replace their compromised appliance." "As of June 8, 2023, approximately 5% of active ESG appliances worldwide have shown any evidence of known indicators of compromise due to the vulnerability," the statement continues. "Despite deployment of additional patches based on known IOCs, we continue to see evidence of ongoing malware activity on a subset of the compromised appliances. Therefore, we would like customers to replace any compromised appliance with a new unaffected device." Rapid7's Caitlin Condon called this remarkable turn of events "fairly stunning," and said there appear to be roughly 11,000 vulnerable ESG devices still connected to the Internet worldwide. "The pivot from patch to total replacement of affected devices is fairly stunning and implies the malware the threat actors deployed somehow achieves persistence at a low enough level that even wiping the device wouldn't eradicate attacker access," Condon wrote. Barracuda said the malware was identified on a subset of appliances that allowed the attackers persistent backdoor access to the devices, and that evidence of data exfiltration was identified on some systems. Rapid7 said it has seen no evidence that attackers are using the flaw to move laterally within victim networks. But that may be small consolation for Barracuda customers now coming to terms with the notion that foreign cyberspies probably have been hoovering up all their email for months. Nicholas Weaver, a researcher at University of California, Berkeley's International Computer Science Institute (ICSI), said it is likely that the malware was able to corrupt the underlying firmware that powers the ESG devices in some irreparable way. "One of the goals of malware is to be hard to remove, and this suggests the malware compromised the firmware itself to make it really hard to remove and really stealthy," Weaver said. "That's not a ransomware actor, that's a state actor. Why? Because a ransomware actor doesn't care about that level of access. They don't need it. If they're going for data extortion, it's more like a smash-and-grab. If they're going for data ransoming, they're encrypting the data itself -- not the machines." In addition to replacing devices, Barracuda says ESG customers should also rotate any credentials connected to the appliance(s), and check for signs of compromise dating back to at least October 2022 using the network and endpoint indicators the company has released publicly . Update, June 9, 11:55 a.m. ET: Barracuda has issued an updated statement about the incident, portions of which are now excerpted above. This entry was posted on Thursday 8th of June 2023 04:17 PM Latest Warnings Time to Patch Barracuda Networks Caitlin Condon CVE-2023-2868 Email Security Gateway International Computer Science Institute Mandiant Nicholas Weaver Rapid7 Post navigation - Service Rents Email Addresses for Account Signups 25 thoughts on "Barracuda Urges Replacing -- Not Patching -- Its Email Security Gateways" 1. Larry June 8, 2023 Who will pay for the replacement hardware? I did not see in the announcement that Barracuda would replace the hardware at no charge to the customer, or that they would reimburse the labor costs required. Reply - 1. White Hat Bob June 8, 2023 Clearly, these devices are likely not "merchantable or fit for its intended purchase." Legally that could mean the manufacturer will be required to replace them without cost to the buyer regardless of warranty limits and exculpatory language in their various agreements. Consequential losses, such as the labor to replace units, are legally easier for the manufacturer to avoid, and they will likely be a matter of some negotiation, regulatory action, or litigation to determine who will pay these costs. The manufacturer may claim a defense of force majeure, a legal term for an act of god or an act of war, but that would be a reach if the attack could have been reasonably foreseen (or we're not soon at war with the state actor initiating the attack). Anyway you look at it; it's an existential crisis for the manufacturer. Reply - 2. Moe June 8, 2023 It's important that they get the pertinent point out that "impacted" devices are to be considered untrusted regardless of patch level after the fact, whether or not they're landfill or refurbishable. If they don't offer some sort of RMA replacement there would be a huge stink so I'd expect that. Reply - 3. KFritz June 9, 2023 Thank you for getting to the heart of the matter and putting into precise language what must have been in most readers' minds by the second paragraph! Reply - 2. Nobby Nobbs June 8, 2023 Any manufacturer of "security equipment" that fails this comprehensibly SHOULD be in existential crisis. If we allow the "Corporations are people too" argument to stand (it shouldn't), this person should receive the death penalty. Reply - 3. Ehud Gavron June 9, 2023 Thanks, Brian, and Krebs. This is good as a start. Looking forward to hearing the full nature of why the hardware should be tossed. While I sympathize with the people above, why is morr important than "who will line my pockets." Reply - 1. Schitts Creek June 9, 2023 Krebs does all the work. Brian is just the face of the operation. Reply - 2. A.A. June 9, 2023 Quite obviously, because the hardware is too compromised to be salvageable. I can't think of any other scenario where they 'd do this. Reply - 1. Moe June 9, 2023 Not necessarily, but at the moment don't trust end user patches to fully remove existing compromise. Patching only (maybe) halts a vector of initial compromise, not detect/remove subsequent payloads. It doesn't mean hardware is "unsalvageable," it means the process for salvaging is more intensive (or at this point unknown) so they can't recommend action other than unit replacement at this time. The advice is important to prevent users from patching over a deeper compromise unknowingly, assuming the device is now secure on that basis. They have to research a secure restoration process. It may be that it's simpler for the company to eat the cost of replacement with new units instead. We'll see. Reply - 2. Xeiran June 9, 2023 Can't help but wonder if the equipment *could* be salvaged but Barracuda has decided this is the perfect opportunity to force people to their cloud SaaS solution. Reply - 4. Dennis June 9, 2023 Sorry, but "Email Security Gateway" sending out malware sounds funny. The question is, who would trust such a vendor in the future? Reply - 5. Gunter Konigsmann June 9, 2023 There are unpatchable hardware problems, sometimes. But if the manufacturer tells me to replace hardware I always wonder if the problem really is hardware being unsafe or if the manufacturer doesn't want to support the old boxes any more/wants me to buy new ones anyway. Reply - 6. PHP June 9, 2023 Guess the boxes can't be customer booted from USB, or possible that they have UEFI bugs that have been left unpatched, so not even the bootloader can be trusted. Maybe there are evidence of hacked UEFI firmware. If we are there, then only hardware replacement can fix the issues, unless you replace the ROM, which is most likley soldered. I have been playing with different microelectronics, and on some the bootloader becomes corrupted, but then there usually is some way to pull pins high/low that will put a microprocessor into flash mode. But this might be impossible when everything is soldered. Reply - 7. Luke June 9, 2023 Its not clear anywhere I search: Are these Barracuda models or software that is past its end-of-life? Or is Barracuda pressing for removal of current and supported models of their technology? Either way, not great. But I have less sympathy for the customer that was using obsolete and no longer supported technology if that's the case. If its supported technology, should be able to RMA the device and get a newer unaffected model in return from the manufacturer. Until they run out anyway.. Reply - 1. BrianKrebs Post authorJune 9, 2023 Barracuda said the vulnerability affects a subset of devices running versions 5.1.3.001 - 9.2.0.006. That seems like a pretty broad range. Reply - 8. Steve June 9, 2023 This is a bold move by a company. Most would hide the issue because of the fear this would tarnish the brand. I love the honesty and integrity of the brand that admits they can't fix something and does what is right for their customers. Much respect! Reply - 1. Houston Vanhoy June 9, 2023 Yes, Steve, I agree. I am reminded of the 1982 Tylenol murders and the prompt, effective and open response by Johnson & Johnson. The murderers had used cyanide to contaminate a legitimate product. They were never caught. Tylenol is still a best-selling pain relief medicine. https://www.pbs.org/newshour/health/tylenol-murders-1982 Barracuda did not intentionally manufacture and sell defective devices. We may never know who created the malware. Reply - 9. Wayne June 9, 2023 Gee, I wonder how many of the corps that own these 11,000 devices are going to be replacing their Barracuda ESGs with ones from the same manufacturer? Personally, I'd be looking for a new brand. Reply - 1. Moe June 9, 2023 If the company offers 1:1 replacement I'd imagine most would take them up on that. Reply - 10. M June 9, 2023 Any indications the malware was introduced in the supply chain? Who will bear the cost of replacing the devices? Would this considered part of the support contract as an uncorrectable defect? Reply - 11. Richard Faulkner, J.D., LL.M., F.C.I.Arb., Dip.Intnl.Com.Arb. June 9, 2023 The victimized businesses probably have insurance claims with their own insurance companies under multiple lines of coverage. Document all replacement activities and costs, including delays, labor expenses, business interruption etc. Of course, any deductibles and Self Insurance Retentions apply. Be aware that Cyber Insurance Policies frequently have multiple sublimits and special deductibles for each category of loss. Have someone very knowledgeable about insurance coverage, especially manuscript Cyber Insurance coverages, review your policies immediately. If your insurance pays the claims, the insurance carrier will be subrogated and also able to assert claims to recover your deductibles and SIRs as well as carriers own payments against Barracuda. Reply - 12. Bob Collins June 9, 2023 Likely they will not replace hardware and push customers to the SaaS solution. Reply - 13. Jorge Pinto June 10, 2023 There were 2 cases similar to this in the past, one with Cisco ESA and the other one with Symantec Email Gateway. I know some people that preferred to replace their appliances instead of patching, however both vendors confirmed the patch was effective. I only know 3 other cases where the hardware was replaced, the Meltdown problem from Intel, NVidia with Spectre and Apple on some laptops because of a problem with bootcamp. Was ever something like this on a security appliance? Reply - 1. Dom June 10, 2023 Firmware compromise is nothing new. The NSA was using this tactic against HDD manufacturer bios/firmware. Basically they used the HDD firmware for persistence and could execute against several HDD major manufacturers. Reply - 14. DM June 10, 2023 My isp is most likely a victim of this issue. I have service and an email address through Windstream. I received an email with my password. Clients of the business I work for received emails with their passwords as well. Many of them suffered password resets of their social media and other accounts. Windstream has yet to own up to being compromised. Reply - Leave a Reply Cancel reply Your email address will not be published. Required fields are marked * [ ] [ ] [ ] [ ] [ ] [ ] [ ] Comment * [ ] Name * [ ] Email * [ ] Website [ ] [Post Comment] [ ] [ ] [ ] [ ] [ ] [ ] [ ] D[ ] Advertisement [7] Advertisement Mailing List Subscribe here Search KrebsOnSecurity Search for: [ ] [Search] Recent Posts * Barracuda Urges Replacing -- Not Patching -- Its Email Security Gateways * Service Rents Email Addresses for Account Signups * Ask Fitis, the Bear: Real Crooks Sign Their Malware * Discord Admins Hacked by Malicious Bookmarks * Phishing Domains Tanked After Meta Sued Freenom Spam Nation Spam Nation A New York Times Bestseller! Thinking of a Cybersecurity Career? Thinking of a Cybersecurity Career? Read this. All About Skimmers All About Skimmers Click image for my skimmer series. Story Categories * A Little Sunshine * All About Skimmers * Ashley Madison breach * Breadcrumbs * Data Breaches * DDoS-for-Hire * Employment Fraud * How to Break Into Security * Latest Warnings * Ne'er-Do-Well News * Other * Pharma Wars * Ransomware * Russia's War on Ukraine * Security Tools * SIM Swapping * Spam Nation * Target: Small Businesses * Tax Refund Fraud * The Coming Storm * Time to Patch * Web Fraud 2.0 The Value of a Hacked PC valuehackedpc Badguy uses for your PC Badguy Uses for Your Email Badguy Uses for Your Email Your email account may be worth far more than you imagine. Donate to Krebs On Security Most Popular Posts * Sextortion Scam Uses Recipient's Hacked Passwords (1076) * Online Cheating Site AshleyMadison Hacked (798) * Sources: Target Investigating Data Breach (620) * Trump Fires Security Chief Christopher Krebs (534) * Why Paper Receipts are Money at the Drive-Thru (530) * Cards Stolen in Target Breach Flood Underground Markets (445) * Reports: Liberty Reserve Founder Arrested, Site Shuttered (416) * Was the Ashley Madison Database Leaked? (376) * DDoS-Guard To Forfeit Internet Space Occupied by Parler (374) * True Goodbye: 'Using TrueCrypt Is Not Secure' (363) Why So Many Top Hackers Hail from Russia [computered-580x389] Category: Web Fraud 2.0 Criminnovations Innovations from the Underground [shreddedID-copy-285x189] ID Protection Services Examined Is Antivirus Dead? Is Antivirus Dead? The reasons for its decline The Growing Tax Fraud Menace The Growing Tax Fraud Menace File 'em Before the Bad Guys Can Inside a Carding Shop Inside a Carding Shop A crash course in carding. Beware Social Security Fraud Beware Social Security Fraud Sign up, or Be Signed Up! How Was Your Card Stolen? How Was Your Card Stolen? Finding out is not so easy. Krebs's 3 Rules... Krebs's 3 Rules... ...For Online Safety. (c) Krebs on Security - Mastodon