https://krebsonsecurity.com/2021/07/dont-wanna-pay-ransom-gangs-test-your-backups/ Advertisement [14] Advertisement [8] Krebs on Security Skip to content * Home * About the Author * Advertising/Speaking Don't Wanna Pay Ransom Gangs? Test Your Backups. July 19, 2021 13 Comments Browse the comments on virtually any story about a ransomware attack and you will almost surely encounter the view that the victim organization could have avoided paying their extortionists if only they'd had proper data backups. But the ugly truth is there are many non-obvious reasons why victims end up paying even when they have done nearly everything right from a data backup perspective. [ransomware] This story isn't about what organizations do in response to cybercriminals holding their data for hostage, which has become something of a best practice among most of the top ransomware crime groups today. Rather, it's about why victims still pay for a key needed to decrypt their systems even when they have the means to restore everything from backups on their own. Experts say the biggest reason ransomware targets and/or their insurance providers still pay when they already have reliable backups is that nobody at the victim organization bothered to test in advance how long this data restoration process might take. "In a lot of cases, companies do have backups, but they never actually tried to restore their network from backups before, so they have no idea how long it's going to take," said Fabian Wosar, chief technology officer at Emsisoft. "Suddenly the victim notices they have a couple of petabytes of data to restore over the Internet, and they realize that even with their fast connections it's going to take three months to download all these backup files. A lot of IT teams never actually make even a back-of-the-napkin calculation of how long it would take them to restore from a data rate perspective." Wosar said the next most-common scenario involves victims that have off-site, encrypted backups of their data but discover that the digital key needed to decrypt their backups was stored on the same local file-sharing network that got encrypted by the ransomware. The third most-common impediment to victim organizations being able to rely on their backups is that the ransomware purveyors manage to corrupt the backups as well. "That is still somewhat rare," Wosar said. "It does happen but it's more the exception than the rule. Unfortunately, it is still quite common to end up having backups in some form and one of these three reasons prevents them from being useful." Bill Siegel, CEO and co-founder of Coveware, a company that negotiates ransomware payments for victims, said most companies that pay either don't have properly configured backups, or they haven't tested their resiliency or the ability to recover their backups against the ransomware scenario. "It can be [that they] have 50 petabytes of backups ... but it's in a ... facility 30 miles away.... And then they start [restoring over a copper wire from those remote backups] and it's going really slow ... and someone pulls out a calculator and realizes it's going to take 69 years [to restore what they need]," Siegel told Kim Zetter, a veteran Wired reporter who recently launched a cybersecurity newsletter on Substack. "Or there's lots of software applications that you actually use to do a restore, and some of these applications are in your network [that got] encrypted," Siegel continued. "So you're like, 'Oh great. We have backups, the data is there, but the application to actually do the restoration is encrypted.' So there's all these little things that can trip you up, that prevent you from doing a restore when you don't practice." Wosar said all organizations need to both test their backups and develop a plan for prioritizing the restoration of critical systems needed to rebuild their network. "In a lot of cases, companies don't even know their various network dependencies, and so they don't know in which order they should restore systems," he said. "They don't know in advance, 'Hey if we get hit and everything goes down, these are the services and systems that are priorities for a basic network that we can build off of.'" Wosar said it's essential that organizations drill their breach response plans in periodic tabletop exercises, and that it is in these exercises that companies can start to refine their plans. For example, he said, if the organization has physical access to their remote backup data center, it might make more sense to develop processes for physically shipping the backups to the restoration location. "Many victims see themselves confronted with having to rebuild their network in a way they didn't anticipate. And that's usually not the best time to have to come up with these sorts of plans. That's why tabletop exercises are incredibly important. We recommend creating an entire playbook so you know what you need to do to recover from a ransomware attack." This entry was posted on Monday 19th of July 2021 05:11 PM Ransomware Bill Siegel Coveware Emsisoft Fabian Wosar kim zetter [119] Post navigation - Microsoft Patch Tuesday, July 2021 Edition 13 thoughts on "Don't Wanna Pay Ransom Gangs? Test Your Backups." 1. Kurt Seifried July 19, 2021 Also would have accepted the title "don't want your boss to yell at you after a server dies or someone fat finger deletes data? Test your backups" Reply - 2. Justin Case July 19, 2021 As a Senior Administrator once told me long before ransomware was a thing: "Until you have tested your backups by regularly verifying the restored data - it's not a backup, it's wishful thinking and a fast path to the unemployment office" Reply - 3. The Sunshine State July 19, 2021 I use 7-ZIP to encrypt and back up my folders and files. I know for a fact that it allows you to test your backup's integrity Reply - 1. Blurp July 19, 2021 I really hope you are joking. Reply - 4. Randy Graham July 19, 2021 Very interesting topic. Thank you Brian Reply - 5. Mark Bennett July 19, 2021 I'm currently engaged in a project of my own to create a local cloud backup using a Synology NAS, MinIO and Retrospect backup software in a walled off and hardened environment. One of the goals is to keep ransomware from touching the actual backup device. The other is to be able to restore everything in a local cloud environment, which should be as fast as possible, but realistically will take several days. It has been a tough project as far as learning curve and testing on a live system, but I plan to have it all working by the end of this month. I 'm also working on specific plans about how to recover an entire network, which is a huge mental exercise. This is the first story I've read that explores the cloud backup/restore problem, which is simply ignored by most people. Kudos. Reply - 6. Dragon Cotterill July 19, 2021 Schrodinger's Backups. The state of any backup is unknown until a restore is attempted. Reply - 7. Damian July 19, 2021 "The Tao of Backup" is my favorite site covering the often-forgotten details of how to do backups properly. Reply - 8. Hans July 19, 2021 There are a few things about on-line backups that make me nervous. One is that some on-line backup company a few years ago shut down and gave everyone a month to retrieve their data. From what I've been told, many didn't get it retrieved. Another is the issue that the ones I've looked at are just trying to mirror the data on each computer. If you lose some files and don't catch them in time, the on-line backup will often assume that after some period of time, they must have been intentionally removed and will delete their copy as well. For example, if you have some accounting files that are only used for the end of the year processing, they could be gone an entire year before you realize that they aren't on your computer any more. The speed issue you mentioned is definitely a good thing to keep in mind. I think that I'll just stick to using borg backup. Reply - 9. JDMurray July 19, 2021 Perhaps "backups" should be renamed "restorables" to drive their purpose into consciousness. Reply - 10. Chris July 19, 2021 Since the internet has not been around 69 years, the 69 year figure to restore isn't possible. There is no reason why it would take longer to restore than to create. Clearly hyperbole, but the implication remains: backups should be local. Reply - 11. Pete2 July 19, 2021 I restore the full C: drive (OS) on the server from local backup (not cloud!). Takes about an hour. Use backup from a few nights prior, to give time for my protections provider to find any trojan that also got restored. Then can get workstations back logged on to domain (or do full restore from the server to them if they were also encrypted, adding 1-2 hours). Then get backup of data drives from local backup (simple high quality USB drive). I do recommend having a cloud backup provider for the data, but that is for backup of the local backup. If there is a lot of data, usually there are a few folders that are needed right away, so I do those first. Only use a backup software that creates proprietary files, and better yet, makes the backup drive invisible (like old, included in Windows Server, Microsoft Backup). Nothing fancy but it works. Reply - 12. John McGing July 19, 2021 I worked at a government agency where the head of the IT services required a periodic disaster recovery over a holiday weekend. The failures were post mortemed and incorporated into the recovery plan. Senior Management always questioned the use of time and resources but he insisted. It's the kind of thing that's too often overlooked or viewed as unnecessary. Reply - Leave a Reply Cancel reply Your email address will not be published. Required fields are marked * [ ] [ ] [ ] [ ] [ ] [ ] [ ] Comment [ ] Name * [ ] Email * [ ] Website [ ] [Post Comment] [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ] Advertisement [111] Advertisement [117] Mailing List Subscribe here Search KrebsOnSecurity Search for: [ ] [Search] Recent Posts * Don't Wanna Pay Ransom Gangs? Test Your Backups. * Microsoft Patch Tuesday, July 2021 Edition * Spike in "Chain Gang" Destructive Attacks on ATMs * Kaseya Left Customer Portal Vulnerable to 2015 Flaw in its Own Software * Microsoft Issues Emergency Patch for Windows Flaw Spam Nation Spam Nation A New York Times Bestseller! All About Skimmers All About Skimmers Click image for my skimmer series. Story Categories * A Little Sunshine * All About Skimmers * Ashley Madison breach * Breadcrumbs * Data Breaches * DDoS-for-Hire * Employment Fraud * How to Break Into Security * Latest Warnings * Ne'er-Do-Well News * Other * Pharma Wars * Ransomware * Security Tools * SIM Swapping * Spam Nation * Target: Small Businesses * Tax Refund Fraud * The Coming Storm * Time to Patch * Web Fraud 2.0 The Value of a Hacked PC valuehackedpc Badguy uses for your PC Badguy Uses for Your Email Badguy Uses for Your Email Your email account may be worth far more than you imagine. Donate to Krebs On Security Most Popular Posts * Sextortion Scam Uses Recipient's Hacked Passwords (1076) * Online Cheating Site AshleyMadison Hacked (798) * Sources: Target Investigating Data Breach (620) * Trump Fires Security Chief Christopher Krebs (534) * Cards Stolen in Target Breach Flood Underground Markets (445) * Reports: Liberty Reserve Founder Arrested, Site Shuttered (416) * Was the Ashley Madison Database Leaked? (376) * DDoS-Guard To Forfeit Internet Space Occupied by Parler (374) * True Goodbye: 'Using TrueCrypt Is Not Secure' (363) * Who Hacked Ashley Madison? (361) Why So Many Top Hackers Hail from Russia [computered-580x389] Category: Web Fraud 2.0 Criminnovations Innovations from the Underground [shreddedID-copy-285x189] ID Protection Services Examined Is Antivirus Dead? Is Antivirus Dead? The reasons for its decline The Growing Tax Fraud Menace The Growing Tax Fraud Menace File 'em Before the Bad Guys Can Inside a Carding Shop Inside a Carding Shop A crash course in carding. Beware Social Security Fraud Beware Social Security Fraud Sign up, or Be Signed Up! How Was Your Card Stolen? How Was Your Card Stolen? Finding out is not so easy. Krebs's 3 Rules... Krebs's 3 Rules... ...For Online Safety. (c) Krebs on Security