https://krebsonsecurity.com/2021/04/ubiquiti-all-but-confirms-breach-response-iniquity/ Advertisement [9] Advertisement [14] Krebs on Security Skip to content * Home * About the Author * Advertising/Speaking Ubiquiti All But Confirms Breach Response Iniquity April 4, 2021 33 Comments For four days this past week, Internet-of-Things giant Ubiquiti did not respond to requests for comment on a whistleblower's allegations the company had massively downplayed a "catastrophic" two-month breach ending in January to save its stock price, and that Ubiquiti's insinuation that a third-party was to blame was a fabrication. I was happy to add their eventual public response to the top of Tuesday's story on the whistleblower's claims, but their statement deserves a post of its own because it actually confirms and reinforces those claims. [ubiquiti] Ubiquiti's IoT gear includes things like WiFi routers, security cameras, and network video recorders. Their products have long been popular with security nerds and DIY types because they make it easy for users to build their own internal IoT networks without spending many thousands of dollars. But some of that shine started to come off recently for Ubiquiti's more security-conscious customers after the company began pushing everyone to use a unified authentication and access solution that makes it difficult to administer these devices without first authenticating to Ubiquiti's cloud infrastructure. All of a sudden, local-only networks were being connected to Ubiquiti's cloud, giving rise to countless discussion threads on Ubiquiti's user forums from customers upset over the potential for introducing new security risks. And on Jan. 11, Ubiquiti gave weight to that angst: It told customers to reset their passwords and enable multifactor authentication, saying a breach involving a third-party cloud provider might have exposed user account data. Ubiquiti told customers they were "not currently aware of evidence of access to any databases that host user data, but we cannot be certain that user data has not been exposed." [ubi-notice] Ubiquiti's notice on Jan. 12, 2021. On Tuesday, KrebsOnSecurity reported that a source who participated in the response to the breach said Ubiquiti should have immediately invalidated all credentials because all of the company's key administrator passwords had been compromised as well. The whistleblower also said Ubiquiti never kept any logs of who was accessing its databases. The whistleblower, "Adam," spoke on condition of anonymity for fear of reprisals from Ubiquiti. Adam said the place where those key administrator credentials were compromised -- Ubiquiti's presence on Amazon's Web Services (AWS) cloud services -- was in fact the "third party" blamed for the hack. From Tuesday's piece: "In reality, Adam said, the attackers had gained administrative access to Ubiquiti's servers at Amazon's cloud service, which secures the underlying server hardware and software but requires the cloud tenant (client) to secure access to any data stored there. "They were able to get cryptographic secrets for single sign-on cookies and remote access, full source code control contents, and signing keys exfiltration," Adam said. Adam says the attacker(s) had access to privileged credentials that were previously stored in the LastPass account of a Ubiquiti IT employee, and gained root administrator access to all Ubiquiti AWS accounts, including all S3 data buckets, all application logs, all databases, all user database credentials, and secrets required to forge single sign-on (SSO) cookies. Such access could have allowed the intruders to remotely authenticate to countless Ubiquiti cloud-based devices around the world. According to its website, Ubiquiti has shipped more than 85 million devices that play a key role in networking infrastructure in over 200 countries and territories worldwide. Ubiquiti finally responded on Mar. 31, in a post signed "Team UI" on the company's community forum online. "Nothing has changed with respect to our analysis of customer data and the security of our products since our notification on January 11. In response to this incident, we leveraged external incident response experts to conduct a thorough investigation to ensure the attacker was locked out of our systems." "These experts identified no evidence that customer information was accessed, or even targeted. The attacker, who unsuccessfully attempted to extort the company by threatening to release stolen source code and specific IT credentials, never claimed to have accessed any customer information. This, along with other evidence, is why we believe that customer data was not the target of, or otherwise accessed in connection with, the incident." [UI-response] Ubiquiti's response this week on its user forum. Ubiquiti also hinted it had an idea of who was behind the attack, saying it has "well-developed evidence that the perpetrator is an individual with intricate knowledge of our cloud infrastructure. As we are cooperating with law enforcement in an ongoing investigation, we cannot comment further." Ubiquiti's statement largely confirmed the reporting here by not disputing any of the facts raised in the piece. And while it may seem that Ubiquiti is quibbling over whether data was in fact stolen, Adam said Ubiquiti can say there is no evidence that customer information was accessed because Ubiquiti failed to keep logs of who was accessing its databases. "Ubiquiti had negligent logging (no access logging on databases) so it was unable to prove or disprove what they accessed, but the attacker targeted the credentials to the databases, and created Linux instances with networking connectivity to said databases," Adam wrote in a whistleblower letter to European privacy regulators last month. "Legal overrode the repeated requests to force rotation of all customer credentials, and to revert any device access permission changes within the relevant period." It appears investors noticed the incongruity as well. Ubiquiti's share price hardly blinked at the January breach disclosure. On the contrary, from Jan. 13 to Tuesday's story its stock had soared from $243 to $370. By the end of trading day Mar. 30, UI had slipped to $349. By close of trading on Thursday (markets were closed Friday) the stock had fallen to $289. [ubistock] Data Breaches Ubiquiti breach [109] Post navigation - New KrebsOnSecurity Mobile-Friendly Site Ransom Gangs Emailing Victim Customers for Leverage - 33 thoughts on "Ubiquiti All But Confirms Breach Response Iniquity" 1. G.Scott H. April 4, 2021 The burden of proof lies with Ubiquiti in regards to proving customer data was not accessed. Without such proof, the safest assumption for security of customer data is that it was accessed. Reply - 1. JamminJ April 4, 2021 Unfortunately, proving a negative looked this is not really possible. It's also one of the main reasons auditors are always adamant about logging everything. Auditors understand that you can't prove a negative, but you can of confidence in the negative if you had robust logging. Reply - 1. Ray April 4, 2021 Proving this negative is absolutely possible, if you do access logging. AWS provides this option, and it's reeeeallly easy to set up. UI was apparently not doing access logging. This is a best practice that is warranted because of the distribution and adoption numbers. Not to mention their own desire to funnel users through the cloud based auth service. Reply - 1. JamminJ April 5, 2021 Like I said... Proving a negative is NOT possible. Having reasonable confidence in the negative would be possible, yes, if logging was enabled. Which it wasn't. According to the whistleblower, it was specific logging for on premises database access, not on general AWS access. So although easy for AWS, perhaps there were reasons Database Access Monitoring wasn't set up. Reply - 1. Me April 5, 2021 "Like you said it" twice already but you're missing the nuance here. Proving "a" negative is very possible despite you saying otherwise. The problem isn't that "it's a negative", although this platitude gets parroted a lot. Plenty of negatives (evidence of absence, negative proof) can in fact be proven. Proving "this" particular negative is not possible because Ubiquiti intentionally set it up in such a way. Disabling access logging is exactly the situation you want to be in in case of a breach because you want to be able to legitimately say "we have no evidence customer data was accessed". Being able to say this has far more value than the access log. Reply - 2. Jim April 5, 2021 Proving a negative is certainly possible. I can prove that there is no coffee in my mug right now, I can prove that 2+2 does not equal 5, I can prove that I didn't get drunk at lunch. While it's true that absence of evidence is not evidence of absence. The opposite is also true that evidence of absence is not an absence of evidence. Reply - 1. Logic tests April 5, 2021 You might want to rethink your examples. Your examples are not examples of "proving a negative". This concept is often misunderstood. Proving a cup empty, is not the same as proving there was never any coffee in the cup's history. After all, can the present condition (which is provable) prove the cup never had coffee? No, because you lack information regarding the entire history. How can you know if the cup was ever washed? The example of getting drunk at lunch, has it's "proof" relying on the biological understanding that blood alcohol content can only decrease so quickly. But again, as the event in question moves from the present state, into the past, "proof" diminishes. You can prove the current state, but proving the negative never happened becomes impossible without retaining perfect knowledge. The math example is often misapplied to this concept as well. "You can't prove a negative" usually does not refer to mathematics or any abstract in which all information is known. So in math, yes proving a negative is possible. But in concrete reality, in which omniscience is not possible or practical, not proving a negative is a reasonably true statement. A good rule of thumb when creating examples of this concept, is to ask yourself, does it require that you know everything? Every variable? -- Now, as the concept of "Cannot prove a Negative" relates to infosec data breaches... It is technically true, there is no absolute proof that something did not happen. So JamminJ is correct when stating "reasonable confidence in the negative" is possible with lots of logs being available. Even with perfect and comprehensive logs, you can only have high confidence, but cannot absolutely prove it. It may be good enough for auditors, but someone could always say, "maybe the logs were wiped". Just like someone can say, "maybe someone cleaned your coffee cup". The path to high confidence is to gather more and more logs, to make it less and less likely the positive exists. But that's not the same as "proving the negative". Reply - 2. Cringe Worthy April 5, 2021 Ubiquiti just doesn't know. Assume the worst. Reply - 3. Alex Curtis April 5, 2021 Since apparently, Ubiquiti didn't keep access logs for its systems, I don't think there is any way for Ubiquiti to "prove the negative" which is to prove that customer data had never been accessed. The truth is that they have no way of knowing. On a related note: I don't understand why more companies don't log everything. Logging costs are trivial. Stored in cloud providers like AWS/Azure/GCP at ~1 cent per gig or less (since they can be stored on low-availability or archive storage). Even if you don't know how to search logs or use them, I recommend that every company at least be creating them and throwing them off to archive storage somewhere. If anything ever happens you can hire a consultant to come in and query or parse the logs for you. Reply - 2. Watching and Waiting April 4, 2021 Thx for update, Brian. My regret is that I didn't short UI! LOL. As a mere low tech user of systems this is more curiosity to me than anything else but it is interesting nevertheless. Reply - 3. Gannon (J) Dick April 4, 2021 At least Ubiquiti had a strategy to protect their stock price. Someone in Law Enforcement should have simply mentioned to Spring-Break crowds that mask-less faces are much faster to check against picture ID's and therefore would be done first. This would have raised the bar on bogus credential quality for under-age drinking etc.. Better living through Social Engineering ! Reply - 4. Jon Marcus April 4, 2021 "We see no evidence of compromise", they say with their eyes tightly closed. Reply - 5. John Pavon April 4, 2021 Please check out CL, they are having trouble with their contact codes, I think they have a big cannot contact posters the have a code generator where they give you a temp email address for a limited time, but now no body can reach posters? They claim they have this solved, but it still is infected? Reply - 1. muck April 4, 2021 CL? Craigslist? Reply - 6. vb April 4, 2021 Follow the chain of events backwards...the "accident" happened when local-only networks were being connected to Ubiquiti's cloud. The gun was loaded, safety off, and aimed at their foot. The breach only pulled the trigger. Reply - 1. Catwhisperer April 4, 2021 True, however that has become de rigueur in the industry. Take Extreme Cloud, Mist, et al, all have management interfaces that are cloud hosted. Ubiquity is just another one that jumped on the wagon. However, the idea that you left the keys to the kingdom available in an ex-employee's account active deserves the responsible individual to, well, that can't be posted online... Reply - 7. Stratocaster April 4, 2021 As usual, anyone who states, "We take the security of your information very seriously" probably hasn't and doesn't, and has created a proof of concept why. Reply - 8. Paul Yeager April 4, 2021 I wonder how this might affect our internally hosted UISP and Unifi instances. Reply - 9. Michael Swenor April 4, 2021 Here is the deal, as an MSP spin up your own cloud controller and don't connect it to your account.UI.com cloud account that way non of this matters. MSP and hobbyists that are serious about security figure out ways to make it more secure. 2nd why did you not have 2FA already turned on your ubiquiti account if it does business?? This would have made this whole breach a non issue for your business as the 2FA of UI was not compromised. Don't blame ubiquiti for users amateur hour skills, krebs, give these lazy people a free week of lessons to show them. Appreciate the heads up on the breach but we knew about this a while ago and already updated firmware and didn't connect to their cloud in the first place. Good read anyway Mike Reply - 1. Trevor Dyck April 5, 2021 It does still matter if they have source code. Reply - 1. alambrito April 5, 2021 No, source code does not matter. Security through obscurity is no security at all. Reply - 10. Mike Lowe April 4, 2021 Negligence. Ubiquiti was in such a great position, record profits, all time high stock price and none of that goes to security. You'd think they had of learned their lesson in the past. Brian it might be a good idea to bring up their past behavior including falling for phishing attacks and the whole ubnt / ubnt user password fiasco. Reply - 11. Hugh Brown April 5, 2021 WSJ puzzle clue, 2021-04-03: 'Wishy-washy response to "Did they really say that?" (5 wds.)' Answer: "Not in so many words" Reply - 12. Jeff April 5, 2021 Brian, wrong link to Ubiquiti forum topic. https://community.ui.com/questions/ Update-to-January-2021-Account-Notification/ 3813e6f4-b023-4d62-9e10-1035dc51ad2e Reply - 13. UnBlinking April 5, 2021 Note to Ubiquiti Board: It's not the crime, it's the coverup. Find out who pressed for the coverup, and replace them. Note to Ubiquiti C-Suite: Someone in your early coverup planning sessions uttered phrases like "We can't lie about this," or "Let's be more honest," or even just "I don't agree with this approach." That person was disinvited from later coverup planning sessions. Find that person, and put them in charge of all future communications about the breach. You're running out of chances to get this right. Reply - 14. Mark Pyle April 5, 2021 Hikvision and Ubiquiti are in a stiff competition for the prosumer camera market. A competition with national security implications - Hikvision products banned from US Government contacts. So are we going to get whistle blower info from Hikvision? Mr Krebs which side are on. For for a moment of editorial fame you undermined the a substantial US corporation. Reply - 1. That is R April 5, 2021 Are you a closeted racist? Sounds like you want to give an American company a pass, just because they aren't Asian. Reply - 1. Read Better April 5, 2021 There's nothing racist in pointing out the China's totalitarian government has a military stake in Hikvision and that said company's vulns worldwide without question imperil national security of non-totalitarian-aligned orgs. Krebs isn't undermining a US corporation, he's reporting on a breech that said corporation was responsible for. Pyle's rhetorical about whistleblowers in China's state-operated security companies is entirely apt however. Because they don't have those, they disappear for years into secret prisons without trials. The difference. Reply - 1. JamminJ April 5, 2021 It's probably not racism... but nationalism, which often overlaps with xenophobia and racism. There is a LOT of this going around. Whether it's overt racism that leads to violent hate crimes against Asian Americans, or just "whataboutism" (what about this Chinese company that had significant vulnerabilities 5 years ago??). Now that an American company has to be held to account... another round of "whataboutism". Hikvision is NOT in competition, in ANY way, with Ubiquiti. They make cheap consumer cameras. Ubiquiti is a much broader network hardware company, closer to Cisco, Huawei and TP-Link. All of which have had serious breaches of security and trust. Much of this has been fueled by rhetoric against China over the past few years. People believing in conspiracy theories about a Sars-CoV2 being engineered in a lab in China. In cybersecurity, there was even a fake news story about SuperMicro, motherboards having secret chips that allowed spying from China. All alleged victims denied this, and even years later, no evidence. Pyle may or not be motivated by racism, but it does sure sound like ultra-nationalism and an axe to grind against China (which has nothing to do with this). Pyle seems to advocate against speaking out (whistleblowing) against American companies doing the same thing. As if it is somehow patriotic to shield companies from accountability if they are American. The ironic thing is... this is how China deals with their own companies. They turn a blind eye. They don't allow journalists to do their job, if they perceive any negative coverage to be "undermining" their own national interests. Reply - 1. Read April 5, 2021 "Hikvision is NOT in competition, in ANY way, with Ubiquiti" - Was not claimed otherwise. The distinction is they're both companies with security concerns and physical products. One is owned/operated on behalf of a totalitarian kleptocratic cabalism directly, whereas comparably the other operates in much a more nuanced system re: nationalism. Your last paragraph is the relevant one regarding blind nationalism, but nigh-equating the US and China governmentally speaking is perhaps a different type of intentional blindness also. Hikvision, Huawei, multiple such individual companies are on specific lists for specific reasons with regard to both national and non-US international security regimes - for good reason. Not a racist reason either, it ought be said and underscored. Intl'y known APT reasons. Reply - 1. JamminJ April 5, 2021 "Was not claimed otherwise" Read again... "Hikvision and Ubiquiti are in a stiff competition..." No, no they are not. "they're both companies with security concerns and physical products" Okay? So what? That is extremely broad and includes many many manufacturers from around the world. Nationality isn't a factor here. There is no reasonable justification for bringing up Hikvision in this discussion, other than "whataboutism". I do agree that the US government should not use foreign hardware or software for government use. Doubly true for military. Just like I think it would be reasonable for China not to use American products for their government use. That really has nothing to do with the type of government (democratic/capitalist or authoritarian/communist). The NSA would be smart to put in backdoors for any US imported network equipment being used in China. Do you think we don't already? I don't believe the original comment was racist. But it does speak to a deeper irrational fear. Some people are trying to make China into an all encompassing boogeyman. Yeah, they are a significant geopolitical adversary for many reasons. But no, they didn't create the virus in a lab. No, they did not plant hidden chips onto motherboards sent to the US. And no, they don't need to be mentioned on every major cybersecurity breach. By the way, why do you call China "cabalism"? Is that supposed to be a new term for fearmongering against China? I know Qanon likes to throw around anti-Semitic terms like that, and many people don't know what the words mean, so it gets mixed in with other conspiracy theories. Although the original comment itself may not be racist... it may indeed be used to fuel racism. Creating irrational fear of the other, blaming them for everything, is how you create a culture of hate. Hate crimes against Asian Americans is on the rise. And this tangent may be a symptom of that trend. Reply - 2. Mark Bennett April 5, 2021 Proof-read your posts please. This is barely coherent. Shooting the messenger is also not a productive line to pursue on this website, which is specifically dedicated to providing information on events such as this. Why are you here? Reply - 1. Greg April 5, 2021 I agree. If these 'in-the-know' IT pros can't write a coherent sentence perhaps they write code in the same haphazard manner. Maybe this is why there are so many security holes in their products. Just sayin'. Reply - Leave a Reply Cancel reply Your email address will not be published. Required fields are marked * [ ] [ ] [ ] [ ] [ ] [ ] [ ] Comment [ ] Name * [ ] Email * [ ] Website [ ] [Post Comment] [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ] Advertisement [7] Advertisement [110] Mailing List Subscribe here Search KrebsOnSecurity Search for: [ ] [Search] Recent Posts * Ransom Gangs Emailing Victim Customers for Leverage * Ubiquiti All But Confirms Breach Response Iniquity * New KrebsOnSecurity Mobile-Friendly Site * Whistleblower: Ubiquiti Breach "Catastrophic" * No, I Did Not Hack Your MS Exchange Server Spam Nation Spam Nation A New York Times Bestseller! All About Skimmers All About Skimmers Click image for my skimmer series. Story Categories * A Little Sunshine * All About Skimmers * Ashley Madison breach * Breadcrumbs * Data Breaches * DDoS-for-Hire * How to Break Into Security * Latest Warnings * Ne'er-Do-Well News * Other * Pharma Wars * Ransomware * Security Tools * SIM Swapping * Spam Nation * Target: Small Businesses * Tax Refund Fraud * The Coming Storm * Time to Patch * Web Fraud 2.0 The Value of a Hacked PC valuehackedpc Badguy uses for your PC Badguy Uses for Your Email Badguy Uses for Your Email Your email account may be worth far more than you imagine. Donate to Krebs On Security Most Popular Posts * Sextortion Scam Uses Recipient's Hacked Passwords (1076) * Online Cheating Site AshleyMadison Hacked (798) * Sources: Target Investigating Data Breach (620) * Trump Fires Security Chief Christopher Krebs (534) * Cards Stolen in Target Breach Flood Underground Markets (445) * Reports: Liberty Reserve Founder Arrested, Site Shuttered (416) * Was the Ashley Madison Database Leaked? (376) * DDoS-Guard To Forfeit Internet Space Occupied by Parler (374) * True Goodbye: 'Using TrueCrypt Is Not Secure' (363) * Who Hacked Ashley Madison? (361) Category: Web Fraud 2.0 Criminnovations Innovations from the Underground [shreddedID-copy-285x189] ID Protection Services Examined Is Antivirus Dead? Is Antivirus Dead? The reasons for its decline The Growing Tax Fraud Menace The Growing Tax Fraud Menace File 'em Before the Bad Guys Can Inside a Carding Shop Inside a Carding Shop A crash course in carding. Beware Social Security Fraud Beware Social Security Fraud Sign up, or Be Signed Up! How Was Your Card Stolen? How Was Your Card Stolen? Finding out is not so easy. Krebs's 3 Rules... Krebs's 3 Rules... ...For Online Safety. (c) Krebs on Security