[HN Gopher] SSD belonging to Euro-cloud Scaleway was stolen,then...
___________________________________________________________________
SSD belonging to Euro-cloud Scaleway was stolen,then turned up on
YouTube
Author : ronalbarbaren
Score : 85 points
Date : 2021-07-28 11:58 UTC (7 hours ago)
(HTM) web link (www.theregister.com)
(TXT) w3m dump (www.theregister.com)
| Nextgrid wrote:
| > The CEO said recovering the disk helped the authorities to
| advance their investigations into the heist, and meant the
| company felt able to publicly disclose the theft.
|
| Aka they could no longer sweep it under the rug.
| jnwatson wrote:
| Why in the world do they ship unencrypted drives around?
|
| Drive encryption is pretty standard these days, isn't it?
| atonse wrote:
| With the big cloud provides its a trivial checkbox.
|
| Can anyone speak to the effort in doing it at the OS level?
| tpxl wrote:
| There's a checkbox to encrypt the disks when installing some
| linux distros.
| jsiepkes wrote:
| > There's a checkbox to encrypt the disks when installing
| some linux distros.
|
| While I'm not going to defend moving around unencrypted
| disks it is not as simple as that.
|
| The difficult part is not the encryption. The difficult
| part is the key management of the encryption. You need to
| solve issue like ensuring you don't lose access to the keys
| (and thereby access to the data), securely providing keys
| to people and machines that need access, dealing with key
| revocation when people leave the company, etc.
|
| Though I guess in this case using a simple (long enough)
| passphrase instead of some form of keys could have saved
| them from this particular threat.
| atonse wrote:
| I thought any of these providers always encrypted with
| their own key whether you encrypted on top of that or
| not. But I guess not.
|
| Yet another reason to just stick with the big providers.
| They seem to have pretty serious compliance needs and
| encrypt no matter what. (Yet you should still encrypt
| your assets, but I believe they all encrypt their volumes
| anyway and have pretty strict policies on how to dispose
| this kind of hardware)
| rkeene2 wrote:
| When I ran a cloud provider, we used a hardware security
| module on storage systems which had public/private key
| pair which could be used to decrypt the header on the
| disk which contained a copy of the symmetric cipher key.
| Each header had the symmetric cipher key encrypted with
| multiple different public keys, including a fallback
| whose private key only existed in on paper in a vault.
|
| Each system could reach out to other storage systems to
| ask them to use their private key to decrypt the header
| (for example if their hardware security module had
| failed), but in some configurations this would require an
| operator to intervene to enter in passphrase to unlock
| the hardware security module to authorize the action.
|
| This means that:
|
| 1. The symmetric cipher key could always be recovered,
| even from paper backup
|
| 2. Having physical access to a disk or any set of disks
| did not allow you to read the data
|
| 3. Having physical access to a disk and a hardware
| security module did not allow you to read the disk
| (unless you knew the passphrase, which was always
| present, and user set)
|
| 4. Having physical access to disks, servers, and hardware
| security module may not allow you to read the data (in
| the more secure configuration where passphrases were not
| cached on disk -- but this meant that rebooting required
| an operator to manually enter the password at boot)
|
| 5. The set of valid public keys could be changed
| frequently (this was indeed automated and only the set of
| currently active hardware security modules could decrypt
| the current disk header)
|
| Of course you could always short circuit this by making a
| copy of the symmetric key when you did have access (e.g.,
| `dmsetup table --showkeys`) but without putting some more
| hardware in the fast/hot path that was unavoidable. The
| symmetric key could not easily be changed, without
| rewriting the entire disk (though since it was a storage
| system designed to accommodate multiple failures, this
| wasn't that hard, but we didn't do it automatically)
|
| The hardware security modules we used were FIPS 140-2
| validated and physically connected to the racks (though
| it would be possible to cut them away). It would also be
| possible to spy on the APDUs sent to the modules to
| capture the decrypted data, since there was not mutual
| authentication (it was in the works though).
| jtchang wrote:
| Is there anything on the consumer level like this?
| michaelt wrote:
| Well, a lot of laptops these days have TPMs that can
| store disk encryption keys. BitLocker on Windows, or an
| exotic dm-crypt setup on Linux.
|
| However for a server, the chance of a thief getting a
| disk and the HSM at the same time is low; and you really
| don't want to have to enter a password every boot. So the
| HSM design delivers some benefits worth having.
|
| Whereas for a laptop, you can easily type in a password;
| and the chance of a thief snatching the disk and TPM at
| the same time is basically 100%. So there hasn't been a
| big push among Linux users to start using the TPM.
| rkeene2 wrote:
| It was all made out of parts with standardized
| interfaces, so components could be consumer equipment.
|
| I also used the same software on my laptop to encrypt it.
| It's nothing fancy, just using the dm-crypt kernel module
| and any kind of hardware security module that talks
| PKCS#11 (which is all of them); On my laptop I just used
| my existing smartcard (which I used to login to the
| system and remote systems).
|
| The disk header was just text occupying the first 4MiB of
| the disk in 2 circular buffers similar to LVM (though 2
| copies for redundancy).
| atonse wrote:
| This is exactly why most of us are more than happy to pay
| a premium to AWS, GCP, and Azure. They reduce all what
| you listed to a handful of checkboxes for us plebs. You
| have to have a few discussions about how you as an org
| will manage your keys, and you're a couple of terraform
| files away for having access to this (budget
| notwithstanding).
|
| This is the level of redundancy/backups/processes the
| large cloud providers have. It's not just "hey is this
| encrypted" - it's "what happens, how do you retire a
| drive, are the modules FIPS 140-2 compliant, how do we
| physically secure our HSMs?" etc
|
| It's just so much more than "hey Digital Ocean is 50%
| cheaper"
|
| AWS's egress bandwidth charges still feel criminal
| though.
| rkeene2 wrote:
| We were a relatively small cloud provider, but all disks
| were encrypted so at least we wouldn't leak someone's
| data if we had to replace a disk.
| jnwatson wrote:
| It is trivial in the Ubuntu installer.
| Nextgrid wrote:
| It's not trivial when it comes to a server. Sure, you can
| tick that checkbox you'll enable encryption, but when you
| reboot you realize that you can't actually SSH in it, as
| it's waiting for a key to be entered on the physical
| console.
|
| Entering (and managing) that key is the hard part.
| [deleted]
| madaxe_again wrote:
| They say they're iso27001 compliant and certified. If they
| aren't encrypting in transit, it's hard to see how they
| actually can be.
| unilynx wrote:
| Are you sure they're actually saying that?
|
| A lot of hosting providers actually say that their _data
| centers_ are ISO 27001 compliant, but that doesn 't make the
| provider itself compliant
|
| (and it seems scaleway is still doing that too on
| https://www.scaleway.com/en/datacenter/)
| fulafel wrote:
| So was there cleartext data on it?
| hdjjhhvvhga wrote:
| Yes, the article clearly says the data was unencrypted.
| Moonlight_TC wrote:
| The timeline of events in Scaleway's blog post is very dubious.
|
| If the SSD was stolen over one year ago why do they only
| acknowledge it now?
|
| In March 2021 [1] they were writing about how great their
| security was:
|
| _> We are proud of our data centers and their security. We
| consider that we have implemented the best solutions to protect
| your most valuable asset: your data. We are well aware of the
| huge responsibility this represents. There can be no compromises
| when it comes to your data._
|
| Why were customers whose data was leaked only informed in June
| 2021? The delay of over 1 year is a huge GDPR issue.
|
| The timeline of the incident from public sources:
|
| 21 May 2021: Micode tweets a screenshot of the directory listing
| [2]
|
| 26 May 2021: First video on the subject [3]
|
| 6 June 2021: Scaleway customer is notified their data was leaked
| [4]
|
| 21 July 2021: Second video on the subject [5]
|
| 24 July 2021: Third video on the subject [6]
|
| 24 July 2021: Scaleway releases a French blog post stating "Over
| a year ago, an SSD was stolen" [7]
|
| Storing the data unencrypted is bad, but IMHO Scaleway's handling
| of the incident creates much bigger questions about their
| credibility.
|
| [1] https://blog.scaleway.com/how-we-protect-your-data/
|
| [2] https://mobile.twitter.com/Micode/status/1395640486715662336
|
| [3] https://www.youtube.com/watch?v=vt8PyQ2PGxI
|
| [4]
| https://www.lowendtalk.com/discussion/comment/3258386/#Comme...
|
| [5] https://www.youtube.com/watch?v=aOBVZUL1iBA
|
| [6] https://www.youtube.com/watch?v=xf_cKTlOYLo
|
| [7] https://blog.scaleway.com/incident-securitaire-video-
| youtube...
|
| Previous discussion:
| https://news.ycombinator.com/item?id=27957471
| detaro wrote:
| 2 days ago: https://news.ycombinator.com/item?id=27957471
| mot2ba wrote:
| I posted that thread and I don't know why it was penalty by
| mod, completely out of front page HN.
| terom wrote:
| Apparently with plaintext customer data.
|
| > Lechelle said Scaleway worked with the YouTuber to recover the
| disk. The French-language video creator has written to Scaleway
| with assurances they have not copied the information contained on
| the disk. It is said some customer data was on the drive,
| unencrypted, including the source code and SSH keys of an Italian
| VPS provider.
| terom wrote:
| Based on links in the other hackernews thread, it was quick-
| formatted, but a filesystem containing qemu disk images was
| recovered: [1]
|
| The YouTube video is in French without English CC subtitles,
| but there's screenshots of exploring what I assume would be one
| of the qcow images: [2]
|
| * 03:59 `/root/.ssh` with SSH id_rsa (private) keys, (public)
| authorized_keys and (hashed) known_hosts
|
| * 07:14 server binaries
|
| * 08:42 server logs
|
| * 12:59 AWS/S3 secret access keys
|
| * 13:33 Source code... in QBASIC ?! must be very valuable
| legacy code :D
|
| * 17:30 brute-forcing the ssh known_hosts
|
| * etc
|
| [1]
| https://mobile.twitter.com/Micode/status/1395640486715662336
|
| [2] https://www.youtube.com/watch?v=xf_cKTlOYLo
| rasz wrote:
| >it was quick-formatted
|
| are there OSes that perform quick format without issuing TRIM
| command to the SSD? TRIM would vanish the data in less than
| 15 minutes.
| djrogers wrote:
| Yes, 15 minutes is not 'quick' by most standards - quick
| formats take seconds, not minutes.
| BBC-vs-neolibs wrote:
| Not if you turn the computer off right after deleting the
| files.
| TonyTrapp wrote:
| Note: QB64 (which is mentioned in the video) is a modern-day
| reimplementation of QBasic. It doesn't have to be legacy.
| Some people still prefer using Basic as their "scripting
| language" for quick tasks where other people would be using
| Python or JS these days.
| alyandon wrote:
| Although I never cared much for the language itself, I
| continued to write small programs in QB for many years
| because of its ability to produce an EXE file that ran in
| DOS and Windows without any dependencies.
| ChrisArchitect wrote:
| Other thread mentioned:
| https://news.ycombinator.com/item?id=27957471
| vajenetehais wrote:
| On their website they refer to multiple certification [0]. One of
| them is the CISPE and on their website, it is stated:
|
| "Requirement for CISP:
|
| (a) Security measures
|
| The CISP will implement and maintain appropriate technical and
| organisational measures for the CISP's data centre facilities,
| servers, networking equipment and host software systems that are
| within the CISP's control and are used to provide the CISP's
| service (the CISP Network). Those technical and organisational
| measures should (a) be designed *to help customers secure
| personal data against unauthorised processing and accidental or
| unlawful loss, access or disclosure*, and (b) address the
| security responsibilities of the CISP as set out in Annex A
| (Security Responsibilities)." [1]
|
| I do not know about others certifications, but this situation
| seems to be a clear violation of one of the requirement for CISP.
| Answer to this requirement is disk encryption. Moreover they are
| authorized to store medical records and data. I can't imagine
| that they do this without providing proper disk encryption. in
| the light of this event, I'm not sure they qualify for all these
| ceertifications.
|
| [0] https://www.scaleway.com/fr/a-propos/
|
| [1] https://cispe.cloud/code-of-conduct/
| throwaway382985 wrote:
| I worked at scaleway.
|
| This whole post is basically bullshit, "secure" transporation
| meant taking a random electric car (renault zoe) at the company
| headquarters or at DC5 and throwing hardware in the trunk.
|
| No security in place whatsoever, servers laying in datacenter
| hallways fully loaded with disks, accessible to anyone.
|
| Most of the company had access to the back office where they
| could just access customers' data without any kind of logging.
| (Internally this was called sudo mode on the online console, we
| had basically to click on a button to log as any customer).
|
| The "funniest" was the corporate vpn network being shared by the
| internal datacenter network which meant any employee had, for
| instance, full access to all the home made switches management
| interfaces which had no access control whatsoever, it happened
| that employees stumbled accross this and wondering what it was
| (LOL), there was even a button to upload and flash (!) a
| firmware.
|
| The upper-management was terribly incompetent and would discard
| any kind of issue that was not affecting sales in the immediate
| term as not important (security or otherwise).
| emteycz wrote:
| What about GDPR, did it change anything?
| Abishek_Muthian wrote:
| There was a post in the scaleway user forum(~2020) asking 'Who
| has access to the data on the disk' to which the founder or CEO
| replied that the Engineers do have access but the datacenters
| are very secure as it's thoroughly monitored.
|
| I remember this because I used fscrypt after seeing the post,
| Scaleway seems to have removed the entire user forum sometime
| last year.
|
| I stopped using their service after they removed their ARM
| severs with short notice, But this is more scary.
|
| Unfortunately this is the reality when it comes to AWS[1] or
| other top-tier cloud providers vs inexpensive ones. The cost
| saving has to happen somewhere.
|
| [1] https://aws.amazon.com/blogs/security/importance-of-
| encrypti...
| raverbashing wrote:
| The things that are not clear to me are:
|
| - Why was the disk being replaced (SMART warnings? HW upgrade? It
| failed in some way)
|
| - Why was the quick format recoverable? Was TRIM issued or the HW
| didn't support it?
|
| - As a follow up, I'm guessing the HW didn't support HW
| encryption?
|
| - Why are they relying on 3rd parties for the destruction of
| data?
|
| Meanwhile, it seems that HDs don't leave Google datacenters in
| one piece
| dracodoc wrote:
| The disk could be simply transferred to another data center
| physically so they can move data faster/cheaper than
| transferring the data by internet.
| capableweb wrote:
| Reminds me of the following quote:
|
| > Never underestimate the bandwidth of a station wagon full
| of tapes hurtling down the highway.
|
| - Andrew S. Tanenbaum
___________________________________________________________________
(page generated 2021-07-28 19:02 UTC)