[HN Gopher] Hetzner Cloud - Data loss incident
___________________________________________________________________
Hetzner Cloud - Data loss incident
Data loss incident (snapshots) Dear Customer, Unfortunately, we
have to inform you that there was a data loss incident that affects
a small amount of your snapshots on Hetzner Cloud. All snapshots
you create are stored on our highly available storage systems. The
snapshot contents are distributed over multiple internal servers
and data is stored in a way that allows up to two separate disks to
fail without impacting data integrity. This means the snapshot can
still be accessed, even if two disks fail at the same time. Due to
a recent, very unfortunate series of events in one of our clusters,
multiple disks failed in short succession and caused a small number
of snapshots to become unavailable. We immediately tried to recover
the affected snapshots but unfortunately the data is lost and we
have exhausted all our options. Affected snapshots in your
account: XXXXXXXXX The snapshots have been removed from our system
as they are no longer accessible. We sincerely hope this doesn't
cause too much trouble for you; we know losing data is the worst-
case scenario. Also, we have added 20EUR as Cloud Credits to your
account (valid for one year). While we know that this will not
bring back your data, we still hope that you will accept the
gesture. In response to this we will re-evaluate our snapshot
cluster data replication strategies as well as our strategies for
replacing disks and rebuilding redundancy after replacement. Best
Regards, Hetzner Cloud
Author : bratao
Score : 138 points
Date : 2022-04-13 15:29 UTC (7 hours ago)
| jasonzemos wrote:
| There is no reason to rely on "triple replication" for data
| integrity. This has long been a solved problem. An appropriate
| erasure encoding can reduce the probability of loss ~ten-fold
| while consuming physically less space (i.e. 2x worth of
| replication). Companies forego this technology because they feel
| confident in their operational ability to address failures
| quickly and competently. That's what we're relying on for data
| integrity, not the math.
| toast0 wrote:
| This email doesn't detail the integrity scheme, only that data
| was protected if no more than two disks failed, and more than
| two disks failed.
|
| That could be three copies at 3x the storage cost, or it could
| be a RAID6 or raidz2 style system where the storage cost is
| essentially two disks out of the storage set size (which we
| don't know).
|
| You could certainly increase the required failure count by
| increasing the parity, but if the problem was something like
| all the disks in the storage system hit the same fatal disk
| firmware bug at the same time (as speculated elsewhere in the
| thread, and is unfortunately plausible), then that doesn't
| really help much.
| [deleted]
| throwaway81523 wrote:
| Erasure codes with lots of redundant drives is how online.net's
| C14 storage product works, but it is much more hassle than
| ordinary replication, since retrieval requires reading data
| from a lot of different drives to reassemble the shares, and
| similarly for splitting storage requests into shares and
| recording them. C14 is a Glacier-like product where you request
| a retrieval and your data is restore as an S3 object sometime
| (up to hours) later. That makes it easier, I expect.
|
| I agree that this would be good as a last-ditch backup for
| stuff like Hetzner cloud snapshots, but the primary storage for
| those snapshots probably has to be ordinary RAID.
|
| Fwiw I have a bunch of data in Hetzner Storagebox and have
| gotten several notices in the past few months that Storagebox
| would be temporarily offline for maintenance, which I assume
| meant raid rebuilds and/or drive swaps. The most recent of
| these was an "urgent" maintenance with less notice time than
| usual. It hasn't caused me any inconvenience, but I wonder if
| they have suffered a spate of drive failures recently.
| cpach wrote:
| Folks, remember the 3-2-1 rule. IMO, it's still extremely
| relevant, even in today's cloud-centric world.
|
| For data you can't afford to loose, please, for the love of
| ${DEITY}, don't store it with just one vendor. You never know
| what happens.
|
| https://www.backblaze.com/blog/the-3-2-1-backup-strategy/
| 1970-01-01 wrote:
| You forgot to explain it:
|
| Make 3 copies of your systems (just data if you can rebuild
| infra within your RTO)
|
| Use 2 different media (compress then encrypt)
|
| Move 1 copy off-site. ( _print the encryption key_ and store
| that in a fireproof safe)
| freemint wrote:
| Maybe not compress and encrypt depending on your thread
| model.
| noodlesUK wrote:
| I generally think not encrypting data is a bad idea, but I
| can see a world where that's a risk trade off you want to
| make (such as a hard drive with family photos). Why
| wouldn't you want to compress it though?
| jl6 wrote:
| Last time I surveyed my data, more than 99% of it was in
| an already compressed format (e.g. JPEG and HEVC), so
| compressing the backups added almost no benefit.
|
| (Now, that's not strictly true any more for JPEGs thanks
| to projects like lepton and JPEG XL which can indeed
| compress already-compressed JPEGs losslessly).
| justsomehnguy wrote:
| Compression by itself:
|
| 1. Doesn't provide a meaningful size tradeoff for /family
| photos/
|
| 2. Doesn't provide a way to recover the data if it was
| damaged, at best you get a notification if a
| decompression routine failed the verification of
| uncompressed data
|
| 10 years ago I had a meaningful dedup ratio, nowhadays it
| is cheaper for me to have another 50Gbs than bother eith
| a dedup
| bombcar wrote:
| I think that people forget that you need _two_ copies of your
| data _under your control_ - and when your data is primarily on
| the cloud it is _not_ data under your control; it is under the
| cloud company 's control. You should have at least _one_ copy
| that is physically in your possession, offline preferably, so
| that you have access to it.
|
| For my cloud data, I sync it down to a local server and then
| sync that to removable offline media.
| Proven wrote:
| jamal-kumar wrote:
| I've never trusted cloud provider snapshots because it's just
| paying more for something that might fail outside of your
| responsibility.
|
| I have a local backup box and a remote one and they both sync
| stuff in from rsync on a cron job from multiple servers. Works
| great. In case of catastrophic failure (Which has happened a few
| times), putting my eggs in more than one basket has been
| extremely successful.
| c7DJTLrn wrote:
| That's not good but you do get what you pay for. Should never put
| all your eggs in one basket like that.
|
| The EUR20 of credit made air accelerate out of my nostrils.
| cle wrote:
| > The EUR20 of credit made air accelerate out of my nostrils.
|
| Also they made sure you knew it's only valid for one year!
| systemvoltage wrote:
| I came across tour of Hetzner datacenter[1] and it looks like a
| completely unprofessional setup, like castle on stilts. Does
| not inspire confidence.
|
| [1] https://www.youtube.com/watch?v=5eo8nz_niiM
| philjohn wrote:
| That's why they're so cheap, compared to a lot of other
| offerings.
| brokenodometer wrote:
| I just watched the whole thing and thought it looked really
| cool! Seems like a scrappy and self-reliant company that is
| focused on cost effectiveness and efficiency.
| native_samples wrote:
| FWIW if you focus on the modern areas, Google datacenters
| don't look much different.
| jamal-kumar wrote:
| If that's your standard for a garbage data centre... hmmm
|
| Yeah I've seen way, way worse. Stuff like power cables zip
| tied to network cables and fibre lines. DUST. delerict
| hardware just sitting everywhere.
|
| This is nice. What are you looking for, IBM dudes in white
| shirts running around?
| jacquesm wrote:
| I've seen far worse. That said, the compute density is pretty
| low and the gear is rather older than what you would expect,
| they clearly are not optimizing for power efficiency but for
| capital efficiency, which means that as long as they can
| operate a box profitable they'd rather add more buildings
| than upgrade their servers. Interesting thermal arrangement,
| I wonder how well this would work in a fire, it looks like a
| giant chimney to me but maybe there is some secret sauce.
| tobltobs wrote:
| Could you tell us some of the red flags you see in this
| video? Considering it is a low cost hoster it looks fine to
| me, but I am not an expert.
| singingboyo wrote:
| Looks like standard machines and not really servers, but
| there's nothing wrong there. Fairly clean wiring, no random
| cables, etc etc. HDD testing is a little sketchy, but they
| need to be quick-swapped, so it makes sense.
|
| Ever been in a firewall/router manufacturers R&D/QA lab room?
| Those are so, so much worse. This is heaven by comparison.
| throwaway71271 wrote:
| looks pretty good
|
| depends how localized the fire extinguishers are and how the
| power is distributed which is hard to see from the video
|
| not sure what you expect, using 1-2U boxes or blades or
| something wont change the setup that much, it pretty much
| boils down to 'how many machines will die in the same time on
| power/fire' and how can we cool them as cheap as possible.
| boxingrock wrote:
| multiple drive failure... i wonder if they were all the same
| brand / model / batch
| uuyi wrote:
| Lost an EBS snapshot on AWS once. The only way to provide some
| assurance in this space is to make sure your data is stored in
| more than one physically and commercially separate location.
| Aachen wrote:
| This is the way. I get that not doing small-scale things
| yourself saves costs and that's why "cloud" exists, but if
| something is important to you or your organization, I cannot
| stress enough how useful having just a simple old data copy is.
| I've lost data to bad drives too often myself. Microsoft can
| erroneously close your paid-for account without recourse, or
| ransomware can try to extort you, but your data will simply not
| be lost if you have an offline (or more advanced: append-only)
| system with an hourly/daily/... backup on it.
|
| There are various options here but the one I happened to have
| read up on (and then contributed documentation to make it
| easier to use safely) is Restic with Rest-server in append-only
| mode. It's not perfect (e.g. no compression yet, only dedup)
| but it's pretty good. If you're an individual or small-medium
| business, put your favorite raspberry pi equivalent in a closet
| and invest a hundred bucks in a hard drive big enough for the
| important documents, pictures of loved ones, etc.
|
| A single drive shouldn't be your only backup, just don't put
| 100% of your trust in magic cloud. See also Atlassian this
| week: it may be down, but if you need to access the data of
| ticket xyz for a lead this week and you have a local data
| export, you can find the needed data.
| mahastore wrote:
| you do get what you paid for
| Aachen wrote:
| Not always. Pay for various games on Xbox, paid onedrive
| storage, pay for office licenses for the whole family, trip
| some false positive and still get your account banned with
| everything in it. You can talk to a wall or ask a judge to
| force Microsoft to give a reason, but then they can't
| actually show the reason because that's against some law in
| another country or something. It's fantastic. (Dutch,
| unfortunately)
| https://tweakers.net/nieuws/194930/microsoft-moet-
| blokkade-n... https://tweakers.net/reviews/9094/account-
| geblokkeerd-wat-nu...
| ckozlowski wrote:
| I would argue that EBS snapshots should not be used as a backup
| solution. EBS is reliable, but isn't engineered for the same
| durability as say, S3. These also become more unwieldy and less
| useful the older they get. Use EBS snapshots for rollback prior
| to making changes to your environment.
|
| For durable backup I'd recommend storing objects in S3/Glacier.
| If you need to image an entire volume, make an AMI, which is
| stored in S3. Much more durable and targeted than a snapshot,
| and in the case of S3 objects, gives you benefits such as
| logging and version control.
| forty wrote:
| Aren't EBS snapshots stored on S3? :)
| ckozlowski wrote:
| They are (and in going to look that up I see that they're
| being pitched as more of a backup solution than they were
| previously) but given they're incremental and based on the
| existing EBS volume, I'd still recommend using something
| without dependencies.
|
| I'm curious now as to how the EBS Snapshot was lost. If I
| had to guess, I suspect it had something to with that
| linkage back to the original volume. I maintain that the
| best backup has no dependencies on another resource
| existing.
| uuyi wrote:
| We never got a straight answer from AWS.
|
| We now run an instance in another region and account and
| rdiff-backup to it.
| forty wrote:
| It's a good point. There is a relatively new feature to
| archive EBS snapshots [1] which stores non incremental
| snapshots
|
| [1] https://aws.amazon.com/fr/blogs/aws/new-amazon-ebs-
| snapshots...
| imtringued wrote:
| For most people Glacier is a terrible solution.
| herpderperator wrote:
| 3-2-1 Backup: https://www.backblaze.com/blog/the-3-2-1-backup-
| strategy/
| sidewndr46 wrote:
| That works until you find out the commercially separate
| location was using Amazon for their storage solution as well.
| timcavel wrote:
| rmetzler wrote:
| Better communication than Atlassian apparently.
| joecool1029 wrote:
| I remember being told years back with Digitalocean to not rely on
| their backups by one of the people that wrote their system. This
| made sense to me as they didn't even charge for it for the
| longest time. We do now use it some of the time and haven't had
| problems but there's always a second place we send images to
| (usually S3) as well as longer term offline archives.
| KptMarchewa wrote:
| Even when you consider that you might lose some data in an
| unlikely event, there still is value proposition in this kind
| of services. Easy restore compared to external services.
| jacquesm wrote:
| > All snapshots you create are stored on our highly available
| storage systems.
|
| Highly available right until they are not. Nice to see that your
| data is worth 20 bucks to Hetzner. I always liked them but this
| is a bit rude to put it mildly.
| xmaayy wrote:
| The storage system is highly available, but the data is not
| guaranteed to be :)
| charcircuit wrote:
| Availability and durability are different things.
|
| S3 for example is designed for an availability such that it can
| fail 1 in every 10,000 requests on average and for a durability
| that can lose one object for every 100,000,000,000 objects you
| have each year on average.
| jacquesm wrote:
| NSS.
| raxxorraxor wrote:
| A snapshot of the current data? So the data is still available
| and another snapshot can be created?
| op00to wrote:
| Snapshots are often used for deployment of systems. What
| happens if the snapshot you rely on as a base image gets lost?
| traspler wrote:
| Create a new one?
| op00to wrote:
| If the base snapshot is gone, all future snapshots that
| rely on that snapshot may be invalid. Depends on exactly
| how snapshots are implemented. Are they a delta of the
| previous snapshot? Are they a full copy of the device?
| Kwpolska wrote:
| Hetzner snapshots are fully independent disk images,
| nothing more, nothing less.
| op00to wrote:
| Thanks! $20 seems kinda fair then, I guess.
| justinholmes wrote:
| Base images should be repeatable anyway using packer etc.
| tetha wrote:
| Even though this is a bit tricky on the hetzner cloud,
| because it requires the addition of a static route[1] to
| access the internet which isn't pushed by the DHCP, and I
| haven't been able to fully automate that in debians preseed
| for the life of me.
|
| Yes it's offtopic, but that's one really frustrating thing.
| Maybe someone here has a solution.
|
| 1: https://docs.hetzner.com/cloud/servers/iso-installation-
| gate...
| sam_lowry_ wrote:
| cloud-init?
| tetha wrote:
| cloud-init does help us, once we have a working linux,
| but we can also bake the static route into a base image.
|
| The problem is that preseed is a very minimal linux
| environment and either relies on DHCP to work, or to use
| the usual network config DHCP sets (address, netmask,
| gateway). Arbitrary routes are not a possibility there.
| And ways to run abitrary commands to just <ip r>
| something come after the network initialization, which
| fails in that setup. The same holds for the kernel
| network bootstrapping parameters, I haven't found a way
| to add an arbitrary route there yet.
|
| And for some reason, kickstart is able to handle this
| with it's centos/RHEL base.
| kro wrote:
| It reads like it indeed does not affect live data in running
| cloudservers, yes.
|
| Aside from backups, the snapshots can also be used as base
| images to provision new nodes though, losing those could indeed
| be inconvenient.
| throwawayboise wrote:
| I was always under the impression that a snapshot should be an
| short-lived thing. E.g. you take a snapshot, make a backup or
| copy from that, then delete the snapshot. Where I work, the VM
| admins will only allow one snapshot at any point in time
| without special request and justification, and they encourage
| deleting the snapshot within 72 hours.
| CodesInChaos wrote:
| You're using a rather narrow/implementation specific
| definition of snapshot.
|
| I believe EBS snapshots are equivalent to you temp-snapshot,
| incremental/deduplicated backup/delete temp snapshot
| combination. Logically it's still a snapshot, because it
| contains all data of the volume at a specific point in time,
| but the cloud provider abstracts away the details of how it
| achieves durability.
| carlhjerpe wrote:
| Depends on the solution, but in many systems snapshots are
| cheap, like zfs or btrfs. I can imagine ceph snapshots being
| quite cheap too.
|
| Edit: people use snapshots to compliment backups too, you can
| do snapshots hourly and back a snapshot up daily. (They don't
| act as backups, more like "oops" revert solutions, humans are
| more likely to fail than machines)
|
| If you're using VMware or Hyper-V the backups rely on a dirty
| "bitmap" of which blocks has been written since last backup
| so it only has to pull those from the system, this doesn't
| work with more snapshots for some reason. Enterprise backup
| solutions will just pull the entire machine and compare
| what's changed anyways, but you usually get a warning about
| it. So grumpy admins
| throwawayboise wrote:
| Yes it's VMware.
| k8sToGo wrote:
| Also old snapshots
| op00to wrote:
| Wow, twenty whole euro for blowing away your data. Let's hope
| Hetzner doesn't go bankrupt offering this immensely generous
| offer.
| MarcoZavala wrote:
| yjftsjthsd-h wrote:
| What would you like them to do?
| op00to wrote:
| Refund the total cost for a month at least?
| 10987654321 wrote:
| Just checked and for the location where I have my vm the
| price is:
|
| > Snapshots cost EUR0.0125/GB/month (incl. 25 % VAT)
|
| so EUR20 would cover a month for about 1.6TB of snapshots.
| Sounds pretty reasonable to me. But I guess it depends on
| how much data OP lost.
| emteycz wrote:
| They refunded much more than that...
| cxcorp wrote:
| Which would be less than the 20EUR they've offered?
| sleepyhead wrote:
| Not sure what OP is paying but I'm paying like EUR3 to Hetzner
| so it could be a sum that is comparative to the payment.
| bennyp101 wrote:
| I wonder if this is related to the emails I've been getting since
| the start of March regarding maintenance on Storage Box Hosts?
| There were urgent maintenance ones on the 11th April as well.
| cersa8 wrote:
| The day I ordered my first Hetzner storage box (May 16) was the
| day I had my first Urgent Maintenance notice. Today the second
| one. "Urgent maintenance on Storage Box hosts XXX to XXX"
|
| Never had any trouble with their vps offerings, but I'm not
| liking it.
| [deleted]
| drfuchs wrote:
| Anybody have more color on the nature of the "recent, very
| unfortunate series of events in one of our clusters [such that]
| multiple disks failed in short succession"? What kind of
| "events"? A failed climate-control system? A berserk employee
| with a sledge hammer? Russian hackers? The Spanish Inquisition?
|
| It seems odd that they'd be quite so vague and circumspect about
| it. Why not just say what it was, so we're not all left to
| speculate, perhaps for the worst?
| [deleted]
| aeyes wrote:
| I used to have a couple of filers with thousands of disks on
| premise, we'd always have a dead disk somewhere. Keeping up
| with dead disks was almost a full time job, you'd open a case
| with the vendor, get them access to the datacenter, then be on
| the phone to guide them to the rack and verify that the new
| disk came online. To not go completely crazy, we usually
| batched these jobs once or twice per week. And even with double
| parity and replicated filers in two locations things can go
| wrong.
|
| Also a favorite of mine is power cycling a machine. With enough
| disks, chances are high that a couple won't come back after
| they spin down.
| throwawayboise wrote:
| A batch of disks in a RAID array that were all from the same
| manufacturing lot. Such setups have been known to have multiple
| drives fail within hours/days of each other. Best practice is
| to mix manufacturers or at least manufacturing lots when you
| build the storage array.
| jerf wrote:
| There's a couple of other well-known ways to screw up
| redundancy too, though I hope they're in far enough to know
| them.
|
| One of my favorite is, if you want to store three copies of
| something on a lot of disks, randomly select which three
| disks you want to send the copies to so that each bit of
| content is stored on three random disks. It superficially
| seems very appealing; I have 3 copies of everything, nothing
| is likely to go wrong!
|
| However, what you've actually created is a situation where if
| _any_ three disks go down, you 're guaranteed to lose data
| because as you scale up, the probability that those three
| disks were the three randomly-chosen disks for _some_ set of
| data goes to 1. The probability of three disks going down at
| some point also likewise trends to 1 as you scale up.
|
| Not saying that was the case here, just sharing it as one of
| the well-known pitfalls I find particularly interesting.
| jl6 wrote:
| How could any approach involving n copies of data mitigate
| against loss of n devices? What is the superior approach
| that avoids this pitfall?
| toast0 wrote:
| If you store three copies of all your files, with copies
| uniformly distributed over the disks, and three disks
| fail simultaneously [1], you are guaranteed to lose data.
|
| If instead, you break your disks into N/3 groups of
| three, and store each file three times in one group, you
| only lose data when all three disks in one group fail
| simultaneously. If you're lucky, you can lose 2/3rds of
| your disks before you lose data.
|
| Edit to add:
|
| If you're using mirroring, it's low cost to do mirroring
| by groups although uniform distribution can help with
| hotspots. With uniform distribution, even if one device
| has two popular files, the other copies of those files
| are likely to be on devices with only one popular file.
| With grouping, if one group gets two popular files, all
| devices in the group have two popular files and therefore
| elevated use.
|
| If you're using a parity scheme (raid6/raidz2/some other
| erasure coding), the cost to do grouping is more
| apparent. Each group has the +X parity cost; If you did
| raidz2, and had 45 drives in a case, you could do one set
| of 45, and get 43 drives worth of storage, or three sets
| of 15 and get 39 drives worth of storage, or five sets of
| 9 to get 35 drives worth of storage, etc. In each of the
| smaller sets, you still can lose data with the first
| three failures, but you need an increasingly large number
| of failures to guarantee data loss. Also, the size of the
| data lost when there's a loss is smaller.
|
| [1] or at least, the third disk fails before either of
| the first two failures is repaired
| jl6 wrote:
| Intuitively, it feels as though grouping is a tradeoff
| that lowers the probability of a data loss event, at the
| expense of increasing the amount of data lost in a data
| loss event?
|
| Edit: "proof" by example...
|
| 9 disks, replication factor of 3. 9 choose 3 = 84 ways to
| allocate a file's replicas across the disks. If 3 disks
| fail, on average 1/84 of the total data is lost (assuming
| >>84 files).
|
| Versus... 3 groups of 3 disks. If 3 disks fail, the
| probability of the second disk failure being in the same
| group as the first disk failure is 2/8, and then 1/7 that
| the third failure is also in the same group.
| (2/8)*(1/7)=1/28.
|
| So a 100% chance of losing 1/84 of the data versus a 1/28
| chance of losing 1/3 of the data. Same total expected
| amount of data lost.
| GauntletWizard wrote:
| I keep hearing this pitfall, and then I keep seeing the
| results of idiots who don't understand that by not picking
| three totally at random, what they have created is a giant
| mess should any of those three fail, instead of a quick and
| easy to clean up mass that you are constantly raid
| rebuilding from.
| HyperRational wrote:
| hoofhearted wrote:
| I believe the more likely scenario was that they lost a disk
| and replaced it.. While the disk array was restriping,
| another disk in the raid array failed because of the
| restriping load, and all data would be lost at that point in
| a raid setup as they described.
| toast0 wrote:
| I dunno, HPE had two issues with disks failing at specific
| power-on hours[1], and that wasn't the first time I've
| heard of similar things, just the one I can remember the
| vendor for. It's a _lot_ more work to mix batches and
| stagger power on times than to build a bunch of disks
| systems at one time and turn them all on at once.
|
| [1] https://www.hpe.com/us/en/services/sas-ssd-
| advisory.html
| wiredfool wrote:
| Hetzner has been doing emergency maintenance on their backup
| drive systems (BX-XX) lately. Both of mine on separate accounts
| have had the notifications.
___________________________________________________________________
(page generated 2022-04-13 23:01 UTC)