[HN Gopher] InfluxDB Cloud shuts down in Belgium; some weren't n...
___________________________________________________________________
InfluxDB Cloud shuts down in Belgium; some weren't notified before
data deletion
Author : PaulAA
Score : 68 points
Date : 2023-07-09 19:11 UTC (3 hours ago)
(HTM) web link (community.influxdata.com)
(TXT) w3m dump (community.influxdata.com)
| raffraffraff wrote:
| Wow. That's amazingly bad.
| plasma wrote:
| Hard to reverse actions need multiple safety switches, for
| example, turning off the machines in that region for 2 weeks
| before deleting them, which would bring support issues to
| attention ahead of the no-going-back step of deleting data.
| lopkeny12ko wrote:
| Wow, the incredibly callous 3-word explanation of the issue by
| pointing to a docs link with no other context. Really gives off
| "it's your fault for not reading the wiki." Is this how InfluxDB
| treats their customers?
|
| Incidentally at work we've been evaluating a new hosted
| observability provider, looks like we can rule out Influx as an
| option.
| Matthias247 wrote:
| This seems really poor. They should have been able to see what
| customers are still using the service in that region, send them
| additional reminders that they need to get off, and only after
| several of those and a grace period remove access. Maybe even
| have a phase of read-only access before full removal.
| simonw wrote:
| This kind of thing really does need a cooling off period.
|
| Assume that your users won't see your emails. How do you help
| them avoid data loss when you shut down a service like this?
|
| One option that I like is to take the service down (hence loudly
| breaking things that were depending on it) but keep backed up
| copies of the data for a while longer - ideally a month or more,
| but maybe just two weeks.
|
| That way users who didn't see your messaging have a chance to get
| in touch and recover any data they would otherwise lose.
|
| I'm not sure how best to handle the liability issues involved
| with storing backups of data for a period of time. Presumably the
| terms and conditions for a service can be designed to support
| this kind of backup storage "grace period" for these situations.
| SSLy wrote:
| you start with reliability brownouts. first fail 0.1% requests,
| then after a week 1%, then after a month 5%.
| remram wrote:
| That seems like the worst of both worlds, during the brown-
| out you have to keep paying for the compute while your
| customers don't get a reliable service, even if they have a
| plan to migrate.
|
| Also you probably can't keep charging customers for that
| period since you offer a crippled service on purpose.
| dfadsadsf wrote:
| Much better is to stop the service but add button "Resume"
| that re-enables service for two more weeks with no data loss.
| That way you give users opportunity to gracefully migrate
| away.
|
| Stopping service and immediately delete data is just callous.
| tredre3 wrote:
| > According to the support, the notification emails to the users
| were sent on Feb 23, Apr 6 and May 15th. However, we did not
| receive those at all.
|
| If true, this is concerning. One message getting lost in spam
| understandable. But three over 6 months would imply they're being
| blacklisted and/or their mail sender is simply broken.
|
| Do serious companies not have canaries or other checks in place
| to ensure their notifications are correctly delivered to
| customers?
| hrpnk wrote:
| A check on the expected migration progress and usage patterns
| in the region should have also rung a bell.
| arp242 wrote:
| If a spam service erroneously marks one email as spam, chances
| are it will also marks other very similar emails as spam. So
| it's not too surprising all three were marked.
|
| For these kind of automated emails getting all emails
| consistently being delivered to everyone is really hard, almost
| impossible.
|
| The problem here isn't really that emails weren't being
| delivered, it's that they seem to have tried only one method to
| contact people, didn't check how successful that was (e.g. by
| seeing how many customers were still on those regions), and
| seemingly never tried anything else (such as notifications on
| the dashboard, a temporary brown-out to alert people, etc.) -
| "we tried one way to contact you and that didn't work, so we
| just deleted your service sucks to be you lol kthxfuckitybye"
| PaulAA wrote:
| Similar reports have also appeared on the InfluxDB Slack.
| shrubble wrote:
| It's tough to believe that the turning off of the service
| couldn't have involved at least a week of 'soak' time, where if
| you contacted them they then helped you move to another location.
| After all the additional cost/benefit ratio of keeping the VMs
| around, but not using CPU or bandwidth vs. retaining a few
| customers would indicate it is the right thing to do for both the
| customers and the business.
| berkle4455 wrote:
| It's almost like using bullshit hype-hype-hype databases,
| especially their cloud offerings, is a horrible idea.
| gemstones wrote:
| I wonder if they're open in other EU regions? If you wanted to
| shut down a region, as a database provider, is it even possible
| to send snapshots to another region without user consent?
|
| It feels like that could be a good practice, or not, depending on
| the laws in question.
| kawsper wrote:
| Their migration guides does include help for customers to
| migrate data from one region to another EU region (Frankfurt).
|
| They might not have been able to do it automatically as the
| region name were hardcoded in the hostname.
| jjgreen wrote:
| "But look, you found the notice, didn't you?"
|
| "Yes," said Arthur, "yes I did. It was on display in the bottom
| of a locked filing cabinet stuck in a disused lavatory with a
| sign on the door saying 'Beware of the Leopard."
___________________________________________________________________
(page generated 2023-07-09 23:00 UTC)