[HN Gopher] Tell HN: Heroku deleted my database with no warning
___________________________________________________________________
Tell HN: Heroku deleted my database with no warning
Last December, Heroku nuked the database on one of my active
projects. I was travelling at the end of the year and did not catch
wind of this until I returned and saw messages about an issue with
the app. Sure enough, I checked and noticed that the database was
gone and detached on December 9th. Before the hate comes out, yes
I know Heroku deprecated free tiers. However, I did not understand
this would affect my projects on paid dynos. The real issue here is
that I never received a single email or notice of any kind to my
email about this. From researching, it appears most people received
SEVERAL notices about this. I did not think there was an issue with
my setup because I received zero communication. Upon reaching out,
Heroku has told me that they cannot recover the database. They also
admitted that there was "an issue" sending out notifications to me,
and confirmed that none were sent. So I guess just a warning to
all - your database might be nuked at any time. I learned my lesson
about not doing an offsite backup regularly. I guess the bigger
lesson though is that Heroku should really be a last resort option
for projects these days. RIP.
Author : fireworks
Score : 274 points
Date : 2023-01-31 17:44 UTC (5 hours ago)
| nineteen999 wrote:
| > I guess the bigger lesson though is that Heroku should really
| be a last resort option for projects these days.
|
| Maybe people shouldn't, I don't know because I've never used it,
| but ...
|
| > I learned my lesson about not doing an offsite backup
| regularly.
|
| This is the bigger lesson, no matter whether you are using third
| party database hosting, or hosting yourself, whether it's a NoSQL
| database or a SQL one.
|
| Once the data is gone, its gone.
| ericpauley wrote:
| This whole debacle has been such short-term thinking from
| Salesforce. That after carrying these free projects for years
| they couldn't stomach more than 30d of data retention is just the
| icing on the cake.
|
| In my view this has caused yet further reputational harm for
| Heroku, and is going to have a long-term effect on the bottom
| line from paid projects. The value prop of Heroku has always been
| being able to sleep at night, but clearly that's gone now.
| alexpetralia wrote:
| Salesforce is like the Midas touch of destruction.
| xp84 wrote:
| I assumed the value prop of being on Heroku in 2023 was purely
| "not having to do the pain in the butt to migrate your legacy
| app that you made on Heroku in the pre-2010 era" - in other
| words, I would be really shocked if Heroku got significant new
| business or any growth at all, now that it's just expensive AWS
| with some basic CI integration points.
|
| I also assume Salesforce only bought them as a cash generator
| and has no interest in investing in it. So if they saved bottom
| line from this move, that's a win for them.
|
| (Feel free to correct my assumptions if I'm very wrong)
| jrochkind1 wrote:
| "Pre-2010"? Heroku only launched in April 2009. So I would
| assume the vast majority of current customers are not
| actually from a "pre-2010 era". Heroku didn't even have a
| postgres add-on until November 2010 -- the salesforce
| purchase went through in Dec 2010. In fact, the heroku
| "golden years" were mostly during the early salesforce
| period. (Heroku put matz, creator of ruby, on staff in July
| 2011; there was no Heroku API until 2014!). (source for
| timeline: https://www.heroku.com/about)
|
| Heroku is still quite a bit of value-added over "AWS with
| some basic CI integration", people chose it and still choose
| it because it requires a lot lot lot less in-house expertise
| and management hours than AWS for most kinds of standard
| apps. (I'm not totally sure what AWS services/architecutres
| you are thinking of when you say that; but I'll say: for
| pretty much all of them.)
|
| (Heroku def has more competition in that space than it did 10
| years ago, opinions differ on the relative merits)
|
| AWS of course already existed when Heroku was launched, so if
| you do consider them just "AWS with some CI integration" then
| I guess they would have been from the start? What would have
| made that true now if it wasn't then?
|
| I think Salesforce thought it would somehow be more
| "synergistic" with their other offerings than it has turned
| out to be. That they'd get Salesforce customers on heroku
| when they needed something beyond the "no-code" tools
| Salesforce already provided, or that they'd do better at
| converting heroku customers to salesforce customers than they
| have been. It does seem to be true that salesforce stopped
| really investing in new heroku features or improvements some
| years ago, and seem to be looking to minimize costs while
| continuing to collect revenue, I agree. (Sometimes I'm not
| even sure how much they care about continuing to collect
| revenue...)
| amenghra wrote:
| I used Heroku's free tier for small projects/prototype. I
| continue to use their cheapest offering for new projects
| because it saves me time. If a project grows, I can
| reevaluate my choice. I don't think anything in Heroku is a
| pain to migrate away from. I would miss the
| intuitive/familiar UI.
| cpursley wrote:
| Check out render.com, it's a solid alternative.
| tevon wrote:
| We currently use render. Its excellent, but is missing a
| couple key features: like Heroku's continuous rollback
| protection. If they offered this they'd be head and
| shoulders better.
| anurag wrote:
| (Render CEO) We're actively working on Point-in-Time
| Recovery and hope to launch it in early access soon.
| glenngillen wrote:
| > Salesforce on bought them as a cash generator
|
| As an early Heroku employee, this gave me quite the chuckle.
| brogrammernot wrote:
| Same - wasn't an employee but used it often in ~2013-2015
| timeframe. I'm pretty sure the valuation was many many many
| multiples of any sort of revenue based metric, wasn't it?
| teaearlgraycold wrote:
| render.com
| throwawaykai wrote:
| Herokai here. Unfortunately we had no choice on the data
| retention front -- once we've disconnected your database, we
| aren't ALLOWED to hold your data for more than 30 days. That's
| part of the data scrubbing protocol that we agree to when you
| sign up. We fought hard for 90+ days internally, but in the end
| couldn't get over the issue that we'd be in violation of our
| contracts with customers.
| rurp wrote:
| If I'm understanding you correctly, the 30 day policy is one
| that Heroku chose to put in the contract. Engineering might
| have fought the terms, and yes they need to be followed once
| set, but it seems totally fair to blame Heroku for creating
| the limitation in the first place.
| jacobsenscott wrote:
| If you enable daily backup those are nuked too?
| fireworks wrote:
| Are you allowed to inform paying customers that you are going
| to do this? This is my primary complaint here. I don't
| understand how this oversight happened. This is going to
| cause an enormous amount of time and energy to recover from
| this.
| yamtaddle wrote:
| > Are you allowed to inform paying customers that you are
| going to do this?
|
| I can't be the only one who's basically completely blind to
| emails from major companies, including SaaS providers,
| because they're so fucking spammy that the SNR is like
| 1:99. Notifying me by email, for one of these places, is
| functionally the same as not notifying me at all.
|
| [EDIT] Sorry, didn't mean to imply the parent wasn't paying
| attention, just that I'd fully expect a very high
| percentage of their users to miss the warning in all the
| noise _even if_ they emailed everyone--even if they emailed
| them a couple times, actually. That 's the cost of every
| company sending out tons of "join our online seminar on
| [product]!" and "hey, look, it's our newsletter you never
| read!" and "it's time for our weekly TOS modification!"
| emails.
| function_seven wrote:
| > _...because they 're so fucking spammy that the SNR is
| like 1:99._
|
| This 1000x. I signed up for an SMS gateway service last
| year. Just for my own hobby use, nothing major. I gave
| them $10 to start service, and they charge like 2C/ or
| something per outgoing message.
|
| They have like 180 different prices for 180 countries,
| territories, provinces, parishes, cantons, prefectures,
| etc. Every week one of those prices changes, and I get an
| email notifying me of that. I tried to turn those off in
| my preferences, but they refused. I can opt out of
| marketing, weekly digests, and "tips and tricks"; but I
| can't opt out of pricing notices.
|
| So I added a rule on my end to hide those. I totally
| understand where they're coming from. They can't NOT give
| pricing change warnings. But at the same time, in the
| flood of constant notices, there may be something major I
| will miss.
|
| It would be nice if they instead gave me the option to
| never spend more than $0.XX per message, and return an
| API error if an attempted send fails for price threshold
| violation. Then the spam wouldn't be needed.
| fireworks wrote:
| I'm not blind, they didn't send one. And they admitted to
| me that they did not send one.
| eptcyka wrote:
| That's a major failure on their part. I did get several
| nagging emails from them.
| yamtaddle wrote:
| Well, that's even worse, of course.
| edgyquant wrote:
| I use heroku at work at they definitely sent out a ton of
| emails. It also said they were going to delete them right
| above all third party plugins (even if you weren't using
| their database service)
| inopinatus wrote:
| This sounds like the legal equivalent of looking the wrong
| way through a telescope.
|
| Whoever fostered that naive interpretation was a nitwit. If
| they're an actual lawyer, they promoted an intentional,
| mutually harmful unilateral reinterpretation of an agreement
| and should be sacked.
|
| Cowering behind T&Cs like this is intellectual bankruptcy.
| There's _always_ another solution. The law is not a
| programming language.
| numpad0 wrote:
| Using the fact that a customer was shown exit as the basis
| for destroying their assets don't look great to me, at least
| on surface...
| tacker2000 wrote:
| [flagged]
| craigkerstiens wrote:
| Having worked at Heroku and had a large part in building
| Heroku Postgres I do not recall this explicit policy, and it
| seems very squirrelly to me. Maybe this came in as a policy
| in recent years and it is the case, but still seems like
| hiding behind a policy as opposed to doing right by
| customers.
|
| You could easily block all incoming connections to the
| database. For a free database of 10k rows there were no SLAs,
| and you would still technically be hosting the database.
|
| Even taking a dump and emailing it to me feels like a safer
| option here.
|
| There were better answers here for sure. If the honest answer
| is we just didn't feel the effort was worth it for this class
| of users at least own that.
| CoastalCoder wrote:
| > Even taking a dump and emailing it to me feels like a
| safer option here.
|
| I genuinely had to read this twice to get the _intended_
| meaning.
| yuubi wrote:
| as someone who doesn't use heroku, so disregard my
| opinion:
|
| i'd probably prefer feces-by-email to surprise database
| deletion
| tptacek wrote:
| Having been through a SOC2 audit: this wouldn't fly. It's
| on the checklist of issues that you get hit with regardless
| of what kind of company you are: when customer accounts are
| terminated, the data retention clock starts ticking.
|
| You can pick an arbitrary time frame for retention, but
| whatever you pick, you have to communicate to users, and
| you can't just change it on a whim. Normal customers want
| this clock _short_. They don 't want you to retain their
| stuff after they cancel.
| shkkmo wrote:
| The customer account wasn't terminated, the free DB being
| used by paid Dynos was deleted without any input from or
| notification to the customer.
|
| I highly doubt normal customers want this clock short
| when the cancellation is not customer initiated.
| tptacek wrote:
| Data retention policies are written by compliance people,
| not product people, so distinctions like overt,
| deliberate cancellation and "cancellation for nonpayment"
| or "abandonment" or "discontinuation" usually aren't
| captured in them. I'm not saying it's great that Heroku
| deleted these databases; I'm saying: the description
| given upthread, that the databases were deleted because
| of contractual requirements, is super plausible.
| shkkmo wrote:
| Plausible, sure, but that doesn't mean it isn't a cop
| out. There are many things that Heroku could have done to
| prevent this, like delaying the disconnection when they
| don't have confirmed delivery of the notifications and/or
| when they are connected to paid dynos/accounts.
|
| Thus didn't happen because of the contract, but because
| the people implementing this transition didn't give a
| crap.
| crote wrote:
| But this isn't an account termination. They deleted the
| data of an _active_ account, without informing the user.
|
| Why could they not turn it into a read-only database
| without access from the Heroku apps instead? Then it'd
| just be a routine change to the service offered, would it
| not?
| [deleted]
| robryan wrote:
| Yeah ideally hold onto a backup for say a year, if the
| owner hasn't come and downloaded it after a year can then
| assume that they don't want it.
| johannes1234321 wrote:
| It's not that easy. Those databases might contain
| personal information which is protected under different
| privacy laws like GDPR, HIPAA and others and
| Heroku/Salesforce can not simply store that for longer
| than agreed upon and Heroku cancelling the account
| enables the retention period as the customer agreed upon
| as part of the T&C.
| tuukkah wrote:
| The account was not cancelled.
| dboreham wrote:
| That's peak idiocy and the product of lawyers taking over the
| assylum.
| mytailorisrich wrote:
| Please. You'd just need to ask if the customer is OK with 90
| days instead of 30. Done.
|
| The company has no commercial interest in doing that, though.
| porpoisemonkey wrote:
| It doesn't sound like this would have helped in this
| particular case since they were unable to contact the
| customer.
| mytailorisrich wrote:
| I was replying to a bullshit claim that they cannot
| retain for more than 30 days no matter what.
|
| That's hiding behind the T&Cs instead of owning their
| decision not to even try because there is nothing in it
| for them.
| joemi wrote:
| So you're in favor of companies breaking their terms and
| conditions at will? I think that would cause quite a lot
| more outrage and problems.
| reaperducer wrote:
| _So you 're in favor of companies breaking their terms
| and conditions at will? I think that would cause quite a
| lot more outrage and problems._
|
| Do you really never get e-mails from various companies
| every few weeks telling you that the terms of service
| have been updated?
|
| I think I get one from eBay alone every other day.
| johannes1234321 wrote:
| Those can not be compared that easily. There are many
| changes you can do easily. Changing data retention
| periods impacts guarantees customers may have given in
| compliance to privacy regulation like GDPR, HIPAA etc.
| and thus is a substantial change to the customers in
| areas with such regulation.
| rurp wrote:
| Contracts can be amended when all parties agree on a
| change.
| mytailorisrich wrote:
| You did not read my comments, did you?
|
| The T&Cs are an agreement between the parties. That
| agreement can be changed at any time if both parties
| agree. So they just need to ask.
| [deleted]
| celestialcheese wrote:
| Yup - this happened to me too. It was wildly frustrating, but
| they did send few emails looking back - it was just a really
| small project and it wasn't on my radar.
|
| What pissed me off most is that I WAS paying for the account. I
| was paying $17/mo for redis and dynos, so there was an active
| card on file.
|
| Why not just start charging for the postgres db, and only delete
| if there's no active billing?
|
| Heroku was already a no-go for me with new projects, I just keep
| old things running in there since it's too much work to migrate
| off. This just cements that for me.
| tchock23 wrote:
| I also didn't receive a notification about it, but some good
| samaritan HN user posted days before to give people a heads up.
| That's the only way I found out.
| dboreham wrote:
| They were charging money for thing A, that connected to (free)
| thing B, and rather than just begin charging the same customer,
| using same credit card, for thing B, they deleted it. I would
| love to see the recording of that zoom meeting...
| kragen wrote:
| does seem like a cautionary story that should rule heroku out for
| anything except quick experiments
| collectedparts wrote:
| This happened to me as well: projects with paid dynos but free
| databases, databases got nuked.
|
| How Heroku missed this is beyond me. They managed to screw over
| paying customers in their broad attempt to stop freeloaders.
|
| These are good accounts with credit cards on file. Why not just
| autocovert me to the lowest tier paid database?
|
| FWIW I was able to get them to restore my databases. But I also
| had free Heroku Redis on one my projects and that, they assured
| me, is gone forever.
| sieabahlpark wrote:
| Redis is a memory store, you're the one who fucked up treating
| a cache as a persistence layer.
| [deleted]
| itake wrote:
| Redis offers disk persistence [0]. Why can't it be used as a
| persistence layer?
|
| [0] - https://redis.io/docs/management/persistence/
| ceejayoz wrote:
| Redis offers disk persistence; Heroku's free Redis offering
| did not.
| robertlagrant wrote:
| You can use Redis as a cache, but it's not fundamentally a
| cache.
| addandsubtract wrote:
| I got "lucky" because I randomly checked the status page on my
| Heroku dashboard. It laid out which services were going to cost
| me after the switch date. I upgraded to the Eco tier and
| noticed that the DBs were still marked with a warning. So yeah,
| I would've been burned as well, had I not done that.
| fireworks wrote:
| Right? Like just charge me and tell me. Also, while I still
| think it is way too aggressive how they deprecated free in
| general, at the very least... maybe uh... I don't know, let me
| know if you are going to delete my database? Absolutely insane.
| favorited wrote:
| Same exact thing happened to me. Out of nowhere, my (paid) app
| started failing. It was only after signing in did I see that the
| entire postgres database had been removed. No warning that this
| was going to happen, no ability to recover a backup. Just...
| deleted.
|
| Not the end of the world for me, because 95% of the data I had in
| there had already been processed, but I did lose some. Complete
| joke of a service.
| bryanrasmussen wrote:
| reading this I keep thinking the real questions are:
|
| 1. probably the notice sending and db deletion is two separate
| teams or responsibilities. Are they?
|
| 2. Did people know there was a bug in notice sending so some
| notices were not being sent, all notices not being sent? I ask
| this because generally in places I've worked where notice sending
| was an important part of things you knew if there was a bug and
| notices were not being sent. But maybe it wasn't that important
| for Heroku. Maybe it was not known that notices were not being
| sent for a while - or was it known immediately but things on
| other parts of business chugged along anyway.
|
| 3. If they knew notices were not being sent and they went ahead
| and deleted db anyway, seems messed up, but that would probably
| be ok with people if they had data retention for people who did
| not get notices sent.
|
| 4. The whole thing your stuff can be deleted at any time without
| telling you is basically probably true almost everywhere in that
| notice sending can have a bug and deletion of stuff is probably
| not adequately tied to notice sending so that if notice not sent
| automatic deletion is stopped. Which I'm thinking is probably
| everywhere - if you work somewhere with automatic deletion and a
| notice sending module - what happens? Is this scenario handled?
|
| 5. answer to this is probably not, but is there a legal issue if
| notices not sent and stuff deleted, issue might be if some
| notices were sent - if account A gets notice about deletion and
| is thus able to act on it and account B does not get notice and
| is thus not able to act on it there might be a ground for action.
| Probably not, but when something seems unfair there might be a
| law that can be stretched to fit it.
| deepsun wrote:
| I remember in Google one project sent some announcement to
| incorrect recipients. Upon investigation it turned on that the
| bug was in all SQL queries that selected relevant customers, but
| it was broken for years and no one complained before our
| investigation.
| MzHN wrote:
| We also had some of our free databases on our Team tier projects
| nuked, which Heroku said would instead be automatically upgraded
| to lowest paid tier. No warning emails either, to anyone on our
| team.
|
| My personal account did get a bunch of warning emails but nothing
| on our business accounts.
| numpad0 wrote:
| Heroku also had weird business with Skeb.jp on 12/23-24 JST that
| was literally solved under the table - after Skeb redeployed to
| AWS. It seemed like a policy decision being made at Salesforce
| which owns Heroku. Whether such behavior adds confidence to their
| business users, I don't know.
| sneak wrote:
| Anything stored in a cloud provider you're not paying a minimum
| of $10k a month should be assumed to be subject to evaporation
| without recourse at any time.
|
| Make offsite backups.
|
| For this use case, a raspberry pi 4 with a 1TB SD card hidden
| somewhere in your home with a cronjob is probably more than
| enough.
| rovr138 wrote:
| Well, I moved everything I had to a VPS for the time being
| because I left it to the end.
|
| So I take this opportunity to ask, what alternatives have people
| moved to? I really haven't gone back to look what's out there.
|
| Not looking for free, just alternatives to review.
| xp84 wrote:
| For what kind of an app? If I was tasked to put an app
| someplace today, the answer would depend if it was easily
| containerized, and what kind of scale you're talking about.
|
| For small apps, if containerized, I would host on Google Cloud
| Platform using App Engine Flexible Environment:
| https://cloud.google.com/appengine/docs/flexible Add on their
| managed database, of course.
|
| But also, random VPSs are so cheap that if it works fine as is,
| you could do a lot worse than just running it on Linode,
| Digital Ocean, Hetzner, etc.
| donnieashok wrote:
| I went full blast into SQLite, agreed there's a migration job
| in between. But it's totally worth it. In fact my app is now 5x
| faster.
| dcow wrote:
| You can host static sites on DigitalOcean's app platform for
| free. Their lowest tier offering for non-static sites is like
| $5/mo. Might be worth checking out.
| robbiep wrote:
| I also had this happen - unbelievable really
| latortuga wrote:
| This happened to me but I filed a ticket and they were able to
| restore it. I guess maybe it was within the 30 day window?
| yellowapple wrote:
| Something similar happened to me on IBM Cloud, back when it was
| called Bluemix. This was early in my career for an early-stage
| startup (that eventually shuttered - not due to this issue, but
| rather due to a lack of investor interest and us running out of
| runway as a result). Easy enough setup: staging and production,
| each with two Docker containers, one for the (Elixir/Sugar) app
| and one with the (Postgres) database.
|
| Well, one day the production Postgres container just... vanishes.
| All the storage is gone. After weeks back and forth with IBM's
| support, they confirm that the loss is permanent. No explanation,
| no refunds, just "lol fuck you". Naturally, we didn't have
| backups yet, so the few users we did have now had to start from
| scratch.
|
| I, too, learned my lesson about not doing an offsite backup
| regularly - and the bigger lesson that Bluemix should've been a
| last resort option. Funny enough, we had migrated to that from
| Heroku; we ended up migrating again to AWS (specifically: Elastic
| Beanstalk).
| marvinkennis wrote:
| Heroku sent me repeated messages that they would shut down my
| account due to inactivity. I was fine with that because finding
| and turning off the supposedly active Dyno was impossible. Guess
| what? Still charging me $27 a month for a server I can't even
| manage.
| dns_snek wrote:
| If this is true I think it deserves spotlight in a separate
| post.
| mtmail wrote:
| For reference here's the email another user received in November.
|
| "Remind HN: Heroku will delete all free dbs and shut down all
| free dynos Monday" https://news.ycombinator.com/item?id=33755651
| lalopalota wrote:
| For me, the messaging in the announcements, dashboard, emails,
| and discussions here was clear and obvious that free databases
| were going to be deleted. If someone managed to miss all that,
| that's on them.
| fireworks wrote:
| No in-app notifications, no emails, and I was travelling and
| not following the news closely. I don't think I should have
| to rely on forums to know if I'm going to have all my data
| deleted without warning as a paying customer anyway.
| yonrg wrote:
| I can just second most of those comments here. One thing should
| be clear: It doesn't matter if you have a payed or free plan with
| someone who stores your data. Always make backups! There are so
| many things what can go wrong
| coffeeblack wrote:
| Never trust any company or hardware. Either can fail you at any
| moment. Always keep your own backup. Still sucks though.
| cameronfraser wrote:
| Same happened to me, I didn't receive any email notification or
| anything, found out when things started failing.
| purpleblue wrote:
| You didn't have a full backup of your database? That's on you.
| Disaster Recovery is an important part of any project. Heroku
| could have blipped out of existence by accident and you would
| still have the same problem.
| marcelaguiar wrote:
| Same here. I have 2 big gripes:
|
| - I did not receive any email warnings that this would happen. I
| have a valid email registered with them because I do receive non-
| promotional emails. However nothing for this. After getting in
| contact with customer service their rebuttal was that I also
| would've known by checking the forums/blog. Who on earth is doing
| that? It's a terrible response.
|
| - They detached the DBs at the start of December. I wasn't going
| to fix my broken personal projects over the holidays. I told
| myself I would take a look come January. Which by that point it
| was past the 30 day grace period.
|
| In the end, I've had to completely stop paying for the dynos. So
| Heroku lost business overall from me.
| menzoic wrote:
| Times like this are great opportunities for growth. This can help
| you build consistent habits in the future to have more durable
| storage practices.
| gwbas1c wrote:
| What frustrated me about Heroku nuking free projects was that the
| paid version was, IMO, too expensive. (Granted, I understand that
| the free tier was probably a money sink for them.)
|
| $5-10 a month is pocket change in some contexts, but for a hobby
| or a one-off project, it's a huge chunk of change.
|
| I really wish they offered a lower-cost tier.
| shortcake27 wrote:
| Or let me share dynos across all my projects. $5-$10 per month
| for all projects, sounds good! $5 per month for each project
| adds up very quickly.
| throwawaykai wrote:
| That was the new "eco" plan that was introduced alongside the
| ending of free plans -- $5/mo for 1000 hours shared across
| all eco dynos.
| gwillen wrote:
| I definitely wish you guys had advertised that earlier /
| better. I feel like I got lucky because I put off migrating
| until just before the sunset, at which point I discovered
| eco dynos. I swear they either didn't exist or were harder
| to find when the sunset was first announced. Thanks for
| making them available at all, though.
|
| (Context: I maintain an app for an event that runs about
| one week a year. The ability to use free dynos the other 51
| weeks means we retain the ability to do one-off analytics
| queries, minor development work, etc. during the off-
| season, without having to delete and recreate the app or
| something every year. Eco isn't quite as good as free, but
| it means we can still have separate staging and production
| instances during the off-season, without paying extra for
| staging to be idle, and without having to destroy and
| recreate staging every year.)
| shortcake27 wrote:
| Wow, I also don't recall seeing this either. It must have
| been marketed poorly. Same for Postgres, looks like there
| are now some more cost effective options which weren't
| available when I migrated my stuff away.
| m00x wrote:
| That's 2 cups of coffee, or 1 latte. It's objectively not a
| huge chunk of change unless you live in a very LCOL area.
|
| You can also go to any competitors, which have their smallest
| tier around the same price. Linode has a $5/m tier.
|
| You could also buy a Raspberry Pi for $50 and run it in your
| closet.
| Towaway69 wrote:
| My personal take is that I'd rather work with heroku than
| dealing with aws or gcs.
|
| Sure there are others out there that provide the same easy of
| use, I am just too lazy to switch ;)
|
| OTOH I've never had them wipe a database on me, so I guess that
| would be the motivation to move on.
| xp84 wrote:
| > I've never had them wipe a database on me,
|
| Give it time, apparently ;)
|
| If you're like me and are really into ease of use, I really
| endorse Digital Ocean. People don't seem to use it in Big
| Important Companies, but I never had a problem with anything
| I hosted there when I was my last, medium-sized company, and
| everything was so easy to manage in the GUI without having to
| learn proprietary APIs and whole stacks of infrastructure-as-
| code stuff like in AWS land.
| Towaway69 wrote:
| DO is also good and less restrictive for a new user than
| aws (I.e. getting more resources). I remember having to beg
| to get more than 1(or 2) servers at aws as a new user.
|
| I haven't looked recently but does DO have the heroku
| _click to add resources and git push to deploy_ offerings?
| jacobsenscott wrote:
| On heroku be sure you enable database backups with heroku
| pg:backups:schedule, and _also_ script syncing those backups on a
| regular bases to some "off site" (non heroku) storage like your
| own s3 buckets. It is easy to get the URLs to you backups with
| `heroku pg:backups:url`.
|
| I have no faith that if a rouge employee clicks the "delete app"
| button (because did you know to give an employee the ability to
| update the ssl cert on your web app you also need to give them
| permission to delete the whole damn app?) you'll ever be able to
| get your database back (although you might have a 30 day window
| to do so, but I wouldn't trust it.)
| JEDI-HACKER wrote:
| Check your junk mail folder.
| [deleted]
| javawizard wrote:
| > They also admitted that there was "an issue" sending out
| notifications to me, and confirmed that none were sent.
|
| Junk mail wasn't the issue in this case.
| hot_gril wrote:
| A bit of a tangent, but my only real problem with Heroku involved
| a premium DB. Turns out that upgrading to premium enables high
| availability (HA) by default, and I don't even remember if you
| can disable it. HA replicates asynchronously to the standby
| master, so a master failover can cause a small amount of data
| loss. For my application, this was unacceptable, and I would have
| preferred unavailability instead (see CAP theorem). Today I have
| enough experience to check the fine print for that kind of
| detail, but anyway such a big change should come with big bold
| letters IMO.
|
| [Edit: To this day I'm still puzzled by what I'm about to
| describe, so idk if it's Heroku's fault or mine.] I got a call
| from my colleague one day saying our database had gone back in
| time. Evidently we lost an hour of records. The code wasn't even
| capable of deleting rows, and nobody had direct DB access but me,
| so after leafing through the docs I suspected a failover event
| caused it. Premium DBs also let you roll back the DB to a
| previous point in time, and we were able to recover most of our
| data this way, like _Back to the Future_. If this really was a
| failover event, it 's super weird if that the backup was more up
| to date than the standby master, and that a whole hour (rather
| than minute) was lost.
| londons_explore wrote:
| I wonder what architecture they use that can lose an hour of
| data?
|
| Most architectures I see might lose a few milliseconds of
| writes in the typical case, and perhaps a second of writes in
| the worst case (which occurs when the master gets islanded with
| a couple of clients).
| hot_gril wrote:
| If it was really the HA causing this, maybe the follower had
| a temp outage before the leader and hadn't yet caught up.
|
| I already don't want HA if there's a chance for even 1 second
| of data loss, but for those who can tolerate that, there
| really should be an upper bound on the staleness. If your
| leader fails, the follower shouldn't take over unless it
| knows it's close to up-to-date.
| nightpool wrote:
| Having a HA follower is the _only_ different between Premium
| and Standard tiers, so I 'm not really sure what else you
| expected them to do in this case. Like, premium-6 is 2x the
| cost of the standard-6 plan explicitly because of the HA
| follower.
| hot_gril wrote:
| Yes. I was inexperienced, saw "high availability," and didn't
| realize the standby could fall behind and lose data.
| hot_gril wrote:
| Also, it's worded like a strict improvement when really
| it's a tradeoff. You're sacrificing the guarantee of
| persistence for more availability. I feel like most people
| who know what this means are not going to want it.
| Andys wrote:
| Yes, AWS is similar with their DB offerings. You can discourage
| it from doing any updates/reboots (which causes a failover),
| but ultimately if they want to failover, they can at any time.
| hot_gril wrote:
| Ouch. If it knows it's about to fail over from an update, it
| really should get the follower totally in sync with the
| leader first.
| codegeek wrote:
| "I learned my lesson about not doing an offsite backup regularly"
|
| Heroku is a shitshow after the Salesforce takeover and not to
| shit on you because I know it really sucks. BUT please everyone,
| do offsite backups and test them. Please people. Please. If you
| have anything that is important, BACK THEM UP on your own outside
| of the provider.
|
| Heck, we wrote our own script to backup RDS databases offsite as
| well even though RDS has backups and restore options. I want that
| database file.
| lamontcg wrote:
| Backups are apparently "too 20th century" for today's cloud-
| focused devops, you can just trust your entire business to
| someone else's procedures and if it all disappears in a puff of
| smoke, so do you.
|
| (or just maybe we shouldn't have thrown out all the sysadmins
| with the bathwater)
| kaushikc wrote:
| Don't mind my curiosity but I wonder if you do offsite backup
| of your emails or just take it for granted that Gmail is a
| reliable enough service ? It is very important to me but I
| never even considered backing up my Google takeout data. The
| point I want to make is that service reliability is a critical
| factor when considering what to backup. It was expected from
| Heroku to do the right thing and be conservative in their
| approach.
| TimWolla wrote:
| Personally I sync the emails in my Gmail account to my server
| at Hetzner using imapsync [1] every once in a while. The data
| on that server is backed up off-site at rsync.net using Borg.
| I'm intentionally not using Hetzner's storage boxes, because
| I want my backup to be entirely independent, e.g. if my
| Hetzner customer account is closed down for whatever reason.
|
| [1] https://imapsync.lamiral.info/
| rurp wrote:
| Anyone using Google products should _absolutely_ backup their
| data regularly.
| codegeek wrote:
| I hear you. I didn't do it for a while but I did 2 things
| more recently:
|
| 1. Setup IMAP backups locally on my computer and everything
| is backed up on computer to 3rd party backup tool
|
| 2. Use Google Takeout from time to time (it is a bit weird at
| times though).
| Max-q wrote:
| After the Salesforce takeover? The one in 2010? So Heroku was
| good in like a year!?
| hot_gril wrote:
| Heh. I always loved Heroku and have complained that
| Salesforce is messing it up recently, but turns out the first
| time I used Heroku was like 5 years after the acquisition. I
| think Salesforce just started inserting their own branding
| more recently.
| justin_oaks wrote:
| I spoke out loudly at a previous employer how it's really dumb
| to have our database and it's backups under a single AWS
| account. A single AWS account compromise, account issue (e.g.
| AWS shuts down account), or a disgruntled employee could result
| in the business being destroyed.
|
| They took some half-hearted efforts to back up the data, but it
| was far from ideal.
|
| Back up your data, and do it in different places so a failure
| in one won't affect any other copy of the data.
| robryan wrote:
| Yeah pretty cheap and easy, at least relative to the value of
| data to most companies to setup a server you have physical
| control of to go download the backup once per day.
| hot_gril wrote:
| It's not exactly cheap to always have a backup that you
| know works. You have to set it up and test it periodically.
| Of course, that doesn't mean you shouldn't do it.
| fishnchips wrote:
| Very much that. As a former SRE responsible for tape
| storage, I saw things like regular backups of an error
| phrase "You have no access to this database". Guess what
| happened when the team accidentally dropped the database?
|
| Unless you're doing regular restores, you don't have a
| backup. You have hope. So yes, doing backups in a way
| that gives you some form of guarantee isn't exactly
| cheap.
| hot_gril wrote:
| Or we do have everything in the backup, but the restore
| process isn't worked out. Someone is losing a weekend
| writing hacky scripts, and every SLO is being violated,
| if we ever have to use it for real.
| tevon wrote:
| The reputation hit to Heroku is real. Its so too bad that they've
| made these changes, and stopped pushing new features.
|
| I built my prior business on heroku and it was wonderful. Now
| with this current company I've switched to Render (through am
| looking for something else). For me the writing was on the wall
| for Heroku and it wasn't worth it to me to dig in there if new
| features and support wasn't a given.
|
| Curious for anyone with an inside perspective. What happened?
| Heroku was so far ahead of the pack for awhile, then suddenly
| stopped staying a step ahead. Was this intentional? Why?
| tflinton wrote:
| Heroku could have done better. At a bare minimum offering an
| option to download the existing data for 1-year or convert it to
| a paid db would have been better options and not brought this
| type of despise from the community.
|
| I still use Heroku on a daily basis and wouldn't say this has
| caused me to re-evaluate my decision to stay with them, but then
| again i'm not using anything free from them.
| benediktdeicke wrote:
| Oh crap! What a horror story.
|
| We recently migrated from Heroku Postgres to Crunchy Bridge and
| can totally recommend it. So maybe set up your new database there
| instead.
| swader999 wrote:
| I'm curious how this affected your projects on paid dynos, can
| you provide more details around that?
| fireworks wrote:
| So I did not realize this at first (would have helped if I
| were, you know, emailed some information about this), but my
| project was still using a free Postgres addon despite being on
| a paid dyno setup. So that I suppose was nuked along with the
| general free tier deprecation.
| m00x wrote:
| Did they explain why the notifications didn't work?
| fireworks wrote:
| "I discussed this with our backend team and upon deeper
| investigation, we were able to confirm that there was some
| issue sending the notifications to your email."
|
| Nope.
| dboreham wrote:
| Guesses 1, 2 and 3 are : their MTA's IP was black listed
| by a reputable mail hosting provider's spam filtering
| system (because they send spam).
| joemi wrote:
| That phrasing doesn't rule out an issue on your email
| server's end, like the email could have bounced. (Though
| I'm guessing that's unlikely.)
| fireworks wrote:
| Would be surprised if that was this case - vanilla GSuite
| setup.
| m00x wrote:
| I don't really understand. You were paying for the dyno, but they
| deleted your instance anyway?
| riebschlager wrote:
| They were likely paying for a web dyno, but were using a free
| database instance.
| jack_riminton wrote:
| Free db but paid dyno in this case I think
| danpalmer wrote:
| They deleted the database.
|
| On Heroku, databases and dynos are completely distinct
| entities, each with their own payment plans.
|
| When you set up a dyno and "add" a database, as most people do,
| it's really easy to think that they are part of the same plan,
| and conclude that the "pro" or paid dyno is actually the combo
| of the dyno and database. They are not.
|
| I understand why they did this, databases can be shared between
| applications, that's handy, but this is a sharp edge.
| favorited wrote:
| The least they could have done is disconnect the database
| without deleting the data, so users would at least have a
| chance to make the decision to either delete or start paying.
| tetha wrote:
| This is something I've put into most workflows at
| $(WORKSPACE=/=heroku) that end up disabling a customers
| systems. It depends a bit on how hard we have to kill a
| system for various factors. But for the softest setting, we
| usually start pulling DNS or similar access configs for a
| minute, wait for a minute, disable for 2, wait for 2, ...
| and the 30 minutes never end.
|
| This gets people on the phone with support quickly if they
| don't react to their account manager or mails. Like, we've
| had a possible security incident coming from a customer
| system and they didn't react - because they tried to hide
| their processes internally. Scream-Testing got us on calls
| with their InfoSec in a hurry.
| brightball wrote:
| Best option is https://www.crunchydata.com/ IMO.
| rurp wrote:
| I remember being quite surprised when I first learned that Heroku
| was owned by Salesforce, because I had such a different
| impression of the two companies. I haven't used it in a few
| years, but Heroku used to be a great platform for certain types
| of projects. Unfortunately, the number of concerning stories I
| have heard about them in recent years has discouraged me from
| ever using Heroku for a future project.
| ioseph wrote:
| Happened to myself as well, luckily it was just our staging
| environment but it was while I was on leave. I feel one should
| reasonably expect their configuration to be stable to 3 months
| and 30 day data retention is just insulting.
___________________________________________________________________
(page generated 2023-01-31 23:00 UTC)