[HN Gopher] Accident Forgiveness
___________________________________________________________________
Accident Forgiveness
Author : piperswe
Score : 154 points
Date : 2024-08-21 18:27 UTC (2 days ago)
(HTM) web link (fly.io)
(TXT) w3m dump (fly.io)
| lagniappe wrote:
| This shouldn't be a selling point, it should just be table
| stakes.
| sealeck wrote:
| You can say this for a whole bunch of new features people role
| out but this doesn't erase the fact that they are very much not
| the stakes and introducing them is great because it raises
| them!
| tptacek wrote:
| We agree.
| morningsam wrote:
| >If you do something luridly stupid and rack up costs, AWS and
| GCP will probably cut you a break. [...] Everyone does.
|
| If the incidents that made the rounds here in the last few months
| are any indication, they'll start out insisting you pay no matter
| what. You'll then have to write a blog post about it, post it to
| Twitter, HN, and Reddit, get a couple hundred comments expressing
| anger at the provider, and wait for someone from their PR
| department to see it. Only then will they finally waive the
| costs.
|
| Good on Fly.io for trying to handle such situations more
| sensibly.
| bdcravens wrote:
| There's plenty of situations you don't hear about.
|
| A few years ago I f'ed up and accidentally pushed keys to a
| public repo, and by the next morning, we racked up $50k in AWS
| charges from crypto miners. We reached out, they gave us a
| security checklist that if we followed, they'd take off the
| charges. We did, and by Monday (my code push was Friday
| evening) the charges were taken off. No public shaming
| required.
| paulgb wrote:
| I'm curious what sorts of things were on the checklist. I
| wonder if it's something they will share proactively?
| bdcravens wrote:
| It's been a few years, so I'm going off of memory, but it
| was mostly best practices stuff (enabling Cloudtrail,
| rotating older keys, etc). Anything to ensure that once the
| attackers no longer had access, removing/monitoring
| anything that would have longer term implications.
| skeeter2020 wrote:
| this makes sense; plug the hole and stop taking on water
| before you ask for a free cleanup.
| perchlorate wrote:
| You likely already know that, but to anyone else interested:
| a good way to prevent these kinds of situations is to run
| 'nosey parker' on your git repo before pushing it to a
| remote. It will dig through your code and configs, looking at
| files and through all the git history, and highlight anything
| that looks like tokens, passwords, keys, etc. You can set it
| as a pre-commit hook to block the offending code from even
| being committed.
|
| https://github.com/praetorian-inc/noseyparker
| paxys wrote:
| > If the incidents that made the rounds here in the last few
| months are any indication
|
| They really aren't. There's a huge world out there beyond the
| HN front page.
| AtlasBarfed wrote:
| Another reason to be cloud agnostic. They are counting that
| switching cost will make you eat the bill.
|
| If you have to be cloud, do dev in one cloud and test/prod in
| another.
|
| I know, I know, easier said than done.
| srockets wrote:
| An argument that is often only made by folks who didn't ran
| anything at scale on a public cloud. If you've ever set in a
| room when a cloud deal was signed, you'd know it cable: you
| don't get to pay less by threatening to switch providers
| (I've seen people try to suggest that and get laughed out of
| the room), but by buying more from a single one.
|
| Hence accident forgiveness is in the same marketing bucket as
| free credits for new customers: reducing the aversion to
| trying the cloud and putting more work on it.
|
| And the switching costs, the biggest line item isn't building
| for multiple clouds (you can't be cloud agnostic:
| abstractions always leak somewhere, so you need to select a
| set of specific clouds to build for), but the cost of moving
| data between clouds. That's the real lock-in.
| hinkley wrote:
| In the IBM, HP days it wasn't about paying less per se by
| threatening to switch providers, it was getting more
| concessions (of which money might be one). Some big
| companies had both systems so they could play them off of
| each other. Time to order more hardware, who will kiss our
| butts more?
|
| I'd be very surprised if that doesn't still exist for the
| big boys. Though most of us are not big boys, and half of
| the biggest boys are cloud providers themselves.
| giraffe_lady wrote:
| I have personal knowledge of two cases of this happening and
| both were immediately dropped. One was a high school student's
| personal project gone awry for like $3k worth the other was a
| startup where it was $120k or so. Two different providers.
| Neither had to particularly argue their case, just ask and wait
| a few days.
| bdcravens wrote:
| To be clear, in the case of those abusing the policy:
|
| > We reserve the right to cut you off.
| paxys wrote:
| It's unfortunate that the solution to cloud pricing complexity
| that all providers are adopting is - add even more complexity on
| top.
|
| The number you see on your bill is increasingly calculated by
| running some black box algorithm on top of the billing events
| your resources generate. Was it accidental or not? What is a
| "weird" deployment vs a normal deployment? By what factor should
| the spikes on your billing graph be smoothened? None of this can
| be deterministically calculated from the pricing page. And
| there's no way for you to do these checks yourself _before_
| deployment because you have no idea what this logic even is. So
| you are entirely at the mercy of the provider for "forgiveness".
|
| Who wants to bet that some provider is going to launch "AI cloud
| billing" within the next year?
| bdcravens wrote:
| It's far from perfect, but the large providers do give you
| tools for categorizing your charges (tagging, etc). There's
| some fuzziness, especially around data transfer, but for the
| most part developers can look at an application and know what
| the cost of each resource is. The biggest risk seems to be out-
| of-control auto scaling turned on without doing reasonable
| analysis first.
| candiddevmike wrote:
| You'll pay exactly the amount they think you can afford to pay
| and not a penny less.
| tptacek wrote:
| That would be what you would pay in the absence of
| competition. This is an _incredibly_ competitive space.
| paxys wrote:
| Competition only matters for new contracts. Once you have
| picked a provider and made a big enough infrastructure
| investment, there's no realistic path to switch to someone
| else.
| javcasas wrote:
| They laugh at me for constructing as much as I can from
| the infra as plain kubernetes objects.
|
| Keep laughing, I can switch clouds in less time than you
| take for figuring out cloud costs.
| immibis wrote:
| Then how do AWS, Azure and GCP charge 100 times the amount
| for bandwidth as other hosting providers and the IP transit
| quote sitting in my email inbox?
| tptacek wrote:
| Why isn't one of those providers roflstomping the other 3
| (add OCI to the mix) with better pricing?
| pclmulqdq wrote:
| Because they land your data in their systems by offering
| you a sweetheart deal in year 1-2 and then they jack the
| price back to the extortionate level once you're stuck.
| Compute is portable between providers, but data is not.
| talkin wrote:
| With such a small group, chances are they all figured out
| that a race to the bottom won't reach the desired long
| term outcome.
| rendall wrote:
| "roflstomping"? What does it mean?
| ShroudedNight wrote:
| A beligerent that so completely outclasses their opponent
| that they can inflict existentially threatening losses
| with so little effort that its expenditure is
| indistinguishable from normal overhead
|
| Alternatively, when Stitches managed to ambush your level
| appropriate character alone in Duskwood
| kibwen wrote:
| If the cost calculation is complex and opaque, that
| successfully prevents anyone from evaluating what their
| costs would be on a competing service. And vendor lock-in
| makes it expensive and nontrivial to simply try it out.
| thenewnewguy wrote:
| I'd recommend thinking of this (and other accident forgiveness
| schemes from competitors) as a gesture of goodwill that rarely
| happens rather than an official part of the billing policy.
|
| If you actually look at your contract, no cloud provider is
| going to contractually obligate themselves to forgive your
| bill, and you shouldn't be planning or predicting your bill
| based on it.
| ToucanLoucan wrote:
| Anyone else remember we had widely available and completely
| understandable options to rent everything from baseline web-
| hosting all the way up to private rack services for actual
| years before AWS came along and apparently all of that was
| forgotten, Warhammer 40K style?
|
| I've rented a VPS from a vendor for going on 20 years now (Holy
| fuck I'm old) and I've never once been surprised at the bill.
| LAC-Tech wrote:
| Remember? Your own comment tells us this is not a distant
| memory! It is still available! And a lot of people use it.
| maliker wrote:
| You know that book "JavaScript: The Good Parts"? There needs
| to be a similar one "AWS: just the good parts". It would
| probably talk about EC2 (VPS), S3 (cheap bulk storage), SES
| (emails), and that's about it. When folks get into the
| elastic-super-beanstalk-container-swarm-v0.3 products, that's
| when they really kill themselves on the bills.
|
| That said, yes, just using a VPS vendor is the easy way to
| stick to the good parts.
| rob wrote:
| Speaking of complexity, Fly.io recently introduced region-
| specific pricing as well:
|
| https://community.fly.io/t/region-specific-machines-pricing/...
|
| https://news.ycombinator.com/item?id=40873001
| tomjen3 wrote:
| I really, really wish there was just a way to put a hard limit on
| how much a cloud could charge me.
|
| I am never going to want to spend 200k of my personal money on
| some project on a cloud. Never. I don't even want my ant-based
| basket viewing project simulator to cost me 1 thousand dollars
| because it went viral and all clouds overcharge for bandwith.
|
| Just let me put in a limit.
| cube2222 wrote:
| I think this sounds easy, but it's actually quite nontrivial in
| practice. You need special handling of such a limit for each
| kind of resource.
|
| E.g. when my limit is reached to they remove the database,
| along with all backups, and all objects on S3? Since storage is
| billed, it should also be stopped when the limit is reached,
| right?
|
| I think in practice among companies paying most of their
| revenue, there'd be zero interest in this, while it would be a
| lot of effort to implement.
|
| This is really something specific to hobby projects, which just
| shouldn't be using those "unlimited potential cost" services.
| candiddevmike wrote:
| I don't think any of these requirements are necessary. A
| basic "if I reach my bill limit, turn off things that are
| billing" toggle would suffice for 80%+ of users, especially
| with better billing controls/per-team billing accounts. I
| think you run into more problems/user frustrations with a
| tenuous conditional-shut-off approach.
|
| My tinfoil hat is that a lot of cloud billing is accidental,
| probably from "lab environments", and they don't want to
| provide a way to budget/limit these.
| tptacek wrote:
| The whole post is about this accidental billing situation.
| cube2222 wrote:
| I mean, those S3 objects, those backups, those databases,
| _are_ billing.
|
| "Turning them off" means deleting them.
| candiddevmike wrote:
| Sounds good to me, that's what offsite backups are for
| (if the data is even necessary)
| jabbany wrote:
| It doesn't have to be though?
|
| The provider could reject further access to them (reads /
| writes) once the limit is reached. The cost of actually
| keeping objects as "cold" storage has a natural cap per
| billing cycle since those are billed based on time.
| bee_rider wrote:
| What if my bug was saving, like, way more data to the cloud
| than I expected, and suddenly having a big bill for the
| hard drive space my data is using up just sitting there? It
| would be a pretty dumb bug, but hey, you never know, right?
| In that case they have to choose to either delete my data
| or keep charging me, so I guess there isn't an easy zero-
| cost "stop" option.
| HeatrayEnjoyer wrote:
| It there was a law about this companies would find a way.
| Where there's any uncertainty they can simply eat those edge
| costs, the extremely fat profit margins cover it easily.
| mappu wrote:
| You might be happier with a fixed-size VPS, instead of a cloud
| provider,
| pocketarc wrote:
| I started writing this comment by saying:
|
| Exactly - the whole point of a cloud provider is scalability.
| If you're doing a personal hobby project, get off big
| scalable clouds and get yourself one (or multiple!) fixed-
| price VPS or dedicated servers.
|
| But as I think of it, I think what people really want, for
| hobby projects, is not so much the scalability, but the
| managed offerings. They want zero-ops, zero-maintenance,
| zero-server-updates hosting, with a fixed price and hard
| limits. It won't be infinitely scalable, but it doesn't need
| to be - it's a hobby project.
|
| They just don't wanna sysadmin a server of their own. Which
| is completely understandable.
|
| There's room in the market for something like this.
| spencerchubb wrote:
| If you want a server that's fixed cost and zero
| maintenance, I recommend replit.
| scop wrote:
| One of the things I _really_ admire about Fly is "how human they
| come across". Fly is a collection of people providing a computing
| service to other people. As a fellow human, there is a very
| primitive part of my brain that will always prefer "tech from
| humans" over "tech".
|
| Given that the very teleology of cloud computing is digital &
| ephemeral resources, having an actual human face associated with
| it makes it tangible in a way that is hard to place.
|
| My little brain can briefly understand complex computer systems
| on a need-to-know basis, with knowledge constantly coming and
| going based on the demands of the day. I can never fully
| "understand" my cloud infrastructure. But hey, Kurt's standing
| there. I like Kurt. I can understand Kurt as he's a human and so
| am I and he's in the same exact place I am. Let's work hard and
| make cool stuff, what do ya say Kurt?
| scop wrote:
| I should add that another way Fly distinguishes itself as a
| "tangible" cloud company is that their blog art is all based on
| physical mediums. It feels very familiar and quantifiable to my
| brain.
| tptacek wrote:
| That's Annie Ruygt, our in-house illustrator. Check out her
| work; she's amazing.
|
| https://annieruygtillustration.com/
| gwbas1c wrote:
| > and may only come within tens of dollars of accurately
| predicting your water bill.
|
| Whenever I've had to water new grass seed I've always been
| surprised by my water bill.
| tptacek wrote:
| There's a note on the draft of this post with like 6 comments
| from people on our team about how the water bill is a bad
| example. I stand by what I wrote!
| hinkley wrote:
| There's a guy on Reddit a couple days ago showing off his
| softball sized watermelon he grew with a $100 watering bill.
| The struggle is real.
| aeternum wrote:
| New startup idea:
|
| Pool unused "accident" credits, automate the forgiveness request,
| and sell it to the highest bidder.
| datadrivenangel wrote:
| I believe that's called a reserved instance.
| joshfraser wrote:
| In other words, they are socializing the costs. Servers and
| electricity aren't free. Wouldn't it be better for the customer
| if they had no accident forgiveness and passed those cost savings
| along? Instead, you are paying for other peoples mistakes plus
| all the extra overhead caused from fraud that they incentivized.
| tptacek wrote:
| No, it would be worse if we "passed these cost savings along".
| trjordan wrote:
| The servers are already bought; the electricity cost is
| negligible for actual accidents.
|
| As mentioned in the post: the hosting providers don't actually
| pay marginal costs for transient mistakes, so neither should
| honest customers.
| joshfraser wrote:
| And what about the extra costs fighting fraud now that
| they've advertised this policy?
| DAlperin wrote:
| We have to invest in fighting fraud and abuse anyway, such
| is the public cloud business. We don't intend to diminish
| user experience in service of fighting it.
| osbulbul wrote:
| the biggest reason of I am not using aws, google cloud, vercel
| etc... for my personal projects is surprise bills. I am not
| earning anything from them so I can only put $50 but can't afford
| $500 or $5000. so still i will feel much more secure if I can put
| hard limit and absolutely sure my bill will never go above $50.
| (or cloud billing insurance :))
|
| but you got my attention, i can try fly.io
| tptacek wrote:
| Except for things like student accounts, which we're working
| on, I don't think hard limits are coming any time soon. Our
| expectation is that the enforcement of hard billing limits
| would mostly make customers furious with us. If you read to the
| end of the post, the direction we're going is preemptive
| detection of weird billing spikes, so you don't even have to
| notice and ask us.
|
| If you're really only looking to spend $50, we should put our
| cards on table and say that we're generally not making product
| pricing decisions with you in mind. If your needs are pretty
| straightforward, there are hosting providers that will do a
| better job of serving that business than we will.
| osbulbul wrote:
| yeah I am pretty sure no cloud provider thinks people who can
| willing to spend small money. but guess what? if one of my
| projects start earning money, then I will continue to what I
| know best :) I think that's why every cloud provider still
| has configs for couple of $
| tptacek wrote:
| We think all the time about people starting small projects
| here. I think a really good line to draw, if you want to
| understand how we think about this stuff, is between people
| who would be OK with us terminating their service because
| they mispredicted what their cap should be, and people who
| need their stuff to stay up and running and will just talk
| to support if they're concerned about billing. A shorter
| way to say this is that we making product pricing decisions
| for people who are doing this work professionally.
| gwbas1c wrote:
| One thing I'm really curious about is why caps are so
| hard? (Perhaps this would result in a more technical blog
| post?)
|
| IE, you clearly don't want to _terminate or shut down_ an
| account if they get too close to a cap. But what about
| things like a warning email, service slowdown, ect?
|
| Likewise, the old "slashdotted" or "hug of death" might
| be an appropriate result when something goes beyond a
| reasonable safety buffer?
|
| Anyway, just curious. It's clear that it's a complicated
| topic, and the real constraints and challenges are
| interesting.
| tptacek wrote:
| We talk about warnings in the post. We'll do that at some
| point. The hard part about caps is what to do when
| someone hits them.
|
| If there was a way to make caps work for our core
| customers, we'd do it. We're open to ideas. A theme of
| our work this past month and these next several months is
| extracting maximal value from ANFWWAONW, our new billing
| system. The thing you have to remember though is that our
| belief about our core customer is that they are averse to
| nothing more fiercely than service disruption.
|
| We're not in principle opposed to caps. We just don't
| have a product story for them that we're comfortable
| with. Keeping you from spending more money than you
| wanted to is an explicit product goal of ours (again: see
| post); we're just very wary of trading availability off
| against that goal.
| Alacart wrote:
| Configurable warnings as webhooks would be pretty cool.
| Then I can automate whatever needs to happen on my side.
|
| I already automate apps, machines, etc with the machines
| API and GraphQL, so my big worries in this area are:
|
| - Woops, some bad logic deployed too many machines
| (sounds like this policy helps) - Some kind of mistake or
| attack that just explodes bandwidth usage suddenly
| rincebrain wrote:
| It "seems" simple, but people have really complicated
| needs, and the simplest answer if you don't have the time
| to build and maintain something to cover enough of the
| state space to make enough people happy is to not support
| it.
|
| Some people are going to want you to go to 1Mb/s if you
| blow your bandwidth limits, or cash spend. Some might
| want you to go to 10 connections at once. Some might want
| you to just disable networking entirely.
|
| And what happens to your storage when you blow your
| billing? Or CPU time?
|
| It all becomes a hairy mess of state machines and
| companies wanting precisely _their_ requirements met, so
| you try to offer as much as you need to still have a
| compelling offering that enough people want to use, and
| no more.
|
| Probably at best you could provide an API to make it easy
| for customers to build their own state machine, but
| that's fraught because then the customer will still blame
| you even if their own code did the wrong thing.
| hinkley wrote:
| If there is a class of people who get steered to your service
| by a small number of organizations, you might. Universities,
| trade schools, a podcaster you're sponsoring.
|
| You can't scale to individualized service for $50 per
| month/quarter/year users, that's true. But you can shape
| policy for a demographic with... shall we call it flocking
| behavior, for lack of a better term?
| ilaksh wrote:
| I love your service, but this erroneous take makes me think
| that even if I was looking to spend up to $500 with a "real"
| business, you don't have me in mind.
|
| There should obviously be multiple types of caps. AWS and
| others have set a precedent that they can get away with
| "gotchas" for anyone who isn't paying attention.
|
| It's really the primary business model. People that aren't
| watching costs have more and more sneak in there.
|
| Does anyone know of a service like fly.io that has a
| perspective that's more friendly to bootstrapped startups?
| tptacek wrote:
| Let me know if you find them!
| Poiesis wrote:
| Thomas, every time I read something you write it's a delight. I
| love your writing style, and it reads to me like you're always
| putting effort into making it succinct yet unambiguous, without
| unnecessary embellishment.
| tptacek wrote:
| The trick is just that I write specifically for this place.
| bitbasher wrote:
| There is a very obvious fix for surprise billing. Enforce a
| billing cap and terminate service if it's met. Even better if you
| send alerts when the cap is approaching.
|
| If I pay $39/month, a default cap should be $39 per-month.
| Otherwise, let me set a cap I am comfortable with.
|
| Surprise billing is never good for customers, only the business.
| tptacek wrote:
| I promise, you are not the first person to have thought of
| this, and, believe it or not, there are reasons other than
| malice and avarice that cloud providers don't terminate service
| based on billing caps. Terminating service is a big deal.
|
| We agree completely about surprise billing.
| karmajunkie wrote:
| The only thing in cloud that's worse than a surprise bill is
| a surprise outage.
| lkbm wrote:
| Depends. For my hobby project that hasn't been monetized,
| being charged $100k is way worse than it going down.
|
| If it were critical infrastructure, or monetized in a way
| that brought in revenue to cover the charge, then maybe I
| don't want it to shut down despite skyrocketing costs, but
| that's hardly the only situation you could be in.
| tptacek wrote:
| Nobody is going to charge you $100k.
| immibis wrote:
| (ignores the multiple hobbyists who accidentally got
| charged $100k)
| tptacek wrote:
| My belief, and you can correct me on this if you have
| evidence to the contrary, is that nobody _actually_ pays
| those bills.
| withinboredom wrote:
| Someone paid those bills to somebody. AWS has costs
| incurred from it (support, opportunity, etc) and they
| paid them.
| bee_rider wrote:
| From that point of view, the cap seems more like a
| common-sense self defense feature that the cloud provider
| would want to implement. But we have cloud providers in
| this thread saying they don't want to implement caps, so,
| I dunno.
| srockets wrote:
| Getting charged is not the same as paying those charges.
| I'm willing to bet real money that all who, in good
| faith, went and asked not to pay those bills ended up not
| paying them (if they didn't ask, that's a separate
| issue).
| bitbasher wrote:
| Perhaps that is true, but the stress and anxiety from
| seeing 100K bill is real and has an impact.
|
| https://www.forbes.com/sites/sergeiklebnikov/2020/06/17/2
| 0-y...
| tobyjsullivan wrote:
| If you're stressed over lighting $100k on fire, then
| you're not the target market. /s
| gary_0 wrote:
| Just in case you don't know, the person you're replying
| to is the author of the blog post, and they mention in it
| that being stressed out by (potential) unexpected large
| bills is something they are aware of.
|
| Although even then, "we will only _threaten_ to charge
| you $100k without meaning it " isn't much of a
| reassurance.
| brundolf wrote:
| But more importantly: just let the customer decide! Let
| them decide whether there's a threshold where an outage
| is less costly than the hosting, and what that threshold
| is
| moring wrote:
| This made me think. There is usually some "hard" cost limit
| X that you cannot / don't want to afford, so terminating
| the service is preferable.
|
| There is also usually some "soft" limit Y < X that you
| don't want to exceed, and don't plan to exceed, but you'd
| rather pay >Y than face an outage.
|
| But a hard limit would have to be set to X to avoid that
| outage, and if it gets exceeded, you'll face a bill of X
| _and_ an outage.
|
| So what a customer would actually need is to specify both X
| and Y, with the rule: If the cost would exceed X, then
| terminate it early so the cost doesn't actually exceed Y.
|
| Sounds complicated to implement, but then, the current
| practice of waiving the bill is complicated too if you
| tried to formalize it.
|
| (For the sake of this discussion, I'm ignoring all the
| technical difficulties of terminating a high-availability
| service at all.)
| bee_rider wrote:
| Is this a soft limit or a trajectory prediction? I think
| there isn't such a thing as a soft limit. Nobody wants to
| spend any money really right? But you need to spend some
| to avoid losing service. That's just a cost you don't
| like but need to pay.
|
| I definitely get the idea of: I don't want to spend X so
| if it looks like I will, terminate service at Y. But I
| think that's a special case of the general situation, I
| want to know how much I'm on track to spend, right?
|
| But I don't know much about this at all. My whole
| experience was accidentally getting my own personal self
| a $500 AWS charge and then deciding they cloud services
| were dumb.
| jameshart wrote:
| It would, of course, be just _mean_ and _unscrupulous_
| for a cloud vendor to look at the number you have set as
| being 'the absolute most I am willing to pay for this
| service' and then optimize their pricing offer to you
| specifically to make sure they go right up to that line
| and no further.
| aidos wrote:
| Having flashbacks to the time where we had paid for a server
| and were paying for rack space for a customer and they were
| refusing to pay their bill. Our lawyers told us in no
| uncertain terms that turning off the server would be a
| terrible idea. "Obstruction of service" is the term that
| comes to mind.
| bitbasher wrote:
| Cloud squatting?
| alpinisme wrote:
| While the parent point about cloud providers having arrived
| at their policies thoughtfully, this particular issue is
| likely not part of the equation. There are plenty of
| services that run on a quota system (chat gpt, sentry,
| etc). There is a difference between shutting off a service
| the customer reasonably expected to be always on and
| shutting off a service when it reaches a threshold set by
| the customer as part of their purchase. The former is more
| like repossessing a physical good willy-nilly if the
| customer misses a payment or you find a check bounces...you
| can't do that.
| hinkley wrote:
| When you build an app for resiliency you end up with classes
| of service where the app fails in stages.
|
| But to extend that to the billing case, you'd have to have a
| partnership with your customers, not just a dashboard where
| they push buttons and an API where you add or delete
| machines.
|
| Maybe the website goes read only except for admin traffic
| when the budget is exceeded, for instance. Not as a bespoke
| process each company has to reinvent, but as functionality
| provided by the vendor.
| tmnvix wrote:
| My preferred option would be to have an optional billing cap
| that I can enable knowing full well that if it is exceeded
| the service would be terminated (obviously with notifications
| as that cap is approached). I could then apply this to simple
| hobby projects and such, while not having the risk of
| termination apply to more serious applications (though a
| 'soft cap' would be nice here so that I could still receive
| notifications as it approaches).
| tonnydourado wrote:
| When you're paying $39/month for something that generates
| $0/month, that is a very sensible policy.
|
| When you're paying $50,000/month for something that generates
| $200,000/month in value, or if an outage can generate
| $100,000/month in costs, or if the people that can fix an
| outage cost $100,000/year, then it's not.
| bitbasher wrote:
| Then that company's threshold for "sensible monthly cost"
| would be a lot higher. 500k? 1m? Give them the option to set
| something.
| lucifargundam wrote:
| I just wanted to say I first heard of Fly.io from "The Changelog"
| podcast
| brundolf wrote:
| I've never understood why providers don't just give you the
| option of setting a cost cap: "If this piece of infra exceeds [5x
| expected usage cost], shut it down and send me an email"
| theusus wrote:
| Their docs are all over the place. I don't understand what their
| default offerings are. And that makes it hard for me to chose
| them.
| tptacek wrote:
| What do you mean by "default offerings"? Thanks!
| theusus wrote:
| https://www.heroku.com/
|
| Checkout their products
| tptacek wrote:
| Y'all really, really want caps. We think that's a crazy feature.
| We don't want to do it. But I just had this conversation on a
| Slack with a bunch of friends at other companies and they were
| just as yell-y about this. There's a threshold of feedback at
| which I think you could get us to do some kind of cap thingo. I'm
| just saying.
|
| Yeesh.
| ustad wrote:
| In your article or in the posts here you have not said why you
| think its a crazy idea.
| tptacek wrote:
| I've said it over and over again: if you have a serious app,
| and someone somehow steals a credential from you and uses it
| to light up a bunch of crypto miners, you don't want us
| shutting down your main app in response. We perceive it
| mostly as a feature that will blow people up.
|
| We agree about the underlying problem! You don't want to
| spend $5000 in a month for services you never wanted. We
| don't want you spending that either. We'd rather just improve
| our billing so that you can fix this after the fact without
| trading off availability.
|
| But I have just planted a flag: we are prepared to be wrong
| about this. I don't think we are, but like, I'm the only one.
| :)
| ustad wrote:
| Crypto miners is the reason for not having a "crazy" option
| for "cap on/off" at sign up?
|
| You know what you can't fix after the fact of getting a
| huge bill? a heart attack.
| tptacek wrote:
| This post is about us not giving you that heart attack.
|
| I'm pushing back, and I'm going to keep pushing back; if
| we do this feature, it is going to be kicking and
| screaming. :)
| bee_rider wrote:
| I think your position is essentially that you mostly want
| to serve people who have serious apps, for whom the cap
| idea doesn't work. And you definitely know more about the
| general trajectory of your customers than all of us sitting
| here randomly speculating. But, is it really so uncommon
| for an application to transition from unserious, I want to
| cap, to serious?
|
| I guess I'm slightly confused because I thought one of the
| nice things about cloud services is that they give you the
| ability to fit your infrastructure to your size while you
| are trying things. If I'm still trying things, I might not
| even know if I'm serious yet, right?
| tptacek wrote:
| Sure, but bear in mind that across the entire industry of
| public clouds, which has been around for something like 2
| decades now, nobody really has this feature. In fact,
| didn't Google have this feature and then pull it?
| bee_rider wrote:
| Yes, clearly is it not the convention to provide this
| feature. But the chain of thought seems so
| straightforward and obvious to me: cloud infrastructure
| is great for exploring what you'll be doing, and
| guardrails make you more willing to explore fearlessly.
| I'm obviously missing something, though!
| belthesar wrote:
| I'll be honest, I think you've got an uphill battle to
| convince so many of us that are concerned about accidental
| pricing that this is exactly the thing we actually want,
| vs. what we feel we want. This position parrots what
| Vercel's leadership said when someone had a massive
| surprise bill, and the story made the rounds both here and
| elsewhere.
|
| To be super honest, you might be right too. You might go
| down a huge engineering effort to build this in only for 5%
| of your customers to ever engage it. I think the real
| question is what percentage of your customers will feel
| better knowing that the have the choice to set those
| limits, and how will that comfort actually improve their
| trust with Fly, and cause them to choose it over another
| cloud provider.
|
| It may end up being a lot like this feature we had in a
| platform that I used to support. It had a just-in-time
| analytics pipeline that at one point required tens of
| thousands of dollars in compute, storage and network
| hardware alone to function. Based on our analytics, it was
| barely used compared to the usage of the rest of our app,
| which made the zounds of resources and fairly frequent
| support attention it needed feel silly in comparison, so I
| advocated to sunset it. Product assured me that, regardless
| of how silly it might be to continue supporting this
| feature, it was a dealmaker, and losing it would be a
| dealbreaker.
|
| So yeah, y'all might be right in that the majority of your
| customers don't actually want it. But maybe what they do
| need is to know that it's there, ready for them if they
| ever need to engage with it.
| dlisboa wrote:
| > You might go down a huge engineering effort to build
| this
|
| This is an overlooked issue: billing caps are hard to
| implement and will likely incur losses for the cloud
| company that does.
|
| Take an object storage service as an example. Imagine
| Company X has a hard cap at US$ 1000 but some bug makes
| their software upload millions of files to a bucket and
| rack up their bill. Since objects are charged in GB/month
| they will not reach the cap until some time later that
| month. Then, when they do, what does termination of
| service mean? Does the cloud provider destroy every last
| resource associated with the account the second the hard
| cap is reached? If they don't, and they still have to
| store those files somewhere in their infra, then they'll
| start taking a loss while Company X just says "oops,
| sorry".
|
| That's what tptacek is talking about: you want to NOT
| destroy the customers' resources because they can quickly
| figure out that something went wrong and then adjust
| while still maintaining service. But the longer you keep
| the resources the more you're paying out of pocket as a
| cloud provider. If you can't bill the overages to the
| customer, which a hard cap would imply, then you're at a
| loss. Reclaiming every resource associated to an account
| the moment a cap is reached is an extreme measure no one
| wants.
|
| A hard cap then becomes only a "soft" cap, a mere
| suggestion, and cloud providers would then say "you hit
| the cap, but we had to keep your resources on the books
| for 12 hours, so here's the supplemental overages
| charges". Which would lead to probably just as many
| charges disputes we have today.
| lilyball wrote:
| I'm pretty sure I'm fairly far from your target customer,
| given that I'm not building products, and right now my only
| usage of fly.io is to host a tiny app that expects to be
| used a few times per _year_ by my spouse and me.
|
| Having said that, the reason why I personally would be
| happier with the ability to set a hard cap is that if I'm
| going to put a project on fly.io, I'm spending my own
| personal money on it. If I'm spending my own personal money
| on it, I want a guarantee that it _cannot_ possibly cost
| more than a given amount. When it 's my own money on the
| line, I absolutely want the service to shut down rather
| than have even 0.01% chance of costing me a lot of money.
|
| The moment I'm actually building something for a business,
| the moment it's company money instead of personal money,
| then priorities change and everything you're saying makes
| sense. But as long as I'm just a single developer playing
| with stuff and billing it to my personal credit card, I
| want that guarantee that I won't accidentally make myself
| go broke.
| amirhirsch wrote:
| Sounds like I should just DM Kurt until it happens?
| tptacek wrote:
| Absolutely you should. I'm intractable, he isn't.
| tedunangst wrote:
| Okay, so now that it's inevitable you add caps, can we wager on
| how long between rollout and the first front page "I told fly I
| wanted a service cap, but not like this!" post?
| knowitnone wrote:
| no matter what, you'll have some fraud. Perhaps a better solution
| is the customer has a know they can turn that sets max limits and
| some alerts on when usage is abnormal.
| csours wrote:
| > " You probably can't tell me how much electricity your home is
| using right now, and may only come within tens of dollars of
| accurately predicting your water bill. But neither of those bills
| are all that scary, because you assume there's a limit to how
| much you could run them up in a single billing interval. "
|
| I had a $600 surprise water bill. It was (partially) forgiven
| because the water department could drive to my house and see
| evidence of the leak next to my water meter. It did turn out to
| be on my side of the meter, so it is my responsibility.
|
| If the water department had driven to my house and seen evidence
| of commercial agriculture (so to speak), then it would not have
| been forgiven.
|
| ---
|
| The parallel here is that the water department can't come into my
| house uninvited - the cloud provider SHOULD NOT have intimate
| access to the running code, but they are able to observe some
| patterns without 'breaking in'.
|
| ---
|
| Side note: the size of the bill and the amount of forgiveness was
| largely driven by waiving an 'excess usage' surcharge - similar
| to how you can get a discount for cloud service reservations.
| ctennis1 wrote:
| Google Cloud will not cut you a break unless you have an
| incredible about of pull. Even then, you won't get much of one.
| smithcoin wrote:
| I'm suffering for this now. They just erroneously charged me
| almost $4,000 out of nowhere because they decided to scan all
| of old container images for no reason and without notification.
| I have a support partner and everybody is just passing the buck
| around, it's super frustrating.
| spencerchubb wrote:
| When I was in highschool, my friends and I were doing a small
| project. My friend accidentally ran Google's image recognition
| service in a while loop for a whole minute, and racked up a $300
| bill. Thankfully GCP waived it for us.
| reaperman wrote:
| I ran up a $4,000 bill with like a week of TPU time bc I didn't
| realize that they kept running after the attached CPU turned
| off (unlike GPU offerings).
|
| GCP also waived the whole thing - even refunding the $4k they
| had already pulled from my payment method.
| spencerchubb wrote:
| That's brutal, and if you're anything like me, probably
| learned a lesson very thoroughly for the future!
| ultra_nick wrote:
| Build everything on Docker, Kubernetes, and VPS.
|
| Run on the cheapest VPS provider with human support.
|
| Make a quick and easy switch if they get too shifty.
| andrewstuart wrote:
| Just run Linux computers on ionos with unlimited traffic and get
| on with the job instead of worrying about silly things like how
| much traffic passed over some internal software system and will
| it bankrupt the company.
|
| https://www.ionos.com/servers/vps
___________________________________________________________________
(page generated 2024-08-23 23:00 UTC)