[HN Gopher] GCP automatically lowered our quota, caused an incid...
       ___________________________________________________________________
        
       GCP automatically lowered our quota, caused an incident, and
       refused to upgrade
        
       Author : teej
       Score  : 168 points
       Date   : 2023-06-10 17:42 UTC (5 hours ago)
        
 (HTM) web link (twitter.com)
 (TXT) w3m dump (twitter.com)
        
       | redman25 wrote:
       | We had a similar thing happen with our company. Google had given
       | a bigquery concurrent query limit extension to 300 queries. In
       | January they removed the extension back down to 100 concurrent
       | queries because they were introducing "query queues" that should
       | cover the difference.
       | 
       | Unfortunately for us, query queues only affect spikes in
       | concurrent queries and not overall throughput. We've been
       | struggling with bigquery support ever since.
        
         | newhouseb wrote:
         | Quota management is indeed nonsensical. We serve large cash
         | assistance programs that have stampedes of people applying all
         | at once where we needed decent geocoding to determine
         | eligibility. We were exceeding the default 50qps quota. I asked
         | nicely (to double it, I think) and explained our use case and
         | they said... no.
         | 
         | So... we left and switched to Smarty and haven't looked back
         | since. We spend tens of thousands on geocoding annually. Really
         | mind-boggling behavior from GCP.
        
       | nathants wrote:
       | the purpose of not aws is two:
       | 
       | - put price/quality pressure on aws
       | 
       | - fill weird niches made by regulatory capture and other nonsense
       | 
       | not using aws is like not using linux. there are valid reasons,
       | but you don't want any of them.
        
         | aftbit wrote:
         | Well unless you just want to run your own hardware. That can be
         | orders of magnitude cheaper (business trajectory changing
         | stuff, think offering a free plan vs not) for the right kind of
         | workloads.
        
           | nathants wrote:
           | optionality. having aws doesn't mean using it for everything.
           | 
           | for most uses cases, netflix model seems like the right one.
           | control plane on cloud, data plane on not cloud.
           | 
           | at a minimum you probably want to backup some high value data
           | in s3 unless you have a more durable store somewhere.
        
       | Demmme wrote:
       | Just an anecdotal counter point: very happy with gcp.
       | 
       | Best network from all and coherent modern ui.
       | 
       | Not the usability hell like azure... (You know when clicking on a
       | often used resource on the start page which let's you jump
       | directly to it but doesn't allow you to jump a level up of all
       | the other resources of the same type which totally works fine
       | when you navigate to it the normal way... Or the huge hassle and
       | complexity of resource groups for f everything...)
       | 
       | But you know the tweet not even states what quota was reduced.
        
         | HatchedLake721 wrote:
         | UI in the world of terraform and pulumi where some companies
         | even block UI access in prod?
        
         | fnordpiglet wrote:
         | You lost me at UI. But I'm one who believes if you don't write
         | your software defined infrastructure as software, you'll regret
         | life pretty soon.
        
           | Waterluvian wrote:
           | A UI for cloud services is super helpful to have when
           | exploring, troubleshooting, and noodling around. But yeah,
           | massive red flag if anyone's using it to deploy production
           | services.
        
           | paulgb wrote:
           | That's a pretty myopic view of what a UI can be used for. I'm
           | all in on IAC but still like having a UI to click around to
           | observe state rather than memorize a bunch of CLI commands.
        
         | textninja wrote:
         | [flagged]
        
           | baxtr wrote:
           | They're the top comment now. Sometimes it's worth to wait and
           | see how it settles.
        
           | bluepizza wrote:
           | OP is being downvoted because his anecdotal defense of GCP
           | due to nice UI is irrelevant to the link posted. And probably
           | to cloud computing in general.
        
           | jiggawatts wrote:
           | A resource group is literally just a folder -- a name -- and
           | is the _best_ feature of Azure.
           | 
           | For comparison, any large AWS account is always a total mess.
           | Just an endless list of randomly named things that are
           | totally unrelated to each other.
           | 
           | If _that_ is his criticism, he deserves the down votes.
           | 
           | I bet his desktop has a hundred icons on it strewn randomly
           | where half of them are named "file.txt" and "document.doc"
        
             | coredog64 wrote:
             | AWS has Resource Groups:
             | https://docs.aws.amazon.com/ARG/latest/userguide/resource-
             | gr...
             | 
             | Resource Groups are supported pretty well within the
             | CloudWatch ecosystem (you can create an AppInsights
             | application from an RG and you can filter CW alarms by RG)
        
             | deathanatos wrote:
             | Well, part of his criticism is that the UI is woefully
             | inconsistent with regards to "upwards" navigation. If you
             | navigate to a resource via the resource group, you can
             | navigate back "up" to the parent RG. If you nagivate to the
             | resource from the home screen, IIRC, you cannot. That's the
             | criticism.
             | 
             | Yes, RG's are a pretty killer feature, and trying to
             | understand the organization of resources in GCP is hard by
             | comparison, and an utter nightmare in AWS. I'm not sure why
             | he's knocking that. (And ... GCP requires projects, which
             | seem equivalent to the complaint against RGs.)
             | 
             | ... that said ... there are so many other things utterly
             | and horrifically wrong with Azure that I wouldn't put them
             | on a pedestal for resource groups. As much as I do like
             | RGs.
             | 
             | (Also, "A resource group is literally just a folder" _sigh_
             | , no, because they're not hierarchical. Azure goofed hard
             | there.)
        
           | LoganDark wrote:
           | They replied to an incident report with "works for me". (With
           | an implied "sounds suspicious" in the last sentence, but I
           | don't pretend to know whether that's what they meant.)
        
           | seanhunter wrote:
           | I think they may be being downvoted not because their opinion
           | is different but because usability of the web UI is possibly
           | the least important attribute in choosing a cloud provider
           | for a lot of people given the use of tools such as terraform.
        
         | TX81Z wrote:
         | I was happy with Azure but just got a bill with "other" charges
         | for $1k on what is a basic VM used solely as a db replica.
        
           | richieartoul wrote:
           | Interzone bandwidth maybe? They used to not charge for that,
           | but they started doing so recently
        
         | elankart wrote:
         | This is certainly a troll post. I use all three cloud
         | providers. GCP is the worst of all, just try their simple text
         | to speech UI. It doesn't work most of the times.
         | 
         | Don't even get me started on a deployment story for GCP their
         | deployment manager is deprecated and redirect you to use
         | terraform.
         | 
         | I hate to swallow it but Azure was more usable and
         | straightforward.
        
           | VirusNewbie wrote:
           | Azure has had multiple global outages for many services. I
           | believe you can look at overall stability and see it's not in
           | the same league as GCP and AWS.
        
           | dimgl wrote:
           | > Azure was more usable and straightforward.
           | 
           | There is no world where Azure is more usable and
           | straightforward. Just my two cents.
        
       | showdeddd wrote:
       | This article seems silly, just add custom metrics to your app for
       | what was fetched from cache vs DB. And label the metric by
       | route/query/pattern. To control costs, don't tick the metric for
       | every single request, instead accumulate locally and post to
       | metrics API every X minutes.
        
       | jtchang wrote:
       | Why don't I hear about these incidents with AWS? Is it just less
       | common or different business takes on quotas and rate limits?
        
         | iot_devs wrote:
         | I personally reply to customers inquiries on a very famous AWS
         | product where the client was asking why the execution was
         | interrupted after the timeout they set themselves expired and
         | why it was a timeout error.
         | 
         | I am an engineer working on such products and it was a routine
         | on-call shift where I got this kind of question.
         | 
         | Admittedly we should have this cover by our support engineers,
         | that are very very good, but this one slip through and I took
         | the time to answer such query.
         | 
         | I am not a fanboy or anything, I could not be further from a
         | fanboy or a blind fan.
         | 
         | But after working on AWS, I do suggest it as very sensible
         | choice. Again, not because it is my employer but because they
         | really take care of operations and customers.
         | 
         | Screw ups can happen but they are very rare in my experience.
        
         | balls187 wrote:
         | AWS is great.
         | 
         | Gotta check out FBA/Amazin Marketplace to find all the nasty on
         | Amazon.
        
           | elankart wrote:
           | In what world is AWS great? A world where they don't have a
           | clean way to group resources and clean them up? Their own
           | twisted identity products in AWS that doesn't integrate
           | outside their cloud or their other products. A world where
           | they leave lingering resources when you cleanup something.
           | 
           | AWS has to much hype riding behind it.
           | 
           | It's not enterprise class and looks very incohesive.
        
         | JoshTriplett wrote:
         | I've seen a lot of incidents reported about AWS involving
         | quotas, in the other direction: "I can't sign up new customers
         | because I'm at a quota limit on resources", sometimes with "and
         | AWS is taking a long time reviewing a quota increase", with the
         | occasional "and AWS denied a quota increase".
        
           | marcinzm wrote:
           | Google wouldn't increase my quota unless I talked to a sales
           | person. I wanted 4 GPUs. 4. That took at least a week to
           | achieve.
        
           | coredog64 wrote:
           | AWS quotas come in hard and soft varieties. A hard quota
           | cannot be changed as it's factored into service operations
           | (e.g. APIGW's 29 second integration timeout). A soft quota
           | may or may not require review before being granted. However,
           | if it is a soft quota, the best thing to do is escalate to
           | your TAM after opening with the business impact of the
           | request. The TAM is paid out of your enterprise support
           | contract, so the least they can do is advocate for you.
        
             | cyclotron3k wrote:
             | My problem with the quote system is that it's largely
             | opaque until you run into it. Yes, you can pore through the
             | documentation and find the relevant quotas, but did you
             | find them all? Spoiler alert: no, you didn't.
             | 
             | I know they are trying to improve visibility on quota
             | limits, and they have a tool now, but in my experience it
             | was half baked and only knew about a handful of the limits
             | we were running into.
        
               | joecool1029 wrote:
               | Some of the AWS services have clear quota limits and it
               | works as expected, you give me 10k widget credits and if
               | I use 10k widget credits it stops working and I need to
               | open a ticket. Example of this would be SES.
               | 
               | Where it ends up being a bit of a nightmare is stuff like
               | SMS on SNS where they would specify a quota but we could
               | still hit it before their system reported it. We never
               | did figure out if they were looking at a rolling monthly
               | average and bursty campaigns could cause it to creep
               | above their projection. This was always a manual review
               | for this product and the only way we could avoid it was
               | getting way higher quotas than we needed approved.
               | Ultimately we ended up moving SMS over to Nexmo (now
               | Vonage) for bulk SMS so we didn't have to have potential
               | outages when the mystery quota of the now was reached.
        
         | adrr wrote:
         | Because Amazon will jump through hoops to make the customer
         | happy. If I file ticket to get a quota raised it usually
         | happens the same day and I have never had them push back on me.
        
           | theptip wrote:
           | Ran my startup on GCP. No regrets. Can't recall a time I
           | waited more than a few hours for a quota bump.
           | 
           | Only time I ever got pushback was trying to get them to make
           | an unreasonable quota increase to work around a bug in early
           | GKE that left stale network backend lying around, and when I
           | explained they approved it.
           | 
           | I've found GCP support to be reasonable, but you have to pay
           | for the enterprise tier. Their pricing model used to be
           | insane ($250/seat??) but they fixed that a few years ago. I
           | suspect a lot (not all!) of the complaints around here are
           | for hobbyist / free tier, which sure, is garbage. If you
           | spend $K/yr on support it's fine though.
        
         | rhtgrg wrote:
         | > Is it just less common or different business takes on quotas
         | and rate limits?
         | 
         | Neither, you're probably not seeking out such stories (there
         | are plenty, even on HN). Google tends to get more negative
         | attention on this particular forum, which might play a role,
         | but the larger component is probably just chance.
        
           | CSMastermind wrote:
           | I disagree that it's simple selection bias.
           | 
           | AWS is _extremely_ customer friendly and if this happened
           | would likely be offering dedicated support to make it right,
           | credits for the business loss, etc.
           | 
           | Google's customer service is the worst I've experienced in
           | the industry (like even speaking to a person is hard). While
           | AWS is some of the best I've received.
        
             | neilv wrote:
             | I'm sure that GCP _could_ turn around their HN customer
             | service reputation, if they wanted to prioritize that.
             | 
             | Though, when it might not be baked into the culture as much
             | as at AWS, I don't know offhand how to reconcile a customer
             | happiness turnaround with the rush to avoid being an also-
             | ran in the AI deployment frenzy.
             | 
             | (Are you going to assign conflicting KPIs in a company that
             | has cultivated a career-driven culture, and hope that
             | individuals strike optimal balances for the company? Or
             | partition the goal assignments to separate teams, when the
             | optimal outcome requires holistic thinking and behavior
             | across teams?)
        
               | etse wrote:
               | Wasn't this supposed to be Kurian's prerogative--sell
               | enterprise services? Google product seem fine to me, but
               | I agree that they have no customer care DNA. I was
               | expecting someone from the outside to bring it in.
        
               | pram wrote:
               | I worked in Kurian's org at Oracle and I was shocked when
               | Google hired him. OCI was a total disaster at the start.
               | Like embarrassingly bad, they threw most of the original
               | platform in the trash.
               | 
               | Literally failing upward.
        
             | rhtgrg wrote:
             | > AWS is _extremely_ customer friendly and if this happened
             | would likely be offering dedicated support to make it
             | right, credits for the business loss, etc.
             | 
             | It doesn't take much legwork to find counterexamples of
             | that claim, even on HN (see below, I spent 2 minutes
             | searching to find those).
             | 
             | > Google's customer service is the worst I've experienced
             | in the industry (like even speaking to a person is hard).
             | While AWS is some of the best I've received.
             | 
             | This is a bit too hyperbolic for me, but on balance I agree
             | that Amazon has better customer experience than Google.
             | That doesn't really answer what GP is asking, especially
             | given all the posts made about AWS on this very forum that
             | only sometimes get attention.
             | 
             | https://news.ycombinator.com/item?id=2478129
             | 
             | https://news.ycombinator.com/item?id=25224220
             | 
             | https://news.ycombinator.com/item?id=35375558
             | 
             | https://news.ycombinator.com/item?id=16283547
        
               | marcinzm wrote:
               | >That doesn't really answer what GP is asking, especially
               | given all the posts made about AWS on this very forum
               | that only sometimes get attention.
               | 
               | If these are your examples of angry AWS posts then I
               | think it just proves that AWS has amazing in customer
               | service.
               | 
               | >https://news.ycombinator.com/item?id=2478129
               | 
               | That's 12 years old but a legitimate complaint about AWS
               | customer service.
               | 
               | >https://news.ycombinator.com/item?id=25224220
               | 
               | Sounds like AWS had an outage, not sure how this is about
               | customer service?
               | 
               | >https://news.ycombinator.com/item?id=35375558
               | 
               | That's a list of AWS customer's having security incidents
               | and, again, nothing about AWS customer service.
               | 
               | >https://news.ycombinator.com/item?id=16283547
               | 
               | Not sure what this is about since the link is dead except
               | due to being 5 years old but, based on the upvotes, it's
               | someone failing to get people to have angry opinions
               | about AWS? Not sure how this is about AWS customer
               | service given the context.
        
               | rhtgrg wrote:
               | Given that this isn't handpicked and just top results
               | from a 2-minute search:
               | 
               | > That's 12 years old but a legitimate complaint about
               | AWS customer service.
               | 
               | Should have at least prompted you to do your own search.
               | The fact that it didn't is the end of this discussion.
        
               | marcinzm wrote:
               | So you made a claim that there's "there are plenty, even
               | on HN" and then utterly failed to provide any proof even
               | after trying. Then when called out you blamed the other
               | person for not doing so on your behalf. I'm going to
               | stick with my response.
        
             | kenhwang wrote:
             | I've seen my org send a silly amount of silly questions
             | that were obviously our fault to AWS support and they
             | always took it in stride. We very rarely needed to follow
             | up with our client executive (but they always offer to jump
             | in if necessary). When we did have serious issues, we had
             | no problem scheduling time with the executive managing the
             | product and the engineers that wrote the code.
             | 
             | Meanwhile we can't even get Google to answer an email about
             | serious incidents that were very obviously their fault,
             | much less assign a dedicated human point person (hell with
             | AWS, we even had several backup contacts assigned to cover
             | for vacation time). We were important enough to be featured
             | on their client success frontpage, but that didn't make a
             | difference in support quality we received.
             | 
             | I can't imagine why anyone would risk their business with
             | GCE. Especially since it tends to cost more than AWS these
             | days.
        
             | slowmovintarget wrote:
             | Amazon worked very hard to build a culture of customer
             | service. Google worked very hard to create tech and create
             | a developer-first culture.
             | 
             | You're seeing culture in operation.
        
             | [deleted]
        
         | oldtownroad wrote:
         | Amazon hate their employees. Google hate their customers.
        
           | neilv wrote:
           | Best of both worlds would be a place that takes great care of
           | both customers and employees.
        
             | yjftsjthsd-h wrote:
             | Well, yeah, obviously if you can have the best of all
             | worlds then by all means do that, but in practice
             | something's going to give; if you _did_ somehow prioritize
             | customer and employee well-being, it would almost certainly
             | manifest in much higher prices, not actually sustaining
             | quality, or both. (Now, charging more for a better outcome
             | does seem like a good idea, but in practice  >90% of
             | customers will go for the cheaper option.)
        
               | robocat wrote:
               | Most people seem to equate employee well-being with more
               | money.
               | 
               | Plenty of people care more about other job qualities than
               | how much it pays, and a well run business can create a
               | great environment without paying more than average. Look
               | for companies where turnover is almost zero (assuming
               | company is not growing) and you often find happy
               | employees.
               | 
               | Places with worse employee conditions have to pay more,
               | everything else held equal.
               | 
               | Anecdotally, I have certainly stayed in jobs with poorer
               | pay because I liked my colleagues and the working
               | conditions; but alternatively I have stayed in another
               | job mostly because the pay was great.
        
       | Animats wrote:
       | Google giveth, and Google taketh away. Blessed be the name of
       | Google.
        
         | civilitty wrote:
         | To our queries, it lights the way. In its Matrix, we are but a
         | doodle.
         | 
         | Blessed be the name of Google, come what may.
        
       | invalidname wrote:
       | From my past experience Google is by far the worst when it comes
       | to support. By far. Even if you pay for their gold support you're
       | lucky if you reach an engineer and even then they blame you for
       | the problem with no "proof" or direction. Most unhelpful ever...
       | See: https://medium.com/hackernoon/why-and-how-we-left-app-
       | engine...
       | 
       | AWS were surprisingly helpful by comparison and even pro-active.
       | Not a fan of theirs overall but much better than Google. This is
       | a deep cultural problem with Google that also expresses itself in
       | mobile development and everywhere:
       | https://dev.to/codenameone/google-play-kafkaesque-experience...
       | 
       | They aren't the cheapest and their service is terrible. I can't
       | think of a good reason to use GCP.
        
         | showdeddd wrote:
         | This medium article seems silly, just add custom metrics to
         | your app for what was fetched from cache vs DB. And label the
         | metric by route/query/pattern. To control costs, don't tick the
         | metric for every single request, instead accumulate locally and
         | post to metrics API every X minutes.
        
           | tyingq wrote:
           | > accumulate locally
           | 
           | Is that an option in AppEngine? The memcache docs seem to
           | indicate the free tier has undocumented eviction policies.
        
             | showdeddd wrote:
             | You have to implement that with your own code but it isn't
             | much more than a dict/map and a timestamp for last update.
        
               | tyingq wrote:
               | Same-ish problem, though. You wouldn't know for sure the
               | instance will run again...your dict/map data can be
               | dropped. I don't see any sort of instance timeout
               | callback where you could guard against that.
        
               | showdeddd wrote:
               | I think it's negligible. The only metrics you lose are on
               | rare scaledowns and they are averaged out anyways. GCP
               | likes to keep instances idle for a long time.
        
         | Spooky23 wrote:
         | Ymmv. I found a bug in their CA/PKI service, sent an example in
         | and got a callback and feedback from a knowledgeable engineer
         | who wrote me a workaround and put in a change request. I think
         | the bug was fixed in a few weeks.
         | 
         | My team works with lots of companies, Google is one of the
         | better ones in our experience. AWS is also excellent. I'm told
         | Azure is good, but most of my experience with Microsoft is with
         | enterprise product support, which is awful.
        
         | fastest963 wrote:
         | I've created several tickets against both AWS and GCP and had
         | good and bad experiences with each. I actually used to have
         | mostly bad experiences (a lot of useless back and forth, wrong
         | answers, etc) with GCP but the last few years it's actually
         | gotten significantly better. For example, we found a bug with
         | autoscaling specific instance types (it's now posted as a known
         | issue) and it was acknowledged and workarounds suggested
         | without much hassle.
        
         | brigadier132 wrote:
         | I'm only considering them because of GKE autopilot and cloud
         | run which both seem like completely painless ways to deploy
         | containers (I've setup prototypes with both).
        
         | textninja wrote:
         | I find the self-serve UI and overall experience with GCP to be
         | superior, so to me it's a bit like the premium you pay for a
         | Mac over a PC. Granted I never needed or wanted to speak to a
         | human, but their infamy in that respect deserves a special nod
         | of acknowledgment.
        
           | zmmmmm wrote:
           | That's sort of the flip side of the coin. Google firmly
           | believes that technology can solve everything and humans
           | shouldn't need to be involved out of principle. So they build
           | a great UI, great tools for devs and engineers to use, and
           | then provide absolutely zero humans to support that.
           | 
           | I have a hilarious situation at the moment where they decided
           | to shut down my Workspace account because they retracted the
           | grandfathered-in free tier. This has 15+years of history of a
           | small business I ran in it. I _tried_ to move it to a paid
           | account, but because the account was created in a different
           | country to my current billing country, the form doesn 't work
           | (can't enter address for credit card). And there's just
           | literally NO way to do it. No avenue to any way to contact a
           | human to quite literally _give them money_. So in the end I
           | gave up : downloaded the emails and resigned myself to losing
           | all the other history associated with the account.
           | 
           | But what's really hilarious is they can't seem to shut it
           | down either, perhaps for the same reason. Every time they set
           | a new billing deadline it warns me with another spate of
           | emails and then the date goes by and its still there. I
           | suspect their software doesn't know how to shut down a hybrid
           | multi-country account either, and there are no humans so they
           | are stuck.
        
       | endisneigh wrote:
       | Tweet has basically no details
        
         | Havoc wrote:
         | That's presumably how they felt too...
        
       | yashg wrote:
       | I keep getting badgered by GCP channel partners to move over to
       | GCP. One guy once said if I can show him our AWS usage he can
       | give me exact configuration in GCP and cost would be 20-25% less.
       | I was intrigued so I shared. He came back with a price estimate
       | higher than our AWS bill! I haven't entertained them ever since.
        
         | neilv wrote:
         | They lost a sale, but they got competitive data.
         | 
         | Maybe that's how that salesperson or their org thinks?
        
           | nordsieck wrote:
           | > Maybe that's how that salesperson or their org thinks?
           | 
           | Nah - it's just a numbers game. Sometimes they'll be wrong;
           | that's a dead end.
           | 
           | Sometimes they'll be right. And some of those times, they'll
           | make a sale.
        
         | yjftsjthsd-h wrote:
         | > One guy once said if I can show him our AWS usage he can give
         | me exact configuration in GCP and cost would be 20-25% less. I
         | was intrigued so I shared. He came back with a price estimate
         | higher than our AWS bill!
         | 
         | Now I have to know more! Did you call him on it? Did he have
         | any sort of explanation/excuse?
        
       | willtemperley wrote:
       | I had to close my GCP account because I was being spammed with
       | "Action Required" emails every day for a non-existent tax admin
       | issue. Bug reports were immediately closed and getting through to
       | a human seems impossible.
        
       | parpfish wrote:
       | I've recently had an issue where GCP keeps shutting down my
       | e2.micro because they think I'm crypto mining. I have no idea
       | what I'm doing to trigger it, and I don't blame them for not
       | sharing their secret anti fraud heuristics, but it's a freakin
       | e2.micro. They think I'm mining on that?!
        
         | missingdays wrote:
         | Why would they care if you mine crypto? You pay for resource
         | usage, you use resources
        
           | aseipp wrote:
           | Fraud vector. It's very easy to turn compute into dollars if
           | the compute is free or stolen. Cloud compute is very cost
           | ineffective for something like coin mining, so any actual
           | legitimate miners on any chain are going to just run
           | everything in-house anyway. In the end, that means the people
           | left in that pool predominantly fall into two groups: A)
           | people using stolen credit cards B) accounts that had hacked
           | credentials or leaked API keys running rampant.
           | 
           | Whatever heuristics cloud providers tend to use to discover
           | and "remediate" such behavior is a totally different thing
           | (e.g. obliterating an established account when an API key
           | might have gotten leaked and a few GPUs let loose is a little
           | overboard), but if I was offering a similar compute service,
           | and its sole purpose _wasn 't_ just coin mining, I'd almost
           | certainly ban it as well for similar reasons.
        
           | cavisne wrote:
           | Its impossible to profitably mine crypto on a public cloud so
           | anyone doing it is either stupid, or has no intention of
           | paying (much more common!).
           | 
           | Historically GCP/GAE has had a more generous free tier
           | without credit card requirements, so they've always been a
           | bit stricter on this than the other clouds.
        
           | herdrick wrote:
           | Providers often just ban all crypto mining. I think it might
           | be because of pressure from authorities fighting money
           | laundering.
        
             | Sebguer wrote:
             | No, it's because it's an extraordinarily common fraud
             | vector, and causes noisy neighbor problem because despite
             | GP's remark about 'paying for resource usage', cloud
             | providers don't actually anticipate anyone using 100% of
             | CPU 24/7/365. And the folks who really _are_ crypto mining
             | are never going to pay their monthly invoice.
             | 
             | Source: Worked at a cloud infra provider that struggled
             | deeply with this problem.
        
               | yjftsjthsd-h wrote:
               | > despite GP's remark about 'paying for resource usage',
               | cloud providers don't actually anticipate anyone using
               | 100% of CPU 24/7/365
               | 
               | Unless that's exposed to the customer (AWS burstable
               | instances are actually okay IMO), that sounds like the
               | cloud provider committing fraud and hoping nobody calls
               | their bluff.
        
               | Sebguer wrote:
               | You don't know what the word fraud means, and I recommend
               | reading these crazy things called 'terms of service'.
        
               | yjftsjthsd-h wrote:
               | > The crime of stealing or otherwise illegally obtaining
               | money by use of deception tactics.
               | 
               | > Any act of deception carried out for the purpose of
               | unfair, undeserved and/or unlawful gain.
               | 
               | (https://en.wiktionary.org/wiki/fraud#Noun)
               | 
               | Is the cloud provider getting money off of this
               | arrangement? => Yes, obviously they're getting paid here.
               | 
               | Is the cloud provider getting that money as a result of
               | deception? => Yes, by selling ex. the use of 1 CPU core
               | when they actually have no intention of letting you use
               | that core. Now, I'll grant that in an actual legal case
               | it might well be possible to get away with this by
               | burying it in the ToS, but this is HN, not a court room,
               | so I'm comfortable setting the bar at "would most actual
               | users expect that to happen based on marketing?",
               | conclude that no, most users would consider that
               | surprising even if there's something buried in the ToS
               | claiming that it's allowed, and call a spade a spade.
        
               | parpfish wrote:
               | the problem is that the terms of service say "don't mine
               | crypto", and I'm not. they've set up an imperfect
               | detection mechanism and I'm one of the false positives
               | but have no meaningful way to prove my innocence.
               | 
               | the worst part is that there's somebody in a GCP office
               | somewhere that might secretly believe that I'm actually a
               | cryptobro. _shudder_
        
               | sigstoat wrote:
               | > the worst part is that there's somebody in a GCP office
               | somewhere that might secretly believe that I'm actually a
               | cryptobro.
               | 
               | you're probably safe on that front, it isn't like the GCP
               | folks are paying any attention to the customers.
        
               | jmaker wrote:
               | So you pay expecting full utilization but are expected to
               | utilize on a fair-share basis only? And if I run my Monte
               | Carlo simulation, which can take a week to complete, my
               | instance is flagged for alleged crypto mining?
        
               | Sebguer wrote:
               | This generally depends on the tier of service you're
               | paying for. Every infra provider offers tiers that
               | provide dedicated resources. AWS handles this via CPU
               | credits, other providers are less sophisticated.
               | 
               | Generally speaking you're not going to get banned unless
               | you spin up max quotas and have other signals for fraud,
               | such as zero payment history, connecting from regions
               | with high fraud rates, etc. More often the provider will
               | just limit your resources - either openly, like AWS does,
               | or more subtly.
        
           | reaperman wrote:
           | It's a _very_ convenient way to convert identity theft into
           | cash. A huge amount of cloud crypto mining is paid for in
           | fraudulent credit cards. Detecting and banning crypto mining
           | greatly reduces % of fraudulent transactions as these
           | attackers will move on to greener pastures.
           | 
           | They can also use an absolutely incredible amount of
           | resources, because it's not their money. Two hundred A100
           | GPU's for a week? Sure, why not?
        
           | TX81Z wrote:
           | Maybe they're waiting to finish the big roll out of NFT on
           | YouTube!
        
           | professorsnep wrote:
           | GCE allows one e2.micro instance in their free tier
        
       | pawelduda wrote:
       | Wonder what does GCP side have to say on this
        
         | slig wrote:
         | Computer says no.
        
         | mmanciop wrote:
         | [flagged]
        
       | uoaei wrote:
       | It is interesting to note the differences in apparent
       | defensibility of companies such as Google and Reddit changing
       | terms of use and behaviors of platforms.
       | 
       | According to the general sentiment on HN, Google is being mean to
       | developers and shouldn't get away with things like this, but
       | Reddit merely has a rug and developers are silly to build things
       | on top of that rug when it can be pulled away at any time.
       | 
       | I wonder why there's a difference here.
        
         | anon84873628 wrote:
         | Well, GCP is literally selling a "platform" to developers.
         | Whereas Reddit is a consumer website?
        
       | holografix wrote:
       | A friend works at GCP and has told me this time and time again:
       | if you don't buy support from GCP _you have no support_.
        
         | azmodeus wrote:
         | Even if you buy support it is not comparable to AWS. I suffered
         | GCP Dataflow support in the past it was: 1. slow 2.
         | condescending 3. not helpful Really don't recommend GCP for
         | long term use. If you get some free credits it's good to use it
         | to train some ML models but AWS is much more customer friendly.
        
       | profwalkstr wrote:
       | It feels as if Google is intentionally trying to sabotage its own
       | cloud business
        
       | mvdtnz wrote:
       | Quota for what?
        
       | exabrial wrote:
       | Once Again: Do not use Google for anything. Given they treat the
       | consumer side of their business like cattle (Gmail, maps, etc)
       | (,and despite the fact that an email address is an essential part
       | of one's daily life: access to banking, investments,
       | communication, cell phone, employment opportunities), they
       | operate with absolutely zero support.
       | 
       | What makes you think they'll treat your small business any
       | differently? You will be thrown under the bus the second you make
       | them lift a finger to help you.
        
       ___________________________________________________________________
       (page generated 2023-06-10 23:01 UTC)