[HN Gopher] Trying to get a bunch of GPUs on Google Cloud
___________________________________________________________________
Trying to get a bunch of GPUs on Google Cloud
Author : MikusR
Score : 76 points
Date : 2022-12-01 20:06 UTC (2 hours ago)
(HTM) web link (twitter.com)
(TXT) w3m dump (twitter.com)
| tveita wrote:
| It's a pain. I recently had a compute job that would take a
| couple of days to run locally. "This is exactly what the cloud is
| for," I thought and went to Google Cloud where I already had an
| account.
|
| First try I'm allowed to spec up a beefy server with AVX-512 and
| click "create", but it never actually starts. Turns out the
| default CPU limits are tiny - I need to apply to bump two
| separate limits and wait for that to go through.
|
| Second try I'm actually able to spin up a couple of servers. I
| set the compute job running and go for lunch. An hour later both
| are terminated and I get a message about "violation of our
| Acceptable Use Policy".
|
| There is a link that says crypto mining is forbidden on free
| instances. I send a reply informing them that a) these are not
| free instances, and b) I am not mining crypto.
|
| I get a reply back saying more or less "Yes it says free
| instances but why would you expect anything we write in the
| documentation to be correct. Also you might _think_ you weren 't
| mining crypto but actually your VM was probably compromised
| because no one would buy compute instances and actually run
| compute jobs on them. We're not giving you back access to your
| instances, maybe delete them and create new ones lol."
|
| I actually find it an easier product to get started with than
| AWS, but man they really really don't want your money. When you
| are actively trying to prevent your customers from scaling up
| resources you are failing at being a cloud.
| zerr wrote:
| @JohnCarmack Can you please ask Google when do they plan to
| provide merchant registration support for leftover countries on
| Play Store? Thank you.
|
| https://support.google.com/googleplay/android-developer/answ...
| jandrese wrote:
| His comment about not being able to pre-load an account with
| enough money to cover the usage to avoid the noob filter seems
| like it should be easy to fix, but I'm not sure it would solve
| the "whoops, I just accidentally wasted 6 figures, can I have it
| back?" problem Google presumably implemented this policy to
| fight. Sure Google won't necessarily be on the hook since they
| already have the money, but at the same time it's hard for
| customer service to go tell someone like that to pound sand. They
| clearly have money to spend even if they managed to waste a ton
| of it this time.
|
| Maybe a flag you could set in your account that says "I know what
| I'm doing and I promise not to ask for refunds for my mistakes."
| Obviously if the money is lost due to a Google screwup then yeah,
| I'm going to want that back, but if I forgot to turn off a few
| thousand max GPU instances over the weekend then that's on me.
| Maybe combine that flag with a prepayment system? And if you blow
| out your account then it halts your instances until you top it
| off, hopefully with a bunch of text messages and emails to the
| admin.
| minhazm wrote:
| I think it's largely about capacity. Even if you have the money
| they don't want you to randomly spin up up a few hundred GPU
| instances for 6 hours because it will end up causing capacity
| issues for all of the other people trying to also allocate
| those instances. GPU's are in crazy high demand and I doubt
| even GCP, Azure, or AWS can get their hands on enough of them.
| Contrary to what cloud providers may have you believe you can't
| actually spin up hundreds of instances unless you already have
| thousands of instances or you've talked to them and have
| already told them you're going to spend millions of dollars
| with them and got the quota pre-approved.
| willcipriano wrote:
| > Contrary to what cloud providers may have you believe you
| can't actually spin up hundreds of instances
|
| I've had three software engineer jobs, learned this lesson on
| the first one and watched others get tripped up by it in the
| other two. Nobody ever believes me when they say "We will
| just spin up a instance for every file in the directory" or
| similar and I respond with "No you won't, they aren't going
| to give you a thousand instances". This is a lesson that can
| only be learned first hand.
| fragmede wrote:
| For GPUs, yeah. If you're talking compute though, you just
| keep requesting your limits be raised until you're happy.
| You'll probably hit your ability to pay before you hit
| actual AWS limits. Talk to your AWS TAM if you have
| questions on what your spend has to be before they'll give
| you that kind of access. If you don't have a TAM then you
| might be too small.
| jiggawatts wrote:
| That's why spot pricing exists.
|
| It's basically an auction market. You set your max price to
| bid on instances, and will get them as long nobody else bids
| higher.
| bombcar wrote:
| AWS famously lets you spend billions overnight, so it can't be
| _that_ big of a problem.
|
| I suspect it's mostly Google being Google and horribly at
| customer service, even when the customers are waving money at
| them.
| llaolleh wrote:
| Google customer service is atrocious. Their ability to solve
| customer support issues is effectively a toss up.
|
| Please do better Google.
| nszceta wrote:
| You shouldn't need to ask Google for mercy. When a company
| gives you bad service you should be free to leave for a
| competitor. If you are not able to you need to ask yourself
| why you are willing to put up with such abuse?
| bvhvhfcgxfdf wrote:
| The more I read about Google the more I am convinced that the
| only reason it proliferated because it's an ad company. Google
| would've gone bankrupt in its preliminary phase if it was a
| company that added any real value.
| type_Ben_struct wrote:
| I think that's an unfair statement. There are good and bad
| parts of Google, but to say they add no value is a stretch.
|
| Can you really say that across YouTube, Search, Gmail, Drive,
| Cloud, Open Source Contributions (Kubernetes, Tensorflow, etc)
| and so on they don't add any real value anywhere at all?
| mkl95 wrote:
| The more I read about Google the more I am convinced it cannot
| be treated as a single organization, but as many organizations
| operating under the same name - some of them brilliant, some of
| them utterly clueless.
| joemi wrote:
| Back when they were just a search engine, they added HUGE
| value. Search was really terrible before them, and when using
| google for the first time after using all the other search
| engines, it really felt like actual magic.
| nmfisher wrote:
| What's also amusing is that their support for advertisers is
| just as bad. The reality is that companies will tolerate just
| about anything as long as you have eyeballs to sell.
| KptMarchewa wrote:
| They don't have that many GPUs to just enable everyone to use it.
| gary_0 wrote:
| Imagine the problems startups have who don't have a Big Name at
| the helm; Google/Amazon/Apple can ignore or impede them
| indefinitely. People in this thread are bringing up fraud and
| customer service issues, but these filters should _not_ be
| something reputable corporations (and famous pioneers like
| Carmack) have to worry about. Something is very wrong indeed.
|
| Google-sized megacorps should not be able to impede innovation
| and economic activity like this. We're all losing because of it.
| denkmoon wrote:
| Google doesn't have to rent their GPUs to anyone if they don't
| want to.
|
| Besides, their incompetence allows companies like Lambda to
| fill the niche, and probably do a better job of it than Google
| do.
| scombridae wrote:
| _Google should not be able to impede economic activity_
|
| Bad producers can only impede as much activity as stupid
| consumers give them.
| jiggawatts wrote:
| Digital Ocean was just obliterating legitimate businesses when
| their GPU or CPU usage _appeared_ to look like crypto mining.
|
| Since they only accept credit, this is exposing Digital Ocean to
| the risk of handing out cash-equivalents for potentially bad
| credit. This "forces" them to be aggressive against abuse.
| Unfortunately, heuristics aren't perfect, so they also
| occasionally execute a hapless business by accident.
|
| John C made the same comment that I did. Other than outright
| crime (child porn, etc...), there ought not be any limits on
| instance usage in clouds, provided that customers pay cash.
|
| Heck, _let them crypto mine if they wish_ , as long as they pay
| cash! Who cares? It's just computation.
| fragmede wrote:
| Because smart money was on the GPU market eventually coming
| back down to Earth due to a crash in the crypto market. Which
| it mostly has. What no one could reasonably predict is _when_
| (market can stay irrational and all that). Cloud providers buy
| obscene amounts of hardware for obscene amounts of money, and
| if their capacity predictions are off, they lose tons of money.
| Disallowing crypto mining on GPUs then just makes sense. You
| just never know when those customers are going to evaporate
| leaving you high and dry with a GPU farm that people aren 't
| paying for.
| brunoTbear wrote:
| Cash doesn't really exist in the payments stack for an online
| business. Credit card chargebacks can roll in months after a
| compute instance is closed down and companies like Google
| have little recourse here.
|
| "Just take cash" isn't an option.
| xxpor wrote:
| I suspect Google isn't worried about the financial aspects of
| fraud so much as they're worried about fraud using extremely
| limited capacity. His point about prepaying makes sense from the
| customer side, but I'd be willing to bet google doesn't want to
| deal with having negative balances in accounts that are backed by
| real cash (as opposed to credits). Maybe they already allow that
| if you overpay? Just guessing here.
___________________________________________________________________
(page generated 2022-12-01 23:01 UTC)