[HN Gopher] Google will let companies run Gemini models in their...
___________________________________________________________________
Google will let companies run Gemini models in their own data
centers
Author : jonbaer
Score : 430 points
Date : 2025-04-09 13:47 UTC (3 days ago)
(HTM) web link (www.cnbc.com)
(TXT) w3m dump (www.cnbc.com)
| rwmj wrote:
| A bit thin on detail, but will this require confidential VMs with
| encrypted GPUs? (And I wonder how long before someone cracks SEV-
| SNP and TDX and pirate copies escape into the wild.)
| unsnap_biceps wrote:
| The number of folks that have the hardware at home to run it is
| going to be very low and the risk of companies for leaking it
| is gonna make it unlikely IMHO.
| notpushkin wrote:
| I think home users would be the least of their concerns.
| RadiozRadioz wrote:
| It only takes one company to leak it
| franga2000 wrote:
| Or one company to get hacked and the hackers leak it
| spacebanana7 wrote:
| Realistically the only people able to run models of this
| size are large enterprises.
|
| Those enterprises won't take the risk of being sued for
| using a model without proper permission.
| nxobject wrote:
| I don't know - if there's still dumb money being thrown
| towards AI in non-tech and non-privacy-heavy industries,
| especially ones traditionally targeted by ransomware,
| there'll always be a chance of datasets getting leaked.
| I'm thinking retail and consumer product-oriented
| companies. (There's always non-Western governments
| without strong security orgs, too.)
| blackoil wrote:
| Nations.
| BiteCode_dev wrote:
| They can get "hacked" and wooops.
| vasco wrote:
| At the pace models improve, the advantage of going the dark
| route shouldn't really hold for long, unless I'm missing
| something.
| miohtama wrote:
| Access to proprietary training data: Search, YouTube, Google
| Books might give some moat.
| maxloh wrote:
| We have Common Crawl, which is also scraped web data for
| training LLMs, provided for free by a non-profit.
| UltraSane wrote:
| The Common Crawl is going to become increasingly
| contaminated with LLM output and training data that is
| more likely to have less LLM output will become more
| valuable.
| kouteiheika wrote:
| I see this misconception all the time. Filtering out LLM
| slop is not much different than filtering out human slop.
| If anything, LLM generated output is of higher quality
| that a lot of human written text you'd randomly find on
| the internet. It's no coincidence that state-of-art LLMs
| increasingly use more and more synthetic data generated
| by LLMs themselves. So, no, just because training data
| was produced by a human doesn't make it inherently more
| valuable; the only thing that matters is the quality of
| the data, and the Internet is full of garbage which you
| need to filter out one way or another.
| SXX wrote:
| Problem with filtering is that LLMs can generate few
| orders of magnitude more slop than humans.
| empiko wrote:
| But the signals used to filter out human garbage are not
| the same the signals that would be needed to filter LLM
| garbage. LLMs generate texts that look high-quality at a
| glance, but might be factually inaccurate. For example,
| an LLM can generate a codebase that is well-formatted,
| contains docstrings, comments, maybe even tests; but it
| will use a non-existent library or be logically
| incorrect.
| hdjjhhvvhga wrote:
| Are the differences between Google Books and LibGen
| documented anywhere? I believe most models outside of
| Google are trained on the latter.
| bjackman wrote:
| > I wonder how long before someone cracks SEV-SNP
|
| https://bughunters.google.com/blog/5424842357473280/zen-and-...
| NoahZuniga wrote:
| I'd expect watermarked model weights plus a lot of liability to
| distinctivise leaking the model.
| blitzar wrote:
| Is The Gavin Belson Signature Edition Box is needed to run these?
| unixhero wrote:
| No, Jack Barkers' revamped version is needed.
| FirmwareBurner wrote:
| I miss that show. Too bad it ended right before the AI hype.
| radicalbyte wrote:
| Fingers crossed that it'll do an Arrested Development.
| candiddevmike wrote:
| Richard has a student with an idea involving AI and joins
| his company as an advisor but can't keep his opinions to
| himself. Ends up ruining the company because everything he
| touches turns to shit.
| hinkley wrote:
| Ah, I'd forgotten how much I wanted to strangle Richard
| by the end of that show. He kind of lived long enough to
| become the villain.
| fonsai wrote:
| Wasn't the last season mainly about AI?
| anshumankmr wrote:
| In fact, an AI that went rogue was the major plotpoint, so
| the satire is still on point.
| j_bum wrote:
| I miss it too.
|
| I think by the end I was far more invested in the characters
| rather than the plot though.
| zer00eyz wrote:
| > Too bad it ended right before the AI hype.
|
| It was the very start of the AI hype cycle, and in fact they
| built the app: https://news.ycombinator.com/item?id=14636228
| swalsh wrote:
| The google search appliance might have been one of the worst
| products I've ever used in my career. If they're going to make
| a box, I hope they put some effort into it.
| mosura wrote:
| Strictly speaking Google still make boxes for people just in
| a different market.
|
| What was so bad about the search appliance though? Physical?
| Software?
| stingraycharles wrote:
| The Netflix appliance is pretty good in my experience. No
| reason Google couldn't pull something similar off themselves,
| unless they're being very Google about it.
| ijustlovemath wrote:
| They have some incredible hardware talent (TPUs, Pixels),
| but I'm guessing this project will not get the polish of
| those more public facing products
| SwamyM wrote:
| Given the myraid of issues they seem to have, I am not
| sure I would classify Pixels as having polish. But yes,
| they definitely have the talent to make some good
| hardware. It's just a matter of whether their priorities
| match those of their users.
| NBJack wrote:
| I can't think of a major phone brand that hasn't had some
| kind of major issue over the years. The batteries of the
| Note 7, iPhone "antenna gate" (and the more recent lack
| of advertised AI debacle), etc.
|
| How would you say it compares to those?
| aftbit wrote:
| I think Pixels are pretty polished, at least compared to
| all the cheapo off-brand Android options out there. Some
| people like Samsung better but I can't stand their UI.
| Apple would be fine if I could sideload...
| lallysingh wrote:
| Samsung with a 3rd party launcher works pretty well
| hinkley wrote:
| Do they only sell those to ISPs or could a housing
| developer or a hotelier get their hands on those?
| jauer wrote:
| They don't sell them. But, if the developer / hotelier
| had a sufficiently large network, think providing service
| equivalent to the number of rooms at a US state
| university system network (multiple universities), then
| they might qualify: https://openconnect.netflix.com/en/
| dmurray wrote:
| There are plenty of hotel groups big enough for that, but
| their properties are geographically distributed and I
| can't imagine they'd benefit from running fibre for their
| own multi-site network. Better to just connect each
| property to a local ISP like everyone else.
|
| Maybe there are some exceptions. Disney World? MGM
| Resorts in Las Vegas?
| er4hn wrote:
| I actually mourn the loss of it. It felt so much better than
| any other accursed on-prem search solution I've seen since.
| cyberpunk wrote:
| fastsearch was pretty good before Microsoft bought it..
| Elastic is good enough.
| phatskat wrote:
| > Elastic is good enough.
|
| Probably the most praise I've ever seen about Elastic.
|
| I do respect the amount of power and utility, and it's
| definitely a workhorse, but it's like a horse with one
| human leg, a bad eye, extra bones but also not enough
| bones, and a French accent but only knows Korean. Once
| you get used to the fact that you can't do what you
| intend to, but you can do what elastic wants, it becomes
| a lot more manageable.
| Workaccount2 wrote:
| The box might have been awful, but the sigh of relief when
| seeing a website was using it was wonderful.
| dehrmann wrote:
| My theory is that the heuristics (PageRank and click-through
| feedback) that made 2008 Google great don't work in corporate
| environments.
| tomjakubowski wrote:
| Hiring a 2010-era SEO expert to spam the company Notion
| with backlinks to the stuff I've written.
| tehjoker wrote:
| I wonder if the improvements in semantic search have
| changed that at all. For a big company though, you might
| need a pretty beefy setup to perform the initial indexing.
| runjake wrote:
| Why? In my limited experience, it was pretty useful, but
| again, my experience is limited.
|
| What made it one of the worst products you've ever used?
| sorokod wrote:
| More like the GSA probably.
|
| https://en.m.wikipedia.org/wiki/Google_Search_Appliance
| larodi wrote:
| which is a black-box more or less, so the "own data center"
| is a bit vague as concept here.
| jgalt212 wrote:
| right, when you return the box. Google can then retrieve
| all the logs.
| lupire wrote:
| It's quite clearly a yellow-box, made from The Chest hide.
|
| https://homestarfanstuff.fandom.com/wiki/The_Cheat
| surajrmal wrote:
| You're going to need a few specialized racks all wired up
| together. A single box won't be sufficient.
| ashoeafoot wrote:
| He personally stands behind the developed haedware, will you
| stand with him ?
| next_xibalba wrote:
| This tactic comes straight out of the Conjoined Triangles of
| Success playbook. It's a classic Action Jack Barker move.
| hinkley wrote:
| On paper, Stephen Tobolowsky seems like he shouldn't be
| successful enough of an actor to warrant an autobiography.
| But man do I love Ned "The Head" Ryerson in all his
| incarnations. What a strange, tall, little man.
|
| That the world does not have a Stanley Tucci, Stephen
| Tobolowsky buddy comedy trilogy has made it all the poorer.
| But it's been a while since someone tried to remake The Odd
| Couple...
| iamacyborg wrote:
| His role in Californication always makes me laugh
| nkassis wrote:
| His signature could be the size of a full size rack on this
| one.
| atum47 wrote:
| I got that reference
| hank808 wrote:
| FTA: As part of the announcement, Google said Nvidia will bring
| Gemini models to the company's Blackwell graphics processing
| units, or GPUs. Companies can buy the chips through Google or
| other channels.
| fxtentacle wrote:
| I believe these are pure word tricks to suggest privacy without
| actually delivering it.
|
| As context, you need to remember that Google deleted their "Don't
| Be Evil" motto and became a defense contractor. The customer will
| most likely receive a black box owned and set up by Google. That
| means they have no way of knowing if the system inside is phoning
| home or being remote controlled by an US government agency, or
| not. You can then say that the model is hosted in your own data
| center, which might make some people feel good, but using it with
| personal information is still a violation of the GDPR.
|
| If Google, however, would make these boxes fully offline capable
| and I was also allowed to wipe all hard disks myself before
| returning it, that would convince me of their good intentions.
| greggsy wrote:
| You're making a lot of assumptions there. It's trivial to
| monitor traffic patterns from modern appliances, even if it's
| encrypted.
|
| Also, companies have been sharing data with cloud security
| organisations for years now. There a robust means of assessing
| the risk. License agreements are a very real thing.
| fxtentacle wrote:
| I don't fully disagree, but the only reason why this product
| is noteworthy is precisely because companies don't trust
| cloud providers with their data anymore. And while you might
| be able to prevent data exfiltration by monitoring the
| traffic patterns, you probably can't prevent sabotage that
| way.
| sitkack wrote:
| I would have more problems with it even being on the
| network before we start talking about exfiltration.
| greggsy wrote:
| I'm confused and not sure what you mean.
|
| Are you implying that Google will sell a product that is
| designed to 'sabotage' their own customer's business? The
| legal and reputational damage far outweigh the value of
| stolen information.
|
| Or do you mean that it could be a vector of attack? That
| can happen with literally any piece of software, hardware,
| or appliance you install in or out of your datacentre.
| michaelt wrote:
| _> Are you implying that Google will sell a product that
| is designed to 'sabotage' their own customer's business?_
|
| The US government is constantly telling us that the likes
| of Huawei and Hikvision are doing precisely that, despite
| being subject to the same risks of reputational damage.
|
| Of course, the same could be said of everything else in
| the data centre. It's not like Google are somehow more
| vulnerable than Juniper or Cisco or Unifi or Dell or
| Intel or whoever.
| pqtyw wrote:
| > don't trust
|
| That and there are various regulatory, political etc.
| reasons. Also I'm not sure about the "anymore" IMHO a lot
| more companies trust cloud providers with their data than
| they did 10-20 years ago .
| surajrmal wrote:
| It's the same folks it always has been. Google is just
| trying to win those customer's business that would never
| have otherwise chosen Google. I'm sure these on prem
| solutions are not nearly as cost efficient as running the
| same workloads in Google data centers. Most companies would
| not pay that difference unless forced to via regulatory
| requirements.
| fhd2 wrote:
| Well, TFA appears to be thin on the details, but who says
| whatever they deploy is phoning home? If you run their model on
| prem, it wouldn't be a difficult feat to monitor its network
| traffic. Not to mention limiting it. It would be tricky if it
| phoned home by design, but if this is all abstracted through
| tool use or something, it can certainly be audited. And the
| kind of company that wants this usually doesn't just run random
| software without understanding and inspecting closely what it
| does.
| positr0n wrote:
| You're talking about Fortune 50 companies here. I don't think
| Google is going to be messing around spying on them in direct
| violation of the no-doubt sophisticated contract that will be
| signed between them.
| threeducks wrote:
| The NSA has been spying through Google since at least 2009.
| https://en.wikipedia.org/wiki/PRISM
| surajrmal wrote:
| That was not with Google's consent and it was quickly shut
| down by enabling encryption between nodes in Google's
| internal networks. Your average company is far more likely
| to be susceptible to state actors than Google is.
| threeducks wrote:
| > That was not with Google's consent
|
| The NSA does not need consent from Google. Google is
| simply ordered to comply. See
| https://policies.google.com/terms/information-
| requests?hl=en...
|
| > FISA orders and authorizations can be used to compel
| electronic surveillance and the disclosure of stored
| data, including content from services like Gmail, Drive,
| and Photos.
|
| If you look at the content requests under FISA, you can
| see that there were over 118000 requests for user data
| between July 2023 and December 2023.
| https://transparencyreport.google.com/user-data/us-
| national-...
| surajrmal wrote:
| That's very different from prism. It's also why Google
| has spent a lot of energy trying to make it impossible
| for them to see the contents of your data. The government
| cannot conpel information Google doesn't have access to.
| I'm also not sure it's relevant for the topic of this
| post.
| lupire wrote:
| That law applies to all of Google's US customers too.
| linkregister wrote:
| According to the documents leaked by Edward Snowden, that
| espionage was sniffed in-transit in plaintext across the
| Internet's trunk and filtered against XKEYSCORE queries
| for eventual collection. Google's surprise came from the
| expectation that cross-datacenter traffic was sent over
| direct circuits and not susceptible to interception.
|
| It was totally unrelated to PRISM, which was more like a
| voluntary law enforcement access portal that autoapproved
| every request. The participating companies since made
| public statements saying they no longer operate the
| portal, thereby forcing intelligence agencies to use
| National Security Letters instead. That's certainly
| closer to the intent of the laws passed by Congress.
| mkl wrote:
| The boxes can run airgapped:
| https://cloud.google.com/blog/products/ai-machine-learning/r...
| surajrmal wrote:
| Why is don't be evil relevant here? If Google never had that
| motto would you care less? It's not even factual that they
| dropped it from the code of conduct. It was just moved to the
| end rather than at the beginning. Moving it wasn't some magical
| event that signaled a change in Google's ethical values. Do the
| right thing was just seen as less ambiguous and placed more
| prominently.
|
| As others have stated, being able to see that the appliance is
| phoning home or not is trivial. No one who is in the market for
| this won't ensure it meets some rigurous bar.
| cavisne wrote:
| This is being sold as an air gapped product, it has to work
| offline by definition.
|
| Sure you could hide some way of phoning home and deploy it into
| the SCIF, but would you really want to risk a firing squad to
| improve some advertising metrics?
| holografix wrote:
| This is obvious government contract baiting. Kudos though they
| might actually move some Google Distributed Cloud this way
| aduffy wrote:
| I don't think so. To my knowledge GCP has no approval for
| classified networks, which is by far the hardest part. Contrast
| with Azure OpenAI has been approved to run on government
| networks for over a year now.
|
| This feels like a play for companies in highly regulated
| industries, GCP has a notable list of biopharma customers.
| ZeroCool2u wrote:
| FedRAMP High is the mark you really want to hit for the US
| Government and GCP's service coverage is surprisingly broad
| in that realm.
| skybrian wrote:
| From Google's blog post:
|
| > Our GDC air-gapped product, which is now authorized for US
| Government Secret and Top Secret missions, and on which
| Gemini is available, provides the highest levels of security
| and compliance.
| Maxious wrote:
| >Today at Google Cloud Next, we're thrilled to announce
| another significant milestone for Google Public Sector: the
| authorization of Google Distributed Cloud Hosted (GDC Hosted)
| to host Top Secret and Secret missions for the U.S.
| Intelligence Community, and Top Secret missions for the
| Department of Defense (DoD).
|
| https://cloud.google.com/blog/topics/public-sector/google-
| pu...
| aduffy wrote:
| You are right, I should've RTFA
| nkassis wrote:
| Banking as well, this is the kind of offering they've been
| looking for a while. Google just saw the demand decided to
| jump in while OpenAI and Anthropic probably calculated they
| don't have the manpower to deal with the support for this.
| reaperducer wrote:
| _This is obvious government contract baiting_
|
| You don't have to be a government agency to not want your
| company's data all over the place.
| connicpu wrote:
| With a few exceptions for companies with highly secretive
| data, you do have to be a government agency or working in a
| highly regulated government-adjacent area for secured private
| clouds to be a requirement carved in stone and therefore
| worth investing a ton of extra money into though.
| reaperducer wrote:
| Just off the top of my head: Healthcare. Banking.
| surajrmal wrote:
| Both of which are encumbered with regulations that want
| them to need this.
| SoftTalker wrote:
| Neither approaches the secrecy needed by government
| installations. Health care and banking leak PII regularly
| and never really suffer any consequences.
| trc001 wrote:
| This is just inaccurate
| kube-system wrote:
| Healthcare and banking have no issue storing data in
| third party datacenters as long as they meet the
| applicable standards.
| vel0city wrote:
| You absolutely don't need this or FedRAMP to do
| healthcare.
| dhorthy wrote:
| i'll add that on-prem is getting 10-100x easier than it
| was 10-20 years ago (still very hard), and "i want to run
| this in my own datacenter" is becoming accessible to much
| smaller companies than just F500 enterprises
| kortilla wrote:
| Huh? 20 years ago on-prem was the norm and the cloud
| didn't exist.
| tuukkah wrote:
| Yeah, AWS was in private beta in 2005 and public in 2006.
| People knew it existed but didn't have access yet.
| https://www.zdnet.com/article/how-amazon-exposed-its-
| guts-th...
| culopatin wrote:
| They'll have to fight Microsoft who's been promising copilot.
| noitpmeder wrote:
| Financial firms with significant on-prem datacenter use will
| love this as well. My company still stays away from the cloud
| -- we have 6 DCs in the building, and run everything else out
| of colocated racks.
| brcmthrowaway wrote:
| Who provides internet
| miohtama wrote:
| Is Gemini tied/benefitting from Google TPU hardware? Because you
| need hardware in the data center to run this, and I feel it is
| somewhat specialised.
| disgruntledphd2 wrote:
| The Google blogpost notes that it's a partnership with Nvidia,
| so using cuda rather than TPUs apparently.
| stingraycharles wrote:
| Makes complete sense, as NVidia has a lot more experience
| building these types of appliances.
| ddingus wrote:
| Some one said it could also mean Google hardware has some
| advantage they would rather stay inside the G-silo.
| MortyWaves wrote:
| Google abandoned Coral in true Google style.
| drdirk wrote:
| Gemini models are written in Jax which through the XLA compiler
| can be compiled either to TPU or GPU hardware.
|
| Performance may differ but Google (and Nvidia) are very
| interested in having good performance on both platforms.
| cavisne wrote:
| The raw computation is just a bunch of matrix multiplications
| in a row, most of the algorithmic complexity/secret stuff would
| be around scaling & efficiency.
|
| For training the model the HW is much more important as you
| need to scale up to as many chips as possible without being
| bottlenecked by the network.
|
| This would just be inference, and it doesn't need to be very
| efficient as its for on prem usage not selling API access. So
| you could strip out any efficiency secrets, and it would
| probably look like a bigger Gemma (their open source model).
|
| I wonder if they would/could try and strip out stuff like
| whatever tricks they use for long context + video support (both
| of which they are a bit ahead of everyone else on).
| summerlight wrote:
| The model itself is likely built upon their own open source
| system JAX so they should be usable in Nvidia. Of course cost
| efficiency is going to be a different story.
| Workaccount2 wrote:
| TPUs are definitely the reason why Gemini models have both
| massive context and very low prices. There is no nvidia tax to
| pay.
| amelius wrote:
| It's still an advertising company you're doing business with.
|
| I mean, would you buy cookies from a brand that is known for
| producing rodenticides?
| crazygringo wrote:
| ...yes?
|
| I mean, the company that makes Raid also makes Saran Wrap and
| Ziploc bags. Corporate conglomerates can do lots of things.
|
| The entire Google Cloud org is funded by regular customers
| paying money, not advertising.
| righthand wrote:
| > The entire Google Cloud org is funded by regular customers
| paying money
|
| That can't be true, how did they bootstrap it? How do they
| pay for R & D for their half baked offerings?
| wiseowise wrote:
| Didn't they start it for themselves and then saw an
| opportunity to make it a business?
|
| In a sense, yes, it was bootstrapped by ads and now pays
| for itself.
| surajrmal wrote:
| Would you prefer VCs to have fronted the money to bootstrap
| it? How is it relevant today if ads are no longer enabling
| their financial viability? Ads largely finance Google's
| consumer offerings, not their enterprise offerings. Most
| enterprise Google customers understand the difference.
| tfsh wrote:
| I don't think you're wanting to converse in good faith, but
| on the off chance this is a question - yes, GCP was revenue
| losing for a number of years, but since Q1 2023 they've
| been profitable. It takes money to bootstrap anything -
| obviously - this is the case for the vast majority of
| companies and their offerings, especially so for one which
| requires vast amounts of compute resources, SREs, legal,
| etc.
| righthand wrote:
| So just to clarify the entire cloud org was funded by
| advertisers for most of it's existence.
| azinman2 wrote:
| What's the problem? Google is trying to diversify their
| revenue streams. I don't understand the relevance. Apple
| TV+ is paid for by iPhones. Ok? And?
| righthand wrote:
| > Ok? And?
|
| This is a thread about using your money for better things
| than paying an ad company. The comment that started this
| argument you want to have pointed out that it's self
| sustaining. But I pointed out that wasn't always true.
| Tfsh backed my claim.
|
| So today maybe there isn't a problem to which your money
| isn't being spent with the ad org but it was that way for
| a very long time to which we can grant the OP some grace
| as it's a rather recent change.
|
| There is even still an argument to be made that while you
| may not be giving money to the ad org you are still
| giving money to Google thereby helping them deflect the
| damage they cause the world in their other orgs.
| crazygringo wrote:
| No, even if you were Google Cloud paying customer #1,
| your money was going to Cloud. It wasn't supporting
| anything to do with ads.
|
| The ads were providing income to Google which allowed
| Google to bootstrap Cloud until it was profitable on its
| own, _not vice-versa_.
|
| When you buy (or bought) Cloud services, that doesn't
| affect Google's ad revenue or advertising behavior at
| all, not for the better and not for the worse. They're
| basically unrelated orgs within the corporation. Using
| Cloud isn't promoting ads or whatever you seem to think,
| not now and not previously.
| jsnell wrote:
| No, it was funded by Google.
|
| Advertisers paid money for Google for totally unrelated
| services. Google invested that money in a number of ways.
| One of them was to build this very profitable non-
| advertising business. The advertisers didn't fund that
| business any more than the advertisers funded US
| treasuries, or the dozens of startups that Google has
| invested in as a VC.
| NicuCalcea wrote:
| I buy Lidl store brand biscuits as well as Lidl store brand
| cleaning products, among many other things, so I guess I would.
| rafaelmn wrote:
| Lidl produces none of those, just brands them. More
| comparable would be something like Raid cookies I guess
| sahila wrote:
| Didn't google just buy deepmind and rebrand it too?
| tomschwiha wrote:
| Lidl produces a lot of products on their own, as well as
| packaging (Schwarz Produktion) [0]
| https://en.m.wikipedia.org/wiki/Schwarz_Group
| paxys wrote:
| Google Cloud had $50B in revenue last year, so clearly plenty
| of companies didn't get your memo.
| aussieguy1234 wrote:
| What is the risk that some hacker could exfliltrate the weights?
| dist-epoch wrote:
| Very low if they use confidential VMs (CPU rooted encryption).
| Just like the Xbox uses and remains unhacked 10 years later.
| onlyrealcuzzo wrote:
| What would be the benefit of hacking the Xbox? What would you
| get?
| DoctorOW wrote:
| Pirated games, cheating/botting in multiplayer rooms, etc.
| macintux wrote:
| And fame.
| cavisne wrote:
| Seems pretty high, this is an air gapped product so at some
| point the employees of whatever government they are giving it
| to would need to SSH into the VM's to load new weights etc.
| Lots of ways to make it tricky/watermark the weights though.
| djoldman wrote:
| Google announcement:
|
| https://cloud.google.com/blog/products/ai-machine-learning/r...
| mkl wrote:
| Interesting that the hardware is NVidia Blackwell, not Google
| TPUs. That means Google will likely have an energy efficiency
| and cost advantage, and keep their proprietary hardware out of
| other people's reach.
| bitexploder wrote:
| Part of the reason for this is likely due to customers
| preference to have CUDA available which TPUs do not support.
| TPU is superior for many use cases but customers like the
| portability of targeting CUDA
| alienthrowaway wrote:
| What are the pros of using CUDA-enabled devices for
| inference?
| bitexploder wrote:
| My limited understanding is that CUDA wins on smaller
| batches and jobs but TPU wins on larger jobs. It is just
| easier to use and better at typical small workloads. At
| some point for bigger ML loads and inference TPU starts
| making sense.
| j5r5myk wrote:
| Which use cases are TPUs superior for?
| re-thc wrote:
| > not Google TPUs
|
| They're in limited supply. Even Google doesn't have enough
| for their own use.
| crowcroft wrote:
| Getting a whole business set up to build TPU hardware for
| third parties (design, build, sell, support, etc.) is
| probably not worth it when there is overflowing demand for
| TPUs in their cloud already.
|
| Businesses running their own hardware probably prefer CUDA as
| well for being more generally useful.
| WalterGR wrote:
| Google doesn't make TPUs available to 3rd parties, right? I
| assume there would be tremendous reverse-engineering risk if
| they were to include them?
| fc417fc802 wrote:
| Not really. Reverse engineering a modern chip is no small
| feat. Any company capable of it is also capable of
| designing their own from scratch. However getting something
| taped out (and debugged) on a modern process is massively
| expensive.
| throwaway48476 wrote:
| Like you can with deep seek? Or will it be more complicated and
| expensive. I don't know who would actually want that.
| yoavm wrote:
| Absolutely many would, especially those with deep pockets. The
| biggest concern I'm hearing from companies adopting AI, for
| basically any use case, is data leaving their network.
| Especially (but not only) in the EU.
| throwaway48476 wrote:
| Deepseek is just the model weights. Nothing about it requires
| network access.
| yoavm wrote:
| Deepseek is not really comparable to Gemini 2.5 Pro.
| phonon wrote:
| DeepSeek-R2 may be...
| surajrmal wrote:
| Folks who would prefer to run deepseek are not in the end
| customer for this product. Deepseek doesn't provide a service
| contract.
| nsriv wrote:
| This might be a great way for them to strengthen their model
| through federated learning.
|
| https://federated.withgoogle.com/
| ein0p wrote:
| The whole point of deploying such things on-prem is air-gapping
| it from Google and its "learning".
| fc417fc802 wrote:
| That's the point of the privacy scheme. It would only be able
| to learn things common to multiple clients. Private data
| wouldn't make it through the noise.
| amarcheschi wrote:
| I did my undergrad internship on federated learning. I was
| tasked with implementing in a simulator different federated
| algorithms, so to have a way to compare them in a meaningful
| way. The last that had to be implemented was FedMA. We didn't
| manage to do it. That algorithm is absolutely devilish. Every
| issue that I solved made other two issue arise, and neither my
| supervisors could help. The sheer idea of matching neurons in
| different networks might (and does) make sense, but the way the
| approximate costs are calculated require other 2/3 math papers
| that I could follow for only the first lines of the abstract.
| I'm happy for the time I spent in my internship there. I'm also
| happy it's over
|
| The general understanding of how it works is surprisingly easy
| though, you can find the paper here
| https://arxiv.org/abs/2002.06440
| jaggs wrote:
| Not a cat's chance in hell that any eu organisation will rush to
| this offer right now. Or maybe ever in fact.
| geodel wrote:
| Huh, how does that matter? Maybe OVH AI is fine for EU.
| tziki wrote:
| I don't understand how Google is willing to do this but won't
| sell TPUs to other days centers. It should be obvious from
| Nvidia's market cap that they're missing a huge opportunity.
| dehrmann wrote:
| The only reasons I can think of is they see them as their
| secret sauce, they don't want to support them for customers
| long-term, or they don't have the foundry capacity.
| paxys wrote:
| It's definitely #3. The GPUs have to first satisfy Google's
| own computing needs, and only then can they start selling
| them to others. Given how much training and inference the
| company is doing and how much demand there is internally it's
| very unlikely they are able to manufacture loads of extras,
| especially not profitably.
| _cs2017_ wrote:
| Curious if this was forced on Google Cloud by Sundar, or was it
| something that Google Cloud as an org wanted to do?
|
| At first glance, it seems Google Cloud might lose some revenue
| from customers who can now deploy Gemini in-house. On the other
| hand, it's not a complete loss, since presumably Google Cloud is
| still involved in providing some underlying tech? Not to mention,
| some customers would never consider using off-premises setup
| anyway.
| wmf wrote:
| I assume Google Distributed Cloud is part of the larger Cloud
| org so they get the revenue either way. The on-prem version may
| even cost more.
| ein0p wrote:
| What a sudden change of heart. Thank you, DeepSeek!
| bushbaba wrote:
| Reminds me of the Google search appliance.
| jonhohle wrote:
| That's the first thing I thought of as well. I had to integrate
| one into our custom CMS early in my career. I vaguely remember
| explaining to management that I was not responsible for the
| order or quality of search results and tweaking queries (now
| prompts?) with hints to restrict searches to certain paths. It
| was such an opaque device, but provided better results than
| MySQL did at the time.
___________________________________________________________________
(page generated 2025-04-12 23:00 UTC)