[HN Gopher] Google TPU v5p beats Nvidia H100
___________________________________________________________________
Google TPU v5p beats Nvidia H100
Author : wslh
Score : 52 points
Date : 2024-01-26 21:18 UTC (1 hours ago)
(HTM) web link (www.techradar.com)
(TXT) w3m dump (www.techradar.com)
| nabla9 wrote:
| Google is one generation behind?
|
| Google announces TPU v5p with specs competing against H100 when
| NVIDIA GH200 Grace Hopper is announced.
|
| >The new pods also have 95GB of high-bandwidth memory (HBM)
|
| GH200 has 282GB of HBM3e memory
|
| >These new pods provide a throughput of 4,800Gbps.
|
| That equals 600 GB/sec. GH200 has 10TB/sec of combined bandwidth
| HBM3e memory.
| miohtama wrote:
| I guess performance per watt is also a major factor?
| bhewes wrote:
| I am more interested in TCO of TPU v5p. Right now it seems only
| Nvidia is making money everyone else loses.
| Y_Y wrote:
| table of contents?
| kirubakaran wrote:
| I think they mean TCO:
| https://en.wikipedia.org/wiki/Total_cost_of_ownership
| bhewes wrote:
| Thanks.
| nabla9 wrote:
| I think Broadcomm designs and builds TPUs for Google. Their
| stock was up 100% in 2023 and their profit margin was 39.31%.
| bhewes wrote:
| They bought VMware for $61B in Nov of last year. Hard to
| think TPUs are driving much of this at Broadcom.
| lordswork wrote:
| The article mentions TPU v5p is 2.1 x perf/TCO of H100.
| bhewes wrote:
| Actual dollars would be nice, since that is what the end
| user understands. Otherwise what are they talking about 1
| million unit orders? But nice catch I didn't see it.
| jsnell wrote:
| I don't think it does?
|
| > Google's v5p TPUs are up to 2.8 times faster at training
| large language models than TPU v4, and offer 2.1-times
| value-for-money.
|
| That's a comparison of TPUv5p to TPUv4, not to H100.
| wmf wrote:
| H100, H200, and GH200 are the same generation although the next
| generation B100 is coming this year.
| modeless wrote:
| A few notes: Nvidia always announces their hardware in advance
| of availability, while Google typically announces some time
| after they've started using it internally. Also TPU has been
| more cost-focused than Nvidia, since Google uses these chips
| internally to serve their web traffic while Nvidia is supply
| constrained and can charge practically anything they want right
| now, plus they don't pay the electricity bills.
|
| There are also a few things wrong with the specs you quoted.
| 282 GB is split between two GPUs, it's 144 GB per GPU (not sure
| where the extra two went). The TPU pod throughput number you
| quoted is interconnect bandwidth which you compared to GH200's
| memory bandwidth. Those numbers are not comparable. I believe
| GH200 does beat TPU v5p per chip, however the differences are
| not anywhere near as large as your comparison suggested. And
| it's likely that TPU v5p is dramatically more cost effective
| but we don't have Google's internal numbers to prove that.
| porphyra wrote:
| It is pretty exciting that various companies are making compute
| for deep learning. I was starting to get worried that the TPUs
| were lagging behind Nvidia. More diversity in hardware is always
| nice.
|
| Meanwhile I am still waiting for Tesla Dojo to be an actual
| thing...
| ShamelessC wrote:
| Tesla's dojo is deployed already I thought? I don't recall them
| ever having plans to make it publicly available.
| stonogo wrote:
| I don't see how we're this far in and so many competitors are
| still trying to ignore the massive market penetration of CUDA. I
| don't care how fast your chip is if I can only run 20% of the
| already tiny amount of software capable of leveraging this
| acceleration.
| collegeburner wrote:
| some stuff is still CUDA-dependent (a lot of scientific code).
| other stuff is not (major ML frameworks and many math
| libraries). nobody said CUDA was obsolete, just that there are
| great options for many use cases that no longer rely on it.
| vessenes wrote:
| I think this was a more accurate assessment a couple of years
| ago. AMD historically under invested in their software stack,
| as is well documented. But, they are catching up. The
| prevalence of pytorch makes it easier for hardware vendors to
| target a single set of libraries for implementation and get
| broad distribution. Even Apple has made major progress getting
| mps support in more broadly, first directly into pytorch, and
| they are now charting their own course with MPX with some early
| interesting successes. Zuck recently used the word "H100
| equivalent" in compute, and my memory is he indicated roughly
| 35% of that compute was not from NVIDIA -- that will all be
| AMD.
|
| There's still work to do -- lots of repositories still contain
| `if device=="cuda"` type language, but my own experience is
| that manging code around to use apple gpus has gotten vastly
| easier this year, and I see more and more AMD GPU owners
| floating around github issues with resolvable problems. A year
| ago they were barely present.
|
| All that said - people aren't ignoring it - and entrepreneurs
| and cloud monopolies are putting real resources into opening
| things up. I think the playing field will continue to level /
| get back to competitive over the next two to three years.
| ShamelessC wrote:
| The main issue is until you solve 99% of all the problems,
| you're still introducing substantial friction to every
| potential user who may have had to go from zero coding
| knowledge needed to very specialized coding knowledge needed.
|
| And while many problems are trivial to fix at eg the PyTorch
| layer, lots of stuff like flash attention, DeepSpeed, etc.
| are coded directly in CUDA kernels.
| glalonde wrote:
| Is that true and is that sustainable? My understanding has been
| that only the relatively low level libraries/kernels are
| written in cuda and all the magical algorithms are in python
| using various ML libaries. It's like how intel BLAS isn't much
| of a moat -- there are several open source implementations and
| you can mix and match.
|
| How is CUDA so sticky when most ML devs aren't writing CUDA but
| something several layers of abstraction above it? Why can't
| intel, AMD, google w/e come along and write an adapter for that
| lowest level to TF, pytorch or whatever is the framework of the
| day?
| 1024core wrote:
| > Why can't intel, AMD, google w/e come along and write an
| adapter for that lowest level to TF, pytorch or whatever is
| the framework of the day?
|
| A long long time ago, i.e. the last time AMD was competing
| with Intel (before this time, that is), we used to use
| Intel's icc in our lab to optimize for the Intel CPUs and
| squeeze as much as possible out of them. Then AMD came out
| with their "Athlon"(?) and it was an Intel-beater at that
| time. But AMD never released a compiler for it; I bet they
| had one internally, but we had to rely on plain old GCC.
|
| These hardware companies don't seem to get that a kick-ass
| software can really add wings to their hardware sales. If I
| were a hardware vendor, I would, if nothing else, make my
| hardware's software open so the community can run with it and
| create better software; which will result in more hardware
| sales!
| benreesman wrote:
| They're not ignoring it, they're eroding it.
|
| The API that matters is Torch, and only the API. Letting NVIDIA
| charge famine prices is both a bad idea and a huge incentive to
| write software bridging the gap.
| kkielhofner wrote:
| I don't know if you do work in this space but this wasn't
| accurate several years ago and it certainly isn't accurate
| now.
|
| The amount of extremely CUDA specific handwritten compute
| architecture code, custom kernels, etc has exploded and other
| than a few things here and there (yes, like torch SDPA) we're
| waaaay past vanilla torch for anything beyond a toy.
|
| The pricing and theoretical hardware specs of these novelties
| doesn't matter when an inferior on paper Nvidia product will
| wipe the floor with the other in the real world.
|
| People have spent 15 years wringing every last penny out of
| CUDA on Nvidia hardware.
|
| There is some light shining through but you'll still see
| things like "Woo-hoo FlashAttention finally supports ROCm! Oh
| wait, what's that, FlashAttention2 has been running on CUDA
| for six months?"
|
| Don't even get me started on the "alternative" software
| stacks and drivers.
| sadhorse wrote:
| More compute can't save you from dumb methods. Memorizing the
| whole of human text while being fooled by a prompt crafted by a
| 14 year old doesn't mean its intelligent.
| crakenzak wrote:
| Plenty of intelligent and educated humans have been fooled by
| "prompts" aka lies/social engineering by others many years
| their younger.
| ilaksh wrote:
| Can we access TPUv4 and/or v5 in Google Cloud? What frameworks
| support them?
| wmf wrote:
| Maybe Google should invent a search engine that can answer such
| questions.
| sxp wrote:
| Is there any good way to program TPUs for non-ML work? The only
| official way appears to involve encoding the compute operation
| into a TF graph and loading that. Is there a better way to
| directly use the hardware?
| counters wrote:
| Just write your code in JAX and run as usual?
| lordswork wrote:
| AFAIK, XLA is the only compiler that can target TPUs, so
| anything written for TPUs would need to lower to HLO. That
| said, why would you want to to run non-ML workloads on such an
| ML-specialized chip?
| sxp wrote:
| I was interested in seeing if raytracing or similar
| embarrassingly parallel workloads could be ported to a TPU.
| renewiltord wrote:
| Can only use on Google Cloud, no? Nvidia is the only self-
| hostable product available.
|
| But Google can build good framework integration. Will wait and
| see.
| wmf wrote:
| Gaudi, MI300, Cerebras...
| jchonphoenix wrote:
| None of this matters if they can't get the hardware stack to work
| correctly.
|
| The media keeps missing the real lock in Nvidia has: CUDA. It's
| not the hardware. It's the ability for someone to use it
| painlessly.
| lordswork wrote:
| What's painful about using TPUs?
| moffkalast wrote:
| And Nvidia does actually sell their hardware. Nobody will ever
| get their hands on one of these outside Google Cloud. It might
| as well not exist.
| kjkjhgkjyj wrote:
| TensorFlow and PyTorch support TPUs. It's pretty painless.
| Mehdi2277 wrote:
| Having used it heavily it is nowhere near painless. Where can
| you get a TPU? To train models you basically need to use GCP
| services. There are multiple services that offer TPU support,
| Cloud AI Platform, GKE, and Vertex AI. For GPU you can have a
| machine and run any tf version you like. For tpu you need
| different nodes depending on tf version. Which tf versions
| are supported per GCP service is inconsistent. Some versions
| are supported on Cloud AI Platform but not Vertex AI and vice
| versa. I have had a lot of difficulty trying to upgrade to
| recent tf versions and discovering the inconsistent service
| support.
|
| Additionally many operations that run on GPU but are just
| unsupported for TPU. Sparse tensors have pretty limited
| support and there's bunch of models that will crash on TPU
| and require refactoring. Sometimes pretty heavy thousands of
| lines refactoring.
|
| edit: Pytorch is even worse. Pytorch does not implement
| efficient tpu device data loading and generally has poor
| performance no where comparable to tensorflow/jax numbers.
| I'm unaware of any pytorch benchmarks where tpu actually
| wins. For tensorflow/jax if you can get it running and your
| model suits tpu assumptions (so basic CNN) then yes it can be
| cost effective. For pytorch even simple cases tend to lose.
| htrp wrote:
| > TensorFlow and PyTorch support TPUs. It's pretty painless.
|
| Unless you physically work next to the TPU hardware team, the
| torch support for TPUs is pretty brittle.
| nisten wrote:
| Meh, the comparison is somewhat pointless when it doesn't account
| for the slowdown that the vast majority of pytorch codebases
| experience on TPU's vs using JAX and it's accelerations specific
| to TPUs, and vice versa.
|
| https://arxiv.org/pdf/2309.07181.pdf
|
| Meaning the TPU v5p is likely slower than H100 for most ML
| workloads that depend on pytorch.
| tgtweak wrote:
| Two things matter at cloud-scale:
|
| Compatability - does using a tpu require reworking significant
| parts of your software stack and application stack? If so, that
| sucks for most companies who's researchers are used to using
| nvidia libraries, tooling and hardware to get their models
| running, and reworking the entire bottom end of that to work on
| an esoteric platform needs to equate to a huge cost savings at
| scale to be worth it.
|
| Cost per tensor flop delivered - likely very low if google has
| optimized the silicon, memory, voltage, power and temperature
| envelope, networking, boards and chassis for the server running
| it, as well as optimizing for optimal process node
| efficiency/cost. They're probably not on bleeding edge tsmc
| process, but instead optimizing for total deployed running cost
| per pflop over 2-3 years.
|
| It's also now public (as of November) that microsoft/azure have
| been working for many years on their own ai chip, dubbed "Maia"
| [1], and appliance with the obvious goal of taking some of that
| nvidia margin in-house (and with openai/bing/copilot being a
| massive consumer of capacity). I think this will become even more
| commonplace with cloud vendors - even medium size ones - than it
| already is. The knowledge and complexity barrier to designing a
| tile processor unit seems pretty low and it looks like most of
| the hard stuff is in the drivers and software - something cloud
| providers designing and integrating internally can bypass and
| control to a great degree.
|
| It's also very hard to benchmark side by side on these since I'm
| sure cuda/nvidia hardware can do compute that a TPU cannot. AMD's
| machine learning accelerators look good on paper too, as do the
| tensor processors in the apple silicon, but on real world
| applications and use cases they don't often measure up save for a
| few optimal workflows.
|
| [1] https://www.geekwire.com/2023/microsoft-unveils-custom-ai-
| ch...
| geor9e wrote:
| Hasn't this always been the case? I remember they released TPUs
| at IO 2018, and they were nice if you build FP16/FP32 models in
| Google Cloud TensorFlow/CoLab and never ever port them to
| anything else. Meanwhile the cool new open source stuff coming
| out every week usually requires a GPU and is rarely compatible
| with TPU without major changes. If you wait a few months a TPU
| compatible copycat appears on the Keras demos page but by then
| you've lost interest.
| 7e wrote:
| There are no benchmarks in this s*t article, just SWAGs. If this
| chip doesn't support FP8, there's no way it's going to beat the
| H100 on workloads that matter.
___________________________________________________________________
(page generated 2024-01-26 23:00 UTC)