[HN Gopher] Nvidia H200 Tensor Core GPU
___________________________________________________________________
Nvidia H200 Tensor Core GPU
Author : treesciencebot
Score : 100 points
Date : 2023-11-13 15:19 UTC (7 hours ago)
(HTM) web link (www.nvidia.com)
(TXT) w3m dump (www.nvidia.com)
| deadballcretin wrote:
| The performance jumps that Nvidia has had in a fairly short
| amount of time is impressive, but I can't help but feel like
| there is a real need for another player in this space. Hopefully
| AMD can challenge this supremacy soon.
| 2OEH8eoCRo0 wrote:
| I'd prefer another player that doesn't rely on TSMC.
| 01100011 wrote:
| Not sure why you were downvoted. Taiwan is in a precarious
| position and diversifying manufacturing away from them makes
| sense.
| xethos wrote:
| > diversifying manufacturing away from them
|
| Diversifying manufacturing away from Taiwan makes their
| position more precarious, not less.
| notact wrote:
| Both parent comments were likely referring to any entity
| other than Taiwan. If you are a fabless chip designer or
| one of their customers, it makes sense to diversify away
| from Taiwan, even if that comes at Taiwan's expense.
| 2OEH8eoCRo0 wrote:
| I worry that a lot of large cap companies either depend
| directly on TSMC (Nvidia, AMD, Apple) or depend on a
| company that depends on TSMC (Microsoft/OpenAI, Arm).
| It's TSMC all the way down and that scares me.
|
| I never thought I'd root for Intel.
| sofixa wrote:
| Doesn't that leave pretty much Samsung and Intel as the only
| options?
| brucethemoose2 wrote:
| And Nvidia has used Samsung before.
| astrodust wrote:
| So basically Samsung.
| brucethemoose2 wrote:
| Or even just offer an alternative, along with Intel:
| https://www.servethehome.com/intel-shows-gpu-max-1550-perfor...
|
| There aren't many many Gaudi/Instinct cloud offerings even
| though the market is accelerator starved.
| singhrac wrote:
| You can use Gaudi2s at the new Intel Developer Cloud[0]. Not
| sure why don't offer it on AWS though, seems a bit odd since
| they have the DL1 instances for the first-gen Gaudis.
|
| [0]: https://developer.habana.ai/intel-developer-cloud/
| brucethemoose2 wrote:
| Interesting, this looks like what I might want:
| https://eduand-alvarez.medium.com/llama2-fine-tuning-with-
| lo...
| meragrin_ wrote:
| I'd rather Intel. People have been pleading with AMD for years
| to compete with Nvidia, but AMD really has not put in a proper
| effort. They still don't look like they are putting in a proper
| effort.
| brucethemoose2 wrote:
| The H200 GPU die is the same as the H100, but its using a full
| set of faster 24GB memory stacks:
|
| https://www.anandtech.com/show/21136/nvidia-at-sc23-h200-acc...
|
| This is an H100 141GB, not new silicon like the Nvidia page might
| lead one to believe.
| latchkey wrote:
| What happened to the H100 NVL?
|
| https://www.anandtech.com/show/18780/nvidia-announces-h100-n...
| brucethemoose2 wrote:
| I dunno. But thats a dual GPU product, so its not _really_
| 180GB.
| jauntywundrkind wrote:
| This _is_ a single-chip H100 NVL. Both are GH100 's with the
| same tweaked 20% wider 6144-bit HBM3e (versus 5120 bit on
| other H100's) running at a higher speed.
|
| The HBM3e loadout is slightly different than H100 NVL's was
| going to be, but this definitely seems like a higher bin
| H100. It's basically as-if AMD had shipped a 7900 XT, then
| latter started selling the 7900 XTX; same chip, but they
| brought up all the memory controllers on this one.
| sva_ wrote:
| It is remarkable how much GPU compute is limited by memory
| speed.
| PartiallyTyped wrote:
| In many cases it's the same for CPU. AMD's cpus with bigger
| caches due to on-die 3D stacked memory are in a different
| league when it comes to perf.
| brucethemoose2 wrote:
| Depends on the workload.
|
| Sometimes things really are compute bound, and sometimes you
| get a "big" workload that still fits nicely in the GPU's L2.
| Generative AI is mostly at the far end of "memory bound."
|
| Some ML startups (like Graphcore) seemed to bet on large
| caches, sparsity and clever preprocessing instead of raw
| memory bandwidth, but I think their strategy was compromised
| when model sizes exploded. Even Cerebras was kinda caught off
| guard when their 40GB pizza was suddenly kind of cramped.
| zozbot234 wrote:
| Current ML architectures tend to be heavily optimized for
| ease of very large scale parallelism in training, even at
| the expense of a bigger model size and compute cost. So
| there may be some hope for different architectures as we
| stop treating idle GPUs as being basically available for
| free and start budgeting more strictly for what we use.
| westurner wrote:
| What would make [HBM3E] GPU memory faster?
|
| High Bandwidth Memory > HBM3E:
| https://en.wikipedia.org/wiki/High_Bandwidth_Memory#HBM3E
| brucethemoose2 wrote:
| Compared to HBM3, you mean?
|
| The memory makers bump up the speed the memory itself is
| capable of through manufacturing improvements. And I guess
| the H100 memory controller has some room to accept the
| faster memory.
| westurner wrote:
| More technically, I suppose.
|
| Is the error rate due to quantum tunneling at so many
| nanometers still a fundamental limit to transistor
| density and thus also (G)DDR and HBM performance per unit
| area, volume, and charge?
|
| https://news.ycombinator.com/item?id=38056088 ; a new QC
| and maybe in-RAM computing architecture like HBM-PM:
| maybe glass on quantum dots in synthetic DNA, and then
| still wave function storage and transmission; scale the
| quantum interconnect
|
| Is melamine too slow for >= HBM RAM?
| westurner wrote:
| "HBM4 in Development, Organizers Eyeing Even Wider
| 2048-Bit Interface" (2023)
| https://news.ycombinator.com/item?id=37859497
| WhitneyLand wrote:
| For anyone wondering how this applies to big LLMs, 144GB is
| big, but you'd need to roughly double this to train Gpt 3.x
| fitting everything in memory at once.
|
| Of course even If 300GB GPUs were available tomorrow, and you
| sold a million house to buy as many as that would allow it'd
| still take years to train once.
| sberens wrote:
| Where does the H200 fit in if the B100 is coming out the same
| year with 2x the performance? Is the H200 just cheaper than the
| B100?
| brucethemoose2 wrote:
| Its a different production line. They can keep producing both
| since they are both in demand anyway.
|
| And the B100 is farther away. Nvidia always doubles the memory
| of their cards like this mid generation.
| Mistletoe wrote:
| Can anyone explain to a layman what exactly I'm looking at in
| that picture? It looks like a neat little city or building from
| Bladerunner.
| brucethemoose2 wrote:
| It's a server motherboard with 8 GPUs crammed on it, facing up.
| The tall towers are the GPU heatsinks. I believe the blade
| looking things on the side are CPU RAM, the heatsinks on the
| back are covering the CPUs, and the little heatsink in the
| middle must be the CPU VRMs. Fans are in the back, and they
| crammed some electrical components on the front where all the
| IO is.
| formerly_proven wrote:
| Looks like an HGX drawer, so there's only GPUs on this. The
| heatsinks towards the front are probably on NVLink switches.
| brucethemoose2 wrote:
| Ah you are right.
| NoMoreNicksLeft wrote:
| Am I the only one that's annoyed by the non-alphabetical model
| numbers? Why not do B100 after the A100, then jump to H
| (supposing there won't be a C100 or D200 at some point)? Like,
| wtf Nvidia.
| robin_reala wrote:
| At least they haven't tried to do a Tesla S, 3, X, Y
| progression.
| Tommstein wrote:
| Wait, did Tesla pick those model names for that reason?
| wmf wrote:
| Yes.
| astrodust wrote:
| It's run by a 13 year old boy. What did you expect?
| brucethemoose2 wrote:
| They name their architectures after scientists (Maxwell,
| Pascal, Turing, Volta, Ampere, Lovelace, Hopper). Thats what
| the GPU initial stands for.
|
| As for the number, the die name counts down to 100 (with GA107,
| for instance, being a small GPU die and GA100 being the big
| one), and the big datacenter GPU as a product inherits the 100.
| semi wrote:
| it'd be nice if they picked them in alphabetical order
| CooCooCaCha wrote:
| I'm surprised nobody at nvidia brought this up
| brucethemoose2 wrote:
| The naming scheme goes back to at least 2004 (Curie), and
| Wikipedia has done the service of alphabetizing it for us:
| https://en.m.wikipedia.org/wiki/List_of_eponyms_of_Nvidia_G
| P...
|
| Also, it occurred to me that Nvidia does sometimes
| increment the die to 200 (EG GM200, as the Maxwell 100
| series was a single small oddball die). Its possible that
| they "refreshed" the GH100 die and are codenaming it GH200.
| formerly_proven wrote:
| At first they used "measures of hotness", but ran out a
| few generations in. Coincidentally, degrees of warmth are
| all named after scientists. So they continued with
| scientists.
|
| https://en.m.wikipedia.org/wiki/Fahrenheit_(microarchitec
| tur... -> https://en.m.wikipedia.org/wiki/Celsius_(microa
| rchitecture) -> https://en.m.wikipedia.org/wiki/Kelvin_(m
| icroarchitecture) -> https://en.m.wikipedia.org/wiki/Rank
| ine_(microarchitecture)
| cout wrote:
| Which scientist is letter B?
| 01100011 wrote:
| https://en.wikipedia.org/wiki/David_Blackwell
| maxekman wrote:
| Bohr?
| scw wrote:
| I recently had occasion to evaluate a database of 1200+ NVIDIA
| GPUs and can tell you that the only thing consistent about the
| model numbers is their inconsistency. For example, what is an
| RTX 4000? It could be the 2018 Quadro RTX 4000, the Quadro RTX
| 4000 Max-Q, or Quadro RTX 4000 Mobile (all Turing cards), but
| it could also be the RTX 4000 Mobile Ada Generation (Ada
| Lovelace card released 2023).
| wolframhempel wrote:
| I'm curious: Do you think there is a realistic chance for another
| chip maker to catch up and overtake NVidia in the AI space in the
| next few years or is their lead and expertise insurmountable at
| this point?
| chaxor wrote:
| I don't think that type of question or logic applies when
| predicting stock markets.
| edgyquant wrote:
| Luckily no one is trying to predict a stock market here
| latchkey wrote:
| AMD is trying.
|
| https://seekingalpha.com/article/4650521-amd-set-to-deliver-...
| dhruvdh wrote:
| This is launched in response to MI300X, and this should still
| not be enough to match AMD's product. This launches 2 quarters
| after MI300X, but B100 should arrive before AMD's MI400
| generation.
| MikeKusold wrote:
| I thought CUDA was NVIDIA's moat. Is that no longer the case,
| or did AMD come up with a good alternative?
| zozbot234 wrote:
| CUDA code can be forward-ported to AMD's HIP, which can be
| used with the ROCm stack. For a more standards-focused
| alternative there's also SYCL, which has implementations
| targeting a variety of hardware backends (including HIP)
| and may also target Vulkan Compute in the future.
| meragrin_ wrote:
| > CUDA code can be forward-ported to AMD's HIP, which can
| be used with the ROCm stack.
|
| Maybe in some cases, but that doesn't even really matter
| since hardware support is poor.
| PartiallyTyped wrote:
| StableHLO[1] and IREE[2] are interesting projects that
| might help AMD here, from [1]:
|
| > Our goal is to simplify and accelerate ML development by
| creating more interoperability between various ML
| frameworks (such as TensorFlow, JAX and PyTorch) and ML
| compilers (such as XLA and IREE).
|
| From there, their goal would most likely be to work with
| XLA/OpenXLA teams on XLA[3] and IREE[2] to make RoCM a
| better backend.
|
| [1] https://github.com/openxla/stablehlo
|
| [2] https://github.com/openxla/iree
|
| [3] https://www.tensorflow.org/xla
| michaelt wrote:
| The vast majority of work in ML isn't people working with
| CUDA directly - people use open source frameworks like
| PyTorch and TensorFlow to define a network and train it,
| and all the frameworks support CUDA as a backend.
|
| Other backends are also available, such as CPU-only
| training. And you can export networks in reasonably-
| standard formats.
|
| nvidia's moat is much more mature framework support than
| AMD's cards; widespread popularity due to that good
| framework support, ensuring everyone develops on nvidia,
| thus maintaining their support lead; much faster
| performance than CPU-only training; and a price that,
| though high, is a lot less than an ML developer's salary.
|
| If you need 24GB of vram and nvidia offers that for $1600
| while AMD offers it for $1300, how many compatibility
| problems do you want to deal with to save a single day's
| wages?
|
| But nvidia's moat is far from guaranteed. Huge users like
| OpenAI and Facebook might find improving AMD support pays
| for itself.
| RcouF1uZ4gsC wrote:
| > Huge users like OpenAI and Facebook might find
| improving AMD support pays for itself.
|
| At that scale they may actually develop their own
| hardware a la Google TPU.
|
| If you want to just focus on the AI problem and not on
| infrastructure, just use NVidia. If you want control and
| efficiency, design your own. AMD kind of falls in a weird
| middle ground with respect to the massive companies.
| Vvector wrote:
| AMD always launches impressive hardware specs. But they are
| way behind in software, which is more important than hardware
| PartiallyTyped wrote:
| StableHLO[1] and IREE[2] are interesting projects that
| might help AMD here, from [1]:
|
| > Our goal is to simplify and accelerate ML development by
| creating more interoperability between various ML
| frameworks (such as TensorFlow, JAX and PyTorch) and ML
| compilers (such as XLA and IREE).
|
| From there, their goal would most likely be to work with
| XLA/OpenXLA teams on XLA[3] and IREE[2] to make RoCM a
| better backend.
|
| [1] https://github.com/openxla/stablehlo
|
| [2] https://github.com/openxla/iree
|
| [3] https://www.tensorflow.org/xla
| stravant wrote:
| If they were allowed to get significantly ahead that status
| quo would likely be disrupted pretty fast.
| photonbucket wrote:
| If AMD launches hardware that is clearly faster, the
| software will move towards it
| pixelpoet wrote:
| That's exactly what the CUDA monopoly is meant to
| prevent, and as a fervent supporter of OpenCL (with two
| commercial apps), this is exactly the case I always make:
| even if some GPU came out tomorrow costing $0 and with
| infinite performance, all these people who paint
| themselves into a corner are hosed.
|
| Not that anyone cares, and everyone keeps using CUDA
| while simultaneously complaining about Nvidia GPU prices,
| as if those two things have nothing to do with each
| other...
| zozbot234 wrote:
| My own understanding is that OpenCL is semi-obsolete at
| the moment (although newer standards revisions are still
| coming out, so this may change in the future) with
| forward-looking projects mostly targeting Vulkan Compute
| or SYCL.
|
| (There are some annoying differences in the low-level
| implementations of OpenCL vs. Vulkan Compute, due to
| their being based on SPIR-V compute "kernels" vs.
| "shaders" respectively, that make it hard for them to
| interop cleanly. So that's why the choice can be
| significant.)
| saltcured wrote:
| Have you had good experience with this for portability
| though? On what classes of hardware and OS?
|
| I did a bit of work in OpenCL almost 10 years ago, and
| found it decently portable on a range of NVIDIA GPUs as
| well as Intel iGPUs. On the high end I used something
| like the Titan X while on the low end it was typical GPUs
| found in business class laptops.
|
| But my limited exposure to AMD was terrible by
| comparison. Even though I am away from that work now, I
| still tend to try to run "clpeak" and one of my simpler
| image processing scripts on each new system. And while I
| liked a Ryzen laptop for general use or even games, it
| seemed like OpenCL was useless there. It seemed my best
| option was to ignore the GPU and use Intel's x86_64 SIMD
| OpenCL runtime.
| zozbot234 wrote:
| AIUI, your current best bet for good OpenCL
| implementation on less-than-cutting-edge AMD hardware is
| the Mesa Project's RustiCL work.
| filterfiber wrote:
| Most people who utilize this hardware aren't programming
| kernels directly for the GPU, they're using abstraction
| layers like pytorch, tensorflow, etc. For the developers
| of those type of frameworks, cuda itself offers a lot of
| libraries like cublas.
|
| There's relatively few people capable of implementing
| these frameworks without a solid cuda-like foundation,
| and those that do exist would need a very strong
| incentive to do it.
| xnx wrote:
| Google's TPUs are competitive, but can only be rented.
| synergy20 wrote:
| based on MLPerf, google is further lagging behind, Intel is
| the only that is catching up a bit, but still.
|
| AMD is trying to catch up too, so far Nvidia still remains to
| be the leader, a few years ahead.
| eklitzke wrote:
| Behind in what dimension? The most expensive Nvidia chips
| are much faster than Google TPUs, but the Google TPUs are
| competitive in terms of end to end training costs (roughly
| you can think of this as FLOPs per dollar).
| brucethemoose2 wrote:
| Anandtech had a good saying. There are no bad products,
| just bad prices.
|
| So if TPU clusters are priced right...
| Mistletoe wrote:
| >I'm curious: Do you think there is a realistic chance for
| another computer maker to catch up and overtake IBM in the
| computer space in the next few years or is their lead and
| expertise insurmountable at this point?
|
| Nothing is insurmountable. :)
|
| https://en.wikipedia.org/wiki/The_Innovator's_Dilemma
|
| >It describes how large incumbent companies lose market share
| by listening to their customers and providing what appears to
| be the highest-value products, but new companies that serve
| low-value customers with poorly developed technology can
| improve that technology incrementally until it is good enough
| to quickly take market share from established business.
| wolframhempel wrote:
| I'm aware, but I'd argue that in addition to the competitive
| moats described by Hamilton Helmer (7 powers guy) there is a
| real moat in unique technological expertise in the chip
| industry. E.g. the chip making machines that ASML makes or
| the 3nm chips that TSMC produces have reached a level of
| sophistication that will take 3+ years for competitors to
| replicate, thus granting them a sort of quasi monopoly for
| the foreseeable future.
| huijzer wrote:
| I's say those are covered by Helmer with scale power.
| Spending millions on tiny process optimizations or other
| research is possible due to the large revenue streams that
| are coming in. For example, in terms of scale economies,
| only when you sell thousands high end GPUs per month you
| can hire people to write highly optimized compilers.
| revel wrote:
| Although no lead is insurmountable, the fixed capital
| investment and mature software ecosystem specific to this
| sector makes it harder to imagine what a competitor would look
| like.
|
| Given how large the prize is, the next chapter of chip
| development is likely to be nvidia vs state sponsored projects.
| China, in particular, will funnel further resources into
| acquiring this technology by any means necessary, including
| (more) industrial sabotage and outright theft. It's going to be
| interesting to see how this will play out. Up until a few years
| ago China was viewed as being a formidable competitor for
| projects of this nature, but as the country has moved to become
| increasingly authoritarian, so too have its decision making and
| execution declined in quality.
| brucethemoose2 wrote:
| Maybe not _overtake_ , but Microsoft and Amazon are going to
| eat some of the pie by pushing their own accelerators inside
| their ecosystems.
| singularity2001 wrote:
| photonics can run light based matrix multiplication for a
| fraction of the current GPUs, it's only a matter of time until
| they initiate a complete paradigm shift
| 1905 wrote:
| Maybe one of the big Chinese chip makers or AMD, but the
| growing popularity of CUDA (not compatible with other GPUs)
| makes that less likely in the near future
| bearjaws wrote:
| "GPU" - zero video output capabilities built in.
| aceazzameen wrote:
| AIPU?
| zeusk wrote:
| It can still process graphics, you just need to do a cross-
| adapter scanout or encode it for transmission over network.
| brucethemoose2 wrote:
| Can it? I thought that capability ended with the A100.
|
| It still has a media encode/decode blocks. A big one, in
| fact.
| christkv wrote:
| Is the limit on the speed on inference a memory bandwidth issue
| or compute?
| thatguysaguy wrote:
| Memory bandwidth/latency, especially when you're at smaller
| batch sizes.
| huac wrote:
| "it depends" https://kipp.ly/transformer-inference-arithmetic/
| brucethemoose2 wrote:
| Depends. One might say its sometimes "cache size limited" too.
| gosub100 wrote:
| Why do they still sell hardware now that practically every other
| business has moved to being a service provider? If we set aside
| the fact that it would be an awful move for end-users, what's to
| stop Nvidia from cornering the market by only renting them in
| their own data centers? Is it the logistics of moving the massive
| training sets?
| constantly wrote:
| What do you think all the other service providers are running
| their services on?
| gosub100 wrote:
| I'm asking why Nvidia doesn't maximize their profits by
| retaining the hardware and selling compute. They could
| capture the market from those other providers if they sold
| more FLOPS/kilowatt (or whatever metric is used). Compared to
| manufacturing GPUs/TPUs, running a datacenter (especially one
| that specializes in Nvidia hw) would seem to be a trivial
| task.
| michaelt wrote:
| Google Cloud Platform hasn't managed to make much of a dent
| in AWS's business, despite being the only place you can get
| 'TPUs' and 'bigquery'.
|
| Becoming a successful cloud provider is far from trivial,
| even if you can offer technology no-one else has.
| 10000truths wrote:
| I surmise that such a strategy would essentially hand their
| market share over to AMD on a silver platter.
| jsnell wrote:
| That would be a highly risky bet on Nvidia becoming AWS faster
| than AWS can become Nvidia.
|
| What they're doing is instead trying to make sure that their
| GPUs continue to be seen as the best option in the short/medium
| term (by having them accessible everywhere), and trying to
| commoditize their complement by giving small cloud providers
| disproportionate GPU allocations, which they hope will drive
| customers from the big providers to the smaller ones that a)
| aren't trying to build their own ML hardware, b) will have less
| negotiating leverage with Nvidia in the long term.
| wedn3sday wrote:
| Many of the largest HPC customers (read: DoD, DoE, NNSA) simply
| will not allow their code to sit on someone else's machine.
| dboreham wrote:
| Semi industry players have a strong cultural memory that
| "competing with your own customers" is a bad plan.
| schrodingerscow wrote:
| This may be a naive question, but all the metrics seem to be for
| inference. Should we expect similar gains on training?
| p1esk wrote:
| Yes. Training would especially benefit from the increased
| memory size.
| schrodingerscow wrote:
| Interesting thanks. I wonder why they aren't marketing that
| more on this page that seems important
| mtw wrote:
| I had a shock when I looked up prices for H100 gpus, wanting to
| use one just for personal experimentation and for an upcoming
| hackathon. How much this one costs? $300,000?
| nacs wrote:
| These are not for consumers -- these are datacenter-grade
| systems.
|
| If you want a consumer GPU, you can go for the RTX 4090 (24GB
| VRAM) or the A6000 Ada (48GB VRAM) if you are building a
| workstation.
|
| If you really need to "experiment" on an A/H100, then you can
| rent it by the hour through a cloud provider like Runpod.
| singhrac wrote:
| To elaborate: you can't really buy these except in specific
| configurations from Supermicro (usually 8x H100) or the like.
| So take whatever chip-specific cost you have in mind, and 8x
| it, and add on the cost of CPU/memory/storage. NVIDIA doesn't
| bother to sell these in a configuration that you can plug
| into your desktop.
___________________________________________________________________
(page generated 2023-11-13 23:01 UTC)