[HN Gopher] ChatGPT Hardware: A look at 8x NVIDIA A100 powering ...
       ___________________________________________________________________
        
       ChatGPT Hardware: A look at 8x NVIDIA A100 powering the tool
        
       Author : giuliomagnifico
       Score  : 143 points
       Date   : 2023-02-14 14:25 UTC (8 hours ago)
        
 (HTM) web link (www.servethehome.com)
 (TXT) w3m dump (www.servethehome.com)
        
       | mk_stjames wrote:
       | I recently saw a rack server of 8x SXM2 P100's (so, the 2016
       | equivalent of this) for sale for $3600 on ebay. And you can buy
       | individual P100 pci-e versions for about $300 now. They retailed
       | for about $7600 each back in 2016.
       | 
       | I wonder how long it'll be until A100's depreciate that amount.
       | 
       | I see articles from "Serve The Home" pop up now often and I've
       | yet to see one discussing hardware that I've ever come across
       | in... someone's home.
        
         | smoldesu wrote:
         | Those P100s are so cheap because they're more trouble than
         | they're worth. They don't mount easily in most cases, will
         | consume a lot of power at idle, generate lots of heat without
         | active cooling and require a Linux machine to use properly.
         | 
         | It's probably possible to engineer a home-scale inferencing
         | network like that, but I've had plenty of success just running
         | GPT-Neo on off-the-shelf ARM chips. It's not worth the extra
         | $800, exploding power bill and engineering effort to get
         | diminishing returns.
        
           | mk_stjames wrote:
           | Oh I know for sure, it's still just amusing to me to look up
           | these just slightly older cards and see them going for so
           | little compared to what they were new;
           | 
           | I do also look at various benchmarks and comparisons over and
           | over because I would like to build a home gpu server for some
           | machine learning fun and I find it interesting that the
           | TFLOPS/$ numbers always come out very much linear... A card
           | that gives 2x performance costs 2x more.... if cards
           | depreciate 50%, it's because new cards have been launched
           | that have double performance for the same price... etc...
           | Even simpler, the CUDA cores-per-dollar is pretty much a
           | constant.
           | 
           | What isn't taken into account there is various optimizations
           | new cards have that speed up training or inference, like
           | better use of the tensor cores. At some point incompatibility
           | with newer CUDA compute levels makes the older cards much
           | less useable.
        
           | dzdt wrote:
           | Not directly applicable for large language models, I was
           | doing some searching for cards with lots of double-precision
           | calc performance. Nvidia has been leaving that out of all of
           | the consumer versions of hardware; cheap P100's seem to be by
           | far the highest DP FLOPS per dollar solution available right
           | now.
        
             | mk_stjames wrote:
             | I have an interest in doing lattice Boltzmann fluid
             | dynamics simulation on GPUs, and this is something I've
             | noticed with the P100's as well. That and memory bandwidth
             | is super important for such calcs, along with total memory.
             | Commercial codes I've seen aren't using any ML ops compute
             | optimizations or tensor cores and anything like that, it's
             | mostly just OpenCL based and thus the P100s throughput per
             | dollar is pretty high.
        
             | bioemerl wrote:
             | MI25
             | 
             | Faster, cheaper, crippled by AMDs poor software support.
        
         | kamikaze675 wrote:
         | Agree, I have 14 V100's in my home but its pretty rare. I use
         | it with fiber backhaul to supply services for marketplaces like
         | vast.ai
        
           | asciimike wrote:
           | With residential power costs, is it economically viable to do
           | this? What's your utilization look like?
        
             | buildbot wrote:
             | In Washington power can be 2.95C/ per kWh so yes, very!
        
               | asciimike wrote:
               | From the utility? That's crazy.
        
             | kamikaze675 wrote:
             | Depends on how many marketplaces and the performance of the
             | entire rig, power is $0.139kwh where I am at.
        
         | Patrick-STH wrote:
         | STH has been around for 14 years and so we have tested these
         | big servers from the P100 days (see our DeepLearning12 build)
         | and newer. Home was the /home/ directory in Linux.
         | 
         | We do have some content for homes. Check out our TinyMiniMicro
         | series as an example.
        
           | trvr wrote:
           | >> Home was the /home/ directory in Linux.
           | 
           | I have never (until reading this) thought of it in that
           | context. Was that the original meaning for the name, or did
           | it shift over time?
        
             | Patrick-STH wrote:
             | It is in the "About" page :-)
             | 
             | I always get confused by why folks ask why we have server
             | content. The Wall Street Journal covers more than just one
             | road in NYC.
        
               | [deleted]
        
               | trvr wrote:
               | Understood. But what about this? https://web.archive.org/
               | web/20110201124803im_/http://www.ser...
        
               | Patrick-STH wrote:
               | Now that is a throwback! That old logo was created with
               | GIMP at a table outside NetApp. STH started at a time
               | when the SMB market (and VERY high-end homes) were using
               | Dell, HP, and IBM rackmount gear off-lease. The original
               | STH idea came from me learning Linux as an alternative to
               | Windows for that segment. STH was what I used to learn
               | the rest of the market outside of work. The first product
               | we were sent for a review was a rackmount case and that
               | started us on the path of reviewing new gear ~2010-2011
               | when it was a part time blog instead of having a team of
               | folks working on the site. I still try to keep 15-20% of
               | our content in the SMB/ high-end home arena.
        
           | metadat wrote:
           | Direct link: https://www.servethehome.com/introducing-
           | project-tinyminimic...
        
         | rsync wrote:
         | "I see articles from "Serve The Home" pop up now often and I've
         | yet to see one discussing hardware that I've ever come across
         | in... someone's home."
         | 
         | You're just not visiting _the right homes_ ...
        
       | hexomancer wrote:
       | Insanely misleading title. The author is basically showing off
       | their A100 GPUs. The ChatGPT part is just clickbait, the only
       | relevance ia that ChatGPT supposedly is also using A100s
       | (definitely not only eight of them though).
        
         | yieldcrv wrote:
         | the real question is whether ChatGPT needs that kind of
         | hardware for a single user. a client side version.
        
           | speedgoose wrote:
           | Yes it does.
        
             | jpeter wrote:
             | When stable diffusion came out, my 8gb gpu could handle 1
             | image at 512x512. Now i can generate 1000x800 with batch
             | size 4. I wonder if Chatgpt has the same unused
             | optimisation potential
        
               | fpgaminer wrote:
               | > I wonder if Chatgpt has the same unused optimisation
               | potential
               | 
               | Doubtful. As an example, OpenAI released Triton, a
               | programming language explicitly for optimizing models.
               | For the scale they're at they'd be crazy not to throw an
               | engineer or two at optimizing the models they're going to
               | send to production.
        
               | speedgoose wrote:
               | Stable Diffusion is a lot smaller than GPT3. It has a bit
               | less than a billion parameters while GPT3 has 175 billion
               | parameters.
               | 
               | I would bet that we can do better than GPT3 with less
               | computing resources but that would need a different model
               | architecture.
        
               | yieldcrv wrote:
               | what packages are you using?
               | 
               | I think I have DiffusionBee installed but I haven't
               | updated it for months
        
               | flangola7 wrote:
               | Make a Google of "hardware overhang"
               | 
               | Our brains outperform GPT-3 with only 30w of power. The
               | potential for software optimization may be many orders of
               | magnitude.
        
       | hyperific wrote:
       | "Over 80% of this assembly is a heatsink to dissipate massive
       | power."
       | 
       | Not what heatsinks do.
        
         | Patrick-STH wrote:
         | Good catch. Fixed
        
       | jhoelzel wrote:
       | Now we just need to let the models get big enough until they can
       | make themselves smaller again ;)
        
       | sxp wrote:
       | Are there articles detailing the tradeoffs between an A100 system
       | and a consumer system in terms of perf/$? E.g, if you're a
       | startup trying to be scrappy and your choice is between a $100k+
       | A100 system and an off-the-shelf RTX 40x0 networked cluster, what
       | would be the tradeoff?. For embarrassingly parallel use cases
       | like ray-tracing or a compute backend that services many clients
       | and doesn't need FP64, the consumer system would win. For other
       | uses cases that have a hard requirement for the NVLinked access
       | to 80GB of RAM per card, the A100 system would win.
       | 
       | Has anyone done benchmarks to show what the tradeoff calculations
       | are for various use cases?
        
         | synergy20 wrote:
         | I was trying to build a ML training machine myself these days.
         | I don't really need the RTX FP64 and its 4K graphic rendering
         | with gaming stuff, all I need is ML training(FP16 mostly with
         | lots of CUDA cores).
         | 
         | There is no such thing, you either use multiple RTX GPU cards
         | and 'bend' them for ML training, or you buy A100 40GB at about
         | 125K or A100 80GB at about 250K, which is far beyond my budget.
         | 
         | Google's TPU could be used to assist ML training, but then it
         | does not have the CUDA ecosystem for software, plus, it kept
         | its new TPU for inhouse use only as far as I can tell.
         | 
         | AWS ML EC2 are very expensive to me which is why I want to
         | build one for mid-sized ML trainings.
        
         | Patrick-STH wrote:
         | There are a few big ones: - The CUDA license does not allow you
         | to use GeForce in the data center. In the US it has become less
         | popular, but if you look at our Inspur AIStation piece, that
         | was a cluster located in China with GeForce cards. So it still
         | happens, but less so. - The memory capacity is another big
         | challenge. Newer models have 80GB which dwarfs the 24GB on a
         | 4090. We just got the RTX 6000 Ada in, so that is an option for
         | more memory. - For higher-end training, one of the big
         | challenges is interconnect, so having NVLink and Infiniband or
         | 100GbE+/ Infiniband NICs is important. The HGX A100 platform is
         | designed for that with its NVSwitch and PCIe switch topology.
         | 
         | With all of that said, you are 100% right that many startups
         | have used consumer cards for years. For example, Andrej
         | Karpathy talked about how our DeepLearning11 build (8x 1080
         | Ti's) had a ~3 month payback period versus AWS
         | https://twitter.com/karpathy/status/924340245478256640
        
           | p1esk wrote:
           | _Andrej Karpathy talked about how our DeepLearning11 build
           | (8x 1080 Ti 's) had a ~3 month payback period versus AWS_
           | 
           | In 2017. Currently you can rent 8xA100 server for $8.8/hr:
           | https://lambdalabs.com/service/gpu-cloud
           | 
           | At this price the payback stretches to about 3 years (taking
           | into account average energy costs in US, and assuming 24/7
           | operation for the whole 3 years).
        
             | TOMDM wrote:
             | A bit of extra context, that's $8.8/hr for the 40GB A100's
             | 
             | The 80GB A100's will run you $12.0/hr for 8 of them.
        
               | fancyfredbot wrote:
               | That's 24k over 3 months which would buy and power 8 4090
               | GPUs by my reckoning. Of course those 8 4090s wouldn't
               | have enough memory to run chatGPT so maybe AWS is good
               | value after all.
        
               | TOMDM wrote:
               | This is Lambda Labs' pricing, which is significantly
               | cheaper than AWS.
        
         | KaiserPro wrote:
         | Unless you are planning to have a full system on all the time,
         | renting a loaded 8xA100 machine is almost certainly the way to
         | go for a startup.
         | 
         | As others have pointed out, vRAM is the killer here. this big
         | of a machine is mostly for vRAM coherence, which is a specific
         | usecase.
         | 
         | For almost every other type of job lots of single GPUs with
         | shared disk storage will do (particle sim, rendering, oil and
         | gas, etc.)
        
         | monkmartinez wrote:
         | Tim Dettmers [0] has been updating the linked article for a
         | while now. It is a great primer -> deep dive into the world of
         | GPU's for ML/DL tasks. I don't need 4x RTX 4090's but I really
         | want them. As it stands now, I use an old M40 that I modified
         | for water cooling. The only tricky part was configuring Win10
         | to run the card in WDDM mode as opposed to TCC mode without
         | onboard graphics. To be honest, I am not entirely sure how it
         | works as the pass-through is an older quadro card.
         | 
         | How do I justify the purchase of DGX-1 as a hobby programmer?
         | Call it my mid-life crisis purchase? I mean, its not that much
         | more than a mid-range 2023 Corvette Stingray.
         | 
         | [0] https://timdettmers.com/2023/01/30/which-gpu-for-deep-
         | learni...
        
         | paulpan wrote:
         | Historically Nvidia's professional-grade and consumer products
         | were artificially segmented. The only difference was in their
         | amount of VRAM, with recent example is the Pascal-generation
         | GeForce 1080 vs. the P100 that someone else has compared in
         | detail: https://medium.com/@alexbaldo/a-comparison-between-
         | nvidias-g...
         | 
         | But in recent years Nvidia has added harder gating in between
         | their products, the biggest being SLI or NVLink support reduced
         | starting with Pascal and now entirely dropped in the latest Ada
         | Lovelace generation. Effectively forcing you to buy the
         | professional chips.
         | 
         | A different angle - how long before Google, Microsoft, and
         | Amazon design and deploy their own "GPUs" instead of buying
         | from Nvidia? They've all started with their own ARM SOCs so
         | surely it's inevitable.
        
         | 0x008 wrote:
         | Not for whole systems but the single GPUs (A100/A40) are
         | significantly better in performance per watt (almost by a
         | factor of 2) than RTX cards for Machine Learning workloads.
         | 
         | You can watch this LTT video to get an idea:
         | https://youtu.be/zBAxiQi2nPc
         | 
         | Probably if you run them full time in a enterprise environment
         | the power consumption will be the bulk of your cost?
        
       | Valgrim wrote:
       | Is this the hardware used to build the model, or to run the
       | model? Does this single 100000$ computer run the entire chatGPT
       | service concurrently for all users around the world? If so this
       | seems incredibly cheap.
        
         | paulluuk wrote:
         | In machine learning, generally the expensive part is training
         | the model, not running the model.
        
           | bobleeswagger wrote:
           | At this point the costs of hosting the trained model probably
           | surpass the training cost.
        
             | paulluuk wrote:
             | It might, I'm not sure. It depends on many factors: how
             | much cloud discount are they getting, for example? And do
             | you count the cost of training all previous models before
             | chatGPT, or just the cost of training chatGPT itself?
             | 
             | But you might be right, chatGPT is insanely popular right
             | now.
        
             | delijati wrote:
             | has anyone numbers on how many queries/second in inference
             | time chatGPT can serve?
        
         | HPsquared wrote:
         | Supercomputer used for training, top 5 in the world apparently:
         | 
         | https://news.microsoft.com/source/features/ai/openai-azure-s...
        
         | 0x008 wrote:
         | The magnificent part is that the 8xA100 is only a blade and you
         | can fill a whole rack with them. Yet one is already incredibly
         | powerful.
         | 
         | Also the successor H100 is already released and at least 2x as
         | powerful.
        
         | anentropic wrote:
         | I would guess that's the minimum needed to run an 'instance' of
         | the model for inference
         | 
         | not sure how many concurrent users that could support, but I'd
         | imagine the public service has a whole bunch of these racks
         | 
         | from what I read... GPT-J at 6B params is too big to load on
         | any consumer GPU (needs ~25GB)
         | 
         | and then GPT-3 is 175B params or ~29x larger, so maybe needing
         | 7-800GB RAM
         | 
         | 8x 80GB A100s would give 640GB RAM, so its approx the right
         | ballpark
        
       | Const-me wrote:
       | A100 has 1555 GB/sec of memory bandwidth. 50% more than Radeon
       | 7900 XTX, but the AMD GPU costs 10x less.
       | 
       | nVidia's top management did amazing job with their strategy. They
       | have spent decades, and millions of dollars, developing and
       | promoting CUDA. Now this motivates people to spend that amount of
       | money on nVidia's hardware.
       | 
       | I don't believe the trend continues for ML inference, at least
       | not for long. Unlike traditional HPC stuff like FEM or fluid
       | dynamics, the GPU code of ML inference is not that complicated.
       | The complexity is limited to the data being processed, these ML
       | models.
       | 
       | It's not terribly hard to port the inference to other hardware
       | architecture or software stack, reusing the models. I think it's
       | only a matter of time until AI companies like OpenAI will start
       | porting their code from CUDA to VulkanCompute or some other
       | vendor-agnostic stack, to avoid buying or renting such $100k
       | computers. Don't get me wrong, these computers are awesome, just
       | very expensive.
        
         | 2OEH8eoCRo0 wrote:
         | > It's not terribly hard to port the inference to other
         | hardware architecture or software stack
         | 
         | Sounds like a job for AI. Universal translator and all that.
        
         | cma wrote:
         | With nvidia's sparsity support memory bandwidth can be
         | effectively doubled for inference, if you can sparsify.
        
         | ericpauley wrote:
         | There's also a problem of dataset size. the 7900XTX maxes out
         | at 24GB when a lot of new models require the higher-end 80GB
         | A100 variants. The A100 is also a server-grade card, so even
         | spec-for-spec it's not a fair comparison.
        
           | TallGuyShort wrote:
           | Could you educate me on what's really different about a card
           | that is "server-grade" vs. one that is not?
        
             | jmalicki wrote:
             | Two of the biggest differences are ECC memory and
             | virtualization support.
        
               | m463 wrote:
               | well, and it's a specialized die. A100 ~ 2x size of 3090
        
               | crabbone wrote:
               | Yeah, also while ECC is mostly transparent to the
               | software layer, virtualization support is in itself a
               | huge software layer that comes with this kind of
               | equipment. MIG (
               | https://docs.nvidia.com/datacenter/tesla/mig-user-guide/
               | ), there's also software made to expose GPU inside
               | containers.
               | 
               | This software is mostly supplied by vendor, and it mostly
               | cares about vendor's tech, and will usually require
               | special features exposed by the hardware, so, won't work
               | with consumer-grade GPUs.
        
             | ericpauley wrote:
             | For these GPUs, it partially comes down to form factor
             | (being able to drop 8 in a chassis with unified cooling).
             | I'd say that's the primary difference here. This applies
             | maybe a little more noticeably to the rest of the computer
             | around the a100s, with ECC ram, hot swappable components,
             | and redundant power supplies. The 8x a100 rigs are absolute
             | beasts, difficult to move even by two people without
             | removing some components. There is also price
             | discrimination here where manufacturers aim for higher
             | margins on server components (as nvidia clearly does).
             | 
             | Between the two mentioned cards, though, it really does
             | come down to performance. The cards are simply not
             | interchangeable for large scale model training.
        
           | godelski wrote:
           | As someone who trains large models, this is pretty much the
           | only number I care about. Batching saves more time, and gets
           | better results, than clock speed.
           | 
           | For the A100, A6000s, and H100s you are paying for the memory
           | and the upcharge for being a server card. AMD isn't really
           | competing in this space, but they are moving in that
           | direction if you look at the exascale machines[0,1]
           | 
           | [0] https://www.top500.org/system/180047/
           | 
           | [1] https://www.amd.com/en/products/server-
           | accelerators/instinct...
        
           | Const-me wrote:
           | > a lot of new models require the higher-end 80GB A100
           | variants
           | 
           | For training, yeah. But at least in some cases, inference can
           | be optimized to use less memory.
           | 
           | For their large Whisper model, OpenAI says they need 10GB of
           | VRAM: https://github.com/openai/whisper#available-models-and-
           | langu...
           | 
           | My DirectCompute port of that uses slightly over 4GB of VRAM,
           | see the last column in that table: https://github.com/Const-
           | me/Whisper/blob/master/SampleClips/... I haven't actually
           | optimized memory usage I only did the DirectCompute port, the
           | memory savings were achieved by Georgi Gerganov in
           | whisper.cpp project, which I ported to Windows.
        
             | anentropic wrote:
             | apparently GPT-J needs ~25GB RAM min (for inference only),
             | i.e. a 32GB card ...and GPT-3 is an order of magnitude
             | bigger again
             | 
             | https://nlpcloud.com/gpt-3-open-source-alternatives-gpt-j-
             | gp...
             | 
             | though TBH just making cards with more RAM to fit these
             | kind of models seems easier than being at absolute cutting
             | edge of performance
             | 
             | I also wonder how this metric translates to something like
             | Apple Silicon 'unified memory'... latest MBP have 32GB RAM,
             | I wonder if anyone managed to run GPT-J there yet?
        
               | trifurcate wrote:
               | I am running GPT-J-6B inference on 13GB of VRAM. FP16 is
               | a must at this scale. I've also loaded GPT-NeoX-20B using
               | int8 quantization onto a single RTX 3090 before (24GB
               | VRAM) however inference latency was quite high.
        
               | anentropic wrote:
               | There's a float16 version of GPT-J rewritten in C++ here
               | that apparently fits into 16GB...
               | 
               | https://github.com/ggerganov/ggml/tree/master/examples/gp
               | t-j
               | 
               | it's CPU-only and runs inference at "about ~6 words per
               | second" on an M1 MBP
        
               | mark_l_watson wrote:
               | They have a link to the bottom for something similar, but
               | with GPT-2. The Hugging Face docs have hints for using
               | FP16, and avoiding double memory requirements for model
               | and checkpoint data.
        
               | smoldesu wrote:
               | That's for the max-size model (6B parameters). GPT-Neo
               | has pruned models 125m and 1.3b that fit into 1.5gb and
               | 6gb of memory respectively.
               | 
               | These models are obviously nerfed, but you can run them
               | on budget ARM hardware and get legible, paragraph-length
               | responses in less than 10 seconds.
        
               | flangola7 wrote:
               | There's a 20B model now
        
               | smoldesu wrote:
               | The 20B model was the original I believe, and the other
               | models were all pruned from it.
        
             | TOMDM wrote:
             | I've been running the stock implementation that OpenAI put
             | out, I didn't know people had made so many improvements!
             | 
             | Thank you for your work.
        
         | mambru wrote:
         | You should probably compare the H100:
         | 
         | https://www.techpowerup.com/gpu-specs/h100-sxm5-96-gb.c3974
         | 
         | With the MI300, which has 128GB and more compute power (at
         | least on paper):
         | 
         | https://www.techpowerup.com/gpu-specs/radeon-instinct-mi300....
         | 
         | A100 can be compared with MI250, perhaps. It also has 128GB.
        
           | Patrick-STH wrote:
           | The MI300 will compete more with Grace Hopper than with the
           | H100 SXM5. We will be covering and showing both later this
           | year.
        
           | metadat wrote:
           | What is the pricing like for the MI300's?
        
             | BizarroLand wrote:
             | I would guess $12k-$16k, but they haven't released the
             | price yet.
        
           | mhio wrote:
           | An Azure 8x MI250X OAM enclosure was out and about at the
           | SC22 conference late last year (second half of the linked
           | article).
           | 
           | https://www.servethehome.com/microsoft-azure-at-sc22-and-
           | the...
        
         | GistNoesis wrote:
         | This TB/sec of memory bandwidth is only when the data is on the
         | GPU. The real problem is that GPU memory is expensive and
         | models are getting bigger.
         | 
         | What we would like to do is what has usually been done with the
         | cache hierarchy where you bring the data as needed to the GPU.
         | As this would allow to only need a lot less memory, as we could
         | stream parameters from RAM or disk as needed.
         | 
         | But there is a bottleneck at the pcie interface ( 32GB/s for
         | pcie 4.0x16 ), and you can't feed the GPU fast enough. Which
         | means you can't leverage cheap memory directly.
        
         | 6gvONxR4sf7o wrote:
         | > 50% more than Radeon 7900 XTX, but the AMD GPU costs 10x less
         | 
         | You could say that about some of nvidia's other GPUs too. A
         | better comparison would be AMD's server offerings, which are
         | priced comparably.
         | 
         | > Unlike traditional HPC stuff like FEM or fluid dynamics, the
         | GPU code of ML inference is not that complicated.
         | 
         | It's complicated enough relative to the problem ML people are
         | trying to solve. That's why in ML, all these tools have emerged
         | specifically to let practitioners not care about the GPU code.
         | People writing pytorch code don't want to think about CUDA vs
         | vulkan at all. It's more of a question for the people writing
         | the foundational libraries like pytorch.
         | 
         | I sincerely hope that tools like openai's triton make it easier
         | to support a variety of GPUs.
        
           | Const-me wrote:
           | > You could say that about some of nvidia's other GPUs too.
           | 
           | Technically, yes. Practically, nVidia forbids to deploy their
           | other GPUs in data centers, in the EULA of their drivers:
           | https://www.datacenterdynamics.com/en/news/nvidia-updates-
           | ge...
           | 
           | May or may not be enforceable, probably depends on
           | jurisdiction, but I'm pretty sure that EULA gonna scare some
           | people.
           | 
           | > People writing pytorch code don't want to think about CUDA
           | vs vulkan at all.
           | 
           | Well, there're also other people. OpenAI's CEO doesn't
           | necessarily write pytorch code, but he described the compute
           | costs of ChatGPT as "eye-watering".
           | 
           | > I sincerely hope that tools like openai's triton make it
           | easier to support a variety of GPUs.
           | 
           | I'll believe it when I see it. Performance portability across
           | GPU architectures ain't trivial. For instance, in my Whisper
           | implementation, the most expensive compute shader is the one
           | which computes product of 2 matrices.
           | 
           | nVidia version: https://github.com/Const-
           | me/Whisper/blob/master/ComputeShade...
           | 
           | AMD version: https://github.com/Const-
           | me/Whisper/blob/master/ComputeShade...
           | 
           | These two compute shaders have little in common, they even
           | use different memory layout for the arguments, despite both
           | compute shaders are doing the same math.
        
         | synergy20 wrote:
         | indeed the CUDA software is the moat for Nvidia, though its
         | hardware is still great.
        
           | fancyfredbot wrote:
           | At the scale of investment OpenAI are making the software
           | shouldn't be the moat anymore - you can hire a lot of very
           | good software engineers with ten percent of that hardware
           | budget.
        
             | synergy20 wrote:
             | it's way more than that, building an ecosystem takes time,
             | like a decade or more, lots of tooling, libraries,
             | optimizations etc have to be done, and they have to be done
             | in a way users think it's either similar to CUDA, or is
             | simpler, the moat is very wide and deep as far as I can
             | tell.
        
               | Const-me wrote:
               | I don't think OpenAI needs a new ecosystem. I think they
               | just need a cost-efficient way to run inference of their
               | ML models, for large userbase.
               | 
               | That smaller task is way more manageable. It shouldn't
               | take a decade, maybe a month or two. Vulkan ain't some
               | new tech, the API arrived years ago and is widely used by
               | game developers.
               | 
               | Thanks to Valve work on DXVK 2.0, there's even an
               | implementation of MS DirectCompute GPU API over Vulkan on
               | Linux. I'm not saying doing that is necessarily the best
               | approach for the job, just one of the many options.
        
               | fancyfredbot wrote:
               | I feel like very few ML researchers actually use CUDA.
               | You'd likely get a long way with an optimised backend for
               | XLA to cover tf/jax, probably something similar for
               | pytorch. That's a lot of work but not sure it's a whole
               | decade.
        
               | synergy20 wrote:
               | Not sure if that's entirely tree. CUDA has 85% or more of
               | the market share last time I checked, intel's OneAPP and
               | AMD's ROMA? are so behind that made their Machine
               | learning hardware totally uninteresting for ML related
               | purchases, sadly.
        
       | pcdoodle wrote:
       | 5KW Peak Power per system pictured. I wonder, how many watt hours
       | per query are used on average? Is this 100k investment enough to
       | roll your own ChatGPT level bot using open source software?
        
         | HPsquared wrote:
         | Not exactly... OpenAI apparently have the 5th most powerful
         | supercomputer in the world. This is probably what they used for
         | training:
         | 
         | https://news.microsoft.com/source/features/ai/openai-azure-s...
         | 
         | https://www.zdnet.com/article/microsoft-builds-a-supercomput...
        
           | pcdoodle wrote:
           | Thanks for the links. I wonder what percentage of the energy
           | expenditure is the initial training vs. operating the
           | service?
        
             | HPsquared wrote:
             | That depends how much it's used, I suppose. Ever-
             | increasing, it looks like. At some point it might be worth
             | investing in more training to reduce operating expense.
             | Pretty standard CAPEX vs OPEX calculation.
             | 
             | On the other hand, we're not counting energy saved by
             | people using it.. What if some scientist uses it and ends
             | up solving fusion power? Or, say, making copywriters and
             | programmers redundant (who I assume would go on to become
             | subsistence farmers) - many energy savings to be had!
        
       ___________________________________________________________________
       (page generated 2023-02-14 23:01 UTC)