[HN Gopher] AMD may get across the CUDA moat
___________________________________________________________________
AMD may get across the CUDA moat
Author : danzheng
Score : 510 points
Date : 2023-10-06 17:35 UTC (1 days ago)
(HTM) web link (www.hpcwire.com)
(TXT) w3m dump (www.hpcwire.com)
| Andrew018 wrote:
| [dead]
| pama wrote:
| There is only limited empirical evidence of AMD closing the gap
| that NVidia has created in the science or ML software. Even when
| considering pytorch only, the engineering effort to maintain
| specialized ROCm along with CUDA solutions is not trivial (think
| flashattention, or any customization that optimizes your own
| model). If your GPUs only need a simple ML workflow all times for
| a few years nonstop, maybe there exist corner cases where the
| finances make sense. It is hard for AMD now to close the gap
| across the scientific/industrial software base of CUDA. NVidia
| feels like a software company for the hardware they produce;
| luckily they make the money from hardware thus cannot lock the
| software libraries.
|
| (Edited "no" to limited empirical evidence after a fellow user
| mentioned El Capitan.)
| falconroar wrote:
| I don't understand why developers of PyTorch and similar don't
| use OpenCL. Open standard, runs everywhere, similar performance
| - what's the downside??
| pama wrote:
| I don't know for sure why the early pytorch team picked it,
| but my guess is due to simplicity and performance. NVidia
| optimizes CUDA better that OpenCL and provides tons of useful
| performance tuning tools. It is hard to match the CUDA
| performance with OpenCL even on the same NVidia GPU hardware,
| and making performant code compatible across different GPU
| with OpenCL is also hard. I know examples of scientific codes
| that became simpler and faster (on nvidia hardware) by going
| from openCL to CUDA but haven't yet heard of examples the
| other way around.
| Certhas wrote:
| The fact that El Capitan is AMD says that at least for
| Science/HPC there definitely is evidence of a closing gap.
| pama wrote:
| Thanks. You are actually right that this new supercomputer
| might move the needle once it is in production mode. I will
| wait and see how it goes.
| fotcorn wrote:
| ROCm has HIP (1) which is a compatibility layer to run CUDA
| code on AMD GPUs. In theory, you only have to adjust #includes,
| and everything should just work, but as usual, reality is
| different.
|
| Newer backends for AI frameworks like OpenXLA and OpenAI Triton
| directly generate GPU native code using MLIR and LLVM, they do
| not use CUDA apart from some glue code to actually load the
| code onto the GPU and get the data there. Both already support
| ROCm, but from what I've read the support is not as mature yet
| compared to NVIDIA.
|
| 1: https://github.com/ROCm-Developer-Tools/HIP
| binarymax wrote:
| And the question for most that remains once AMD catches up: will
| the duopoly result in lower prices to a reasonable level for
| hobbyists or bootstrapped startups, or will AMD just gouge like
| NVidia?
| evanjrowley wrote:
| AMD prices will go up because of the newfound ability to gouge
| for AI/ML/GPGPU workloads. Nvidia's will likely go down, but I
| don't expect it will be by much. The market demand is high, so
| the equilibrium price will also be high. Supply isn't at
| pandemic / crypto-rush lows, but the supply of cards useful for
| CUDA/ROCm still is.
| stjohnswarts wrote:
| In general I think it will lower prices, certainly not as much
| as if there were 4+ on the market where it's hard to anticipate
| your rivals. a 2 body system is pretty straight forward, 3 body
| can be stable for a while with some restrictions, a 4 body
| problem is really damn hard...
| klysm wrote:
| A simplistic economic take would suggest that the competition
| would result in lower prices, but given two players in the
| market who knows.
| sumtechguy wrote:
| It is oligopoly pricing.
|
| https://www.investopedia.com/terms/o/oligopoly.asp
|
| With that few competitors pricing would not change much.
| ad404b8a372f2b9 wrote:
| Prices seemed to have lowered when AMD came out with CPUs
| competitive with Intel's.
| tibbydudeza wrote:
| Price difference between 13900K and AMD Ryzen 9 7950x is
| not big - the latest 7950X3D is about on par with the
| higher clocked 13900KS as well.
| redeeman wrote:
| because intel lowered their prices
| tibbydudeza wrote:
| I was on the market last month - Intel was the better
| choice because AM5 boards and DDR5 was too expensive.
|
| Ryzen 9 7950X -- $799 on release Intel 13900K - $589.
| AnthonyMouse wrote:
| That's mostly when there isn't a lot of price elasticity of
| demand. If you're Comcast and Verizon, each customer wants
| one internet connection and you're not going to change the
| size of the market much by offering better prices.
|
| If you're AMD and NVIDIA and lowering the price would
| double the number of customers, you might very well want to
| do that, unless you're supply constrained -- which has been
| the issue because they're both bidding against everyone
| else for limited fab capacity. But that should be
| temporary.
|
| This is also a market with a network effect. If all your
| GPUs are $1000 and nobody can afford them then nobody is
| going to write code for them, and then who wants them? So
| the winning strategy is actually to make sure that there
| are kind of okay GPUs available for less than $300 and make
| sure lots of people have them, then sell very expensive
| ones that use the same architecture but are faster.
|
| That has been the traditional model, but the lack of
| production capacity meant that they've only been making the
| overpriced ones recently. Which isn't actually in their
| interests once the supply of fab capacity loosens up.
| ngcc_hk wrote:
| Actually there is already a market like this they are in
| - game. Most Gpu used are low to mid-range see steam. The
| AI has to and will go down to that level for using or
| gaming. You cannot just have game for intel ... you did.
| Then steam work hard and realize the steam deck. You can
| have total different software like j and a did. Hence you
| really can't have 1 N to rule for long. Do thank for it
| and all the fish, without it we might be still doing Gpu
| for numerical computing research.
| Qwertious wrote:
| AI doesn't have all that much application to gamedev -
| people think "videogames have lots of AI", and don't
| realize that "game AI" and "stable diffusion/GPT AI" are
| about as related as Java and Javascript.
|
| Game "AI" is meant to be fun to play (and _win_ )
| against, they're not meant to be smart; that's why zombie
| games are so successful. Most "game AI" are finite state
| machines, throwing a neural network at the issue would be
| absurd overkill.
|
| I'm sure there will be some AI applications in games
| (like procedural world generation or such, perhaps) but
| it's not the obvious connection that most people think.
| hackerlight wrote:
| Looking at the CPU market, competition did lead to lower
| prices: AMD are the best, but their CPUs are very reasonably
| priced because Intel is close behind.
|
| In the gaming market for GPUs, Nvidia has no competition
| except in some niche areas. Overall, their lead in upscaling
| software is too commanding so they can price how they want.
| Customers are paying 15-20% premiums for the same raw
| hardware performance, all to access Nvidia's DLSS, because
| there's no good competition.
| binarymax wrote:
| My intuition is along the lines that if AMD had a competing
| product earlier, then it would have kept prices down. But
| since Nvidia has shown what the market will pay, AMD won't be
| able to resist overcharging. It will probably come down a
| little, but nowhere near to the point of affordability.
|
| I sure hope I'm wrong.
| tyre wrote:
| AMD might have to charge less to break into customers that
| are already bought into Nvidia. There has to be a discount
| to cover the switching costs + still provide savings (or
| access).
| zirgs wrote:
| AMD will have to provide a REALLY steep discount to
| convince me to come back.
| wil421 wrote:
| Why would their investors allow anything else? I'm sure they
| see it as a huge loss like intel and mobile.
| quitit wrote:
| I think in this case the changes needed to make AMD useful will
| open the market to other players as well (e.g. Intel).
|
| PyTorch is already walking down this path and while CUDA-based
| performance is significantly better, that is changing and of
| course an area of continued focus.
|
| It's not that people don't like Nvidia, rather it's just that
| there is a lot of hardware out there that can technically
| perform competitively, but the work needs to be done to bring
| it into the circle.
| binarymax wrote:
| Last I checked I saw the H100 was about two gens more
| advanced for certain components (tensor cores, bfloats,
| cache, mem bandwidth) - but my research may have been wrong
| as admittedly I'm not as familiar with AMDs offerings for
| GPU.
| FuriouslyAdrift wrote:
| They are not behind...
| https://www.tomshardware.com/news/amd-expands-mi300-with-
| gpu...
|
| You can also actually buy them as opposed to the nVidia
| offerings which you are going to have to fight for.
| johngossman wrote:
| When AMD caught up to Intel in CPUs, prices went down (at least
| compared to when Intel had a complete monopoly). The same was
| true when AMD gaming cards were more competitive. Chip
| manufacturers have shown themselves willing to both raise
| prices when they can and lower them when they must.
| adamsvystun wrote:
| This is not a binary question. Two players, while not ideal,
| are better then just one.
| rdsubhas wrote:
| Demand will push AMD prices up by couple hundred bucks and
| Nvidia cards down by couple hundred bucks. A hobbyist customer
| will be neither better or worse.
| rafaelmn wrote:
| If the margins and demand is there Intel will eventually show
| up
| Havoc wrote:
| Is either in doubt?
| rafaelmn wrote:
| Wouldn't be surprised if a bunch of investment is hype
| bubble and demand correction forces price correction. Maybe
| not immediately but at Intel's pace - they managed to miss
| out on mining bubble, wouldn't be surprised for them to
| release in a correction.
| wmf wrote:
| Intel already showed up three or four times but their
| software is as bad as AMD's used to be.
| ilc wrote:
| Thankfully, software can be fixed over time as AMD has
| shown. Lack of another competitor can't be fixed as easily.
| nologic01 wrote:
| If the AI hype persists the CUDA moat will be less relevant in ~2
| yrs.
|
| Historically HPC was simply not sufficiently interesting (in
| commercial sense) for people to throw serious resources in the
| direction of making it a mass market capability.
|
| NVIDIA first capitalized on the niche crypto industry (which
| faded) and was then well positioned to jump into the AI hype. The
| question is how much of the hype will become real business.
|
| The critical factor for the post-CUDA world is not any
| circumstantial moat but who will be making money servicing
| stable, long term computing needs. I.e., who will be buying this
| hardware not with speculative hot money but with cashflow from
| clients that regularly use and pay for a HPC-type application.
|
| These actors will be the long term buyers of commercially
| relevant HPC and they will have quite a bit of influence on this
| market.
| benreesman wrote:
| I know a lot of people don't like George, I dislike plenty of
| people who are doing the right thing thing (including by some
| measures sama and siebel while they were pushing YC forward).
|
| But not admitting the tinygrad project is the best Rebel Alliance
| on this is just a matter of letting vibe overcome results.
| ris wrote:
| I don't understand the author's argument (if there is one) -
| pytorch has existed for ages. AMD's Instinct MI* range has
| existed for years now. If these are the key ingredients why has
| it not already happened?
| physicsguy wrote:
| Don't agree at all. PyTorch is one library - yes, it's important
| that it supports AMD GPUs but it's not enough.
|
| The ROCm libraries just aren't good enough currently. The
| documentation is poor. AMD need to heavily invest in their
| software ecosystem around it, because library authors need decent
| support to adopt it. If you need to be a Facebook sized
| organisation to write an AMD and CUDA compatible library then the
| barrier to entry is too high.
| weebull wrote:
| Disagree that the Rocm libraries are poor. Their integration
| with everything else is poor because everything else is so
| highly Nvidia centric, and AMD can't just write to the same API
| because it's copyright Nvidia (see Oracle's Java case).
|
| The adoption of CUDA has been such a coop for Nvidia, it's
| going to take some time to dismantle it.
| physicsguy wrote:
| I don't use high level frameworks like PyTorch because my
| work is in computational physics so I do actually use the
| lower level libraries. The documentation doesn't even come
| close although it has got better. But they're just not at
| feature parity, and that's not on anyone but AMD currently.
| They need to invest more in the core libraries.
|
| Just look at cuFFT vs rocFFT for e.g... they aren't even
| close to being at feature parity - things like multi GPU is
| totally missing and callbacks are still "experimental". These
| are pretty basic features - bear in mind that when people
| ported from CPU codes CUDA had to support these because they
| existed in FFTW (transforms over multiple CPUs rather than
| GPUs though via MPI).
| jiggawatts wrote:
| Can I buy an MI300 or even rent one in a cloud?
| arcanus wrote:
| Soon. The card is coming in Q4. The early shipments are likely
| all going to LLNL's El Capitan Exascale computer:
| https://www.tomshardware.com/news/amds-instinct-mi300-moves-...
| voz_ wrote:
| The amount of random wrong stuff about pytorch in this thread is
| pretty funny.
| pjmlp wrote:
| Unless they get their act together regarding CUDA polyglot
| tooling, I seriously doubt it.
| hankman86 wrote:
| I suspect that AMD will use their improved compatibility with the
| leading ML stack for data center deals. Presumably by offering
| steep discounts over NVIDIA's GPUs. This might help them to break
| into the market.
|
| Individual ML practitioners will probably not be tempted to
| switch to AMD cards anytime soon. Whatever the price difference
| is: it will hardly offset the time that is subsequently sunk into
| working around remaining issues resulting from a non-CUDA (and
| less mature) stack underneath PyTorch.
| javchz wrote:
| CUDA is the only reason I have an Nvidia card, but if more
| projects start migrating to a more agnostic environment, I'll be
| really grateful.
|
| Running Nvidia in Linux isn't as much fun. Fedora and Debian can
| be incredibly reliable systems, but when you add an Nvidia card,
| I feel like I am back in Windows Vista with kernel crashes from
| time to time.
| kombine wrote:
| I use a rolling distro (OpenSUSE Tumbleweed) and have had zero
| issues with my NVIDIA card despite it pulling the kernel and
| driver updates as they get released. The driver repo is
| maintained by NVIDIA itself, which is amazing.
| filterfiber wrote:
| Do you use wayland, multiple monitors, and/or play games or
| is it just for ML/AI?
| smoldesu wrote:
| I do all of those things with my 3070 and it works just
| fine. Most of them will depend on your DE's Wayland
| implementation.
|
| I'm not here to desparage anyone experiencing issues, but
| my experience on the NixOS rolling-release channel has also
| been pretty boring. There was a time when my old 1050 Ti
| struggled, but the modern upstream drivers feel just as
| smooth as my Intel system does.
| orangetuba wrote:
| Nvidia on Linux is more like running Windows 95 from the gulag,
| and you're covered in ticks. I absolutely detest Nvidia because
| of the Linux hell they've created.
| smoldesu wrote:
| Those problems might just be GNOME-related at this point. I've
| been daily-driving two different Nvidia cards for ~3 years now
| (1050 Ti then 3070 Ti) and Wayland has felt pretty stable for
| the past 12 months. The worst problem I had experienced in that
| time was Electron and Java apps drawing incorrectly in
| xWayland, but both of those are fixed upstream.
|
| I'm definitely not against better hardware support for AI, but
| I think your problems are more GNOME's fault than Nvidia's.
| KDE's Wayland session is almost flawless on Nvidia nowadays.
| arsome wrote:
| If GNOME can tank the kernel, it ain't GNOME's fault.
| kombine wrote:
| I really hope that with KDE 6 I can finally switch to
| Wayland!
| Zardoz84 wrote:
| I'm using KDE on Debian 12 with AMD GPU with Wayland, and
| works. it keeps being a bit annoying compared with X11 with
| a few programs (Eclipse, Dbeaver... I need to launch both
| with flags to not use Wayland backend). But even I can play
| AAA games without problems
| PH95VuimJjqBqy wrote:
| I see these complains from time to time and I never understand
| them.
|
| I've literally been running nvidia on linux since the TNT2 days
| and have _never_ had this sort of issue. That's across many
| drivers and many cards over the many many years.
| LtWorf wrote:
| I've had kernel panics that disappeared when I started using
| the on board intel graphics instead of the nvidia.
|
| Your statement makes no sense. It's like a smoker claiming
| that since he didn't die of lung cancer, smoke is 100% safe.
| kkielhofner wrote:
| Describing kernel panics and general nightmare scenarios as
| the general course with Nvidia doesn't make sense either.
|
| Nvidia has 80% market share of the discrete GPU desktop
| market and at least 90% market share of cloud/datacenter.
|
| Nvidia GPUs are used almost exclusively for every cloud
| powered AI service and to train virtually every ML model in
| existence. Almost always on Linux.
|
| Do you really think any of this would be possible if what
| you are describing was anything approaching the typical
| experience starting at the /driver/ level?
|
| Nvidia would have never achieved their market dominance nor
| held on to it this long if the issues you've experienced
| impacted anything approaching a statistically significant
| number of users or applications.
|
| Nvidia gets a lot of hate on HN and elsewhere (much of it
| fair) but I will never understand the people who claim it
| doesn't work and get the job done (often very well).
| mr_toad wrote:
| People use flakey software all their time. As long as it
| mostly works most of the time most people put up with it.
| Examples: Windows in the 90's and 00's, or any AAA game
| on first release in the last 10 years.
| kkielhofner wrote:
| I have a friend at the Facebook AI Research lab and I
| assure you they would not tolerate any level of
| fundamental flakiness from their 8,000 GPU cluster.
| Talent, opportunity cost, and time to market in general
| is so crucial in AI no one has any time or patience for
| the "oddball Linux desktop" experiences people are
| describing here.
|
| Gaming users may tolerate some flakiness for their hobby
| but these AI companies dealing in the nine-figure range
| (minimum) absolutely do not.
| pixl97 wrote:
| My guess is when FB does run into such flakiness they
| email ____.____@nvidia.com as part of some support
| contract they have and go "Yo, we see this issue, figure
| it out and fix it".
|
| But I can promise you after reading things like the LKML
| for decades and a number of different Microsoft blogs,
| that everyone on this planet experiences flakiness issues
| at times and has to figure out how to adjust their
| workload to avoid it until the issue is discovered and
| fixed.
| kkielhofner wrote:
| He has described to me, in detail, some of the challenges
| they have had. I'm not saying it's exhaustive but I'm
| pretty sure if their experience with the fundamental
| software stack was what people here are claiming I would
| never hear the end of it.
|
| Actually, no. Obviously they have Nvidia support but in
| one especially obscure issue he was describing Meta took
| it as an internal challenge and put three teams on it in
| competition. Naturally his team won (of course) ;).
|
| Of course all software has flakiness - I'm not taking the
| ridiculous position that Nvidia is the first company in
| history to deliver perfect anything.
|
| What I am saying is these anecdotal reports (primarily
| from Linux desktop hobbyists/enthusiasts) of "It's
| broken, it doesn't work. Nvidia sucks because it locked
| up my patched kernel ABC with Wayland XYZ on my bleeding
| edge rolling release and blah blah blah" (or whatever)
| are extreme edge cases and in no way representative of
| 99% of the Nvidia customer base and use cases.
|
| Show me anything (I don't care what it is) and I'll find
| someone who has a horror story about it. Nvidia gets a
| lot of heat from the Linux desktop situation over the
| years and some people clearly hold an irrational hatred
| and grudge.
|
| Nvidia isn't perfect but it's very hard to argue they
| don't deliver generally working solutions - actually best
| of breed in their space as demonstrated by their
| overwhelmingly dominant market share I highlighted
| originally.
| iopq wrote:
| Nvidia is bad when combined with Wine/Firefox/Chrome on
| Wayland
|
| Which is literally only 1% of users anyway
| ant6n wrote:
| Well tnt2 should be pretty well supported by now ;-)
| PH95VuimJjqBqy wrote:
| lmao, touche :)
| temp0826 wrote:
| I understand it, but I also haven't had any trouble since I
| figured out the right procedure for me on fedora (which
| probably took some time, but it's been so long that I can't
| remember). Whenever I read people having issues it sounds
| like they are using a package installed via dnf for the
| driver/etc. I've always had issues with dkms and the like and
| just install the latest .run from nvidia's website whenever I
| have a kernel update (I made a one-line script to call it
| with the silent option and flags for signing for secure boot
| so I don't really think about it). No issues in a very long
| time even with the whackiness of prime/optimus offloading on
| my old laptop.
| PH95VuimJjqBqy wrote:
| actually, it's a good point because that's how I always
| install nvidia drivers as well. Never from the local
| package manager.
| bootsmann wrote:
| So you don't recommend going the rpm-fusion route?
| einpoklum wrote:
| I have been NVIDIA cards for compute capabilities only, both
| personally and at work, for nearly a decade. I've had dozens
| and dozens of different issues involving the hardware, the
| drivers, integration with the rest of the OS, version
| compatibilities, ensuring my desktop environment doesn't try
| to use the NVIDIA cards, etc. etc.
|
| Having said that - I (or rarely, other people) have almost
| always managed to work out those issues and get my systems to
| work. Not in all cases though.
| jjoonathan wrote:
| Same but linux experience is a steep and bumpy function of
| hardware.
|
| My guess: something like laptop GPU switching failed badly in
| the nvidia binary, earning it a reputation.
| HideousKojima wrote:
| That was my experience, Nvidia Optimus (which is what
| allows dynamic switching between the integrated and
| dedicated GPU in laptops) was completely broken (as in a
| black screen, not just crashes or other issues) for several
| years, and Nvidia didn't care to do anything about it.
| PH95VuimJjqBqy wrote:
| I don't run laptops except when work requires it and that
| tends to be windows so that may explain the difference in
| experience.
| lhl wrote:
| Yeah, Optimus was a huge PITA. I remember fighting with
| workarounds like bumblebee and prime for years. Also
| Nvidia dragged their feet on Wayland support for a few
| years too (and simultaneously was seemingly intent on
| sabotaging Nouveau).
| distract8901 wrote:
| I tried bumblebee again recently, and it works shockingly
| well now. I have a thinkpad T530 from 2013 with an
| NVS5400m.
|
| There is some strange issue with some games where they
| don't get full performance from the dGPU, but more than
| the iGPU. I have to use optirun to get full performance.
|
| It also has problems when the computer wakes from sleep.
| For whatever reason, hardware video decoding doesn't work
| after entering standby. Makes steam in home streaming
| crash on the client, but flipping to software decoding
| usually works fine.
|
| The important part is that battery life is almost as good
| with bumblebee as it is with the dGPU turned off. No more
| fucking with Prime or rebooting into BIOS to turn the GPU
| back on.
| chaostheory wrote:
| Yeah with my CUDA setup, it feels like I just ducktaped my
| deployment. I am very hesitant to make changes and it's not
| easy to replicate
| wubrr wrote:
| Yeah, nvidia linux support is meh, but still much better than
| amd.
| silisili wrote:
| In the closed source days of fglrx or whatever it's called
| I'd agree. Since they went open source, hard disagree. AMD
| graphics work in Linux about as well as Intel always has.
| phkahler wrote:
| >> Yeah, nvidia linux support is meh, but still much better
| than amd.
|
| Can not confirm. I used nvidia for years when it was the only
| option. Then used the nouveau driver on a well supported card
| because it worked well and eliminated hassle. Now I'm on AMD
| APU and it just works out of the box. YMMV of course. We do
| get reports of issues with AMD on specific driver versions,
| but I can't reproduce.
| bryanlarsen wrote:
| Not my experience. The open source AMD drivers are much more
| pleasant to deal with than the closed source Nvidia ones.
| acomjean wrote:
| As someone who was tasked with trying to get nvidia working
| on Ubuntu, it's a pretty terrible experience.
|
| I have a nvidia laptop with popos. That works well.
| Zambyte wrote:
| Is it better than AMD? I have had literally no graphics
| issues on my 6650 XT with swaywm using the built in kernel
| drivers.
| christkv wrote:
| I think the problems are pro drivers and the issues with
| ROCm being buggy not the open source graphics drivers.
| treprinum wrote:
| I never had an issue with nVidia drivers on Linux in the
| past 5 years, but recently bought a laptop with a 4090 and
| AMD CPU. Now I get random freezes, often right after I
| login into Cinnamon but can't really tell if it's the
| nVidia driver for 4090, AMDGPU driver for integrated RDNA,
| kernel 6.2 or Cinnamon issue. The laptop just hangs and
| stops responding to keyboard so I can't login to console
| and dmesg it.
| SoftTalker wrote:
| The main issue with Nvidia on Linux AIUI is that they
| don't release the source code for their drivers.
| treprinum wrote:
| That might be a philosophical problem that never
| prevented me from training models on Linux. The half-
| baked half-crashing AMD solutions just lead to wasting
| time I can spend on ML research instead.
| 65a wrote:
| I literally gave away my last laptop with a discrete
| nVidia card because it wasted so much of my time.
| aseipp wrote:
| This week I upgraded my kernel on a 2017 workstation to
| 6.5.5 and when I rebooted and looked at 'dmesg' there were
| no less than 7 kernel faults with stack traces in my
| 'dmesg' from amdgpu. Just from booting up. This is a no-
| graphical-desktop system using a Radeon Pro W5500, which is
| 3.5 years old (I just had the card and needed something to
| plug in for it to POST.)
|
| I have come to accept that graphics card drivers and
| hardware stability ultimately comes down to whether or not
| ghosts have decided to haunt you.
| HansHamster wrote:
| Guess I'm also doing something wrong. Never had any serious
| issues with either Nvidia or AMD on Linux (and only a few
| annoyances on RNDA2 shortly after release)...
| gymbeaux wrote:
| I often have issues booting to the installer or first boot
| after install with an NVidia GPU.
|
| Pop_OS, Fedora and OpenSUSE work out of the box. Those are all
| Wayland I believe. Debian/Ubuntu distros are a bad time. I
| think they're still X11. It's ironic because X11 is supposed to
| be the more stable window manager.
| Flameancer wrote:
| I think they moved to Wayland on 23.04 or 23.10. I just
| recently installed both to try and get a 7800xt working with
| PyTorch and the default was Wayland.
| anthk wrote:
| X11 is not a window manager.
| gymbeaux wrote:
| Xorg
| distract8901 wrote:
| My Arch system would occasionally boot to a black screen. When
| this happened, no amount of tinkering could get it back. I had
| to reinstall the whole OS.
|
| Turns out it was a conflict between nvidia drivers and my (10
| year old) Intel integrated GPU. But once I switched to an AMD
| card, everything works flawlessly.
|
| Ubuntu based systems barely worked at all. Incredibly unstable
| and would occasionally corrupt the output and barf colors and
| fragments of the desktop all over my screens.
|
| AMD on arch has been an absolute delight. It just. Works. It's
| more stable than nvidia on windows.
|
| For a lot of reasons-- but mainly Linux drivers-- I've totally
| sworn off nvidia cards. AMD just works better for me.
| aftbit wrote:
| As a counter-argument, I ran Arch Linux + nvidia GPUs + Intel
| CPUs between 2012 and 2020, and still run Arch + nvidia (now
| with AMD CPU) to this day. I won't say it has been bug free
| at all, but it generally works pretty well. If you find a
| problem in Arch that you cannot fix without reinstalling, you
| do not sufficiently understand the problem or Arch itself.
| "Installing" Arch is refreshingly manual and "simple"
| compared to the magic that is other Linux distros or the
| closed source OSes.
| distract8901 wrote:
| I'm sure that I _could_ have fixed it, but I gave up after
| spending multiple evenings on it. Have you ever spent hours
| debugging a system exclusively in text mode? It isn 't fun.
| Reinstalling the OS takes less than 30 minutes. It's a
| clear choice for me
| iopq wrote:
| I tried using an Nvidia card with OBS to record my screen
| and it kind of freezes in Wine. I switched from x11 to
| Wayland and now Wine shows horizontal lines (!) and
| performs like crap.
|
| Even my 4GB RX 570 from years ago gives a better experience
| doing this. You just install OBS from flathub, Wayland
| works, everything works without any setup or tinkering. You
| click record and you can record your gameplay footage.
| __rito__ wrote:
| I use OBS on Linux with NVIDIA card fairly regularly.
|
| It works flawlessly.
|
| Never used Wine + OBS, though.
| wildzzz wrote:
| I ran a laptop with the swappable dedicated Nvidia and
| integrated Intel GPU for a decade with no issues. Used to
| use something called Bumblebee to swap between them
| depending on workload, actually worked surprisingly well
| given the circumstances. Eventually I just dropped back to
| integrated only when I stopped doing anything intensive
| with the machine.
| MegaDeKay wrote:
| I run Arch as well and AMD is only "good". I would have a
| problem every now and then where my RX560 would lose its mind
| coming out of sleep and I'd have to reboot.
|
| But the other problem that _really_ bugs me is the "AMD
| reset bug" that you trip over with most AMD GPUs. This is
| when you pass through a second GPU through to another OS
| running under KVM, and is what lets you run Linux and (say)
| Windows simultaneously with full GPU hardware acceleration on
| the guest. The reset bug means the GPU will hang upon
| shutdown of the guest and only a reboot will let you recover
| the card. This is a silicon level bug that has existed for
| many years across many generations of cards and AMD can't be
| arsed to fix it. Projects like "vendor-reset" help for some
| cards, but gnif2 has basically given up (he mentioned he even
| personally raised the issue with Lisa Su). Even AMDs latest
| cards like the 7800 XT are affected. NVidia works flawlessly
| here.
| __rito__ wrote:
| I have used Pop OS and Ubuntu with NVIDIA card, and honestly,
| I never faced any serious problem.
|
| After every kernel upgrade, I just have to reinstall the
| nvidia drivers and the cuda toolkit.
|
| Everything works as before after I do that. I don't face any
| problems at all.
| nextaccountic wrote:
| > CUDA is the only reason I have an Nvidia card, but if more
| projects start migrating to a more agnostic environment, I'll
| be really grateful.
|
| What AMD really needs is to have 100% feature parity with CUDA
| without changing a single line of code. Maybe for this to
| happen it needs to add hardware features or something (I see
| people saying that CUDA as an API is very tailored to the
| capabilities of nvidia GPUs), I don't know.
|
| If AMD relies on people changing their code to make it
| portable, it already lost.
| mrweasel wrote:
| > I see people saying that CUDA as an API is very tailored to
| the capabilities of nvidia GPUs
|
| I'm wondering how true that is, because that could give
| NVidia issues in the future if they need to redesign their
| GPU should they hit some limit with the current designs.
| Dependence on certain instruction makes sense, but there's
| not technical preventing AMD from implementing those
| instructions, only legal mumbo jumbo.
| JonChesterfield wrote:
| The idea was supposed to be people convert cuda to hip, which
| is a pretty similar language, either by hand or by running a
| tool called 'hipify' that comes with rocm. You can then
| compile that unmodified for amdgpu or for nvptx.
|
| I think where that idea goes wrong is in order to compile it
| unmodified for nvptx, you need to use a toolchain which knows
| hip and nvptx, which the cuda toolchain does not. Clang can
| mostly compile cuda successfully but it's far less polished
| than the cuda toolchain. ROCm probably has the nvptx backend
| disabled, and even if it's built in, best case it'll work as
| well as clang upstream does.
|
| What I'm told does work is keeping all the source as cuda and
| using hipify as part of a build process when using amdgpu -
| something like `cat foo.cu | hipify | clang -x hip -` -
| though I can't personally vouch for that working.
|
| The original idea was people would write in opencl instead of
| cuda but that really didn't work out.
| pjmlp wrote:
| Both ideas are already lost before starting, Hip isn't
| polyglot as CUDA, and OpenCL is mostly stuck in C.
| mschuetz wrote:
| Not just feature parity, but proper UX. Things need to just
| work, without spending hours or days to make them work.
| weebull wrote:
| Blame Nvidia. They are the ones the got the industry hooked
| on a proprietry API.
| javchz wrote:
| I think that could work too. I wonder if they could do a
| translation layer, something like Apple with the M1 chips
| that translates JIT x86 to ARM.
| JonChesterfield wrote:
| That's a fun idea. Qemu parses a binary into something very
| like a compiler IR, optimises it a bit, then writes it out
| as a binary for the same or another target in JIT like
| fashion. So that sort of thing can be built. Apple's
| rosetta is functionally similar, I expect it does the same
| sort of thing under the hood. Valgrind is another from the
| same architecture.
|
| It would be a painful reverse engineering process - the
| cuda file format is sort of like elf, but with undocumented
| bonus constraints, and you'd have to reverse the
| instruction encoding to get sass, which isn't documented,
| or try to take it directly to ptx which is somewhat
| documented, and then convert that onward.
|
| It would be far more difficult than compiling cuda source
| directly. I'm not sure anyone would pay for a cuda->amdgpu
| conversion tool, and it's hard to imagine AMD making one as
| part of ROCm.
| withwarmup wrote:
| CUDA is the result of years of NVIDIA supporting the ecosystem,
| some people likes to complain because they bought hardware that
| was cheaper but can't use it for what they want to use it, when
| you buy NVIDIA, you aren't buying only the hardware, but the
| insane amount of work they have put into the ecosystem, the same
| goes for Intel, mkl and scikit-learn intelex aren't free to
| develop.
|
| AMD has the hardware but the support for HPC is non-existent
| outside of the joke that is bliss and AOCL.
|
| I really wish for more competitors to enter the market in HPC,
| but AMD has a shitload of work to do.
| aiunboxed wrote:
| Exactly, with NVIDIAs core focus on AI way before it was cool
| has lead to them being in this advantageous position. For AMD
| just being a price friendly competitor to Intel and Nvidia was
| the motto.
| arcanus wrote:
| > AMD has the hardware but the support for HPC is non-existent
| outside of the joke that is bliss and AOCL.
|
| You are probably two years behind the state of the art. The
| world's largest supercomputer, OLCF's Frontier, runs AMD CPUs
| and GPUs. It's emphatically using ROCm, not just BLIS and AOCL.
| See for example:
| https://docs.olcf.ornl.gov/systems/frontier_user_guide.html
|
| That's hardly non-existent support for HPC.
| 65a wrote:
| Agreed...the main gap is support on consumer and workstation
| cards, which is where nVidia made headway, but that is
| starting erode super recently. ROCm works pretty well for me,
| I have had a lot more problems with specific packagers than
| the ROCm layer.
| runiq wrote:
| Yeah, that's a pretty shortsighted take of things. Do you
| really believe that Nvidia hasn't taken steps do make sure
| their moat is as wide as possible?
| Blammar wrote:
| The thing about owning the CUDA spec is that Nvidia can add
| new features quickly without having to argue with other
| hardware vendors. I find that a positive thing overall.
|
| Also, I choose to pay the ~$120 Windows tax once (per box),
| everything works very well, and I don't have the driver
| issues that some fraction of other users seem to have with
| Linux and Nvidia cards. Seems like a good use of my time.
| anon291 wrote:
| Literally never had an issue with Nvidia and Linux in
| decades. Despite this, my windows installs have all sorts
| of issues.. as always
| fluxem wrote:
| I call it the 90% problem. If AMD works for 90% of my projects, I
| would still buy NVIDIA, which works for 100%, even though I'm
| paying a premium
| hot_gril wrote:
| I'm lazy, so it's 99% for me. I don't even mess with AMD CPUs;
| I know they're not _exactly_ the same instruction set as Intel,
| and more importantly they work with a different (and less
| mainstream) set of mobos, so I don 't want em. If AMD manages
| to pull more customers their way, that's great, it just means
| lower Intel premium for me.
| hot_gril wrote:
| Forgot to also mention iGPU and other on-chip accelerators
| being different and Intel usually having the edge there.
| Flameancer wrote:
| What mainstream board company is intel only? Maybe a decade
| ago on AM3(+) but on AM5/AM5 I haven't seen a main board
| partner not offer the same board SKU that works with Intel
| and AMD.
| 65a wrote:
| As an owner of some Sapphire Rapids parts, let me just direct
| you to:
| https://edc.intel.com/content/www/us/en/design/products-
| and-...
| hot_gril wrote:
| To see errata tracked _by Intel_ is a good sign.
| anon291 wrote:
| I have no idea what you're talking about. Amd and Intel match
| on the isa in any case you'd see typically. Moreover, Intel
| is currently using AMDs instruction set. X86_64 was designed
| my amd and used to be called AMD64
| bornfreddy wrote:
| That's an interesting take. AMD mobos are no "less
| mainstream" than Intel ones are... When you choose a CPU you
| are also choosing a compatible mobo chipset. The companies
| that make motherboards are mostly the same, so there should
| be no big difference between those.
|
| Also, while the CPU instruction sets are not exactly equal,
| the same is true for Intel processors of different
| generations too. And it doesn't matter one bit... Unless
| there is a bug in CPU you will never notice the difference,
| because it is taken care of at the compiler / kernel level.
|
| Intel does have some advantages (and disadvantages too) over
| AMD, just not those.
| [deleted]
| Zetobal wrote:
| They are just too late even if they catch up. Until they make a
| leap like they did with ryzen nothing will happen.
| Havoc wrote:
| >They are just too late even if they catch up.
|
| Late certainly, too late I don't think so.
|
| If you can field a competitively priced consumer card that can
| run llama fast then you're already halfway there because then
| the ecosystem takes off. Especially since nvidia is being
| really stingy with their vram amounts.
|
| H100 & datacenter is a separate battle certainly, but on
| mindshare I think some deft moves from AMD will get them there
| quite fast once they pull their finger out their A and actually
| try sorting out the driver stack.
| dylan604 wrote:
| >If you can field a competitively priced consumer card
|
| if this unicorn were to show up, what's to say that all the
| non-consumers won't just scarf up these equally performant
| yet lower priced cards causing the supply-demand situation
| we're in now? the only difference would be a sudden supply of
| the expensive Nvidia cards that nobody wants because of their
| price.
| AnthonyMouse wrote:
| The thing that causes it to be competitively priced is
| having enough production capacity to prevent that from
| happening.
|
| One way to do that may be to produce a card on an older
| process node (or the existing one when a new one comes out)
| that has a lot of VRAM. There is less demand for the older
| node so they can produce more of them and thereby sell them
| for a lower price without running out.
| Havoc wrote:
| >if this unicorn were to show up
|
| A unicorn like that showed up a couple hours ago. Someone
| posted a guide for getting llama to run on a 7900xtx
|
| https://old.reddit.com/r/LocalLLaMA/comments/170tghx/guide_
| i...
|
| It's still slow and janky but this really isn't that far
| away.
|
| I don't buy that AMD can't make this happen if they
| actually tried.
|
| Go on fiverr, get them to compile a list of top 100 people
| in the DIY LLM space, send them all free 7900XTXs. Doesn't
| matter if half of it is wrong, just send it. Next take 1.2m
| USD, post a dozen 100k bounties against llama.cpp that are
| AMD specific - support & optimise the gear. Rinse and
| repeat with every other hobbyist LLM/stable diffusion
| project. A lot of these are zero profit open source /
| passion / hobby projects. If 6 figure bounties show up
| it'll absolute raise pulses. Next do all the big youtubers
| in the space - carefully on that one so that it doesn't
| come across as an attempted pay-off...but you want them to
| know that you want this space to grow and are willing to
| put your money where your mouth is.
|
| That'll cost AMD what 2m 3m? To move the needle on a multi
| billion market? That's the cheapest marketing you've ever
| seen.
|
| As I said the datacenter & enterprise market is another
| beast entirely full of moats and strategy, but I don't see
| why a suitably motivated senior AMD exec can't tackle the
| enthusiast market single handedly with a couple of emails,
| a cheque book and a tshirt that has the nike slogan on it.
|
| >what's to say that all the non-consumers won't just scarf
| up these equally performant yet lower priced cards
|
| It doesn't matter. They're in the business of selling
| cards. To consumers, to datacenters, to your grandmother.
| From a profit driven capitalist company the details don't
| matter as long as there is traction & volume. The above -
| opening up even the possibility of a new market - is gold
| in that perspective. And from a consumer perspective
| anything that breaks the nvidia cuda monopoly is a win.
| lhl wrote:
| llama.cpp, ExLlama, and MLC LLM have all had ROCm
| inferencing for months (here are a bunch of setup
| instructions I've written up, for Linux and Windows:
| https://llm-tracker.info/books/howto-guides/page/amd-gpus
| ) - but I don't think that's the problem (and wouldn't
| drive lots of volume or having downstream impact in any
| case).
|
| The bigger problem is on the training/research support.
| Eg, here's no official support for AMD GPUs for
| bitsandbytes, and no support at all for
| FlashAttention/FA2 (nothing that 100K in hardware/grants
| to Dettmers or Dao's labs wouldn't fix I suspect).
|
| The real elephant though is that AMD still having the
| disconnect that lack of support for consumer cards and
| home/academic devs in general has been disastrous (while
| Nvidia supports CUDA on basically every single GPU
| they've made since 2010) - just last week there was this
| mindblowing thread where it turns out an AMD employee is
| paying out of pocket for AMD GPUs to support build/CI for
| drivers on Debian. I mean, WTF, that's stupidity that's
| beyond embarrassing and gets into negligence terriroty
| IMO: https://news.ycombinator.com/item?id=37665784
| bornfreddy wrote:
| Wow, that is really awkward... AMD should be donating the
| cards and even paying extra for the privilege - this is
| an important step for getting satisfied consumers. I hope
| they notice and rectify this situation so that Debian
| (and with it all downstream distros, like Ubuntu) can
| provide better support for their cards. I mean, that's a
| no-brainer...
| dylan604 wrote:
| >an AMD employee is paying out of pocket for AMD GPUs
|
| I hope he's at least getting an employee discount! I
| guess AMD is not a fan of the 20% concept either
| 65a wrote:
| I was running llama on a w7900 a month ago, with 48gb of
| VRAM and excellent performance. ROCm support got a lot
| better really recently.
| omneity wrote:
| I was able to use ROCm recently with Pytorch and after pulling
| some hair it worked quite well. The Radeon GPU I had on hand was
| a bit old and underpowered (RDNA2) and it only supported matmul
| on fp64, but for the job I needed done I saw a 200x increase in
| it/s over CPU despite the need to cast everywhere, and that made
| me super happy.
|
| Best of all is that I simply set the device to
| `torch.device('cuda')` rather than openCL, which does wonders for
| compatibility and to keep code simple.
|
| Protip: Use the official ROCM Pytorch base docker image [0]. The
| AMD setup is so finicky and dependent on specific versions of
| sdk/drivers/libraries and it will be much harder to make work if
| you try to install them separately.
|
| [0]:
| https://rocm.docs.amd.com/en/latest/how_to/pytorch_install/p...
| incognition wrote:
| Fp64??
| latchkey wrote:
| https://en.wikipedia.org/wiki/Double-precision_floating-
| poin...
|
| NVIDIA fp32 (H100) has 2x more TFLOPS than AMD's fp32 (MI250)
| and AI doesn't need fp64 precision.
| fransje26 wrote:
| Hardware limitation.
| wyldfire wrote:
| > Best of all is that I simply set the device to
| `torch.device('cuda')` rather than openCL, which does wonders
| for compatibility
|
| Man oh man where did we go wrong that cuda is the more
| compatible option over OpenCL?
| NavinF wrote:
| This has always been the case. OpenCL is a shit show
| KeplerBoy wrote:
| It must be a misnomer on PyTorch's side. Clearly it's neither
| CUDA nor OpenCL.
|
| AMD should just get it's shit together. This is ridiculous.
| Not the name, but the fact that you can only do FP64 on a
| GPU. Everybody is moving to FP16 and AMD is stuck on doubles?
| omneity wrote:
| I believe the fp64 limitation came from the laptop-grade
| GPU I had rather than inherent to AMD or ROCm.
|
| The API level I could target was at least two or three
| versions behind the latest they have to offer.
| KeplerBoy wrote:
| Might very well be true. I don't blame anyone for not
| diving deeper into figuring out why this stuff doesn't
| work.
|
| But this is one of the great strengths of CUDA: I can
| develop a kernel on my workstation, my boss can demo it
| on his laptop and we can deploy it on Jetsons or the
| multi-gpu cluster with minimal changes and i can be sure
| that everything runs everywhere.
| iopq wrote:
| Sorry, still trying to install some dependencies for DNN
| and CUDA, not sure why it says my Clang version is too
| new (!)
| brutus1213 wrote:
| There is indeed something excellent about CUDA from a
| user perspective that is hard to beat. I do high-level
| DNN and it is not clear to me what it is or why that is.
| Anytime I have worked on optimizing to mobile hardware
| (not Jetson, but actual phones or accelerators), it is
| just a world of hurt and incompatibilities. This notion
| that operators or subgraphs can be accelerated by lower
| level closed blobs .. I wonder if that is part of the
| issue. But then why doesn't OpenCL not just work? I
| thought it gave a CUDA kernel like general purpose
| abstraction.
|
| I just don't understand the details enough to understand
| why things are problematic without CUDA :(
| JonChesterfield wrote:
| FP64 is what HPC is built on. F32 works on the cards too
| (same rate or faster). I don't know the status of F16 or
| F8.
|
| Some architectures provide fast F16->F32 and F32->F16
| conversion instructions so you can DIY the memory bandwidth
| saving - that always seemed reasonable to me, but I don't
| know if the AMD hardware people are/will go down that path.
| londons_explore wrote:
| More importantly, if you specify FP16, yet the hardware
| only supports FP32, then the library should emit a
| warning but _work anyway_ , doing transparent casts
| behind your back as necessary.
| KeplerBoy wrote:
| Sure but Radeon cards are not HPC accelerators. A modest
| 7800XT for example, which would be a great card for SD,
| has 76 TFlops@FP16, 37TF@FP32 and 1.16TF@FP64.
|
| Keeping all those FPUs busy is another problem and not
| easy, but in cases where it can be done FP32 is clearly
| desirable.
| RockRobotRock wrote:
| Have you gotten it to work with Whisper by any chance?
| kkielhofner wrote:
| Whisper is actually a great example of why Nvidia has such a
| stronghold on ML/AI and why it's so difficult to compete.
|
| There's getting something to "work", which is often enough of
| a challenge with ROCm. Then there's getting it to work well
| (next challenge).
|
| Then there's getting it to work as well as Nvidia/CUDA.
|
| With Whisper, as one example, you should be running it with
| ctranslate2[0]. Of all the platforms on their supported list
| you won't find ROCm.
|
| When you really start to look around you'll find that ROCm is
| (at best) still very much in the "get it to work (sometimes)"
| stage. In most cases it's still a long way away from getting
| it to work well, and even further away from making it
| actually competitive with Nvidia for serious use cases and
| applications.
|
| People get excited about the progress ROCm has made getting
| basic things to work with PyTorch and this is good - progress
| is progress. But saving 20% on the hardware when the
| equivalent Nvidia product is often somewhere between 5-10x as
| performant (at a fraction of the development time) because of
| vastly superior software support you realize pretty quickly
| Nvidia is actually a bargain compared to AMD.
|
| I'm desperately rooting for Nvidia to have some actual
| competition but after six years of ROCm and my own repeated
| failed attempts to have it make any sense overall I'm only
| more and more skeptical that real competition in the space
| will come from AMD.
|
| [0] - https://github.com/OpenNMT/CTranslate2
| errnoh wrote:
| While I agree that it's much more effort to get things
| working on AMD cards than it is with Nvidia, I was a bit
| surprised to see this comment mention Whisper being an
| example of "5-10x as performant".
|
| https://www.tomshardware.com/news/whisper-audio-
| transcriptio... is a good example of Nvidia having no
| excuses being double the price when it comes to Whisper
| inference, with 7900XTX being directly comparable with
| 4080, albeit with higher power draw. To be fair it's not
| using ROCm but Direct3D 11, but for performance/price
| arguments sake that detail is not relevant.
|
| EDIT: Also using CTranslate2 as an example is not great as
| it's actually a good showcase why ROCm is so far behind
| CUDA: It's all about adapting the tech and getting the
| popular libraries to support it. Things usually get
| implemented in CUDA first and then would need additional
| effort to add ROCm support that projects with low amount of
| (possibly hobbyist) maintainers might not have available.
| There's even an issue in CTranslate2 where they clearly
| state no-one is working to get ROCm supported in the
| library. ( https://github.com/OpenNMT/CTranslate2/issues/10
| 72#issuecomm... )
| kkielhofner wrote:
| > While I agree that it's much more effort to get things
| working on AMD cards than it is with Nvidia, I was a bit
| surprised to see this comment mention Whisper being an
| example of "5-10x as performant".
|
| It easily is. See the benchmarks[0] from faster-whisper
| which uses Ctranslate2. That's 5x faster than OpenAI
| reference code on a Tesla V100. Needless to say something
| like a 4080 easily multiplies that.
|
| > https://www.tomshardware.com/news/whisper-audio-
| transcriptio... is a good example of Nvidia having no
| excuses being double the price when it comes to Whisper
| inference, with 7900XTX being directly comparable with
| 4080, albeit with higher power draw. To be fair it's not
| using ROCm but Direct3D 11, but for performance/price
| arguments sake that detail is not relevant.
|
| With all due respect to the author of the article this is
| "my first entry into ML" territory. They talk about a
| 5-10 second delay, my project can do sub 1 second
| times[1] even with ancient GPUs thanks to Ctranslate2. I
| don't have an RTX 4080 but if you look at the performance
| stats for the closest thing (RTX 4090) the performance
| numbers are positively bonkers - completely untouchable
| for anything ROCm based. Same goes for the other projects
| I linked, lmdeploy does over 100 tokens/s in a single
| session with LLama2 13b on my RTX 4090 and almost 600
| tokens/s across eight simultaneous sessions.
|
| > EDIT: Also using CTranslate2 as an example is not great
| as it's actually a good showcase why ROCm is so far
| behind CUDA: It's all about adapting the tech and getting
| the popular libraries to support it. Things usually get
| implemented in CUDA first and then would need additional
| effort to add ROCm support that projects with low amount
| of (possibly hobbyist) maintainers might not have
| available. There's even an issue in CTranslate2 where
| they clearly state no-one is working to get ROCm
| supported in the library. ( https://github.com/OpenNMT/CT
| ranslate2/issues/1072#issuecomm... )
|
| I don't understand what you're saying here. It (along
| with the other projects I linked here[2]) are fantastic
| examples of just how far behind the ROCm ecosystem is.
| ROCm isn't even on the radar for most of them as your
| linked issue highlights.
|
| Things always get implemented in CUDA first (ten years in
| this space and I've never seen ROCm first) and ROCm users
| either wait months (minimum) for sub-par performance or
| never get it at all.
|
| [0] - https://github.com/guillaumekln/faster-
| whisper#benchmark
|
| [1] - https://heywillow.io/components/willow-inference-
| server/#ben...
|
| [2] -
| https://news.ycombinator.com/item?id=37793635#37798902
| pedrovhb wrote:
| I've had luck with an RX5700XT and whisper.cpp built with
| clblast. Works like a charm, not entirely a scarring
| experience getting it to work (easier than most other stuff
| which was surprising to me).
|
| One arcane detail is that whereas for PyTorch I have to set
| the env var HSA_OVERRIDE_GFX_VERSION to 10.3.0, getting it to
| run with whisper.cpp and llama.cpp requires setting it to
| 10.1.0. Good luck and may it cost you less hair than it did
| me.
| mikepurvis wrote:
| Sigh. It's great that these container images exist to give
| people an easy on-ramp, but they definitely don't work for
| every use case (especially once you're in embedded where space
| matters and you might not be online to pull multi-gb updates
| from some registry).
|
| So it's important that vendors don't feel let off the hook to
| provide sane packaging just because there's an option to use a
| kitchen-sink container image they rebuild every day from
| source.
| fwsgonzo wrote:
| I feel the same way, especially about build systems. OpenSSL
| and v8 are among a large list of things that have horrid
| build systems. Only way to build them sanely is to use some
| randos CMake fork, then it Just Works. Literally a two-liner
| in your build system to add them to your project with a sane
| CMake script.
| mikepurvis wrote:
| I was part of a Nix migration over the past two years, and
| literally one of the first things we checked is that there
| was already a community-maintained tensorflow+gpu package
| in nixpkgs because without that the whole thing would have
| been a complete non-starter, and we sure as heck didn't
| have the resources or know-how to figure it out for
| ourselves as a small DevOps team just trying to do basic
| packaging.
| amelius wrote:
| > So it's important that vendors don't feel let off the hook
| to provide sane packaging just because there's an option to
| use a kitchen-sink container image they rebuild every day.
|
| Sadly if e.g. 95% of their users can use the container, then
| it could make economical sense to do it that way.
| ngcc_hk wrote:
| Better to come if the tide shift so we can have compatible
| layer. The key is the tide. Obviously would n try to sue ...
| it would be a sign that finally we have real competition. Gar
| is where innovation do.
|
| X86 cannot do 64 bit let us do this and that so the market
| can use only our cpu. Repeat with me x86-64 is impossible.
|
| Not sure Apple is in this otherwise the real great
| competition come.
| xahrepap wrote:
| I know it's still different than what you're looking for, so
| you probably already know this, but many projects like this
| have the Dockerfile on github which shows exactly how they
| set up the image. For example:
|
| https://github.com/RadeonOpenCompute/ROCm-
| docker/blob/master...
|
| They also have some for Fedora. Looks like for this you need
| to install their repo: curl -sL
| https://repo.radeon.com/rocm/rocm.gpg.key | apt-key add - \
| && printf "deb [arch=amd64]
| https://repo.radeon.com/rocm/apt/$ROCM_VERSION/ jammy main" |
| tee /etc/apt/sources.list.d/rocm.list \ && printf
| "deb [arch=amd64]
| https://repo.radeon.com/amdgpu/$AMDGPU_VERSION/ubuntu jammy
| main" | tee /etc/apt/sources.list.d/amdgpu.list \
|
| then install Python, a couple other dependencies (build-
| essential, etc) and then the package in question: rocm-dev
|
| So they are doing the packaging. There might even be
| documentation elsewhere for that type of setup.
| mikepurvis wrote:
| Oh yeah, I mean... having the source for the container
| build is kind of table stakes at this point. No one would
| accept a 10gb mystery meat blob as the basis of their
| production system. It's bad enough that we still accept
| binary-only drivers and proprietary libraries like
| TensorRT.
|
| I think my issue is more just with the _mindset_ that it 's
| okay to have one narrow slice of supported versions of
| everything that are "known to work together" and those are
| what's in the container and anything outside of those and
| you're immediately pooched.
|
| This is not hypothetical btw, I've run into real problems
| around it with libraries like gproto, where tensorflow's
| bazel build pulls in an exact version that's different from
| the default one in nixpkgs, and now you get symbol
| conflicts when something tries to link to the tensorflow
| c++ API while linking to another component already using
| the default gproto. I know these problems are solveable
| with symbol visibility control and whatever, but that stuff
| is far from universal and hard to get right, especially if
| the person setting up the build rules for the library
| doesn't themselves use it in that type of heterogeneous
| environment (like, everyone at Google just links the same
| global proto version from the monorepo so it doesn't
| matter).
| weebull wrote:
| > I think my issue is more just with the _mindset_ that
| it 's okay to have one narrow slice of supported versions
| of everything that are "known to work together" and those
| are what's in the container and anything outside of those
| and you're immediately pooched.
|
| I hear you. I think docker has been a plague on the
| quality of software. It's allowed "works for me" to
| become the norm, except it's now pronounced "works on the
| official docker image". It seems to be especially true in
| the ML sphere where compiling things is so temperamental
| that there's a lot of binaries being distributed.
|
| Docker was meant to be a deployment platform, not a
| distribution medium.
| hotstickyballs wrote:
| If anything, the situation with tensor rt shows that
| companies are absolutely willing to accept a multi gig
| meat blob
| pixl97 wrote:
| > No one would accept a 10gb mystery meat blob as the
| basis of their production system
|
| Heh, if only. When working with F100's I've seen many
| terrible, terrible things.
| JonChesterfield wrote:
| > No one would accept a 10gb mystery meat blob as the
| basis of their production system
|
| Well, except for cuda. Which is a massive pile of
| proprietary software that people are using in production
| anyway.
| mgaunard wrote:
| I don't know what world you live in, but this is a
| problem for any software development.
|
| You need to ensure that there is only one version of any
| library used globally throughout the code and that the
| set of versions is compatible with each other, and
| preferably you also want everything to be built against
| the same toolchain with the same flags.
|
| That usually means onboarding third-party libraries into
| your own build system.
| iopq wrote:
| In NixOS, I can install multiple versions of libraries
|
| Or rather, I install no versions of libraries because
| NixOS will put them all in the store in different
| folders, and will compile the executable to use the
| correct path (or patch the elf when needed)
|
| it has an issue with pip because it's allergic to just
| randomly executing things as part of package management,
| but pip in general is wtf
| anuraaga wrote:
| I'd say with semver becoming far better known, this is
| not a problem for "any" software development. The
| developer gets the choice to pick libraries that are
| stable, often also influencing language choice. Mistakes
| happen, Guava broke the Java ecosystem for about two
| years, but it's never something that is accepted as just
| a fact of software development, it is a mistake.
|
| Wanting to hold Python+C ecosystem more accountable is
| fair I think, at least from my own experience around half
| a year ago, Anaconda doesn't work and you need a
| Dockerfile for any sort of reproducibility, which can
| have issues since GPU with docker isn't that easy. And
| this means developers from the vendors working with
| Anaconda, for example, on solving the issue rather than
| just hoping for contributors to do it. If AMD were to
| make easy, reproducible builds without root or VM a
| reality, that would be reason enough to try their
| hardware. If not, hopefully Nvidia does and then there
| really would be no way across the moat for me at least.
| anthk wrote:
| This would be the work for Guix. Much better than docker,
| and exportable to a lot of formats. Or just build a vm
| from the CLI, an ad-hoc environment, a Docker export or a
| direct rootfs to deploy and run in any compatible
| machine.
| mgaunard wrote:
| Semver is a joke and doesn't work. Languages like C and
| C++ can easily have problems if you link code built with
| different versions together (even if you aim for them to
| be compatible, or even if they are indeed the same source
| version but with subtly different flags), and there are
| no good solutions for this, except not doing it.
|
| A docker container is not really any different from any
| other process; the main difference is that it runs in a
| chroot pretty much.
| anthk wrote:
| That's trivial with Guix.
| josephg wrote:
| It's not a universal problem. A lot of modern languages
| allow multiple versions of a library to be pulled in to
| the same code base, through different dependency paths.
| (Eg nodejs, rust). It's not a perfect answer by any
| means, but it's nice not needing to worry about some
| package pulling in an inconvenient version of one of its
| dependencies.
|
| Also, just to name it, it's ridiculous that a specific
| graphics card manages to restrict the version of gproto
| that you're using. You don't have this problem with
| nvidia drivers, since cuda stuff is much less fiddly. AMD
| needs to pull a finger out and fix the bugs in their
| stack that make it so fragile like this.
| [deleted]
| mathisfun123 wrote:
| > especially once you're in embedded
|
| is this a real problem? exactly which embedded platform has a
| device that ROCm supports?
| mikepurvis wrote:
| Robotic perception is the one relevant to me. You want to
| do object recognition on an industrial x86 or Jetson-type
| machine, without having to use Ubuntu or whatever the one
| "blessed" underlay system is (either natively or implicitly
| because you pulled a container based on it).
| mathisfun123 wrote:
| >industrial x86 or Jetson-type machine
|
| that's not embedded dev. if you
|
| 1. use underpowered devices to perform sophisticated
| tasks
|
| 2. using code/tools that operate at extremely high levels
| of "abstraction"
|
| don't be surprised when all the inherent complexity is
| tamed using just more layers of "abstraction". if that
| becomes a problem for your cost/power/space budget then
| reconsider choice 1 or choice 2.
| mikepurvis wrote:
| Not sure this is worth an argument over semantics, but
| modern "embedded" development is a lot bigger than just
| microcontrollers and wearables. IMO as soon as you're
| deploying a computer into any kind of "appliance", or
| you're offline for periods of time, or you're running on
| batteries or your primary network connection is
| wireless... then yeah, you're starting to hit the
| requirements associated with embedded and need to seek
| established solutions for them, including using distros
| which account for those requirements.
| serf wrote:
| fwiw CompTIA classifies an embedded engineer/developer as
| " those who develop an optimized code for specific
| hardware platforms."
| mathisfun123 wrote:
| > IMO as soon as you're deploying a computer into any
| kind of "appliance", or you're offline for periods of
| time, or you're running on batteries or your primary
| network connection is wireless
|
| yes and in those instances you do not reach for
| pytorch/tensorflow on top of ubuntu on top of x86 with a
| discrete gpu and 32gb of ram. instead you reach for C and
| micro or some arm soc that supports baremetal or at most
| rtos. that's embedded dev.
|
| so i'll repeat myself: if you want to run extremely high-
| level code then don't be "surprised pikachu" when your
| underpowered platform, that you chose due to concrete,
| tight budgets doesn't work out.
| nightski wrote:
| Clearly you've never used a Nvidia Jetson and have no
| idea what it is. You don't need a discrete GPU, it has a
| quite sophisticated GPU in the SoC. It's Nvidia's
| embedded platform for ML/AI.
| Const-me wrote:
| The hardware can be fast, actually. Here's an example of
| relatively modern industrial x86:
| https://www.onlogic.com/ml100g-41/ That thing is probably
| faster than half of currently sold laptops.
|
| However, containers or Ubuntu Linux don't perform great
| in that environment. Ubuntu is for desktops, containers
| are for cloud data centers. An offline stand-alone device
| is different. BTW, end users don't typically aware that
| thing is a computer at all.
|
| Personally, I usually pick Alpine or Debian Linux for
| similar use cases, bare metal i.e. without any
| containers.
| cannonpalms wrote:
| > Ubuntu is for desktops
|
| Tell that to their (much larger, more profitable, and
| better-funded) server org. This is far from true.
| iopq wrote:
| It also works much better as a server. Snaps work really
| well for things like certbot
|
| On Desktop you have to worry about things like... UIs,
| sound, Wine, etc.
| ngcc_hk wrote:
| That is the moat they tried to cross. Imagine you have a
| PyTorch app and run on iOS, arm based, amd based and
| intel ... cloud, or embedded. just imagine. You scale and
| embed as your business case, not as any one firm current
| strategy is.
|
| Or at least you have some case as heaven never come. Or
| come just we do not aware now like internet. Can you need
| to use ibm to rub sna to provide a token ring based
| network. In 1980 ...
|
| Imagine and let us or they competite ...
| rcxdude wrote:
| Not that I want to encourage gatekeeping in the first
| place, but you'll have more success if you have a clue
| what the other person is talking about in the first place
| (and some idea of what embedded looks like outside of
| tiny micros, and how the concerns about abstractions
| extend beyond matters of how much computational power is
| available).
| IronWolve wrote:
| Yup, thank the hobbyists. Pytorch is allowing other hardware.
| Stable diffusion working on m chips, intel arc, and Amd.
|
| Now what I'd like to see is real benchmarks for compute power.
| Might even get a few startups to compete in this new area.
| mattnewton wrote:
| Re: startups, Geohotz raised a few million for this already.
| https://tinygrad.org/
| nomel wrote:
| Obligatory Lex Fridman podcast, where he discusses it:
| https://youtu.be/dNrTrx42DGQ?t=2408
| IntelMiner wrote:
| Didn't he do what he always does. Rake in a ton of money,
| fart around and then cash out exclaiming it's everyone else's
| fault?
|
| The way he stole Fail0verflow's work with the PS3 security
| leak after failing to find a hypervisor exploit for months
| absolutely soured any respect I had for him at the time
| ShamelessC wrote:
| > The way he stole Fail0verflow's work with the PS3
| security leak after failing to find a hypervisor exploit
| for months absolutely soured any respect I had for him at
| the time
|
| That sounds interesting. I tried googling about it but
| can't really find much other than that failoverflow found a
| key and didn't release it, and then geohot released his own
| subsequently. I'd love to hear more about how directly he
| "stole" the work from the Fail0verflow team.
|
| edit: Reading some sibling comments here, it seems you are
| either mistaken and/or were exaggerating your claim about
| the "theft" here. As far as I can tell, he simply took
| their findings and made his own version of an exploit that
| they had detailed publicly. That may be in poor taste in
| this particular community but it's certainly not theft. I
| do agree that his behavior there was lacking in decency,
| but not to the degree implied here where I was thinking he
| _literally_ stole their exploit by hacking them, or
| something similar to that.
| cyrux004 wrote:
| People here generally try to bash people who are much
| smarter than them, throwing shade at their background.
| They will say that he abandoned his first company, gave
| up on tiny grad but both of them are very much alive
| projects
| adastra22 wrote:
| Wow, TIL
| throwitawayfam wrote:
| Yep, did exactly that. IMO he threw a fit, even though AMD
| was working with him squashing bugs. https://github.com/Rad
| eonOpenCompute/ROCm/issues/2198#issuec...
| [deleted]
| nomel wrote:
| To be fair, kernel crashes from running an AMD provided
| demo loop isn't something he should have to work with
| them on. That's borderline incompetence. His perspective
| was around integration into his product, where every AMD
| bug is a bug in his product. They deserve criticism, and
| responded accordingly (actual resources to get their shit
| together). It's not like GPU accelerated ML is some new
| thing.
| JonChesterfield wrote:
| That's a tough issue to read through, thanks for the
| link. 'Your demo code on a system setup exactly as you
| describe dereferences null in the kernel and falls over'.
| Fuzz testing + a vaguely reasonable kernel debugging
| workflow should make things like that much harder to
| find.
| aeyes wrote:
| He's back on it after getting AMD's CEO to commit
| resources to this:
|
| https://twitter.com/realGeorgeHotz/status/166980346408248
| 934...
|
| https://twitter.com/LisaSu/status/1669848494637735936
| kinematikk wrote:
| Do you have a source on the stealing part? A quick Google
| search didn't result in anything
| IntelMiner wrote:
| Marcan (of Asahi Linux fame) has talked about it _many_
| times before. But an abridged version
|
| Fail0verflow demoed how they were able to derive the
| private signing keys for the Sony Playstation 3 console
| at I believe CCC
|
| Geohot after watching the livestream raced into action to
| demo a "hello world!" jailbreak application and
| absolutely stole their thunder without giving any credit
| ryanjshaw wrote:
| If they demod something then they released it publically
| and it was fair game?
|
| In any case he absolutely did credit them, it's easily
| verifiable: https://web.archive.org/web/20110104040706/ht
| tp://geohot.com...
|
| Sony sued them both, afterall!
| aftbit wrote:
| This apparently worked pretty well for him, as I still
| remember him primarily as "that guy who hacked PS3". Some
| people let someone else do the hard technical core, then
| do all the other easy but boring stuff and claim 100%
| credit.
| 22c wrote:
| I remember geohot as being one of the people who
| developed a fairly successful jailbreak for iPhone. I
| understand that iPhone jailbreaking is often standing on
| the shoulders of predecessors, but I believe he does
| deserve significant credit for at least one popular
| iPhone jailbreak.
| mandevil wrote:
| It isn't the hobbyists who are making sure that PyTorch and
| other frameworks runs well on these chips, but teams of
| engineers who work for NVIDIA, AMD, Intel, etc. who are doing
| this as their primary assigned jobs, in exchange for money from
| their employer, who are paying those salaries because they want
| to sell chips into the enormous demand for running PyTorch
| faster.
|
| Hobbyist and open-source are definitely not synonyms.
| Eisenstein wrote:
| People don't usually get employed to make things with no
| demand, and people who work for companies with a budget line
| don't really care how much the nVidia tax is. You can thank
| hobbyists for creating a lot of demand for compatability with
| other cards.
| roenxi wrote:
| There is huge demand for AMD cards that can efficiently
| multiply matrices together. The issue is that while there
| are currently isolated cases where people can make them do
| that, it doesn't seem to be possible at the scale that it
| needs to happen at.
|
| AMD are being dragged along by the market. Willingly, they
| aren't fighting it, but their focus has been on other
| areas.
| [deleted]
| viewtransform wrote:
| <but their focus has been on other areas.>
|
| They've shifted a large pool of experienced engineers
| from legacy software projects to AI and moved the team
| under a veteran Xilinx AI director. Fingers crossed we
| should see significant changes in 2024.
| Flameancer wrote:
| As a new owner of a 7800XT I'm excited.
| iopq wrote:
| Look at the earnings call:
|
| https://www.fool.com/earnings/call-
| transcripts/2023/08/01/ad...
|
| it's literally ALL AI, server, enterprise talk - AI is
| mentioned 64 times
|
| AMD literally doesn't care about gaming anymore, server
| is their primary focus
| johngossman wrote:
| I can only point you to cloud financial results and the
| huge cost of the AI race. Note also the story recently
| about OpenAI looking at building their own chips. Companies
| absolutely care immensely about the cost of GPUs. It's
| billions of dollars.
| kiratp wrote:
| There are so many billions of dollar being spent on this
| hardware that everyone other than Nvidia is doing
| everything they can to make competition happen.
|
| Eg: https://www.intel.com/content/www/us/en/developer/video
| s/opt...
|
| https://www.intel.com/content/www/us/en/developer/tools/one
| a...
|
| https://developer.apple.com/metal/tensorflow-plugin/
|
| Large scale opensource is, outside of a few exceptions,
| built by engineers paid to build it.
| janalsncm wrote:
| Special mention to Facebook and Google AI research teams that
| maintain PyTorch and Tensorflow respectively. And also to
| ptrblck on the PyTorch forums [1] who has the answer to
| basically every question it seems. He alone is probably
| responsible for hundreds of millions of dollars of
| productivity gain.
|
| [1] https://discuss.pytorch.org/u/ptrblck/summary
| jauntywundrkind wrote:
| Pytorch is just using Google's OpenXLA now, & OpenXLA is the
| actual cross platform thing, no? I'm not very well versed in
| this area, so pardon if mistaken.
| https://pytorch.org/blog/pytorch-2.0-xla-path-forward/
| fotcorn wrote:
| You can use OpenXLA, but it's not the default. The main use-
| case for OpenXLA is running PyTorch on Google TPUs. OpenXLA
| also supports GPUs, but I am not sure how many people use
| that. Afaik JAX uses OpenXLA as backend to run on GPUs.
|
| If you use model.compile() in PyTorch, you use TorchInductor
| and OpenAIs Triton by default.
| jauntywundrkind wrote:
| Thank you for saying something useful here. I was vaguely
| under the impression that pytorch 2.0 had fully flipped to
| defaulting to openxla. That seems to not be the case.
|
| Good to hear more than a cheap snub. OpenAI Triton as the
| reason other GPUs work is a real non-shit answer, it seems.
| And interesting to hear JAX too. Thank you for being
| robustly useful & informative.
| mathisfun123 wrote:
| > Pytorch is just using Google's OpenXLA now
|
| this is so far from accurate it should be considered
| libelous; from the link
|
| > PyTorch/XLA is set to migrate to the open source OpenXLA
|
| so PyTorch on the XLA backend is set to migrate to use
| OpenXLA instead of XLA. but basically everyone moved from XLA
| to OpenXLA because there is no more OSS XLA. so that's it. in
| general, PyTorch has several backends, including plenty of
| homegrown CUDA and CPU kernels. in fact the majority of your
| PyTorch code runs through PyTorch's own kernels.
| voz_ wrote:
| Wrong.
| nabla9 wrote:
| > Crossing the CUDA moat for AMD GPUs may be as easy as using
| PyTorch.
|
| Nvidia has spent huge amount of work to make code run smoothly
| and fast. AMD has to work hard to catch up. ROCm code is slower ,
| has more bugs, don't have enough features and they have
| compatibility issues between cards.
| latchkey wrote:
| Lisa has said that they are committed to improving ROCm,
| especially for AI workloads. Recent releases (5.6/5.7) prove
| that.
| einpoklum wrote:
| > Nvidia has spent huge amount of work to make code run
| smoothly and fast.
|
| Well, let's say "smoother" rather than "smoothly".
|
| > ROCm code is slower
|
| On physically-comparable hardware? Possible, but that's not an
| easy claim to make, certainly not as expansively as you have.
| References?
|
| > has more bugs
|
| Possible, but - NVIDIA keeps their bug database secret. I'm
| guessing you're concluding this from anecdotal experience?
| That's fair enough, but then - say so.
|
| > ROCm ... don't have enough features and
|
| Likely. while AMD has both spent less in that department (and
| had less to spend I guess); plus, and no less importantly - it
| tried to go along with the OpenCL initiative, as specified by
| the Khronos consortium, while NVIDIA has sort of "betrayed" the
| initiative by investing in it's vendor-locked, incompatible
| ecosystem and letting their OpenCL support decay in some
| respects.
|
| > they have compatibility issues between cards.
|
| such as?
| kkielhofner wrote:
| I wouldn't say ROCm code is "slower", per se, but in practice
| that's how it presents. References:
|
| https://github.com/InternLM/lmdeploy
|
| https://github.com/vllm-project/vllm
|
| https://github.com/OpenNMT/CTranslate2
|
| You know what's missing from all of these and many more like
| them? Support for ROCm. This is all before you get to the
| really wildly performant stuff like Triton Inference Server,
| FasterTransformer, TensorRT-LLM, etc.
|
| ROCm is at the "get it to work stage" (see top comment, blog
| posts everywhere celebrating minor successes, etc). CUDA is
| at the "wring every last penny of performance out of this
| thing" stage.
|
| In terms of hardware support, I think that one is obvious.
| The U in CUDA originally stood for unified. Look at the list
| of chips supported by Nvidia drivers and CUDA releases.
| Literally anything from at least the past 10 years that has
| Nvidia printed on the box will just run CUDA code.
|
| One of my projects specifically targets Pascal up - when I
| thought even Pascal was a stretch. Cue my surprise when I got
| a report of someone casually firing it up on Maxwell when I
| was pretty certain there was no way it could work.
|
| A Maxwell _laptop_ chip. It also runs just as well on an
| H100.
|
| THAT is hardware support.
| whywhywhywhy wrote:
| Anyone who has to work in this ecosystem surely thinks this is a
| naive take
| freedomben wrote:
| For someone who doesn't work in this ecosystem, can you
| elaborate? What's the real situation currently?
| tails4e wrote:
| AMD playing catch up is a good thing, their SW solution is
| intended to run on any HW, and with hip being basically line for
| line compatible with cuda it makes porting very easy. They did it
| with FSR,and they are doing it with rocm. Hopefully it takes off
| as it's a more open ecosystem for the industry. Necessity is the
| mother of invention and all that.
| superkuh wrote:
| >There is also a version of PyTorch that uses AMD ROCm, an open-
| source software stack for AMD GPU programming. Crossing the CUDA
| moat for AMD GPUs may be as easy as using PyTorch.
|
| Unfortunately since the AMD firmware doesn't reliably do what
| it's supposed to those ROCm calls often don't either. That's if
| your AMD card is even still supported by ROCm: the AMD RX 580 I
| bought in 2021 (the great GPU shortage) had it's ROCm support
| dropped in 2022 (4 years support total).
|
| The only reliable interface in my experience has been via opencl.
| zucker42 wrote:
| Do you mean OpenCL using Rusticl or something else? And what DL
| framework, if any?
| superkuh wrote:
| I should clarify that I mean for human person uses. Not
| commercial or institutional. But, clBLAST via llama.cpp for
| LLM currently. Or far in the past just pure opencl for things
| with AMD cards.
| 65a wrote:
| ROCm works fine on my 2016 Vega Frontier edition, for what it's
| worth.
| htrp wrote:
| has opencl actually improved enough to be competitive?
| orangepurple wrote:
| I thought ONNX is supposed to be the ultimate common
| denominator for machine learning model cross platform
| compatibility
| [deleted]
| spandextwins wrote:
| That's like saying Ford is gonna catch Tesla.
| cantaloupe wrote:
| Do you see that as an inevitability or an impossibility?
| tpmx wrote:
| No, not really. They have similar enough silicon, they "just"
| need some software to make it work.
| the__alchemist wrote:
| When coding using Vulkan, for graphics or compute (The latter is
| the relevant one here), you need to have CPU code (Written in
| C++, Rust etc), then serialize it as bytes, then have shaders
| which run on the graphics card. This 3-step process creates
| friction, much in the same way as backend/serialization/frontend
| does in web dev. Duplication of work, type checking not going
| across the bridge, the shader language being limited etc.
|
| My understanding is CUDA's main strength is avoiding this. Do you
| agree? Is that why it's such a big deal? Ie, why this article was
| written, since you could always do compute shaders on AMD etc
| using Vulkan.
| KingLancelot wrote:
| [dead]
| atemerev wrote:
| Nope. PyTorch is not enough, you have to do come C++ occasionally
| (as the code there can be optimized radically, as we see in
| llama.cpp and the like). ROCm is unusable compared to CUDA (4x
| more code for the same problem).
|
| I don't understand why everyone neglects good, usable and
| performant lower-level APIs. ROCm is fast, low-level, but much
| much harder to use than CUDA, and the market seems to agree.
| alecco wrote:
| Regurgitated months-old content. blogspam
| einpoklum wrote:
| TL;DR:
|
| 1. Since PyTorch has grown very popular, and there's an AMD
| backend for that, one can switch GPU vendors when doing
| Generative AI work.
|
| 2. Like NVIDIA's Grace+Hopper CPU-GPU combo, AMD is/will be
| offering "Instinct MI300A", which improves performance over
| having the GPU across a PCIe bus from a regular CPU.
| bigcat12345678 wrote:
| Cuda is the foundation
|
| NVIDIA moat is the years of work built by oss community, big
| corporations, research insistute
|
| They spend all time building for cuda, a lot of implicit designs
| are derived from cuda's characteristic
|
| That will be the main challenge
| mikepurvis wrote:
| It depends on the domain. Increasingly people's interfaces to
| this stuff are the higher level libraries like tensorflow,
| pytorch, numpy/cupy, and to a lesser degree accelerated
| processing libraries such as opencv, PCL, suitesparse, ceres-
| solver, and friends.
|
| If you can add hardware support to a major library _and_
| improve on the packaging and deployment front while also
| undercutting on price, that 's the moat gone overnight. CUDA
| itself only matters in terms of lock-in if you're calling
| CUDA's own functions.
| bigcat12345678 wrote:
| what I meant is that all these stuff have 15 years of
| implicit accumulation of knowledge and tips and even hacks
| builtin in the software
|
| No matter what you depends on, you'll have a slew of larger
| or minor obstacles or annoyance
|
| That collectively is the most itself
|
| As you said, already it's clear that replacing cuda itself is
| not that daunting
| ddtaylor wrote:
| It's worth noting that AMD also has a ROCm port of Tensorflow.
| ginko wrote:
| When I try to install rocm-ml-sdk on Arch linux it'll tell me
| the total installed size would be about 18GB.
|
| What can possibly explain this much bloat for what should
| essentially be a library on top of a graphics driver as well as
| some tools (compiler, profiler etc.)? A couple hundred MB I
| could understand if they come with graphical apps and demos,
| but not this..
| tomsmeding wrote:
| A regular TensorFlow installation, just the Python library,
| is an 184 MB wheel that unpacks to about 1.2 GB of stuff. I
| have no clue what mess goes in there, but it's a lot.
|
| Still, if you're right that this package seems to take 18 GB
| disk size, something weird is going on.
| Flameancer wrote:
| He's not wrong. I did a new arch install to try and get a
| 7800XT working with ROCm and PyTorch and was concussed on
| how I ran out of space but saw that ROCm was 18GB.
| slavik81 wrote:
| There's a lot of kernels that are specialized for
| particular sets of input parameters and tuned for improved
| performance on specific hardware, which makes the libraries
| a couple hundred megabytes per architecture. The ROCm
| libraries are huge because they are fat binaries containing
| native machine code for ~13 different GPU architectures.
| RcouF1uZ4gsC wrote:
| I am not so sure.
|
| Everyone knows that CUDA is a core competency of Nvidia and they
| have stuck to it for years and years refining it, fixing bugs,
| and making the experience smoother on Nvidia hardware.
|
| On the other hand, AMD has not had the same level of commitment.
| They used to sing the praises of OpenCL. And then there is ROCm.
| Tomorrow, it might be something else.
|
| Thus, Nvidia CUDA will get a lot more attention and tuning from
| even the portability layers because they know that their
| investment in it will reap dividends even years from now, whereas
| their investment in AMD might be obsolete in a few years.
|
| In addition, even if there is theoretical support, getting
| specific driver support and working around driver bugs is likely
| to be more of a pain with AMD.
| AnthonyMouse wrote:
| This is what people complain about, but at the same time there
| aren't enough cards, so the people with AMD cards want to use
| them. So they fix the bugs, or report them to AMD so they can
| fix them, and it gets better. Then more people use them and
| submit patches and bug reporters, and it gets better.
|
| At some point the old complaints are no longer valid.
| pixelesque wrote:
| Does AMD have a solution to forward device combatibility (like
| PTX for NVidia)?
|
| Last time I looked into ROCm (two years ago?), you seemed to have
| to compile stuff explicitly for the architecture you were using,
| so if a new card came out, you couldn't use it without a
| recompile.
| mnau wrote:
| Not natively, but AdaptiveCpp (previously hiSycl, then
| OpenSycl) has a single source single compiler pass, where they
| basically store LLVM IR as an intermediate representation.
|
| https://github.com/AdaptiveCpp/AdaptiveCpp/blob/develop/doc/...
|
| Performance penalty was within ew precents, at least according
| to the paper (figure 9 and 10)
| https://cdrdv2-public.intel.com/786536/Heidelberg_IWOCL__SYC...
| einpoklum wrote:
| I don't know what they do with ROCm, but with OpenCL, the
| answer is: Certainly. It's called SPIR:
|
| https://www.khronos.org/spir/
| jeffreygoesto wrote:
| I am hoping for SYCL and SPIR-V to gain traction...
| raggi wrote:
| Can we just get wgsl compute good enough and over the line
| instead, and do away with these moats?
| mschuetz wrote:
| Not happening. WGSL wants to support the lowest common
| denominator, so it'll always mainly be a 5-year old mobile-
| phone API. Also if you want to beat CUDA, you'll need some
| functionality that's completely missing in compute shaders,
| especially WGSL. Like pointers and pointer casting (and that
| glsl buffer reference extension is the worst emulation of that
| feature I've every seen).
| raggi wrote:
| The language extensions feature is designed to provide these
| kinds of facilities is it not?
| tormeh wrote:
| For LLM inference, a shoutout to MLC LLM, which runs LLM models
| on basically any API that's widely available:
| https://github.com/mlc-ai/mlc-llm
| ur-whale wrote:
| > AMD May Get Across the CUDA Moat
|
| I really wish they would, and properly, as in: fully open
| solution to match CUDA.
|
| CUDA is a cancer on the industry.
| mschuetz wrote:
| What's wrong with CUDA? I avoided it for years because it's
| proprietory but about one year ago I started using it because
| all the alternatives (OpenGL/Vulkan compute, OpenCL, WebGPU,
| ...) couldn't quite do what I wanted, and it turned out to be a
| game changer. Nothing comes close to it. Now I'm hooked because
| there simply isn't an alternative that's as easy to use, yet
| powerfull and fast.
|
| I wish there was an open alternative, but NVIDIA did several
| things right that others, especially Khronos, do not: The UX is
| top-notch. It makes the common cases easy yet still fast, and
| from there you can optimize to your hearts content. Khronos,
| however, usually completely over-engineers things and makes the
| common case hard and cumbersome with massive entry barriers.
| mark_l_watson wrote:
| NVidia hardware/CUDA stack is great, but I also love to see
| competition from AMD, George Hotz's Tiny Corp, etc.
|
| Off topic, but I am also looking with great interest at Apple
| Silicon SOCs with large internal RAM. The internal bandwidth also
| keeps getting better which is important for running trained LLMs.
|
| Back on topic: I don't own any current Intel computers but using
| Colab and services like Lambda Labs GPU VPSs is simple and
| flexible. A few people here mentioned if AMD can't handle 100% of
| their workload they will stick with Intel and NVidia -
| understandable position, but there are workarounds.
| upbeat_general wrote:
| This article doesn't address the real challenge [in my mind].
|
| Framework support is one thing, but what about the million
| standalone CUDA kernels that have been written, especially common
| in research. Nobody wants to spend time re-writing/porting those,
| especially when they probably don't understand the low-level
| details in the first place.
|
| Not to mention, what is the plan for comprehensive framework
| support? I've experienced the pain of porting models to different
| hardware architectures where various ops are unsupported. Is it
| realistic to get full coverage of e.g., PyTorch?
| blueboo wrote:
| Research kernels mostly turn to ash upon publication anyway.
| The wheel turns and the next post-doc gives ROCm a try and we
| move on
| bdowling wrote:
| Someone could reimplement CUDA for AMD hardware. That would be
| legal because copying APIs for compatibility purposes is not
| copyright infringement. (See _Google LLC v. Oracle America_ ,
| Inc., 593 U.S. ___ (2021)).
|
| AMD is unlikely to do this, however, because it would commodify
| their own products under their competitor's API.
|
| A third party could do it though. It may make sense as an open
| source project.
| Roark66 wrote:
| I think the article claiming "PyTorch has dropped the drawbridge
| on the CUDA moat" is way over optimistic. Jest pytorch is widely
| used by researchers and by users to quickly iterate various over
| various ways to use the models, but when it comes to inference
| there are huge gains to be had by going a different route.
| Llama.cpp has showed 10x speedups on my hardware for example
| (32gb of gpu ram + 32gb of cpu ram)for models like
| falcon-40b-instruct, for much smaller models on the cpu (under
| 10b) I saw up to 3x speedup just by switching to onnc and
| openvino.
|
| Apple has showed us in practice the benefits of CPU/GPU memory
| sharing, will AMD be able to follow in their footsteps? The
| article claims AMD has a design with up to 192gb of shared ram.
| Apple is already shipping a design with the same amount of RAM(if
| you can afford it). I wish them-and) success, but I believe they
| need to aim higher than just matching apple in some unspecified
| future.
| sharonzhou wrote:
| ROCm is great. We were able to get run and finetune LLMs on AMD
| Instincts with parity to NVIDIA A100s - and built an SDK that's
| as easy to use as HuggingFace or easier (Lamini). Or at the very
| least, our designer is able to finetune/train the latest LLMs on
| them like Llama 2 - 70B and Mistral 7B with ease. The ROCm
| library isn't as easy to use as CUDA because as another poster
| said, the ecosystem was built around CUDA. For example, it's even
| called ".cuda()" in PyTorch to put a model on a GPU, when in
| reality you'd use it for an AMD GPU too.
| hot_gril wrote:
| People complain about Nvidia being anticompetitive with CUDA, but
| I don't really see it. They saw a gap in the standards for on-GPU
| compute and put tons of effort into a proprietary alternative.
| They tied CUDA to their own hardware, which sorta makes technical
| sense given the optimizations involved, but it's their choice
| anyway. They still support the open standards, but many prefer
| CUDA and will pay the Nvidia premium for it because it's actually
| nicer. They also don't have CPU marketshare to tie things to.
|
| Good for them. We can hope the open side catches up either by
| improving their standards, or adding more layers like this
| article describes.
| zirgs wrote:
| CUDA was released in 2007 and the development of it started
| even earlier - possibly even in the 90s. Back then nobody else
| cared about GPU compute. OpenCL came out 2 years after that.
| killerstorm wrote:
| Not true. People got interested in general-purpose GPU
| compute (GPGPU) in early 2000s when video cards with
| programmable shaders became available.
| https://en.wikipedia.org/wiki/General-
| purpose_computing_on_g...
|
| People made a programming language & a compiler/runtime for
| GPGPU in 2004: https://en.wikipedia.org/wiki/BrookGPU
| hot_gril wrote:
| Everything has old beginnings that the specialists will
| remember, but GPU compute really reached mass popularity
| and became a large selling point for Nvidia in the 2010s.
| frnkng wrote:
| As a former ETH miner I learned the hard way that saving a few
| bucks on hardware may not be worth operational issues.
|
| I had a miner running with Nividia cards and a miner running with
| AMD cards. One of them had massive maintenance demand and the
| other did not. I will not state which brand was better imho.
|
| Currently I estimate that running miners and running gpu servers
| has similar operational requirements and finally at scale similar
| financial considerations.
|
| So, whatever is cheapest to operate in terms of time expenditure,
| hw cost, energy use,... will be used the most.
|
| P.s.: I ran the mining operation not to earn money but mainly out
| of curiosity. And it was a small scale business powered by a pv
| system and a attached heat pump.
| latchkey wrote:
| I ran 150,000+ AMD cards for mining ETH. Once I fully automated
| all the vbios installs and individual card tuning, it ran
| beautifully. Took a lot of work to get there though!
|
| Fact is that every single GPU chip is a snowflake. No two
| operate the same.
| rottencupcakes wrote:
| Have you ever written about this enterprise? This sounds
| super unique and I would be very interested in hearing about
| how it was run and how it turned out.
| latchkey wrote:
| It was unique, not many people on the planet, that I know
| of, who've run as many GPUs as I have. Especially not
| working for a giant company with large teams of people. For
| the tech team, it was just me and one other guy. Everything
| _had_ to be automated because there was no way we could
| survive otherwise.
|
| I've put a bunch of comments here on HN about the stuff I
| can talk about.
|
| It no longer exists after PoS.
| freedomben wrote:
| what type of cards did you have? what did you do with
| them after PoS? How did you even buy so many cards?
| Sorry, like the other commenter I'm extremely curious
| latchkey wrote:
| Primarily 470,480,570,580. We also ran a very large
| cluster of PS5 APU chips too.
|
| Got the chips directly from AMD. Since these are 4-5 year
| old chips, they were not going to ever be used. It is
| more ROI efficient with ETH mining to use older cards
| than newer ones.
|
| Had a couple OEM manufacture the cards specially for us
| with 8gb, heatsinks instead of fans (lower power usage)
| and no display ports (lower cost).
|
| They will be recycled as there isn't much use for them
| now.
|
| I'm also no longer with the company.
| xcdzvyn wrote:
| Cool! Were the PS5 APUs actually attached to a PS5
| motherboard, or were they repurposed entirely?
| latchkey wrote:
| Asrock bc-250. This is some hardware that I wouldn't have
| purchased, if given the choice, especially that close to
| ETH PoS.
|
| That said, I made it work, which was an insane amount of
| work, and it mined really well.
| falconroar wrote:
| Is there any reason OpenCL is not the standard in implementations
| like PyTorch? Similar performance, open standard, runs everywhere
| - what's the downside?
| LoganDark wrote:
| IIRC, ease of implementation (for the GPU kernels), and cross-
| compatibility (the same bytecode can be loaded by multiple
| models of GPU).
| ealloc wrote:
| How is CUDA-C that much easier than OpenCL? Having ported
| back and forth myself, the base C-like languages are
| virtually identical. Just sub "__syncthreads();" for
| "barrier(CL_MEM_FENCE)" and so on. To me the main problem is
| that Nvidia hobbles OpenCL on their GPUs by not updating
| their CL compiler to OpenCL 2.0, so some special features are
| missing, such as many atomics.
| jacobgorm wrote:
| The ease of implementation using CUDA means that your code
| because effed for life, because it is no longer valid C/C++,
| unless you totally litter it with #ifdefs to special case for
| CUDA. In my own proprietary AI inference pipeline I've ended
| up code-generating to a bunch of different backends (OpenCL
| SpirV, Metal, CUDA, HLSL, CPU w. OpenMP), giving no special
| treatment to CUDA, and the resulting code is much cleaner and
| builds with standard open source toolchains.
___________________________________________________________________
(page generated 2023-10-07 23:01 UTC)