[HN Gopher] CUDA Ontology
___________________________________________________________________
CUDA Ontology
Author : gugagore
Score : 241 points
Date : 2025-11-16 18:56 UTC (4 days ago)
(HTM) web link (jamesakl.com)
(TXT) w3m dump (jamesakl.com)
| ArcHound wrote:
| That is a great reference, explains a lot of small inaccuracies
| between various tutorials when you're trying to debug some of
| these issues. Saved and printed, thanks a lot!
| visarga wrote:
| Wondering why a $4T company can't afford a smart installation
| assistant that can auto-detect problems and apply fixes as
| needed. I wasted too many days chasing driver and torch versions.
| It's probably the worst part of working in ML. Combine this with
| Python's horrible package management and you got a perfect combo
| - like the cough and the stitch.
| numbers_guy wrote:
| They provide containers to cater to those needs:
| https://catalog.ngc.nvidia.com/search
| threeducks wrote:
| After being once again frustrated by the CUDA installation
| experience, I thought that I should give those containers a
| try. Unfortunately, my computer did not boot anymore after
| following the installation instructions for the NVIDIA
| container toolkit as outlined on the NVIDIA website.
| Reinstalling everything and following the instructions from
| some random blog post made it work, but I then found that the
| container with the CUDA version that I needed had been
| deprecated.
|
| There were other problems, such as the research cluster of my
| university not having Docker, but that is a different issue.
| YetAnotherNick wrote:
| Containers don't include drivers which is the primary reason
| for issues.
| torginus wrote:
| Containers afair rely on the _exact_ driver version
| matching between the host system and the container itself.
|
| We were on AWS when we used this so setting up seemed easy
| enough - AWS gave you the driver, and a matching docker
| image was easy enough to find.
| kcb wrote:
| That's not the case, CUDA containers user space does not
| have to match the host drivers CUDA capability. The
| container needs to be the same major version or lower. So
| a system with a CUDA 13 capable driver should be able to
| run all previous versions.
|
| For some versions there's even sometimes compat layers
| built into the container to allow forward version
| compatibility.
| fragmede wrote:
| Just have claude code fix it
| ux266478 wrote:
| I'm wondering how a $4T company got away with shipping the
| absolute state of the toolchain to begin with. They have total
| and complete sovereignty on everything on the outside of the OS
| and PCIe boundaries with a bottomless pool of top class labor.
| There's no reason it has to be cruftier or more fragile than
| any other low latency networked computation... and yet here we
| are. AMD isn't any better. I'm almost interested to see if
| Intel has done any better with L0, but I highly suspect it
| suffers from the exact same ecosystem hell problems that plague
| the other two.
|
| The idea that getting a PCIe FPGA board to crunch numbers is
| less headache prone than a GPU is laughable, but that's the
| absurd reality we live in.
| w-m wrote:
| This is a good resource. But for the computer vision and machine
| learning practitioner most of the fun can start where this
| article ends.
|
| nvcc from the CUDA toolkit has a compatibility range with the
| underlying host compilers like gcc. If you install a newer CUDA
| toolkit on an older machine, likely you'll need to upgrade your
| compiler toolchain as well, and fix the paths.
|
| While orchestration in many (research) projects happens from
| Python, some depend on building CUDA extensions. An innocently
| looking Python project may not ship the compiled kernels and may
| require a CUDA toolkit to work correctly. Some package management
| solutions provide the ability to install CUDA toolkits
| (conda/mamba, pixi), the pure-Python ones do not (pip, uv). This
| leaves you to match the correct CUDA toolkit to your Python
| environment for a project. conda specifically provides different
| channels (default/nvidia/pytorch/conda-forge), from conda 4.6
| defaulting to a strict channel priority, meaning "if a name
| exists in a higher-priority channel, lower ones aren't
| considered". The default strict priority can make your
| requirements unsatisfiable, even though there would be a version
| of each required package in the collection of channels. uv is
| neat and fast and awesome, but leaves you alone in dealing with
| the CUDA toolkit.
|
| Also, code that compiles with older CUDA toolkit versions may not
| compile with newer CUDA toolkit versions. Newer hardware may
| require a CUDA toolkit version that is newer than what the
| project maintainer intended. PyTorch ships with a specific CUDA
| runtime version. If you have additional code in your project that
| also is using CUDA extensions, you need to match the CUDA runtime
| version of your installed PyTorch for it to work. Trying to bring
| up a project from a couple of years ago to run on latest hardware
| may thus blow up on you on multiple fronts.
| anotherpaul wrote:
| Yes, this is the actual lived reality. Thank you for outlining
| it so well.
| eapriv wrote:
| Sounds like most of these problems come from using Python.
| mellosouls wrote:
| You imply these problems would go away (or wouldn't be
| replaced by new ones) with another language.
| eapriv wrote:
| Removing layers usually improves stability.
| alecco wrote:
| > nvcc from the CUDA toolkit has a compatibility range with the
| underlying host compilers like gcc. If you install a newer CUDA
| toolkit on an older machine, likely you'll need to upgrade your
| compiler toolchain as well, and fix the paths.
|
| Conversely, nvcc often stops working with major upgrades of
| gcc/clang. Fun times, indeed.
|
| This is why a lot of people just use NVIDIA's containers even
| for local solo dev. It's a hassle to set up initially
| (docker/podman hell) but all the tools are there and they work
| fine.
| embedding-shape wrote:
| > This is why a lot of people just use NVIDIA's containers
| even for local solo dev. It's a hassle to set up initially
| (docker/podman hell) but all the tools are there and they
| work fine.
|
| Yeah, which I feel like is fine for one project, or one-offs,
| but once you've accumulated projects, having individual 30GB
| images for each of them quickly adds up.
|
| I found that most of my issues went away as I started
| migrating everything to `ux` for the python stuff, and nix
| for everything system related. Now I can finally go back to a
| 1 year old ML project, and be sure it'll run like before, and
| projects share a bit more data.
| jcelerier wrote:
| Yep, right now nvidia libs are broken with clang-21 and
| recent glibc due to stuff like rsqrt() having throw() in the
| declaration and not in the definition
| the__alchemist wrote:
| What trouble have you had specifically? On both Win and
| Linux, installing the CUDA toolkit (e.g. v13) _just works_
| for me. My use case is compiling kernels (or cuFFT FFI) using
| nvcc for FFI in rust programs and libs.
| billti wrote:
| > Also, code that compiles with older CUDA toolkit versions may
| not compile with newer CUDA toolkit versions. Newer hardware
| may require a CUDA toolkit version that is newer than what the
| project maintainer intended.
|
| This is the part I find confusing, especially as NVIDIA doesn't
| make it easy to find and download the old toolkits. Is this
| effectively saying that just choosing the right --arch and
| --code flags isn't enough to support older versions? But that
| as it statically links in the runtime library (by default) that
| newer toolkits may produce code that just won't run on older
| drivers? In other words, is it true that to support old
| hardware you need to download and use old CUDA Toolkits,
| regardless of nvcc flags? (And to support newer hardware you
| may need to compile with newer toolkits).
|
| That's how I read it, which seems unfortunate.
| zvr wrote:
| Great explanation!
|
| It should probably also add that everything CUDA is owned by
| NVIDIA, and "CUDA" itself is a registered trademark. The official
| way to refer to it is that the first time you spell it out as
| "NVIDIA(r) CUDA(r)" and then subsequently refer to just CUDA.
| threeducks wrote:
| Why should the author use the registered trademark symbol?
| xpe wrote:
| I am not a layer (IANAL), but here is what Gemini 3 Pro says:
| "You generally do not need to use the trademark symbol for CUDA
| in a blog post, unless you have a specific commercial
| relationship with NVIDIA."
|
| Now direct from actual sources... From [1]
|
| > Intended users of this Brand Guideline are members of the
| NVIDIA Partner Network (NPN), including original equipment
| manufacturers (OEMs), solution advisors, cloud partners,
| solution providers, distributors, solutions integrators, and
| service delivery partners.
|
| From [2]:
|
| > Always include the correct trademark ((tm) vs (r)) by
| referring to the content documents provided or using the list
| of common NVIDIA products and technologies. After the first
| mention of the NVIDIA product or technology, which includes the
| appropriate trademarks, the trademark does not need to be
| included in future mentions within the same document, article,
| etc.
|
| > CUDA(r)
|
| [1]: https://brand.nvidia.com/d/wGtgoY2mtYYM/nvidia-partner-
| netwo...
|
| [2]: https://brand.nvidia.com/d/wGtgoY2mtYYM/nvidia-partner-
| netwo...
| pjmlp wrote:
| Great overview, with lots of effort place into it.
|
| However, it misses the polyglot part (Fortran, Python GPU JIT,
| all the backends that support PTX), the library ecosystem
| (writing CUDA kernels should be the exception not the rule), the
| graphical debugging tools and IDE integration.
| virajk_31 wrote:
| thanks for the kernel nomenclatures
| bbx wrote:
| For reference: CUDA means "Compute Unified Device Architecture".
| montyanderson wrote:
| this is fantastic
| coffeeaddict1 wrote:
| I wish GPU vendors would stick to a standard terminology, at
| least for common parts. It's really confusing having to deal with
| warps vs wavefronts vs simd groups, thread block vs workgroup,
| streaming multiprocessor vs compute unit vs execution unit,
| etc...
| RYJOX wrote:
| Interesting, does this approach change with out-of-order cores?
| In fact maybe I misunderstand lol
| NullCascade wrote:
| What is the cheapest CUDA-enabled VM providers one can use to
| learn CUDA?
| eamag wrote:
| Lightning.ai
| dahart wrote:
| This article has good info, but is the overloading premise
| slightly contrived? Maybe I don't talk to enough CUDA beginners.
| I work with CUDA a lot but I'm not exactly a CUDA expert, and
| from my perspective, in practice there are default assumptions
| one can safely make for the base terms, and people do qualify the
| alternatives almost always. For example, if someone says "CUDA
| version", they always mean the toolkit, and never mean compute
| capability, runtime, or language. The term "driver" when used
| without qualification always means the display driver, and never
| means the driver API, there really is no overload there.
| einpoklum wrote:
| I actually find it is pretty easy to get confused between the
| different kinds of versions. For example:
|
| "The CUDA "driver version" looks like the CUDA runtime version
| - so what's the difference?"
| https://stackoverflow.com/q/40589814/1593077
|
| or consider the version you get when you run nvidia-smi, versus
| the version you get when you run nvcc --version. Those are very
| different numbers...
|
| The compatibility between different versions of the driver and
| the toolkit is also a cause for some headaches in my
| experience.
| dahart wrote:
| Oh yeah, knowing the difference between the runtime API and
| driver API is definitely an issue, and there is common
| confusion around that. But that's not an overloaded word
| problem, right? I wasn't trying to say there's no confusion,
| and I do think understanding the terms in the article is
| super helpful. To your point, I think there's justification
| for needing a codex like this article has without framing it
| as an overloaded terminology problem.
| the__alchemist wrote:
| I suspect you're colored by your experience, despite your
| modesty about it. To you or I, "CUDA version" probably means
| something like 'v13' or w/e of the "CUDA toolkit", which you
| know means the user running your code needs an "nvidia driver"
| = "580" or higher.
|
| I wouldn't have been able to tell you this a few months ago,
| and it was confusing! Machine that compiles vs machine that
| runs, CUDA toolkit which includes both vs nvidia driver which
| just includes one part of it etc... The article explicitly
| describes this.
| einpoklum wrote:
| > _CUDA Runtime: The runtime library (libcudart) that
| applications link against._
|
| That library is actually a rather poor idea. If you're writing a
| CUDA application, I strongly recommend avoiding the "runtime
| API". It provides partial access to the actual CUDA driver and
| its API, which is 'simpler' in the sense that you don't
| explicitly create "contexts", but:
|
| * It hides or limits a lot of the functionality.
|
| * Its actual behavior vis-a-vis contexts is not at all simple and
| is likely to make your life more difficult down the road.
|
| * It's not some clean interface that's much more convenient to
| use.
|
| So, either go with the driver, or consider my CUDA API wrappers
| library [1], which _does_ offer a clean, unified, modern (well,
| C++11'ish) RAII/CADRe interface. And it covers much more than the
| runtime API, to boot: JIT compilation of CUDA (nvrtc) and PTX
| (nvptx_compiler), profiling (nvtx), etc.
|
| > _Driver API ... provides direct access to GPU functionality._
|
| Well, I wouldn't go that far, it's not that direct. Let's call
| it: "Less indirect"...
|
| [1] : https://github.com/eyalroz/cuda-api-wrappers/
| nickysielicki wrote:
| If you do this, you forego both backwards and forwards
| compatibility. You must follow the driver release cadence
| exactly, and rebuild all of your code for every driver you want
| to support when a new release happens, or you risk subtle
| breakage. NVIDIA guarantees nothing in terms of breakage for
| you.
|
| Probably the worst part of this: for the most part, in
| practice, it will work just fine. Until it doesn't. You will
| have lots of fun debugging subtle bugs in a closed-source black
| box, which reproduces only against certain driver API header
| versions, which potentially does not match the version of the
| actual driver API DSO you've dlopened, and which only produces
| problems when mixed with certain Linux kernel versions.
|
| (I have the exact opposite opinion; people reach too eagerly
| for the driver API when they don't need it. Almost everything
| that can be done with the driver api can be done with the
| runtime API. If you absolutely must use the driver API, which I
| doubt, you should at least resolve the function pointers
| through cudaGetDriverEntrypointByVersion.)
| Nydhal wrote:
| This is a classic lesson. You can write almost the same article
| for Java: language vs bytecode vs JVM vs JDK vs libs ...
| scotty79 wrote:
| > This article provides a rigorous ontology of CUDA components: a
| systematic description of what exists in the CUDA ecosystem, how
| components relate to each other, their versioning semantics,
| compatibility rules, and failure modes.
|
| That's the first instance in my life when somebody coherently
| described what the word 'ontology' means. I'm sure this
| explanation is wrong, but still...
___________________________________________________________________
(page generated 2025-11-20 23:01 UTC)