[HN Gopher] Nvidia DGX Spark: When benchmark numbers meet produc...
___________________________________________________________________
Nvidia DGX Spark: When benchmark numbers meet production reality
Author : RyeCatcher
Score : 115 points
Date : 2025-10-26 17:53 UTC (5 hours ago)
(HTM) web link (publish.obsidian.md)
(TXT) w3m dump (publish.obsidian.md)
| RyeCatcher wrote:
| Would love to hear from others using the spark for model training
| and development.
| stuckinhell wrote:
| I'm utterly shocked at the article saying GPU inference
| (PyTorch/Transformers)isn't working. Numerical instability
| produces bad outputs, Not viable for real-time serving, Wait for
| driver/CUDA updates!
|
| My job just got me and our entire team a DGX spark. I'm impressed
| at the ease of use for ollama models I couldn't run on my laptop.
| gpt-oss:120b is shockingly better than what I thought it would be
| from running the 20b model on my laptop.
|
| The DGX has changed my mind about the future being small
| specialized models.
| jasonjmcghee wrote:
| > I'm utterly shocked at the article saying GPU inference
| (PyTorch/Transformers)isn't working
|
| Are you shocked because that isn't your experience?
|
| From the article it sounds like ollama runs cpu inference not
| GPU inference. Is that the case for you?
| RyeCatcher wrote:
| Totally agree. I've been training nanochat models all morning.
| Hit some speed bumps. I'll share more later in another article.
| Buts it's absolutely amazing. I fine tuned a Gemma3 model in a
| day yesterday.
| jsheard wrote:
| No mention of the monstrous 200GbE NIC, seems like a waste if
| people aren't finding a use for it.
| RyeCatcher wrote:
| Need to buy 2 and connect em. :-)
| RyeCatcher wrote:
| I absolutely love it. I've been up for days playing with it. But
| there are some bleeding edge issues. I tried to write a balanced
| article. I would highly recommend for people that love to get
| their hands dirty. Blows away any consumer GPU.
| furyofantares wrote:
| Since the text is obviously LLM output, how much prompting and
| editing went into this post? Did you have to correct anything
| that you put into it that it then got wrong or added incorrect
| output to?
| NathanielK wrote:
| Definitely reeks of someone who doesn't know what makes a
| readable blogpost and hoped the LLM did.
|
| I was not familiar with the hardware, so I was disappointed
| there wasn't a picture of the device. Tried to skim the
| article and it's a mess. Inconsistent formatting and emoji
| without a single graph to visualize benchmarks.
| furyofantares wrote:
| I read the whole thing now and it's filled with slop. I was
| being too nice to ask them politely what they put into it.
| It's garbage and they've wasted our time posting it.
|
| I don't really care about the emojis and the marketing
| voice too much. I do care that it's impossible to tell what
| the author cared about what they didn't, or if any of it is
| made up or extrapolated.
|
| I bet the input to the LLM would have been more
| interesting.
| furyofantares wrote:
| > Training Performance is Real (When It Works)
|
| It looks like it worked? Why's it say this?
|
| > Verdict: Inference speed scales proportionally with
| model size.
|
| Author only tried one model size and it's faster than
| NVIDIA's reported speed at a larger model. Not really a
| "Verdict".
|
| > Verdict: 4-bit quantization is production-viable.
|
| That's not really something you can conclude from messing
| around with it and saying you like the outputs.
|
| > GPU Inference is Fundamentally Broken
|
| Probably not? It probably just doesn't work in llama.cpp
| right now? Takes a while reading this to work out they
| tried ollama and then later llama.cpp, which I'd guess is
| basically testing llama.cpp twice. Actually I don't even
| believe that, I'm sure author ran into errors that might
| be a pain to figure out, but there's no evidence it's
| worse than that.
|
| But then it says this is the "root cause":
| ARM64 + Blackwell + CUDA 13.0 = Bleeding Edge |
| Limited production testing | Edge cases
| in numerical precision (inference) |
| Memory management issues (training)
|
| Am I to believe GPU inference is really fundamentally
| broken? I'm not seeing the case made here, just claims.
| At this point the LLM seems to have gotten confused about
| whether it's talking about the memory fragmentation issue
| or the GPU inference issue. But it's hard to believe
| anything from this point on in the post.
| enum wrote:
| +1
|
| I have H100s to myself, and access to more GPUs than I know
| what to do with in national clusters.
|
| The Spark is much more fun. And I'm more productive. With two
| of them, you can debug shallow NCCL/MPI problems before hitting
| a real cluster. I sincerely love Slurm, but nothing like a
| personal computer.
| latchkey wrote:
| Your complaint sounds more like the way that you have to
| access the HPC (via slurm), not the compute itself. After
| having now tried slurm myself, I don't understand the love
| for it at all.
|
| As for debugging, that's where you should be allowed to spin
| up a small testing cluster on-demand. Why can't you do that
| with your slurm access?
| enum wrote:
| I'm not complaining. The clusters are great. The non-Slurm
| H100s are great. The Spark is more fun.
| latchkey wrote:
| What makes it more fun?
| enum wrote:
| I think that personal computing is more fun than time-
| shared computing. :)
|
| It's remarkable what can now be done on a whisper-quiet
| little box. I hope the Strix Halo's will be just as much
| fun, and they should be, so long as Flash Attention
| works.
| yunohn wrote:
| 100% - slurm is aimed at job maintenance and resource
| management on HPC clusters. Thus being a pain in the ass
| for the kind of fast adhoc iteration and testing that AI/ML
| requires.
| yunohn wrote:
| Thanks for this bleeding edge content!
|
| But please have your LLM post writer be less verbose and
| repetitive. This is like the stock output from any LLM, where
| it describes in detail and then summarizes back and forth over
| multiple useless sections. Please consider a smarter prompt and
| post-editing...
| veber-alex wrote:
| The llama.cpp issues are strange.
|
| There are official benchmarks of the Spark running multiple
| models just fine on llama.cpp
|
| https://github.com/ggml-org/llama.cpp/discussions/16578
| RyeCatcher wrote:
| Cool I'll have a look. All reflections I made were first pass
| stuff.
| CaptainOfCoit wrote:
| There wasn't any instructions how the author got
| ollama/llama.cpp, could possibly be something nvidia shipped
| with the DGX Spark and is an old version?
| moffkalast wrote:
| Llama.cpp main branch doesn't run on Orins so it's actually
| weird that it does run on the Spark.
| eadwu wrote:
| There are bleeding edge issues, everyone dials into transformers
| so that's generally pain proof.
|
| I haven't exactly bisected the issue but I'm pretty sure
| convolutions are broken on sm_121 after a certain size, getting
| 20x memory blowup from a convolution from a 2x batch size
| increase _only_ on the DGX Spark.
|
| I haven't had any problems with inference, but I also don't use
| the transformers library that much.
|
| llama.cpp was working for openai-oss last time I checked and on
| release, not sure if something broke along the way.
|
| I don't exactly know if memory fragmentation is something fixable
| on the driver side - this might just be the problem with kernel's
| policy and GPL, it prevents them from automatically interfering
| with the memory subsystem to the granularity they'd like - see
| zfs and their page table antics - or so my thoughts on it is.
|
| If you've done stuff on WSL, you have similar issues and you can
| fix it by running a service that normally compacts and clean
| memory, I have it run every hour. Note that this does impact at
| the very least CPU performance and memory allocation speeds, but
| I have not have any issue with long training runs with it (24hr+,
| assuming that is the issue, I have never tried without it and put
| that service in place since getting it due to my experience on
| WSL).
| suprjami wrote:
| So I can spend thousands of dollars to have an unstable training
| environment and inference performance worse than a US$200 3060.
|
| Wow. Where do I sign up?
| vardump wrote:
| 3060 doesn't have 128 GB RAM.
| moffkalast wrote:
| 128GB / 12 GB = ~11, * 200EUR = only 2200EUR plus mining rig
| mobo.
|
| It would be cheaper to buy up a dozen 3060s and build a
| custom PC around them than to buy the Spark.
| pjmlp wrote:
| Except the Spark was designed to have everything nicely
| working.
| suprjami wrote:
| And as this post shows, it doesn't.
| pjmlp wrote:
| More than most AMD stuff.
| suprjami wrote:
| And a 14B model running at 22tg/s means you won't be using
| that 128G RAM for inference either.
| yunohn wrote:
| Yeah I'm honestly unclear on Nvidia's thinking here -
| inference speed is unbelievably slow for the price.
|
| Given the extreme advantage they have with CUDA and the
| whole AI/ML ecosystem, barely matching Apple's M-ultra
| speeds is a choice...
| airspresso wrote:
| Definitely a choice to give it low memory bandwidth.
| Probably to avoid customers thinking it can replace any
| data center GPU for inference use-cases.
| thehamkercat wrote:
| The memory bandwidth on this thing is absolute trash, better
| buy a mac mini/studio with this much ram if you're throwing
| this much money, it'll be faster (M4 Max)
| suprjami wrote:
| Agree, any Max or Ultra should walk all over this thing, and
| has the advantage of many years of already-working software.
|
| Apple benchmarks: https://github.com/ggml-
| org/llama.cpp/discussions/4167
| bigyabai wrote:
| It really depends, the metrics are kinda all over the place
| right now:
| https://docs.google.com/spreadsheets/d/1SF1u0J2vJ-ou-
| R_Ry1JZ...
|
| (cited from https://lmsys.org/blog/2025-10-13-nvidia-dgx-
| spark/)
| MaKey wrote:
| Why would you get this when a Ryzen AI Max+ 395 with 128 GB is a
| fraction of the price?
| d3m0t3p wrote:
| Because the ML ecosystem is more mature on the NVidia side.
| Software-wise the cuda platform is more advanced. It will be
| hard for AMD to catch up. It is good to see competition tho.
| shikon7 wrote:
| But the article shows that the Nvidia ecosystem isn't that
| mature either on the DGX Spark with ARM64. I wonder if Nvidia
| is still ahead for such use cases, all things considered.
| bigyabai wrote:
| On the DGX Spark, yes. On ARM64, Nvidia has been shipping
| drivers for years now. The rest of the Linux ecosystem is
| going to be the problem, most distros and projects don't
| have anywhere near the incentive Nvidia does to treat ARM
| like a first-class citizen.
| pjmlp wrote:
| Complete computer with everything working.
| simjnd wrote:
| The complete Framework Desktop with everything working
| (including said Ryzen AI Max 395+ and 128 GB of RAM) is 2500
| EUR. In Europe the DGX Spark listings are at 4000+ EUR.
| pjmlp wrote:
| Framework doesn't sell in Europe and they are sponsoring
| the wrong kind of folks nowadays.
| simjnd wrote:
| Framework does absolutely sell in several countries in
| Europe.
| zamadatix wrote:
| The vast majority of Ryzen AI Max+ 395s (by volume at least)
| are sold as complete system offerings as well. About as far
| as you can go the other way is getting one without an SSD, as
| the MB+RAM+CPU are an "all or nothing" bundle anyways.
| pjmlp wrote:
| Including a Linux distribution with working drivers?
| zamadatix wrote:
| Needing a customized spin of Ubuntu to have working video
| drivers is an Nvidia thing. One can also choose a Windows
| option, if they like, and run AI from there as it's just
| a standard x86 PC. That might actually be the best option
| for those worried about pre-installed OSs for AI
| tinkering.
|
| The userspace side is where AI is difficult with AMD.
| Almost all of the community is build around Nvidia
| tooling first, others second (if it all).
| overfeed wrote:
| Fortunately, AMD upstreams its changes so no custom
| distro is required for Strix Halo boxes. The DGX is the
| platform more at risk of being left behind on Linux -
| just like Jetson before it, which also had a custom, now-
| abandoned distro.
| TiredOfLife wrote:
| Amd works with recent kernels oob. DGX runs on custom
| Ubuntu with a year old kernel
| simlevesque wrote:
| CUDA
| zamadatix wrote:
| Theoretically it has slightly better memory bandwidth, (you are
| supposed to get) the Nvidia AI software ecosystem support out
| of the box, and you can use the 200G NIC to stick 2 together
| more efficiently.
|
| Practically, if the goal is 100% about AI and cloud isn't an
| option for some reason, both options are likely "a great way to
| waste a couple grand trying to save a couple grand" as you'd
| get 7x the performance and likely still feel it's a bit slow on
| larger models using an RTX Pro 6000. I say this as a Ryzen AI
| Max+ 395 owner, though I got mine because it's the closest
| thing to an x86 Apple Silicon laptop one can get at the moment.
| aseipp wrote:
| I'm not yet using mine for ML stuff because there are still a lot
| of various issues like this post outlined. But I am using mine as
| an ARM dev system in the meantime, and as a "workstation" it's
| actually quite good. The Cortex-X925 cores are Zen5 class in
| performance and it is overall an absolute unit for its size, I'm
| very impressed that a standard ARM core is pushing this level of
| performance for a desktop-class machine. I thought about buying a
| new Linux desktop recently, and this is good enough I might just
| plug it into a monitor and use it instead.
|
| It is also a standard UEFI+ACPI system; one Reddit user even
| reported that they were able to boot up Fedora 42 and install the
| open kernel modules no problem. The overall delta/number of
| specific patches for the Canonical 6.17-nvidia tree is pretty
| small when I looked (the current kernel is 6.11). That and the
| likelihood the consumer variant will support Windows hopefully
| bodes well for its upstream Linux compatibility, I hope.
|
| To be fair, most of this also true of Strix Halo from what I can
| tell (most benchmarks put the DGX furthest ahead at prompt
| processing and a bit ahead at raw token output. But the software
| is still buggy and Blackwell is still a bumpy ride overall, so it
| might get better). But I think it's mostly the pricing that is
| holding it back. I'm curious what the consumer variant will be
| priced at.
| eitally wrote:
| One of my colleagues wrote a first impressions blog post last
| week. It's from our company's perspective, but is a solid
| overview of the product and intended capabilities, from the POV
| of an AI developer or data scientist.
|
| https://www.anaconda.com/blog/python-nvidia-dgx-spark-first-...
| victor106 wrote:
| < The CPU memory is the same as the GPU memory and is much
| larger than any other discrete GPU available in a desktop. That
| means much larger datasets and bigger models can be run locally
| than would be possible otherwise.
|
| Isin't this the same architecture that the Mx from Apple
| implements from a memory perspective?
| MomsAVoxell wrote:
| So, it seems like this makes the DGX a viable ARM-based
| workstation, for those of us who need/want such a thing, while
| also offering a relatively decent AI/ML environment.
|
| Two things need to happen for me to get excited about this:
|
| 1. It stimulates other manufacturers into building their own DGX-
| class workstations.
|
| 2. This all eventually gets shipped in a decent laptop product.
|
| As much as it pains me, until that happens, it still seems like
| Apple Sillicon is the more viable option, if not the most
| ethical.
| gjsman-1000 wrote:
| NVIDIA, ethical?
| bigyabai wrote:
| My heart goes out to all the gamers who discovered they were
| chopped liver during the crypto boom.
|
| Besides that though, I don't see how Nvidia is particularly
| non-ethical. They cooperate with Khronos, provide high-
| quality Linux and BSD drivers free of charge, and don't
| deliberately block third parties from writing drivers to
| support new standards. From a relativist standpoint that's as
| sanctimonious as server hardware gets.
| cramsession wrote:
| They make significant investments in Israel and even said
| they'd build a new factory there. It doesn't get any less
| ethical than that!
| bigyabai wrote:
| American tech leaders often have no other choice. In most
| states you can be sued for boycotting, divesting or
| sanctioning Israel for any reason. If you acquire a
| company with outstanding obligations to Israel, your only
| option is to fulfill them.
|
| Specifically WRT Mellanox, Nvidia's behavior was more
| petty than callous.
| cramsession wrote:
| Who would sue them for boycotting and divesting from
| Israel? This is a false statement (or our democracy is in
| a _much_ dire state than advertised!).
| CamperBob2 wrote:
| Google _anti-boycott laws_.
|
| And yes... yes it is.
| agoodusername63 wrote:
| I thought about how to reply to this for a minute and then
| realized that I'm so desensitized by American tech companies
| that all the nonsense NVIDIA gets up to to maintain their
| economic position barely registers to me anymore.
| semessier wrote:
| Nvidia products including from the GPU/CUDA libraries world, the
| NICs and switches tend to feel like MVP frequently. It works in
| some cases, hopefully in the end but they are far from polished
| products without rough edges.
| pertymcpert wrote:
| This article is AI garbage:
|
| ARM64 Architecture: Not x86_64 (limited ML ecosystem maturity) No
| PyTorch wheels for ARM64+CUDA (must use Docker) Most ML tools
| optimized for x86
|
| No evidence for any of this whatsoever. The author just asked
| Claude/claude code to write their article and it just plain
| hallucinated some rubbish.
| bradfa wrote:
| Aarch64 and CUDA has been a thing for many years on Jetson
| boards. Claiming CUDA is immature on arm is very strange.
| furyofantares wrote:
| We're getting slopped every day now and upvoting it.
___________________________________________________________________
(page generated 2025-10-26 23:00 UTC)