[HN Gopher] NVIDIA DGX Spark In-Depth Review: A New Standard for...
___________________________________________________________________
NVIDIA DGX Spark In-Depth Review: A New Standard for Local AI
Inference
Author : yvbbrjdr
Score : 79 points
Date : 2025-10-14 01:07 UTC (21 hours ago)
(HTM) web link (lmsys.org)
(TXT) w3m dump (lmsys.org)
| SethTro wrote:
| Article doesn't seem to mention price which is $4,000 which makes
| it comparable to a 5090 but with 128GB of unified LPDDR5x vs the
| 5090's 32GB DDR7.
| CamperBob2 wrote:
| And about 1/4 the memory bandwidth, which is what matters for
| inference.
| threeducks wrote:
| More precisely, the RTX 5090 has a memory bandwidth of 1792
| GB/s, while the DGX Spark only has 273 GB/s, which is about
| 1/6.5.
|
| For inference, the DGX Spark does not look like a good
| choice, as there are cheaper alternatives with better
| performance.
| CamperBob2 wrote:
| My understanding is that the Jetson Thor is just as good a
| platform, and likely more readily available.
|
| Then there's the Mac Studio, which outdoes them in all
| respects except FP8 and FP4 support. As someone on Reddit
| put it: https://old.reddit.com/r/LocalLLaMA/comments/1n0xoj
| i/why_can...
| altspace wrote:
| I've been thinking the same... I have jetson Thor and
| only difference I can imagine is the capability to
| connect two DGX sparks together... but then I'd rather go
| for RTX pro 6000 instead of buying two DGX spark units,
| because I prefer the higher memory bandwidth, more Cuda
| cores, tensor cores and RT cores over 256 GB memory for
| my use case.
| nialse wrote:
| Well, that's disappointing since the Mac Studio 128GB is
| $3,499. If Apple happens to launch a Mac Mini with 128GB RAM it
| would eat Nvidia Sparks' lunch every day.
| newman314 wrote:
| Agreed. I also wonder why they chose to test against a Mac
| Studio with only 64GB instead of 128GB.
| yvbbrjdr wrote:
| Hi, author here. I crowd-sourced the devices for
| benchmarking from my friends. It just happened that one of
| my friend has this device.
| ggerganov wrote:
| FYI you should have used llama.cpp to do the benchmarks.
| It performs almost 20x faster than ollama for the gpt-
| oss-120b model. Here are some samples results on my
| spark: ggml_cuda_init: found 1 CUDA
| devices: Device 0: NVIDIA GB10, compute
| capability 12.1, VMM: yes | model
| | size | params | backend | ngl | n_ubatch |
| fa | test | t/s | |
| ------------------------------ | ---------: | ---------:
| | ---------- | --: | -------: | -: | --------------: |
| -------------------: | | gpt-oss 20B MXFP4 MoE
| | 11.27 GiB | 20.91 B | CUDA | 99 | 2048 |
| 1 | pp4096 | 3564.31 +- 9.91 | |
| gpt-oss 20B MXFP4 MoE | 11.27 GiB | 20.91 B
| | CUDA | 99 | 2048 | 1 | tg32 |
| 53.93 +- 1.71 | | gpt-oss 120B MXFP4 MoE |
| 59.02 GiB | 116.83 B | CUDA | 99 | 2048 | 1
| | pp4096 | 1792.32 +- 34.74 | | gpt-
| oss 120B MXFP4 MoE | 59.02 GiB | 116.83 B |
| CUDA | 99 | 2048 | 1 | tg32 |
| 38.54 +- 3.10 |
| yvbbrjdr wrote:
| I see! Do you know what's causing the slowdown for
| ollama? They should be using the same backend..
| alecco wrote:
| Dude, ggerganov is the creator of llama.cpp. Kind of a
| legend. And of course he is right, you should've used
| llama.cpp.
|
| Or you can just ask the ollama people about the ollama
| problems. Ollama is (or was) just a Go wrapper around
| llama.cpp.
| ilc wrote:
| Was. They've been diverging.
| __mharrison__ wrote:
| Curious to how this compares to running on a Mac.
| xs83 wrote:
| TTFT on a Mac is terrible and only increases as the
| context increases, thats why many are selling their M3
| Ultra 512GB
| rajatgupta314 wrote:
| Is this the full weight model or quantized version? The
| GGUFs distributed on Hugging Face labeled as MXFP4
| quantization have layers that are quantized to int8
| (q8_0) instead of bf16 as suggested by OpenAI.
|
| Example looking at blk.0.attn_k.weight, it's q8_0 amongst
| other layers:
|
| https://huggingface.co/ggml-org/gpt-
| oss-20b-GGUF/tree/main?s...
|
| Example looking at the same weight on Ollama is BF16:
|
| https://ollama.com/library/gpt-oss:20b/blobs/e7b273f96360
| xs83 wrote:
| Now this looks much more interesting! Is the top one
| input tokens and the second one output tokens?
|
| So 38.54 t/s on 120B? Have you tested filling the context
| too?
| ggerganov wrote:
| Yes, I provided detailed numbers here:
| https://github.com/ggml-org/llama.cpp/discussions/16578
| nialse wrote:
| Makes sense you have one of the boxes. What's your take
| on it? [Respecting any NDAs/etc/etc of course]
| moondev wrote:
| Just don't try to run a NCCL
| pjmlp wrote:
| Only if it runs CUDA, MLX / Metal isn't comparable as
| ecosystem.
|
| People that keep pushing for Apple gear tend to forget Apple
| has decided what industry considers industry standards,
| proprietary or not, aren't made available on their hardware.
|
| Even if Metal is actually a cool API to program for.
| thom wrote:
| It depends what you're doing. I can get valuable work done
| with the subset of Torch supported on MPS and I'm grateful
| for the speed and RAM of modern Mac systems. JAX support is
| worse but hopefully both continue to develop.
| omneity wrote:
| CUDA is equally proprietary and not an industry standard
| though, unless you were thinking of Vulcan/OpenCL which
| doesn't bring much in this situation.
| NewsaHackO wrote:
| CUDA isn't the industry standard? What is then?
| pjmlp wrote:
| Yes it is an industry standard, there is even a technical
| term for it.
|
| It is called _De facto standard_ , which you can check in
| your favourite dictionary.
| EnPissant wrote:
| A 5090 is $2000.
| sandworm101 wrote:
| Msrp, but try getting your hands on one without a bulk order
| and/or camping out in a tent all weekend. I have seen people
| in my area buying pre-biult machines as they often cost less
| than trying to buy an individual card.
| EnPissant wrote:
| It's not that hard to come across MSRP 5090s these days. It
| took me about a week before I found one. But if you don't
| want to put any effort or waiting into it, you can buy one
| of the overpriced OC models right now for $2500.
| adrian_b wrote:
| But you put in a $1500 PC (with 128 GB DRAM).
|
| Still, a PC with a 5090 will give in many cases a much better
| bang for the buck, except when limited by the slower speed of
| the main memory.
|
| The greater bandwidth available when accessing the entire 128
| GB memory is the only advantage of NVIDIA DGX, while a
| cheaper PC with discrete GPU has a faster GPU, a faster CPU
| and a faster local GPU memory.
| bilekas wrote:
| $4,000 is actually extremely competitive. Even for an at-home
| enthusiast setup this price is not our of reach. I was
| expecting something far higher, that said, nVidia's MSRP is
| something of a pipe dream recently so we'll see when it's
| actually released and the availability. Curious also to see how
| they may scale together.
| Xss3 wrote:
| A warning to any home consumer throwing money at hardware for
| AI (fair enough if you have other use cases)...
|
| Things are changing rapidly and there is a non insignificant
| chance that it'll seem like a big waste of money within 12
| months.
| eadwu wrote:
| For this form factor it will be likely ~2 years for the
| next one based on Vera CPU and whatever GPU. The 50W CPU
| will probably improve power efficiency.
|
| If SOCAMM2 is used it will still probably be at most near
| the range of 512/768 GB/s bandwidth, unless LPDDR6X /
| LPDDR7X or SOCAMM2 is that much better, SOCAMM on the DGX
| Station is just 384 GB/s w/ LPDDR5X.
|
| Form factor will be neutered for the near future, but will
| probably retain the highest compute for the form factor.
|
| The only way there will be a difference is if Intel or AMD
| pump their foot on the gas, which this makes maybe 2/3
| years of it, with another 2 years unless they have
| something cooking it isn't going to happen.
| Xss3 wrote:
| Software driven changes could occur too! Maybe the next
| model will beat the pants off of this with far inferior
| hardware. Or maybe itll be so amazing with higher
| bandwidth hardware that anyone running at less than
| 500gbs will be left feeling foolish.
|
| Maybe a company is working on something totally different
| in secret that we cant even imagine. The amount of PS
| thrown into this space at the moment is enormous.
| Tepix wrote:
| They're in a different ballback in memory bandwidth. The right
| comparison is the Ryzen AI Max 395 with 128GB DDR5-8000 which
| can be bought for around $1800 / 1750EUR.
| pixelpoet wrote:
| I wonder why they didn't test against the broadly available Strix
| Halo with 128GB of 256 GB/s memory bandwidth, 16 core full-fat
| Zen5 with AVX512 at $2k... it is a mystery...
| yvbbrjdr wrote:
| Hi, author here. I crowd-sourced the devices for benchmarking
| from my friends. It just happened that none of my friend has
| this device.
| EnPissant wrote:
| Something is wrong with your numbers: gpt-oss-20b and gpt-
| oss-120b should be much much faster than what you are seeing.
| I would suggest you familiarize yourself with llama-bench
| instead of ollama.
|
| Running gpt-oss-120b with a rtx 5090 and 2/3 of the experts
| offloaded to system RAM (less than half of the memory
| bandwidth of this thing), my machine gets ~4100tps prefill
| and ~40tps decode.
|
| Your spreadsheet shows the spark getting ~94tps prefill and
| ~11tps decode.
|
| Now, it's expected that my machine should slaughter this
| thing in prefill, but decode should be very similar or the
| spark a touch faster.
| yvbbrjdr wrote:
| We actually profiled one of the models, and saw that the
| last GeMM, which is completely memory bound, is taking a
| lot of time, which reduces the token speed by a lot.
| lostmsu wrote:
| The parent is right, the issue is on your side.
| hnuser123456 wrote:
| Your system RAM is probably 1/20th the VRAM bandwidth of
| the 5090 (way way less than half) unless you're running a
| workstation board with quad or 8 channel RAM, then it's
| only about 1/10th or 1/5th respectively.
| EnPissant wrote:
| I'm saying its less than half of this DGX Spark: dual
| channel DDR5-6000 vs quad channel LPDDR5-8000.
| EnPissant wrote:
| Strix Halo has the problem that prefill is incredibly slow if
| your context is not very small.
|
| The only thing that might be interesting about this DGX Spark
| is it's prefill manages to be faster due to better compute. I
| haven't compared the numbers yet, but they are included in the
| article.
| pixelpoet wrote:
| There are some benches on reddit:
| https://old.reddit.com/r/LocalLLaMA/comments/1o6163l/dgx_spa...
|
| tl;dr it gets absolutely smashed by Strix Halo, at half the
| price.
| mortsnort wrote:
| The strix halo can also be used as a capable gaming/dev Pac
| with your OS of choice.
| hank808 wrote:
| You guys that continue to compare DGX Spark to the Mac Studios,
| please remember two things:
|
| 1. Virtually every model that you'd run was developed on Nvidia
| gear and will run on Spark. 2. Spark has fast-as-hell
| interconnects. The sort of interconnects that one would want to
| use in an actual AI DC, so you can use more than one Spark at the
| same time, and RDMA, and actually start to figure out how things
| work the way they do and why. You can do a lot with 200 Gb of
| interconnect.
| pavlov wrote:
| It would be very interesting to read a tutorial on case 2.
| hank808 wrote:
| @pavlov here's the tutorial that you wanted.
| https://youtu.be/rKOoOmIpK3I?si=WgLTee3Kc1SnUbDZ
| m00x wrote:
| At best this is a cheap setup to test distributed
| training/inference code.
| nialse wrote:
| Also remember that the Mx Ultras have 2-3x the memory
| bandwidth. Looking at the benchmarks even Strix Halo seems to
| beat the Spark. Buying a 200 Gbps switch is $10k-$100k+ so
| don't imagine anyone actually will use the interconnect. The
| logical thing for Nvidia would be to sell a kit with three
| machines and cabling, and make it a ring with the dual ports
| per machine. Helps for some scenarios but not others with the
| 10 times slower network than memory bandwidth.
| miladyincontrol wrote:
| On another note to remember, you can also ring topology mac
| studios using TB5 for 120Gbps per link with four such ports,
| all using cheaply available cable
| moondev wrote:
| | Buying a 200 Gbps switch is $10k-$100k+
|
| $1,295.00
|
| https://www.balticnetworks.com/products/mikrotik-
| crs812-ddq-...
| limoce wrote:
| > ollama gpt-oss 120b mxfp4 1 94.67 11.66
|
| This is insanely slow given its 200+GB/s memory bandwidth. As a
| comparison, I've tested GPT OSS 120B on Strix Halo and it obtains
| 420tps prefill and >40tps decode.
| nialse wrote:
| Probably the quants have higher perplexity, but the Sparks
| performance seems to be lack lustre. The reviewer videos I've
| seen so far tries their best not to offend Nvidia or, rather,
| not break their contracts.
| andrewgleave wrote:
| Looks like MLX is not a supported backend in Ollama so the
| numbers for the Mac could be significantly higher in some cases.
|
| It would be interesting to swap out Ollama for LM Studio and use
| their built-in MLX support and see the difference.
| mwilcox wrote:
| Just get 5 Mac minis.
| OliverGuy wrote:
| How representative is this platform of the bigger GB200 and GB300
| chips?
|
| Could I write code that runs on Spark and effortlessly run it on
| a big GB300 system with no code changes?
| xs83 wrote:
| If you mean CUDA specific then yes. The biggest benefit of
| these machines over the others is the CUDA ecosystem and tools
| like cuDF, cuGraph etc
| egeres wrote:
| All three (GB10, GB200 and GB300) are part of the Blackwell
| family, which means they have Compute Capability >= 10.X. You
| could potentially develop kernels to optimize MoE inference
| (given the large available unified memory, 128Gb, it makes the
| most sense to me) with CUDA >= 12.9 then ship the fatbins to
| the "big boys". As many people have pointed out across the
| thread, the spark doesn't really has the best perf/$, it's
| rather a small portable platform for experimentation and
| development
| themgt wrote:
| M5 Macs may be launching as early as today. Inference should see
| a significant boost w/ matmul acceleration.
| aurareturn wrote:
| It isn't that good for local LLM inferencing. It's not designed
| to be as such.
|
| It's designed to be a local dev machine for Nvidia server
| products. It has the same software and hardware stack as
| enterprise Nvidia hardware. That's what it is designed for.
|
| Wait for M5 series Macs for good value local inferencing. I think
| the M5 Pro/Max are going to be very good values.
| NaomiLehman wrote:
| because of possible hardware-accelerated matmul in GPU cores?
| bn-l wrote:
| I wish I could run Linux on them (the m5)
| andrewstuart wrote:
| Nvidia always short changes its own products and stunts them in
| some way.
|
| No doubt that's present here too somehow.
|
| Gotta cut off something important so you'll spend more on the
| next more expensive product.
| whitehexagon wrote:
| I think my 2001 MBP M1 Pro is ~200GB/s memory bandwidth, but it
| handles qwen3:32b quite nicely, albeit maxed out at ~70W.
|
| I somehow expected the Spark to be the 'God in a Box' moment for
| local AI, but it feels like they went for trying to sell multiple
| units instead.
|
| I'd be more tempted by a 2nd hand 128GB M2 ultra at ~800GB/s but
| the prices here are still high, and I'm not sure the Spark is
| going to convince people to part with those, unless we see some
| M5 glutenous RAM boxes soon. An easy way for Apple to catch up
| again.
| incomingpain wrote:
| That memory bandwidth choked out their performance. How can you
| claim 1000 tflops if it's not capable of delivering it. Seems
| they chose to sandbag the spark in favour of the rtx pro 6000.
|
| I guess my next one I'm looking out for is the Orange Pi AI
| studio pro. Should have 192gb of ram, so able to run qwen3 235b,
| even though it's ddr4, it's nearly double the bandwidth of the
| spark.
| ta12653421 wrote:
| Two questions:
|
| a) what is the noise level? In that small box, it should be
| immense?
|
| b) how many frames do we get in Q3A at max. resolution and will
| it be able to run Crysis? ;-) LOL (SCNR)
| incomingpain wrote:
| GPT 120B is your goto model:
|
| DGX Spark
|
| pp - 1723.07/s
|
| tg - 38.55/s
|
| Ryzen AI Max+ 395
|
| pp - 711.67/s
|
| tg - 40.25/s
|
| Is it worth the money?
| jerlam wrote:
| "Metal foam" sounds cool but it just looks like a steel wool pad
| you would use for cleaning dishes.
| harias wrote:
| Helps with the cooling is my guess. Increased surface area
| jerlam wrote:
| Possibly if the case is being used as a heat sink, but in
| that case it would be unsafe to touch. I think it's just
| being used instead of a traditional mesh panel.
| egeres wrote:
| I'm pretty sure they just want to be coherent with the which
| has that "steel scrubber finish" on the hardware
|
| (photo for reference: https://www.wwt.com/api-
| new/attachments/5f033e355091b0008017...)
| richardczl wrote:
| Memory indeed would be an issue
___________________________________________________________________
(page generated 2025-10-14 23:02 UTC)