[HN Gopher] Complete hardware and software setup for running Dee...
___________________________________________________________________
Complete hardware and software setup for running Deepseek-R1
locally
Author : olalonde
Score : 214 points
Date : 2025-01-29 14:56 UTC (8 hours ago)
(HTM) web link (twitter.com)
(TXT) w3m dump (twitter.com)
| _giorgio_ wrote:
| > the generation speed on this build is 6 to 8 tokens per second
|
| > ...if you want Q8 you'll need >700GB of GPU memory, which will
| probably cost $100k+
| alias_neo wrote:
| I clicked through because that $6000 price tag seemed insane,
| achievable even.
|
| Now it makes sense.
|
| Still undecided how I feel about having the ability to use all
| that quality in the full size model if one could only retrieve
| it at 6-8 tokens per second.
| permanent wrote:
| -- copied
|
| Complete hardware + software setup for running Deepseek-R1
| locally. The actual model, no distillations, and Q8 quantization
| for full quality. Total cost, $6,000. All download and part links
| below:
|
| Motherboard: Gigabyte MZ73-LM0 or MZ73-LM1. We want 2 EPYC
| sockets to get a massive 24 channels of DDR5 RAM to max out that
| memory size and bandwidth. https://t.co/GCYsoYaKvZ
|
| CPU: 2x any AMD EPYC 9004 or 9005 CPU. LLM generation is
| bottlenecked by memory bandwidth, so you don't need a top-end
| one. Get the 9115 or even the 9015 if you really want to cut
| costs https://t.co/TkbfSFBioq
|
| RAM: This is the big one. We are going to need 768GB (to fit the
| model) across 24 RAM channels (to get the bandwidth to run it
| fast enough). That means 24 x 32GB DDR5-RDIMM modules. Example
| kits: https://t.co/pJDnjxnfjg https://t.co/ULXQen6TEc
|
| Case: You can fit this in a standard tower case, but make sure it
| has screw mounts for a full server motherboard, which most
| consumer cases won't. The Enthoo Pro 2 Server will take this
| motherboard: https://t.co/m1KoTor49h
|
| PSU: The power use of this system is surprisingly low! (<400W)
| However, you will need lots of CPU power cables for 2 EPYC CPUs.
| The Corsair HX1000i has enough, but you might be able to find a
| cheaper option: https://t.co/y6ug3LKd2k
|
| Heatsink: This is a tricky bit. AMD EPYC is socket SP5, and most
| heatsinks for SP5 assume you have a 2U/4U server blade, which we
| don't for this build. You probably have to go to Ebay/Aliexpress
| for this. I can vouch for this one: https://t.co/51cUykOuWG
|
| And if you find the fans that come with that heatsink noisy,
| replacing with 1 or 2 of these per heatsink instead will be
| efficient and whisper-quiet: https://t.co/CaEwtoxRZj
|
| And finally, the SSD: Any 1TB or larger SSD that can fit R1 is
| fine. I recommend NVMe, just because you'll have to copy 700GB
| into RAM when you start the model, lol. No link here, if you got
| this far I assume you can find one yourself!
|
| And that's your system! Put it all together and throw Linux on
| it. Also, an important tip: Go into the BIOS and set the number
| of NUMA groups to 0. This will ensure that every layer of the
| model is interleaved across all RAM chips, doubling our
| throughput. Don't forget!
|
| Now, software. Follow the instructions here to install llama.cpp
| https://t.co/jIkQksXZzu
|
| Next, the model. Time to download 700 gigabytes of weights from
| @huggingface! Grab every file in the Q8_0 folder here:
| https://t.co/9ni1Miw73O
|
| Believe it or not, you're almost done. There are more elegant
| ways to set it up, but for a quick demo, just do this. llama-cli
| -m ./DeepSeek-R1.Q8_0-00001-of-00015.gguf --temp 0.6 -no-cnv -c
| 16384 -p "<|User|>How many Rs are there in
| strawberry?<|Assistant|>"
|
| If all goes well, you should witness a short load period followed
| by the stream of consciousness as a state-of-the-art local LLM
| begins to ponder your question:
|
| And once it passes that test, just use llama-server to host the
| model and pass requests in from your other software. You now have
| frontier-level intelligence hosted entirely on your local
| machine, all open-source and free to use!
|
| And if you got this far: Yes, there's no GPU in this build! If
| you want to host on GPU for faster generation speed, you can!
| You'll just lose a lot of quality from quantization, or if you
| want Q8 you'll need >700GB of GPU memory, which will probably
| cost $100k+
| ijk wrote:
| I'd assume that the existing llama.cpp ability to split layers
| out to the GPU still applies, so you could have some fraction
| in VRAM and speed up those layers.
|
| The memory bandwidth might be an issue, and it would be a
| pretty small percentage of the model, but I'd guess the speedup
| would be apparent.
|
| Maybe not worth the few thousand for the card + more
| power/cooling/space, of course.
| SlavikCA wrote:
| 2x CPU system may be slower for LLM than 1x CPU system.
|
| Because in 2x CPU system, the model may have to be passed via
| NUMA, which has 10% - 30% of memory bandwidth bandwidth
| ryao wrote:
| It would be interesting to see the performance if someone
| built a single socket version. I have a parts list here if
| anyone wants to try it:
|
| https://news.ycombinator.com/item?id=42868360
| kfcjligmom wrote:
| Does it really take these things that long to tell me how many Rs
| are in strawberry?
| RistrettoMike wrote:
| Amen.
| the_sleaze_ wrote:
| behind the internet bubble was the modern internet.
| recursive wrote:
| Sick burn bro
| kfcjligmom wrote:
| It's a genuine question. I haven't used AI before. The linked
| video is maybe the first time I've seen it in action. I'm
| underwhelmed.
|
| Is this at all a fair representation, or did it lock up or
| something? Or is it particularly bad at this type of question
| for some reason?
|
| Surely this can't be the mighty AI that has the whole world
| going bananas..?
| recursive wrote:
| I can't find the video, but I have an idea what's going on.
| The quality of the output from these things is very
| inconsistent. Sometimes it seems to have surprising
| "insight". Sometimes it's incoherent nonsense. People that
| want to be impressed cherry-pick the good results. People
| that expect things to "just work" notice the duds more. You
| can find very good and very bad results. If you're a
| starry-eyed technologist, you'll publicize the good
| outliers because of the potential they represent. If you're
| a skeptic, you'll point out the seemingly brain-dead
| failures.
| __MatrixMan__ wrote:
| This was posted yesterday:
| https://news.ycombinator.com/item?id=42860770
|
| And since X sucks today just about as much as it sucked
| yesterday:
| https://nitter.poast.org/carrigmat/status/188424436990727810...
| alias_neo wrote:
| I don't know if it's just me but I can't seem to view the
| nitter link, says the tweet isn't found, I can't use the OP
| Twitter link because I can't see anything without logging in.
| __MatrixMan__ wrote:
| I ran across a thread once indicating that the guy who hosts
| that instance has a bit of a fight on his hands keeping it
| usable. Frequently fights hoards of bots, that sort of thing.
| Your signal probably got mistaken for some noise.
|
| We need Nitter and BitTorrent to have a baby, p2p so we can
| share the load.
| ForOldHack wrote:
| They did, and it works well ipfs.
| __MatrixMan__ wrote:
| IPFS would be a good layer to build this on top of, but
| IPFS itself doesn't handle the correspondence between URL
| and CID. As it is, web DOMs are too variable to handle
| via content addressing.
|
| This would involve eliminating parts of the page that are
| unlikely to be stable (i.e. ads, pagination based on
| screen size, session specific details like usernames),
| and use that for the CID. That way the only time that
| users end up storing separate copies of the page is when
| it has actually been changed substantially. You'd
| probably also need some code to fluff the normalized
| content back up into something that makes sense for the
| device you're reading it on.
| alias_neo wrote:
| Ah that would make sense because the CSS also didn't load
| in the page
|
| It's probably for the same reason I can't view many other
| sites like Reddit, Imgur etc, because our VPN exit IP is
| from a cloud hosting provider.
| PixelForg wrote:
| The xcancel instance works for me
| https://xcancel.com/carrigmat/status/1884244369907278106
| johnmaguire wrote:
| twiiit.com (three i's) will act as a round-robin to a working
| Nitter instance.
| mrinterweb wrote:
| Thank you for the alternative to X. I feel somehow complicit in
| supporting someone likely aligned with something unspeakable
| any time I click an X link.
| tossit444 wrote:
| The irony is this mirror's host is a far-right fediverse
| instance.
|
| https://www.dailydot.com/debug/poast-hack-leaked-emails-dms/
| horsawlarway wrote:
| 6 to 8 t/s is decent for a GPU-less build, but speaking from
| experience... it will feel slow. Usable - but slow.
|
| Especially because these models <think> for a long bit before
| actually answering, so they generate for a longer period. (to be
| clear - I find the <think> section useful, but it also means
| waiting for more tokens)
|
| Personally - I end up moving down to lower quality models
| (quants/less params) until I hit about 15 tokens/second. For
| chat, that seems to be the magical spot where I stop caring and
| it's "fast enough" to keep me engaged.
|
| For inline code helpers (ex - copilot) you really need to be up
| near 30 tokens/second to make it feel fast enough to be helpful.
| k__ wrote:
| Would DeepSeek-V3 be cheaper?
| sigmoid10 wrote:
| Only in the sense that it will generate fewer tokens.
| guerrilla wrote:
| This is the same I get from a Core i5-9400 for smaller models.
| Is there no prosumer board that can take that much RAM? There
| must be a ThreadRipper that can do it, right? Why did he need
| EPYC?
| hollerith wrote:
| It's not just the amount of RAM: RAM bandwidth also matters.
| UncleOxidant wrote:
| RAM channels is key here. That board has 24 channels of DDR5
| RAM. A lot of lowend boards only have 2 channels. Most come
| in at 4. Some high end boards have 8, so 24 channels is quite
| wide.
| UncleOxidant wrote:
| Is a token a character or a word? I don't think I can read 30
| words/second so 8/second would be fine. But if it's characters
| then yes, it would be slow.
| neoromantique wrote:
| Something in between. example: https://gpt-tokenizer.dev/
| UncleOxidant wrote:
| Looks like for the most part it's words, but some longer
| words get broken at prefix/suffix boundaries.
| ryao wrote:
| It typically somewhere between 3 to 5 characters.
| horsawlarway wrote:
| It varies a lot by model, tokenizer, and language.
|
| For DeekSeek 1 token ~= 3 English characters.
|
| See: https://api-docs.deepseek.com/quick_start/token_usage/
|
| It comes out to around 1-3 words/second. This is not so slow
| that it's maddening (ex 2 token/second is frustratingly slow,
| like walk away and make coffee while it's answering slow),
| but it's still slow enough to make it hard to functionally
| use and not break flow state. You get bored and distracted
| reading at that pace.
| SkyPuncher wrote:
| I've been with RooCode the slower token speeds are acceptable.
|
| I'll ask it to do a task, then it will do it in a few steps and
| notify me when it's done. I use that time to take care of
| something else.
| sgt101 wrote:
| Hmm - if only macpro's had memory that could be upgraded like in
| the olden days.
| SV_BubbleTime wrote:
| If it could, it wouldn't be so tightly coupled and then
| identical to everything else out there.
| bitwize wrote:
| It also wouldn't be as fast. Soldered RAM is faster than
| socketed RAM. RAM that is in the CPU package is faster still.
|
| Apple is going in the direction of total integration. Louis
| Rossmann and iFixit will hem and haw, but soon there will be
| a MacBook whose motherboard, besides cooling, PSU, and ports,
| consists simply of a single component that houses CPU, GPU,
| RAM, storage, I/O port controllers, radios (wifi, Bluetooth,
| etc.), firmware for all of the above, and a security module
| plus keys, all directly on the CPU bus, and it will be
| glorious. It will absolutely lap any PC laptop, and in
| single-core performance will smoke even high-end AMD Epyc
| beastbox builds because of the aggressive elimination of
| inter-component latency.
|
| But there won't be a need to fix it. If it breaks you just
| recycle it and buy new, being sure to sync your data back
| from Apple Cloud -- but it probably won't break. Kinda like
| how unibody cars are both safer and more reliable than the
| much more fixable cars of the 60s, even if they crumple like
| tinfoil and must be totaled upon experiencing any sort of
| impact.
| sgt101 wrote:
| I wonder if that's really set in stone or just a stance that
| suited Apples marketing for a few years. Maybe there's a
| business case for some bigger memory Macs now!
| aurareturn wrote:
| Part of the reason it has high memory bandwidth is that it is
| soldered.
| qingcharles wrote:
| Wow, I had to go confirm this for myself. This sucks. They max
| out at 192GB on the current M2 line, which is absurd.
|
| Hopefully this year's M4 Ultra systems will at least allow a
| much higher top spec.
| drooby wrote:
| The amount of compute required for R1 to determine that
| strawberry has 3 Rs is hilarious..
|
| I'm not so convinced that the Nvidia panic is justified.
| disgruntledphd2 wrote:
| I mean, to be fair, given the way tokenization works, this
| isn't that surprising.
|
| I do agree that it's very funny though.
| cookingrobot wrote:
| I think it's like asking someone "how many times does your
| pen change direction when writing the word strawberry on
| paper".
|
| You need to think really hard to get to an answer, because
| that's more fine grained than the way you usually think about
| words and letters.
| layer8 wrote:
| I wondered how I should feel about that thought process.
| However, I might have a similar thought process when
| inebriated.
| ceejayoz wrote:
| The Nvidia panic stems, in part, from the possibility that this
| is just the first of many potential significant optimization
| leaps.
| drooby wrote:
| Idgi, optimization leaps are virtually guaranteed.
|
| I think this is a deeper question about the bounds of human
| desire... which seem virtually limitless. We seem to have an
| unlimited appetite for answering questions on the complexity
| of existence. Pair that with arms race issues, and you have
| an obvious need for massive compute regardless of how
| efficient the algorithms are.
| kovacs wrote:
| That's fair but look at the assumptions currently built
| into NVDA's stock price. Hard to say for certain but to
| some it's priced as if it has a monopoly on all things AI
| for a decade. If you can spend $6K (and less in the future)
| on a full system that runs a model for you where does that
| leave the assumptions baked into that stock price? I'm too
| lazy to have come up with a model myself but off the cuff
| it seems like there might be some dislocation in a lot of
| assumptions. I dunno. I'm kind of a luddite because of the
| .com bubble and this has the same feel.
| ryao wrote:
| Jevons' paradox seems relevant:
|
| https://en.wikipedia.org/wiki/Jevons_paradox
| UncleEntity wrote:
| > The amount of compute required for R1 to determine that
| strawberry has 3 Rs is hilarious..
|
| How about the amount of compute to take a buggy hextree
| implementation I've been poking at for a few years and
| completely rewrite it into a fully functioning implementation
| (with test cases) without me having to write a single line of
| code? Well...other than me having to break out the printf
| debugger to help with tracking down some deep bugs that is.
|
| I've been trying to get the original implementation to work
| correctly for so long I don't even remember what I wanted to
| use it for in the first place, I just mess with it a bit here
| and there when I have nothing better to do.
|
| And that's just me, a half-assed self-taught junior woodchuck
| coder, playing around to see what all the hype is about.
|
| I can only imagine that I'm giving them valuable training data
| as some of the bugs were _very_ deep and took a whole lot of
| 'thinking' to track down the root cause. It does take a bit of
| prodding to get it to look in the right place but so far it has
| found and fixed them all. The last bug was an overflow in the
| tree iterator's stack it uses to track state across iterations
| that I was concerned would time out as it was thinking for a
| long, long time.
|
| I'm not really one to defend the robots but this one is
| actually uesful.
|
| --edit--
|
| Oh... I guess that's a meme now. Nothing to see here, move
| along...
| indeed30 wrote:
| So, can somebody in the know speculate about how Deepseek (or
| OpenAI, or whoever really) is actually running their API?
|
| If I wanted to run a production-grade service using the full
| Deepseek model, with good tokens/sec and the ability to serve
| concurrent requests, what sort of hardware are we looking at?
| MurkyLabs wrote:
| Racks and Racks of servers (likely nVidia HGX H100/H200 8-GPU
| server) connected at at least 100GB (but more likely 400gb and
| 800gb) links. The servers alone start at about $350k. Then you
| need to supply power, cooling, networking and a technical team
| to support the program.
| ssahoo wrote:
| I have been using deepseek-r1:1.5b/8b on a MacBook m1 pro max.
| The performance has been pretty good, at par with the o1. So last
| 2 days i have been running them side by side. I'm satisfied with
| the results and performance. That's a not very cheap 2k hardware
| and it does the job.
| kgwgk wrote:
| It does the job of running something which is derived from -
| but it's not - the deepseek r1 model. They say the real thing
| is way better (as it should).
| jibbers wrote:
| M1 Pro or M1 Max?
| mysteria wrote:
| Would adding a single GPU help with prompt processing here? When
| I run a llama.cpp GPU build with no layers offloaded prompt
| processing is still way faster as all the matrix multiplies are
| done on the accelerator. The actual memory bound inference
| continues to run on the CPU.
|
| Since tensor cores are so fast you still come out ahead when you
| send the weights over PCIE to the GPU and return the completed
| products back to main memory.
| ryao wrote:
| Maybe. It likely depends on whether GEMM can operate at/near
| full speed with streaming weights over PCI-E. I am not sure how
| to stream the weights over PCI-E for use by CUDA/PTX code
| offhand. It would be a R&D project.
| ComputerGuru wrote:
| CPU-only is, very unfortunately, infeasible for reasoning models.
| This setup would be great for deepseek v3 or (more fittingly) the
| 405B llama 3.1 model, but 6-7 tokens per second on a reasoning
| model is 100% getting (well) into seconds-per-token territory if
| you consider only the final answer.
|
| (You don't have to take it from me: if CPU were good enough,
| AMD's valuation would be 100x its current value.)
| samvher wrote:
| Given what we just saw in terms of the DeepSeek team squeezing
| a lot of extra performance out of more efficient implementation
| on GPU, and the model still being optimized for GPU rather than
| CPU - is it unreasonable to think that in the $6k setup
| described, some performance might still be left on the table
| that could be squeezed out with some better optimization for
| these particular CPUs?
| telotortium wrote:
| Maybe a little, but FLOPs and memory bandwidth don't lie.
| snovv_crash wrote:
| No, because the bottleneck is RAM bandwidth. This is already
| quantized and otherwise is essentially random so can't be
| compressed in any meaningful way.
| ryao wrote:
| The answer to your question is yes. There is an open issue
| with llama.cpp about this very thing:
|
| https://github.com/ggerganov/llama.cpp/issues/11333
|
| The TLDR is that llama.cpp's NUMA support is suboptimal,
| which is hurting performance versus what it should be on this
| machine. A single socket version likely would perform better
| until it is fixed. After it is fixed, a dual socket machine
| would likely run at the same speed as a single socket
| machine.
|
| If someone implemented a GEMV that scales with NUMA nodes
| (i.e. PBLAS, but for the data types used in inference), it
| might be possible to get higher performance from a dual
| socket machine than we get from a single socket machine.
| qingcharles wrote:
| It honestly depends on your use case. I often run bigger,
| slower models on my PC and let them just tootle along in the
| background grinding out their response while I work on
| something else.
| grahamj wrote:
| Yeah this is what I was thinking, or maybe use smaller models
| to work on the prompt then fire it off to the biggie while
| you do something else.
| brandall10 wrote:
| "This setup would be great for deepseek v3 or (more fittingly)
| the 405B llama 3.1 model"
|
| v3 yes w/ 37B activated params, yes, but terrible on 405B as
| it's a dense model.
| niwtsol wrote:
| Is anyone aware of a site that shows various builds or off the
| shelf systems (new and old) and how they handle various models?
| Like I'd love to see the above vs a Mac Studio 198gb vs an old m1
| studio vs other models. I don't have $6000, but what is a good
| happy medium I could get to on a used system with a smaller
| model.
| mv4 wrote:
| not everyone has X access, so
|
| [0] direct link with login
| https://x.com/carrigmat/status/1884244369907278106
|
| [1] alt link without login but with ads
| https://threadreaderapp.com/thread/1884244369907278106.html
|
| Edit: someone posted xcancel link above - no ads
| https://xcancel.com/carrigmat/status/1884244369907278106
| nexus_six wrote:
| What would the context length look like for this setup? How quick
| would the 6-7 tps degrade once you hit say 20k tokens?
| Jotalea wrote:
| My Ryzen 7 3700U is giving it all with the 7b model.
| 999900000999 wrote:
| I'm so hyped for this.
|
| It's going to take some time, but the farce is gone. We'll have
| parity to Chat GPT on consumer hardware soon enough. 6k is still
| too much. I suspect the community will be able to get this down
| to 2K.
|
| I'm tempted to cancel my Chat GPT subscription!
| seanp2k2 wrote:
| Orders of magnitude make the difference. "Consumer-level" will
| be once it's around $300-700ish in a nice little box like an
| Intel NUC or similar. Ubiquiti just did this with their AI-Key
| to help classify stuff on their video camera surveillance
| platform.
| 999900000999 wrote:
| 2K is reasonable.
|
| Not every single person needs to have it. But if someone in
| your circle has the needed hardware...
| binary132 wrote:
| My theory is that in the future we will have much more "friends
| circle cloud" type ops, where that hardware cost is spread out
| among a small community and access is private. What it won't
| look like is every Tom, Dick, and Harry running their own $10k
| hardware to have a chuckle at the naughty jokes and grade-C+
| programmer IDE assistance offered by open-source LLMs.
| swiftcoder wrote:
| Does anyone have the performance delta between running this on a
| 768 GB setup like this where the whole thing fits in RAM, versus
| running it on an M4 Mac with the maxed out 128 GB?
| qingcharles wrote:
| My other question is.. can you jam 768GB of RAM into an M2
| Ultra Mac Pro?
|
| And M4 Ultra Mac Pros are probably only weeks away too.
| qingcharles wrote:
| OK, well, turns out you can't upgrade the RAM in the M2 Mac
| Pros and they top out at 192GB spec from factory, so that's
| that.
| ryao wrote:
| It might be possible to desolder the chips and solder
| larger capacity ones, but it is a risky thing to do,
| especially since there is no guarantee that it will work
| out.
|
| That said, a similar upgrade has been done on the raspberry
| pi 4, so it is theoretically possible:
|
| https://hackaday.com/2023/03/05/upgrade-ram-on-your-
| pi-4-the...
| twoodfin wrote:
| No, you cannot. All M-series Macs to date--including the Mac
| Pro--have RAM fixed at manufacturing.
| oynqr wrote:
| Running the ollama 671b 4 bit quant on a 7950X3D with 128GiB
| RAM, I get like 1-2 t/s.
| monobot12 wrote:
| If you don't mind a speed of 1 token per second, you can run the
| largest R1 model on a 2021 iMac, as I just did.
| jeffbee wrote:
| Are we speaking of a 2020-edition Intel 27" iMac or a 2021 M1?
| btbuildem wrote:
| Largest R1, as in the 671B? How do you accomplish that feat?
| oynqr wrote:
| Just do it? Llama.cpp doesn't load the entire thing into ram.
| It mmaps the file and the kernel takes care of the rest.
| 42772827 wrote:
| Is deepseek r-1 as fashioned here censored?
| m3kw9 wrote:
| This is more like an experiment more than practical use with the
| stated 6 toks/s. Paying 6gs for that and days of setup when the
| next model may come in a month
| JonChesterfield wrote:
| Working through this now. The directions are to download the
| contents of the Q8_0 at
| https://huggingface.co/unsloth/DeepSeek-R1-GGUF/tree/main. That
| turns out to be a git lfs repo. `git clone` followed by `git lfs
| pull` is downloading all of it which will have to do.
| (fetchinclude = DeepSeek-R1-Q8_0 seems to be limiting it to the
| directory of interest). If there's a cleverer way to get the
| files please reply - I looked for a torrent and failed to find
| one.
|
| Not completely clear what changing numa nodes per socket from 1
| to 0 does, possibly gives linux less information about when to
| migrate threads across x64 cores? (didn't upset llvm compile time
| so I'll leave it on nsp0)
| kgwgk wrote:
| Can't you use the download icons in each object at
| https://huggingface.co/unsloth/DeepSeek-R1-GGUF/tree/main/De...
| ?
| JonChesterfield wrote:
| The icons resolve to things like https://huggingface.co/unslo
| th/DeepSeek-R1-GGUF/resolve/main... which wget understands.
| Presumably there's a greater-than-average risk of corruption
| in transit when the files are big and git does some sort of
| integrity checking that one would lose out on? It's the
| verify-local-data feature I'm really missing from torrent
| here.
| kgwgk wrote:
| I've never seen any << corruption in transit >> when
| downloading big files.
| ryao wrote:
| TCP checksums usually prevent corruption, unless the data
| is already corrupt prior to the checksum computation (which
| does happen rarely).
| wg0 wrote:
| Or - Have your own "OpenAI" at home. Train, fine-tune, distill in
| the cloud if and when necessary.
|
| Now this is basically the moment of Apache/ngnix being free and
| open source.
|
| You then have share hosting phenomenon out of it.
| morphle wrote:
| As this is HN, I'm curious if there is anyone here on HN who is
| interested in starting a business hosting these large open source
| LLM's?
|
| I just finished a test of running this Deepseek-R1 768 GB model
| locally on a cluster of computers with 800 GB/s memory bandwidth
| (faster than the machine in the twitter post) and I can now
| extrapolate to a cluster with 6000 GB/s aggregate memory
| bandwidth and I'm sure we can reach higher speeds than Groq and
| Cerebras [1] on these large models.
|
| We might even be cheap enough in OPEX to retrain these models.
|
| Would anyone with cofounder or commercial skills be willing to
| set up this hosting service with me, it will take less than $30K
| investment but could be profitable in weeks?
|
| [1]
| https://hc2024.hotchips.org/assets/program/conference/day2/7...
| ForOldHack wrote:
| The Azure(tm) and AWS version of rent-a-second are in the works
| as we speak. So yes, rent-a-brain/vegetable and no, I will bet
| you $40k you will not beat either AWS ot Microsoft to the
| punch. Zero chance of that. They will have their excess
| computational power with extremely discounted electric rates in
| place before Friday morning.
| morphle wrote:
| I think the important metric will be if we can compete
| against the price of AWS or Microsoft in running large LLMs,
| not their time to market. Competing on cost against
| overpriced hyperscalers is not very hard, and $30K is a small
| investment, not a gamble. If it would fail, worst case you
| would only loose $3000-$6500 or so.
| DoingIsLearning wrote:
| > $40K is a small investment, not a gamble. If it would
| fail, worst case you would only lose $3000-$6500 or so.
|
| As someone not familiar with investment sourcing or SME
| financing. Could you break down the maths/accounting? How
| do you go from sinking 40k in a business to losing 6.5k if
| you turn the lights off at the end?
| morphle wrote:
| You buy the hardware (48 servers), rent part of a
| colocation rack with a 10 Gbps or 100 Gbps internet
| transit link, get a payment processor, make a webpage and
| GitHub demo with the API. Break down: $3000 labour,
| $20.5K hardware, $800 monthly rental fees, $376 car fees.
| When you shut down within a year, the $20.5K popular off
| the shelf hardware can easily be sold for $17K, a fact
| you can check from 25 years of data.
|
| I would invest more than the initial $30K on optimization
| after the servers have found paying customers and thus
| have proven commercial viability. I would invest in
| software development, finetuning, retraining and above
| all reverse engineering GPU and neural engine instruction
| sets and adapting these open source models to the more
| than 2 quadrillion operations per second that these 48
| servers can do.
| seanp2k2 wrote:
| So, $0 budget for software dev / sales / support?
| morphle wrote:
| I broke down the first $30K investment cost for release
| of the online API product, that does not need further
| software development, sales or support.
|
| You would be wise to do the software development I
| mentioned, do more sales and support than was covered
| under my initial $3000 labour fee. But that you can pay
| for with the revenues, it would not be the initial
| investment to see if it is viable as a business.
| gloflo wrote:
| That's what the AI is for, no? /s
| tonyhart7 wrote:
| Well, when you run an AI company, you must test your
| product, right? What better way to test it than by
| building your own webpage, admin panel, etc.?
| DrScientist wrote:
| I wonder if the real market is actually bringing this stuff
| inhouse.
|
| Given the propensity for these big tech companies to hoover
| up/steal any information they can gather, running these
| models locally, with local fine tuning looks quite
| attractive.
| rrix2 wrote:
| > Given the propensity for these big tech companies to
| hoover up/steal any information they can gather
|
| at the end of the day you still have to sell this product
| to the sorts of companies that are far and away all
| microsoft 365/google workspace clients and we're gonna have
| to figure that out one day or another
| antupis wrote:
| Pretty much especially in Europe there is lots of big
| companies and public sector institutions that would pay
| serious EUR if they could run these.
| morphle wrote:
| Spot on! I concur most European business and public
| sector institutions would be eager to rent this because
| they are not allowed by law to use US datacenters like
| AWS or Azure.
| kiviuq wrote:
| That's not the only issue. They want a guarantee that the
| model wasn't trained on copyrighted material.
| TeMPOraL wrote:
| Now that _is_ a real feature for now. A lot of hesitation
| in embracing generative AI in large enterprises stems
| from uncertainty about copyright issue. Anyone who
| trained an o1-level model from scratch on public
| /properly licensed data only would be able to provide a
| _very_ valuable service to those enterprise customers.
|
| _However_ , if both training and operating costs of a
| DeepSeek-like model are as small as they are, the
| companies best able to offer this service are...
| Microsoft, Amazon and Google. And second best are...
| _teams inside the would-be customer enterprises
| themselves_. $6M to train and $6K to run is effectively
| _free_ for such companies; there is no moat here. The
| services that enterprise customers would happily buy
| instead of building are... operations, and assuming legal
| liability if the model turns out not to be safe from
| copyright infringement lawsuits. But those are exactly
| the services those companies are _already buying from
| Microsoft, Amazon and Google_.
| fulafel wrote:
| This would result in some refreshing models, I guess they
| would be trained mostly on out-of-copyright stuff from
| 75+ years ago and wouldn't have knowledge of the modern
| world.
|
| Maybe they could skin the robotic bureucrats in vintage
| scifi appearance as well to have the whole consistent
| experience when you go to the building permits bot, there
| could be small talk about the latest Beatles record etc.
| jhoechtl wrote:
| Not true for two reasons! Azure data center in europe is
| fine if thr data stays there and generally there is the
| EU-US Data Privacy Framework https://en.m.wikipedia.org/w
| iki/EU%E2%80%93US_Data_Privacy_F...
| themanmaran wrote:
| I'd wager this is the real market. Ship some company a
| server rack with Deepseek R1 for a $1M annual rental fee +
| upgrades to the latest models.
|
| _think inside the box_
| morphle wrote:
| Cerebras already does this.
| spacemanspiff01 wrote:
| I thought cerebras had moved to a cloud model so that
| they could more easily manage/patch their systems?
| morphle wrote:
| both. The cloud model is also for renting models across
| several Cerebras Wafer Scale Integrations.
| spacemanspiff01 wrote:
| Oh, they will deploy the racks at customers sites?
| morphle wrote:
| I don't know what their policy is now, but they talk
| about it in the Hot Chips 2024 presentation.
|
| I myself just have proof of a single customer having
| their own private Cerebras rack. There are rumors about
| several more customers with on-prem Cerebras.
| willseth wrote:
| AWS already has platforms for running and fine tuning OSS
| models that can run privately inside a VPC. If Azure and
| GCP don't have equivalent capabilities already, it is
| surely imminent. Seems pretty hard or impossible to beat
| cloud providers at their own game.
| HeatrayEnjoyer wrote:
| If the hardware isn't in your physical possession you
| can't know that your data isn't being hoovered up. You
| can't end to end encrypt compute tasks (homomorphic
| processing is fiendishly uneconomical).
| TeMPOraL wrote:
| True, but at this point we're leaving the realm of
| cryptography and theoretical infosec, and enter the realm
| of _real-world security_. In this realm, permissions are
| established by armies of lawyers across organizations and
| governments defining who can or cannot do things, and
| what happens when transgressions occur; here, "defense
| in depth" carries all the way to the threat of men with
| guns escorting you to jail.
|
| So it's true that you can't encrypt compute tasks of this
| type end-to-end, so you can't know if unauthorized
| parties mine your data. However, Microsoft is _very
| unlikely_ to mine your data (for "you" being e.g. any of
| the many multinational corporations that already run all
| their office work through Azure-hosted Outlook, Office,
| SharePoint, etc.), or to let others mine it, because if
| it ever came out, your customers' lawyers would be after
| you, your lawyers would be after Microsoft, and the whole
| thing would explode into a multiple-billion-dollars
| shitshow and might even get a government or two involved.
|
| That's the working assumption that makes Microsoft well-
| positioned to eat any fledgling self-hosted DeepSeek
| market in the business space. They already have things
| set up at a level that is trusted by governments as well
| as corporations in critical industrial sectors, with huge
| financial and legal exposure.
|
| (Presumably Google and Amazon are in a similar position
| here, though I've only seen this personally with
| Microsoft/Azure, so that's what I can comment on.)
| zie wrote:
| > "defense in depth" carries all the way to the threat of
| men with guns escorting you to jail.
|
| For contract breach civil crime like this, there is zero
| chance it ends with jail time.
| TeMPOraL wrote:
| That's the typical case, true - but for many (most?) of
| the big multinationals, the worst case scenario for a
| hack involves people dying or some piece of critical
| infrastructure exploding.
|
| On top of that, "everything is securities fraud" - and
| since that _does_ carry potential jail time, corporations
| generally try to avoid pissing off parties that would be
| able to frame a contract breach (and its consequences) in
| terms of investment fraud.
|
| EDIT:
|
| For starters, almost all data a multinational corporation
| generates and processes is subject to export control
| regulations, which are broad, full of special cases, vary
| over time, space and politics, and most importantly,
| _violations of them come with huge fines and criminal
| penalties_ [0] for both businesses and _individuals_
| involved. The only reason Microsoft can get a corporation
| like this to migrate to O365 and run their back-office in
| Azure cloud is by solid, tested contractual guarantees
| that the data will be processed in ways that will keep
| the customer compliant with applicable regulations. Now,
| I 'm not a lawyer, but it's not particularly hard to draw
| a line from "Microsoft snooping on enterprise customers"
| to securities fraud.
|
| I mean, even in context of hosting a DeepSeek derivative,
| we're talking about a cloud service offering enterprise
| customers secure training on company data. "Company data"
| may involve, e.g. detailed documentation or specs for
| software for designing advanced optical systems, which
| may sound benign until you make the connection[1]:
| "advanced optics" includes applications in advanced laser
| systems, which basically means weapons (e.g. ranging,
| missile targeting, anti-missile countermeasures).
| Obviously, regulators around the world (and the US in
| particular) would be _very unhappy_ to see such
| information crossing through the wrong borders. For both
| the affected customers and the cloud service, this is
| high stakes game; a random startup isn 't in a position
| to enter it.
|
| --
|
| [0] - E.g. in US, up to $1M per violation _and_ up to 20
| years in prison, possibly _at the same time_ ; see https:
| //www.bis.doc.gov/index.php/enforcement/oee/penalties.
|
| [1] - This was a real intro example used in export
| control training I went through some years ago.
| zie wrote:
| Yes technically you can go to prison for securities
| fraud, and everything _could_ be securities fraud, if you
| have multiple share holders and play in that sandbox.
|
| A small random startup is unlikely to play in the
| securities sandbox until they have enough resources to
| hire enough lawyers to keep themselves out of prison and
| the fines "reasonable"(i.e. not enough to incentivize
| actually doing something about the fine being imposed
| other than to at least temporarily stop doing the
| action).
|
| When was the last time securities fraud ended in jail
| time by any S&P 500 company? My quick web search returned
| no instances ever(but I could be wrong).
| threeseed wrote:
| > you can't know that your data isn't being hoovered up
|
| There is no evidence of this happening in the last 20
| years. None.
|
| And if there was it would be the complete unravelling of
| the entire cloud concept.
|
| So you're talking about solving a problem no one has.
| kristjansson wrote:
| If your data is too sensitive for AWS, you're in a
| different realm that most enterprise users.
| valiant55 wrote:
| This is what has me most excited. AI has it's limited uses
| for now but with the current requirement of handing over
| all your data to big brother it was not even worth
| considering. Now that on prem is reasonable and doesn't
| require you to beg Nvidia for H100s it might actually be
| usable.
| ryao wrote:
| Did you implement token generation for Deepseek R1 using PBLAS?
| rainclouds wrote:
| Hardware requirements? I think I can hit the memory bandwidth
| building from parts I have in my house. Maybe even 2x. Asking
| for fun not profit.
| morphle wrote:
| I'd love to visit your house then. You have 768-1400 GB DRAM
| with 6000 GB/s memory bandwidth? Nice house.
|
| In my house I currently have almost 900 GB/S memory bandwidth
| in aggregate but only 132 GB total DRAM.
| ForOldHack wrote:
| $6000. Oh that's impressive concedering my gaming PC cost all of
| $1100. What a deal! Can I get two? Or three? How about training?
| Is that free? Air? Is air free? Does the economic model ( profit,
| profita and profit ) depend on stealing published works? Is that
| intelligent?
|
| Once the brain dead greedy MBAs get involved, is just how much
| you can steal. It should all be sold short, as we watch the world
| burn.
| culi wrote:
| Hope you're doing okay...
|
| $6k is much less than the millions that would be required to
| run anything by OpenAI. And it's a first pass. It could get
| much lower by the end of the year
| erichocean wrote:
| Unless I'm missing something, the full 1TB bandwidth isn't used
| because the memory layout is wrong.
|
| But that's fixable.
|
| Since it's memory bound, it might be possible to reach 15 tok/sec
| with this build.
| r14c wrote:
| I'm curious, what's wrong with the memory layout? You mean the
| ollama settings?
| tantalor wrote:
| Off topic, but I'd be fine with not seeing direct links to
| Twitter on here anymore. It's not very useful or user friendly.
| Similar to Pinterest, Instagram, or TikTok.
|
| Screenshots or mirrors (without the login requirement) are okay.
| dkjaudyeqooe wrote:
| A "rollup" of a twitter thread through a 3rd party service is a
| good solution
| ryao wrote:
| Like this?
|
| https://threadreaderapp.com/thread/1884244369907278106.html
| faeyanpiraat wrote:
| This has more ads than content
| tingle wrote:
| Replacing 'twitter' in the URL by 'xcancel' can be helpful:
|
| https://xcancel.com/carrigmat/status/1884244369907278106
| joecool1029 wrote:
| poast's nitter instance works as well: https://nitter.poast.o
| rg/carrigmat/status/188424436990727810...
|
| (I use this firefox extension to automatically rewrite
| twitter links into it: https://addons.mozilla.org/en-
| US/firefox/addon/privacy-redir... )
| QuantumGood wrote:
| XCancel.com Chrome redirection extension:
| https://chromewebstore.google.com/detail/xcancelcom-
| redirect...
|
| Nitter Chrome redirection extension:
| https://chromewebstore.google.com/detail/nitter-
| redirect/moh...
| aceazzameen wrote:
| Or use the bluesky link:
| https://bsky.app/profile/carrigmat.bsky.social/post/3lgsoqsx...
| MisterTea wrote:
| Is nitter still a thing?
| ryao wrote:
| A better setup on paper would be this CPU and motherboard
| combination with 12x64GB DIMMs:
|
| https://www.newegg.com/p/N82E16819113866
|
| https://www.newegg.com/supermicro-h13ssl-nt-amd-epyc-9004-se...
|
| As for memory, these two kits _should_ work (both are needed for
| the full 12 DIMMs):
|
| https://www.newegg.com/owc-256gb/p/1X5-005D-001G0
|
| https://www.newegg.com/owc-512gb/p/1X5-005D-001G4
|
| Since it would be a 2DPC configuration, the memory would be
| limited to 4400MT/sec unless you overclock it. That would give
| 422.4GB/sec, which should be enough to run the full model at 11
| tokens per second according to a simple napkin math calculation.
| In practice, it might not run that fast. If the memory is
| overclocked, getting to 16 tokens per second might be possible
| (according to napkin math).
|
| The subtotal for the linked parts alone is $5,139.98. It should
| stay below $6000 even after adding the other things needed,
| although perhaps it would be more after tax.
|
| Note that I have not actually built this to know how it works in
| practice. My description here is purely hypothetical.
| bildung wrote:
| I think the point of the two socket solution is the doubled
| memory bandwith. You propose using just a single one of the
| same CPU, or am I missing something?
| ryao wrote:
| llama.cpp's token generation speed does not scale with
| multiple CPU sockets just like it does not scale with
| multiple GPUs. Matthew Carrigan wrote:
|
| > Also, an important tip: Go into the BIOS and set the number
| of NUMA groups to 0. This will ensure that every layer of the
| model is interleaved across all RAM chips, doubling our
| throughput. Don't forget!
|
| This does not actually make sense. It is well known that
| there is a penalty for accessing memory attached to a
| different CPU. You don't get more bandwidth from disabling
| the NUMA node information and his token generation
| performance reflects that. If there was a doubling effect
| from using two CPU sockets, he should be getting twice the
| performance, but he is not.
|
| Additionally, llama.cpp's NUMA support is suboptimal, so he
| is likely taking a performance hit:
|
| https://github.com/ggerganov/llama.cpp/issues/11333
|
| When llama.cpp fixes its NUMA support, using two sockets
| should be no worse than using one socket, but it will not
| become better unless some new way of doing the calculations
| is devised that benefits from NUMA. This might be possible
| (particularly if you can get GEMV to run faster using NUMA),
| but it is not how things are implemented right now.
| bildung wrote:
| Thanks! TIL.
| freeqaz wrote:
| Do you get more bandwidth at the cost of latency?
|
| Also how much would stuffing a GPU or 3 (3090/4090) improve
| speeds, even with heavy CPU layer offloading, or would the
| penalty be too big? I know in some cases you're swapping
| data into the GPU, but in others you're just doing parts on
| the CPU. I'm curious what the comparison for speed would
| be.
| ryao wrote:
| I would suspect the infinity fabric links are already
| saturated with the local RAM's memory bandwidth such that
| you will not get more by accessing another socket's RAM.
|
| Chips and Cheese suggests things are even worse than this
| as the per CCD bandwidth is limited to around 120GB/sec,
| which probably ruins the idea of using the 9015, as that
| only has 2 CCDs:
|
| https://old.chipsandcheese.com/2024/10/11/amds-turin-5th-
| gen...
|
| https://www.techpowerup.com/cpu-specs/epyc-9015.c3903
|
| Anyway, leveraging both sockets' memory bandwidth would
| require splitting the layers into partitions for each
| NUMA node and doing that partition's part of each GEMV
| calculation on the local CPU cores. PBLAS might be useful
| in implementing something like that.
|
| As for a speed up from using 3090/4090 cards, that is a
| bit involved to estimate. The model has 61 layers. The
| way llama.cpp works is that it will offload layers and
| the computation will move from device to device depending
| on where the layers are in memory. You would need to
| calculate roughly how long it takes for each device to do
| a layer. Then multiple by the number of layers processed
| by that device and sum across the devices. Finally,
| normalize to get the number of tokens per second and you
| will have your answer. DeepSeek R1 has 61 layers
| (although I think llama.cpp will say 62 due to the
| embedding layer if it counts for DeepSeek like it does
| for llama 3). It has 37GB of activated weights, so you
| can do 37GB / 61 / memory bandwidth to get the time per
| layer. You probably want to multiply by 1.25 as a fudge
| factor to account for the fact that these things never
| run at the full speed that these calculations predict.
| Then you can plug in these numbers into the earlier
| calculation I described to get your answer.
| mrbonner wrote:
| Is there a build that would allow me to run llama 3.3 local?
| Something around $2500 or below.
| wmf wrote:
| If you're talking about 3.3 70B Q4, any PC with 64 GB RAM could
| run it.
| mrbonner wrote:
| I can use RTX to speed the inference up. My budget is up to
| $2500.
| wmf wrote:
| Maybe you could run some layers on a 3090; I'm not sure how
| much speedup it would give.
| sheepscreek wrote:
| Update: Scratch that. Two of them together would only be able to
| run something half as big (~400B parameters) and cost as much as
| this rig. Maybe the next gen of DIGITS could do it. Keeping that
| in mind, this rig is pretty darn impressive for $6k!
|
| This is going to drop by half when Nvidia starts shipping DIGITS.
| I think we're all going to want one. It'll probably have a much
| bigger impact than Apple VisionPro, that costs the same.
|
| I can already think of using it as a much more intelligent local
| Siri/Alexa to control devices. It's something that can actually
| keep the kids engaged with useful trivia/knowlege (better than
| watching mindless trash on YT) or it can just humour me whenever
| I want - all without needing to worry about privacy.
| UncleOxidant wrote:
| > "Complete hardware + software setup for running Deepseek-R1
| locally. The actual model, no distillations, and Q8 quantization
| for full quality. Total cost, $6,000."
|
| These guys say they got really good results with 2.51-bit
| quantization of the original R1. The original has 671B params
| weighing in at 720GB - that's what they're running on this $6000
| setup. According to these guys[0] they get really good results at
| 2.51bit quantization which would be this model[1] which is still
| 671B params, but weighs in at 212GB.
|
| [0] https://unsloth.ai/blog/deepseekr1-dynamic [1]
| https://huggingface.co/unsloth/DeepSeek-R1-GGUF/tree/main/De...
| segmondy wrote:
| That's not what they are running. the dynamic quants are Q1,
| Q2. The $6000 is running Q8.
| UncleOxidant wrote:
| Yes, exactly. What I'm saying is that the blog referenced in
| [0] says you can get good results with 2.51 quantization. The
| $6000 rig is running Q8. You can probably get similar results
| with a lesser rig if you use the quantized model.
| benlivengood wrote:
| Do you actually need dual EPYCs? You can get Lenovo T620s on eBay
| for $1500 and fill them up with 1TB of RAM
| hollerith wrote:
| Yes, you need dual epycs:
| https://news.ycombinator.com/item?id=42868067
| Larrikin wrote:
| Bluesky link:
| https://bsky.app/profile/carrigmat.bsky.social/post/3lgsoqsx...
___________________________________________________________________
(page generated 2025-01-29 23:01 UTC)