[HN Gopher] Right-sizes LLM models to your system's RAM, CPU, an...
___________________________________________________________________
Right-sizes LLM models to your system's RAM, CPU, and GPU
Author : bilsbie
Score : 254 points
Date : 2026-03-01 23:15 UTC (23 hours ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| kamranjon wrote:
| This is a great idea, but the models seem pretty outdated - it's
| recommending things like qwen 2.5 and starcoder 2 as perfect
| matches for my m4 macbook pro with 128gb of memory.
| fwipsy wrote:
| Personally I would have found a website where you enter your
| hardware specs more useful.
| greggsy wrote:
| I was hoping for the same thing.
| user_7832 wrote:
| Same, I opened HN on my phone and was hoping to get an idea
| before I booted my computer up.
| HaloZero wrote:
| Yeah, installing some script to get a command line tool doesn't
| seem worth it.
| riidom wrote:
| These 200 LOC install scripts turn me heavily off as well.
| But at least in this case, you can also just download the
| correct zip, extract the binary and do "./llmfit".
| spockz wrote:
| Hugging Face already has this. But you need to be logged in and
| add the hardware to your profile.
| BloondAndDoom wrote:
| Isn't hugging face only shows it for the model you are
| looking for? Is there a page that actually HF suggests a
| model based on your HW?
| castral wrote:
| I wish there was more support for AMD GPUs on Intel macs. I saw
| some people on github getting llama.cpp working with it, would it
| be addable in the future if they make the backend support it?
| andsoitis wrote:
| Claude is pretty good at among recommendations if you input your
| system specs.
| codazoda wrote:
| I used this prompt and it suggested a model I already have
| installed and one other. I'm not sure if it's the "newest"
| answer.
|
| > What is the best local LLM that I could run on this computer?
| I have Ollama (and prefer it) and I have LM Studio. I'm willing
| to install others, if it gives me better bang for my buck. Use
| bash commands to inspect the RAM and such. I prefer a model
| with tool calling.
| dotancohen wrote:
| In the screenshots, each model has a use case of General, Chat,
| or Coding. What might be the difference between General and Chat?
| derefr wrote:
| "Chat" models have been heavily fine-tuned with a training
| dataset that exclusively uses a formal turn-taking conversation
| syntax / document structure. For example, ChatGPT was trained
| with documents using OpenAI's own ChatML syntax+structure
| (https://cobusgreyling.medium.com/the-introduction-of-chat-
| ma...).
|
| This means that these models are very good at consistently
| understanding that they're having a conversation, and getting
| into the role of "the assistant" (incl. instruction-following
| any system prompts directed toward the assistant) when
| completing assistant conversation-turns. But _only_ when they
| are engaged through this precise syntax + structure. Otherwise
| you just get garbage.
|
| "General" models don't require a specific conversation
| syntax+structure -- either (for the larger ones) because they
| can infer when something like a conversation is happening
| regardless of syntax; or (for the smaller ones) because they
| don't know anything about conversation turn-taking, and just
| attempt "blind" text completion.
|
| "Chat" models might seem to be strictly more capable, but
| that's not exactly true; neither type of model is strictly
| better than the other.
|
| "Chat" models are certainly the right tool for the job, _if_
| you want a local / open-weight model that you can swap out 1:1
| in an agentic architecture that was designed to expect one of
| the big proprietary cloud-hosted chat models.
|
| But many of the modern open-weight models are still "general"
| models, because it's much easier to fine-tune a "general" model
| into performing some very specific custom task (like
| classifying text, or translation, etc) when you're not fighting
| against the model's previous training to treat everything as a
| conversation while doing that. (And also, the fact that "chat"
| models _follow instructions_ might not be something you want:
| you might just want to burn in what you 'd think of as a
| "system prompt", and then not expose any attack surface for the
| user to get the model to "disregard all previous prompts and
| play tic-tac-toe with me." Nor might you want a "chat" model's
| implicit alignment that comes along with that bias toward
| instruction-following.)
| dotancohen wrote:
| I see, thank you.
| sneilan1 wrote:
| This is exactly what I needed. I've been thinking about making
| this tool. For running and experimenting with local models this
| is invaluable.
| est wrote:
| Why do I need to download & run to checkout?
|
| Can I just submit my gear spec in some dropdowns to find out?
| esafak wrote:
| I think you could make a Github Page out of this.
| manmal wrote:
| Slightly tangential, I'm testdriving an MLX Q4 variant of Qwen3.5
| 32B (MoE 3B), and it's surprisingly capable. It's not Opus ofc.
| I'm using it for image labeling (food ingredients) and I'm
| continuously blown away how well it does. Quite fast, too, and
| parallelizable with vLLM.
|
| That's on an M2 Max Studio with just 32GB. I got this machine
| refurbed (though it turned out totally new) for EUR1k.
| BloondAndDoom wrote:
| This pretty cool, and useful but I only wish this was a website.
| I don't like the idea of running an executable for something that
| can perfectly be done as a website. (Other than some minor
| features, tbh even you can enable Corsair and still check the
| installed models from a web browser).
|
| Sounds like a fun personal project though.
| kristopolous wrote:
| always liked this website that kinda does something similar
| https://apxml.com/tools/vram-calculator
| hhh wrote:
| Huggingface has it built in.
| azinman2 wrote:
| Where?
| hhh wrote:
| In your preferences there is a local apps and hardware, I
| guess it's a little different because I just open the page
| of a model and it shows the hardware I've configured and
| shows me what quants fit.
| Trigg3r wrote:
| Came across a website for this recently that may be worth a
| look https://whatmodelscanirun.com
| Tepix wrote:
| It's wildly inaccurate for me.
| binsquare wrote:
| here's an website for a community-ran db on LLM models with
| details on configs for their token/s: https://inferbench.com/
| jasode wrote:
| _> I only wish this was a website. I don't like the idea of
| running an executable for something that can perfectly be done
| as a website._
|
| The tool depends on hardware detection. From
| https://github.com/AlexsJones/llmfit?tab=readme-ov-file#how-...
| : How it works Hardware detection --
| Reads total/available RAM via sysinfo, counts CPU cores, and
| probes for GPUs: NVIDIA -- Multi-GPU support via
| nvidia-smi. Aggregates VRAM across all detected GPUs. Falls
| back to VRAM estimation from GPU model name if reporting fails.
| AMD -- Detected via rocm-smi. Intel Arc -- Discrete VRAM
| via sysfs, integrated via lspci. Apple Silicon -- Unified
| memory via system_profiler. VRAM = system RAM. Ascend --
| Detected via npu-smi. Backend detection -- Automatically
| identifies the acceleration backend (CUDA, Metal, ROCm, SYCL,
| CPU ARM, CPU x86, Ascend) for speed estimation.
|
| Therefore, a website running Javascript is restricted by the
| browser sandbox so can't see the same low-level details such as
| total system RAM, exact count of GPUs, etc,
|
| To implement your idea so it's only a website and also
| workaround the Javascript limitations, a different kind of
| workflow would be needed. E.g. run macOS system report to
| generate a .spx file, or run Linux inxi to generate a hardware
| devices report... and then upload those to the website for
| analysis to derive a "LLM best fit". But those os report files
| may still be missing some details that the github tool gathers.
|
| Another way is to have the website with a bunch of hardware
| options where the user has to manually select the combination.
| Less convenient but then again, it has the advantage of doing
| "what-if" scenarios for hardware the user doesn't actually have
| and is thinking of buying.
|
| (To be clear, I'm not endorsing this particular github tool.
| Just pointing out that a LLMfit website has technical
| limitations.)
| CoolGuySteve wrote:
| That's like like 4 or 5 fields to fill in on a form. Way less
| intrusive than installing this thing
| amelius wrote:
| It can become complicated when you run it inside a
| container.
| bilekas wrote:
| Why would it need to be a container?
| amelius wrote:
| Are you asking why people run things in a container?
| bilekas wrote:
| No, I'm asking why a website that someone could fill in a
| few fields and result in the optimized llm for you would
| need to run in a container? It's a webform.
| riddley wrote:
| My ollama and GPU are in k8s.
| seemaze wrote:
| I just discovered the other day the hugging face allows you to
| do exactly this.
|
| With the caveat that you enter your hardware manually. But are
| we really at the point yet where people are running local
| models without knowing what they are running them on..?
| roxolotl wrote:
| The other nice part of huggingface's setup is you can add
| theoretical hardware and search that way too.
| mmmlinux wrote:
| People out there are probably vibecoding their username /
| passwords for websites. Don't under estimate dumb people.
| hidelooktropic wrote:
| The whole point is to measure your hardware capability. How
| would you do that as a website?
| windex wrote:
| What I do is i ask claude or codex to run models on ollama and
| test them sequentially on a bunch of tasks and rate the outputs.
| 30 minutes later I have a fit. It even tested the abliterated
| models.
| codazoda wrote:
| Can you share the prompts?
| ff00 wrote:
| Found this website, not tested https://www.caniusellm.com/
| fwipsy wrote:
| Seems broken. When I changed my auto-detected phone specs to
| manually entered desktop specs the recommendations didn't
| change at all.
| onion2k wrote:
| That site says my 24GB M4 Pro has 8GB of VRAM. Browsers can't
| really detect system parameters. The Device Memory API
| 'anonymizes' the value returned to stop browser fingerprinting
| shenannigans. Interesting site, but you'll need to configure it
| manually for it to be accurate.
| Hamuko wrote:
| You have a whole 8 GB of VRAM? My 32 GB M1 Max has 8 GB of
| RAM and ~4 GB of VRAM according to this website.
| onion2k wrote:
| You have 32GB of unified ram. It's not split between RAM
| and VRAM. The website cannot tell this using the browser's
| APIs.
| asimovDev wrote:
| as someone who's very uneducated when it comes to LLMs I am
| excited about this. I am still struggling to understand
| correlation between system resources and context, e.g how much
| memory i need for N amount of context.
|
| Been recently into using local models for coding agents, mostly
| due to being tired of waiting for gemini to free up and it
| constantly retrying to get some compute time on the servers for
| my prompt to process like you are in the 90s being a university
| student and have to wait for your turn to compile your program on
| the university computer. Tried mistral's vibe and it would run
| out of context easily on a small project (not even 1k lines but
| multiple files and headers) at 16k or so, so I slammed it at the
| maximum supported in LM studio, but I wasn't sure if I was
| slowing it down to a halt with that or not (it did take like 10
| minutes for my prompt to finish, which was 'rewrite this C
| codebase into C++')
| omneity wrote:
| This is a great project. FYI all you need is the size of an LLM
| and the memory amount & bandwidth to know if it fits and the
| tok/s
|
| It's a simple formula:
|
| llm_size = number of params * size_of_param
|
| So a 32B model in 4bit needs a minimum of 16GB ram to load.
|
| Then you calculate
|
| tok_per_s = memory_bandwidth / llm_size
|
| An RTX3090 has 960GB/s, so a 32B model (16GB vram) will produce
| 960/16 = 60 tok/s
|
| For an MoE the speed is mostly determined by the amount of active
| params not the total LLM size.
|
| Add a 10% margin to those figures to account for a number of
| details, but that's roughly it. RAM use also increases with
| context window size.
| zozbot234 wrote:
| > RAM use also increases with context window size.
|
| KV cache is very swappable since it has limited writes per
| generated token (whereas inference would have to write out as
| much as llm_active_size per token, which is way too much at
| scale!), so it may be possible to support long contexts with
| quite acceptable performance while still saving RAM.
|
| Make sure also that you're using mmap to load model parameters,
| especially for MoE experts. It has no detrimental effect on
| performance given that you have enough RAM to begin with, but
| it allows you to scale up gradually beyond that, at a very
| limited initial cost (you're only replacing a fraction of your
| memory_bandwidth with much lower storage_bandwidth).
| 0xbadcafebee wrote:
| Well mmap can still cause issues if you run short on RAM, and
| the disk access can cause latency and overall performance
| issues. It's better than nothing though.
|
| Agree that k/v cache is underutilized by most folks. Ollama
| disables Flash Attention by default, so you need to enable
| it. Then the Ollama default quantization for k/v cache is
| fp16, you can drop to q8_0 in most cases.
| (https://mitjamartini.com/posts/ollama-kv-cache-
| quantization/)
| (https://smcleod.net/2024/12/bringing-k/v-context-
| quantisatio...)
| kittikitti wrote:
| This is a good rule of thumb. I would also include that in most
| cases, RAM use exponentially increases with context window
| size.
| escapeteam wrote:
| Thanks for the formula, I wasn't aware of it.
| AndrewAndrewsen wrote:
| Awesome project! I recently ran a (semi-)crowdsourced quality
| benchmarking for models <=20b
|
| How do you benchmark them? This would be awesome to implement at
| the page as well. I will link to this project at
| https://mlemarena.top/
| mittermayr wrote:
| this is visually fantastic, but while trying this out, it says I
| can't run Qwen 3.5 on my machine, while it is running in the
| background currently, coding. So, not sure what the true value of
| a tool like this is other than getting a first glimpse, perhaps.
| Also, with unsloth providing custom adjustments, some models that
| are listed as undoable become doable, and they're not in the
| tool. Again, not trying to be harsh, it's just a really hard
| thing to do properly. And like many other similar tools, the
| maintainer here will also eventually struggle with the fact that
| models are popping up left and right faster than they can keep up
| with it.
| kittikitti wrote:
| You might be swapping out neural weights between disk and RAM.
| I think people in a year or two will realize why their disks
| have been failing prematurely, or perhaps you too.
| api wrote:
| Read the headline and thought it rescaled LLMs down for your
| hardware. That would be fascinating but would degrade
| performance.
|
| Any work on that? Like let's say I have 64GB memory and I want to
| run a 256 parameter model. At 4 bit quantized that's 128 gigs and
| usually works well. 2 bits usually degrades it too much. But if
| you could lose data instead of precision? Would probably imply a
| fine tuning run afterword, so very compute intensive.
| riidom wrote:
| LM Studio has an option on model load that I believe does what
| you describing here: "K Cache Quantization Type" (and similar
| for "V"). It's marked as experimental and says the effect is
| basically hard to predict. Never tried myself, though.
| throwaway2027 wrote:
| More params and lower quant or higher quant and less params?
| bobmcnamara wrote:
| ISA dependent quant?
| railka wrote:
| Congratulations on the launch! It's useful for Ollama users, for
| example. And LM Studio has built-in hints in the interface.
| 0xbadcafebee wrote:
| This is probably catching ~85% of cases and you can possibly do
| better. For example, some AMD iGPUs are not covered by ROCm, so
| instead you rely on Vulkan support. In that case you can
| sometimes pass driver arguments to allow the driver to use system
| RAM to expand VRAM, or to specify the "correct" VRAM amount. (on
| iGPUs the system RAM and VRAM are physically the same thing) In
| this case you carefully choose how much system RAM to give up,
| and balance the two carefully (to avoid either OOM on one hand,
| or too little VRAM on the other). But do this and you can pick
| models that wouldn't otherwise load. Especially useful with layer
| offload and quantized MoE weights.
| lacoolj wrote:
| As a few others have noted already - this should just be a
| website, not a CLI tool. We can easily enter our CPU, RAM, GPU
| specs into a form to get this info.
| minchok wrote:
| Thanks, it is helpful and easy to use!
___________________________________________________________________
(page generated 2026-03-02 23:01 UTC)