[HN Gopher] Show HN: macOS GUI for running LLMs locally
___________________________________________________________________
Show HN: macOS GUI for running LLMs locally
Hello HN, I've been working on this project for a while, and it
has been in an "open" beta for some time. I finally believe it's
ready for its first release. I hope you like it. Here are some
potential questions that may arise: 1. How does it compare to LM
Studio? It's likely that if you're already using LM Studio, you'll
continue to do so. This project is designed to be more user-
friendly. 2. Is it open-source? No, it is not. 3. Does it use any
open-source libraries? Yes, it uses llama.cpp and a few others, as
indicated in the license information included with the application.
4. Why is not using electronjs? Two reasons, I wanted total control
over the whole tech-stack and second, I wanted to be able to send
this to my friends over iMessage. 5. Does it support Intel macs?
It should, but I couldn't test it. 6. Does it support older macOS?
12.6 is the lowest version at the moment. 7. Is XXX a bug?
Probably :)
Author : cztomsik
Score : 33 points
Date : 2023-09-18 19:51 UTC (3 hours ago)
(HTM) web link (www.avapls.com)
(TXT) w3m dump (www.avapls.com)
| syntaxing wrote:
| Do the models run in a docker container? Probably one of my
| favorite things about Ollama.
| mchiang wrote:
| disclaimer: I'm one of the maintainers working on Ollama.
|
| I would love to hear how you are using Ollama. One of the
| upcoming releases will involve an official release of Ollama on
| Linux with CUDA support of the box. From there, we will publish
| Ollama Docker images to enable GPU support as well.
| syntaxing wrote:
| Thank you for your amazing work! I more or less use it as a
| llama cpp replacement cause I honestly can't figure out a
| good prompt structure so I get bad results. On my M1 MBP, I
| use it for stuff that can't go to GPT but can only use upto
| 13B. I also run a 34B model with CUBLAS enabled on my Linux
| server by modifying the go files mentioned in one of the
| issue tickets.
| swyx wrote:
| very cool. why did you work on it? curious on motivations.
| congrats on shipping.
| cztomsik wrote:
| Thanks :) love your podcast BTW.
|
| See https://news.ycombinator.com/item?id=37562287 - I can't
| tell much more yet, but I feel like LLMs are way more useful
| than for a chat-bot or a vscode extension.
| swyx wrote:
| thanks very much! will keep a look out as you build your app.
| alexstore06 wrote:
| How does this compare to GUIs like llamero
| (https://github.com/mpociot/llamero) and gpt4all
| (https://gpt4all.io/index.html)?
| cztomsik wrote:
| It has playground with prompt saving and (simple) templates,
| chat saving, builtin model downloader. It's also very small,
| most people don't care about that, but I do.
| marcellus23 wrote:
| > What is a language server?
|
| > A language server is a specialized program that processes
| language-related tasks. This includes activities like text
| generation, grammar correction, rephrasing, summarization, data
| extraction, and more.
|
| This confused me initially. I've never heard language server in
| this context. To me, it has always been in the context of the
| LSP[0]. But maybe this is a common usage of the term and I've
| just missed it?
|
| Even if so, it seems like an odd term, since this is a self-
| contained desktop application and not really a server, right?
|
| 0: https://en.wikipedia.org/wiki/Language_Server_Protocol
| duskwuff wrote:
| [delayed]
| cztomsik wrote:
| No, you are right, it's a new term I made up. The idea is that
| I see this as a RAD tool (for non-programmers). So you would
| have this app open all the time and it would serve different
| tasks you have prepared before. But that part is not finished
| yet and it will be limited in the free version.
| CharlesW wrote:
| > _...it 's a new term I made up._
|
| On the bright side, now that you know what a language server
| is, you can rename your thing so it can be searched for.
| [deleted]
| krm01 wrote:
| Is there a central place where most downloadable models are
| available for quick download?
| SparkyMcUnicorn wrote:
| https://huggingface.co/TheBloke
| cztomsik wrote:
| It works with current version of GGUF models for llama.cpp -
| you can find them on huggingface, or you can convert them
| manually.
|
| Only a few download links are baked-in at the moment but
| whatever *.gguf file you put in your Downloads folder should
| appear in the dropdown.
| achrono wrote:
| >Is it open-source? No, it is not.
|
| Why not?
|
| As to Why --> this one actually has a decent GUI and one that's
| not (visibly) based on a locally running webserver -- so it seems
| like it's got a shot at getting 100x popular with just a YouTube
| video (or _gasp_ Tiktok). With the popularity of open-source
| today I find it a little hard to trust that something is
| "ensuring maximum privacy" as you say without having the open-
| source badge (yes, I know I shouldn't be so naive, but no, I am
| not going to look at network level logs to find out if the app is
| truly contained to local).
| cztomsik wrote:
| Fair point, but if this was open-source, would you really go
| and read all of the source code in several programming
| languages and also audit all of the dependencies?
|
| IMHO, at some point, it's just about trust.
|
| Now, to be 100% honest, I'd love to release this as OSS, but my
| current business model is based on freemium. And, sell it as
| white-label for companies.
|
| EDIT: if you have better idea, I'd love to hear about it.
| achrono wrote:
| >it's just about trust.
|
| Yes, and a big part of establishing trust is sending the
| right signals. The risk vs reward differential is way higher
| for open-source compared to closed source, so all other
| things being equal I would trust open-source more. Note that
| all other things are actually _not_ equal in this case, e.g.
| your closest competition is open-source: Ollama, GPT4All,
| Llamero.
| lagniappe wrote:
| I resent having to step in and say something here, but the
| way you describe things, I effectively don't exist. I read
| the app code. I write stdlib code. I grokk every line of
| every library I use. I share alike.
|
| I'm not saying change your ways, just consider being a bit
| less flippant when you say things like "IMHO, at some point,
| it's just about trust". As someone who seeks to profit from
| this, that's about the worst example you can give a
| prospective user like me.
|
| My face is hot just having to type this.
___________________________________________________________________
(page generated 2023-09-18 23:01 UTC)