[HN Gopher] Show HN: AnythingLLM - Open-Source, All-in-One Deskt...
___________________________________________________________________
Show HN: AnythingLLM - Open-Source, All-in-One Desktop AI Assistant
Hey HN! This is Tim from AnythingLLM (https://github.com/Mintplex-
Labs/anything-llm). AnythingLLM is an open-source desktop assistant
that brings together RAG (Retrieval-Augmented Generation), agents,
embeddings, vector databases, and more--all in one seamless
package. We built AnythingLLM over the last year iterating and
iterating from user feedback. Our primary mission is to enable
people with a layperson understanding of AI to be able to use AI
with little to no setup for either themselves, their jobs, or just
to try out using AI as an assistant but with *privacy by default*.
From these iterations & feedback, we have a couple of key learnings
I wanted to share: - "Chat with your docs" solutions are a dime-a-
dozen - Agent frameworks require knowing how to code or are too
isolated from other tools - Users do not care about benchmarks,
only outputs. The magic box needs to be magic to them. - Asking
Consumers to start a docker container or open a terminal is a non-
starter for most. - Privacy by default is non-negotiable. Either
by personal preference or legal constraints - Everything needs to
be in one place From these ideas, we landed on the current state
of AnythingLLM: - Everything in AnythingLLM is private by default,
but fully customizable for advanced users. - Built-in LLM
provider, but can swap at any time to the hundreds of other local
or cloud LLM providers & models. - Built-in Vector Database, most
users don't even know that it is there. - Built-in Embedding
model, but of course can change if the user wants to. - Scrape
websites, import Github/GitLab repos, YouTube Transcripts,
Confluence spaces - all of this is already built in for the user.
- An entire baked-in agent framework that works seamlessly within
the app. We even pre-built a handful of agent skills for customers.
Custom plugins are in the next update and will be able to be built
with code, or a no-code builder. - All of this just works out of
the box in a single installable app that can run on any consumer-
grade laptop. Everything a user does, chats, or configures is
stored on the user's device. Available for Mac, Windows, and Linux
We have been actively maintaining and working on AnythingLLM via
our open-source repo for a while now and welcome contributors as we
hopefully launch a Community Hub soon to really proliferate users'
abilities to add more niche agent skills, data connectors, and
more. *But there is even more* We view the desktop app as a
hyper-accessible single-player version of AnythingLLM. We publish a
Docker image too
(https://hub.docker.com/r/mintplexlabs/anythingllm) that supports
multi-user management with permissioning so that you can easily
bring AnythingLLM into an organization with all of the same
features with minimal headache or lift. The Docker image is for
those more adept with a CLI, but being able to comfortably go from
a single-user to a multi-user version of the same familiar app was
very important for us. AnythingLLM aims to be more than a UI for
LLMs, we are building a comprehensive tool to leverage LLMs and all
that they can do while maintaining user privacy and not needing to
be an expert on AI to do it. https://anythingllm.com/
Author : tcarambat1010
Score : 181 points
Date : 2024-09-05 15:40 UTC (7 hours ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| ranger_danger wrote:
| Finally.
| vednig wrote:
| How do you ensure "privacy by default" if you are also providing
| cloud models?
| hluska wrote:
| Good question - this was an excellent write up and AnythingLLM
| looks great. But I'm really curious about that too.
|
| Regardless of the answer, OP you've done a hell of a lot of
| work and I hope you're as proud as you should be.
| Congratulations on getting all the way to a Show HN.
| Tostino wrote:
| It doesn't seem like they are "providing" cloud models. They
| have their backend able to interface with whatever endpoint you
| want, given you have access. It's plainly obvious when
| interacting with 3rd party providers that it depends on their
| data / privacy policy. When has this ever not been the case?
|
| I could just start up a vLLM instance with llama 3.1 and
| connect their application to it just as easily though.
| Perfectly secure (as I am able to make it).
|
| This seems like such a pedantic thing to complain about.
| tcarambat1010 wrote:
| It's not my position to "impose" a preference on a user for an
| LLM.
|
| Privacy by default means "if you use the defaults, it's
| private".
|
| Obviously, if your computer is low-end or you really have been
| loving GPT-4o or Claude you _can_ use just that external
| component while still using a local vector db, document
| storage, etc.
|
| So its basically like opt-in for external usage of any
| particular part of the app, whether that be an LLM, embedder
| model, vector db, or otherwise.
| phren0logy wrote:
| I have been really impressed with AnythingLLM as a no-fuss way to
| use LLMs locally and via APIs. For those of us who want to
| tinker, there's a solid range of choice for embedders and vector
| stores.
|
| The single install desktop packaging is very slick. I look
| forward to the upcoming new features.
| candiddevmike wrote:
| Have you tried openwebui? What are your thoughts on it vs this?
| phren0logy wrote:
| I had a much smoother experience with the desktop version of
| AnythingLLM. There's more in openwebui, but the things that
| are in AnythingLLM are more polished (IMO).
| tcarambat1010 wrote:
| We will have plugins for: - Auth - Agent skills - Data
| connectors
|
| soon, agent skills are first and this should help plug the
| gaps between offerings since we can't build everything for
| everyone
| anonymous344 wrote:
| what kind of pc does it need ? ram, etc?
| tcarambat1010 wrote:
| Depends on the model you want to run! That is the beauty of it.
| If you can muster that ability to run a Q4 Llama3.1 8B - great,
| if your specs are really low end you can always outsource to a
| cloud provider.
|
| Maybe you want to run privately, but can only manage a Gemma-2B
| - then great, you can use that also.
| nunobrito wrote:
| There was an error while installing on Linux, was solved with:
|
| ''' sudo chown root:root /home/hn/AnythingLLMDesktop/anythingllm-
| desktop/chrome-sandbox sudo chmod 4755
| /home/hn/AnythingLLMDesktop/anythingllm-desktop/chrome-sandbox
| '''
|
| Other than that it worked really well.
| yjftsjthsd-h wrote:
| HN doesn't use normal markdown; to do code blocks you indent
| with 4 spaces, ex. sudo chown root:root
| /home/hn/AnythingLLMDesktop/anythingllm-desktop/chrome-sandbox
| sudo chmod 4755 /home/hn/AnythingLLMDesktop/anythingllm-
| desktop/chrome-sandbox
| nunobrito wrote:
| Thanks!
| hannofcart wrote:
| Downloaded and checked it out. Looks great so far. Tried using it
| to read a bunch of regulatory PDFs using GPT-4o.
|
| Some quick and early feedback:
|
| 1. The citations seem to be a bit dicey. The response seems to
| give largely correct answers but the citations window seems to
| show content that's a bit garbled.
|
| 2. Please please add a text search to search within existing chat
| content. Like if I searched for something about giraffes in one
| of the chats, search chat history and allow switching to it.
| fallinditch wrote:
| For your 2nd suggestion, this would be solved if previous chats
| were (automatically) added to the vector database. Is this
| possible in this or similar apps?
| 101008 wrote:
| LLM will become like web frameworks in the future, in the sense
| that they will be free, open source, and everybody will be able
| to build on them. Sure, there will be paid options, such as there
| are paid web frameworks, but most of the time free options will
| be more than good enough for most of the jobs.
| samdung wrote:
| I really hope this becomes the norm.
| dartos wrote:
| So... just open source?
| smokel wrote:
| What is your argumentation for that? It seems that building
| LLMs is very costly, and that will probably remain so for quite
| some time.
|
| I think it is equally likely that LLMs will perform worse in
| the future, because of copyright reasons and increased privacy
| awareness, leading to less diverse data sets to train on.
| bboygravity wrote:
| Existing free to use (offline) models are already really
| really good and open a ton of possibilities for new apps.
|
| The only barrier right now is that laptop/phone hardware
| can't run them (inference and/or training) fast enough.
| That's a problem that won't really exist 3 to 10 years from
| now.
|
| That's ignoring the fact that creation of new much much
| better models is going full steam ahead.
| 101008 wrote:
| Building software was very costly and hard 30 years ago.
| Things get cheaper and simpler with time. LLM will become a
| tool to use in your work, such as frameworks or libraries.
| Some people will, other people won't (mostly those to
| maintain projects without them).
|
| I hope to be right or I will be jobless lol
| furyofantares wrote:
| This looks sweet.
|
| Totally irrelevant but ... "Language Learning Model" ? Probably
| just some brainfart or I'm missing something but it would be
| hilarious if the authors of this did this whole project without
| knowing what LLM stands for.
| mattygee wrote:
| Large Language Model All good
| rafram wrote:
| The linked GitHub page mentions "Language Learning Models",
| so GP is saying that it seems like the authors of this
| project wrote the whole thing while not knowing what LLM
| actually stands for.
| cholantesh wrote:
| Or it was a simple typo.
| tcarambat1010 wrote:
| It was a typo haha, someone just PR'd the fix for it. God
| knows how long it has been there - whoops
| qup wrote:
| Maybe the assistant can look in the repo and compute
| that, just sayin'
| tcarambat1010 wrote:
| It was indeed a brainfart on my end that has then been in there
| since the dawn of time. Pretty funny honestly
| santamex wrote:
| How does this differ from chatboxai?
|
| https://github.com/Bin-Huang/chatbox
| Ringz wrote:
| Chatboxai is a _client_ for multiple AIs like OpenAi, Cloude,
| etc. It does not work on your local files and documents.
| philipjoubert wrote:
| This looks really great. Are you planning on adding shortcut keys
| anytime soon?
| tcarambat1010 wrote:
| We actually are planning to really lean into the "Desktop
| assistant" style of things, so yes absolutely. Like how on Mac
| you can cmd+space to launch spotlight, we can offer the same
| functionality alongside everything else.
|
| WIP!
| ranger_danger wrote:
| $ docker pull mintplexlabs/anythingllm Using default tag: latest
| Error response from daemon: Get
| "https://registry-1.docker.io/v2/": net/http: request canceled
| while waiting for connection (Client.Timeout exceeded while
| awaiting headers)
| tcarambat1010 wrote:
| This is a connection issue for your docker cli - we have seen
| this issue before and it had to do with client proxy.
|
| https://stackoverflow.com/a/48066666
| ranger_danger wrote:
| I am not behind a proxy, do not have any of those related
| variables set, and all other attempts at net connections
| (curl, telnet, browser, etc.) work fine...
| politelemon wrote:
| > AnythingLLM packages as an AppImage but you will not be able to
| boot if you run just the AppImage.
|
| The 'boot' seems to indicate it will affect the computer's
| startup process, I think you meant to say you will not be able to
| 'start the application'
| CuriouslyC wrote:
| I noticed you put LiteLLM in your list of providers. Was that
| just marketing, or did you re-implement model support for all the
| models LiteLLM already supports separately?
| tcarambat1010 wrote:
| You can use LiteLLM _as the LLM provider_ which is just a relay
| to a ton of other models. For people who already have it set up
| connecting to x,y,z providers can keep that work and use it for
| inferencing as well
| A4ET8a8uTh0 wrote:
| This definitely makes it super easy for less technical folks to
| access it ( got it up and running in less than 5 minutes ).
| Initial reaction is positive with just Ollama. Everything is
| automatically detected and if you want to manually set it up, you
| still can. Lets see how it does after adding huggingface ( quick
| and painless ).
| md3911027514 wrote:
| this is super sweet for developers (or anyone else) who like to
| have granular control over their LLM set up
|
| - ability to edit system prompt
|
| - ability to change LLM temperature
|
| - can choose which model to use (open-source or closed)
| DebtDeflation wrote:
| This is really nice. My first reaction was "oh great, another
| Ollama/WebGenUI wrapper on Llama.cpp" but it's actually much more
| - supporting not only the LLM but also embedding models, vector
| databases, and TTS/STT. Everything needed to build a fully
| functioning voice chatbot.
| tencentshill wrote:
| As someone who doesn't know what an Embed or Vector is, this has
| been the only offline AI tool I've been able to install and start
| using on my standard office PC.
| tcarambat1010 wrote:
| That's the point! Shouldn't have to know and naturally over
| time you'll learn more and the controls are there when you feel
| comfortable tuning them
| egamirorrim wrote:
| I've got an open ai API key, and I pay for chatgpt. I'd imagine
| switching to this and using openai would end up costing quite a
| lot? How are people running it relatively cheaply?
| monkmartinez wrote:
| Openrouter... you get all the models and its not as expensive
| as you would think. I spent $3 with aider the other day in like
| the blink of an eye with Anthropic. I am working on a FASTHTML
| thingy and loaded all the docs, plus a few huge replicate api
| files into the vector database. Most of my back and forth usage
| averaged about $0.02 for each turn with Claude 3.5 Sonnet. To
| give you an idea: My context + prompt were around 18000 tokens
| with completions around 1500 tokens.
| aznumeric wrote:
| One way people keep costs down when using OpenAI with an
| offline RAG system is by limiting the number of text snippets
| sent to the API. Instead of sending the whole database, they'll
| typically retrieve only the top 10 (or so) most relevant
| snippets from the vector database and just send those to OpenAI
| for processing. This significantly reduces the amount of data
| being processed and billed by OpenAI.
___________________________________________________________________
(page generated 2024-09-05 23:00 UTC)