[HN Gopher] Build Personal ChatGPT Using Your Data
___________________________________________________________________
Build Personal ChatGPT Using Your Data
Author : raghavankl
Score : 81 points
Date : 2023-07-08 21:07 UTC (1 hours ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| eminent101 wrote:
| Is it going to send my personal data to OpenAI? Isn't that a
| serious problem? Does not sound like a wise thing to do, not at
| least without redacting all sensitive personal data from the
| data. Am I missing something?
| cloudking wrote:
| Am I the only one who doesn't need to search across my data? What
| are the use cases here
| csjh wrote:
| Why have the OpenAI dependency when there's local embeddings
| models that would be both faster and more accurate?
| yawnxyz wrote:
| Which ones?
| minimaxir wrote:
| all-MiniLM-L6-v2 from SentenceTransformers is the most
| popular one as it balances speed and quality well:
| https://www.sbert.net/docs/pretrained_models.html
| zikohh wrote:
| Also what does this do that llamaindex doesn't?
| chaxor wrote:
| The most frustrating thing about the many, _many_ clones of this
| exact type of idea is that pretty much _all_ of them require
| OpenAI.
|
| Stop doing that.
|
| You will have way more users if you make OpenAI (or anything that
| requires cloud) the 'technically possible but pretty difficult
| art of hoops to make it happen' option, instead of the other way
| around.
|
| The best way to make these apps IMO is to make them work
| _entirely_ locally, with an easy string that 's swappable in a
| .toml file to any huggingface model. Then if you _really_ want
| OpenAI crap, you can make it happen with some other docker secret
| or `pass` chain or something with a key, while changing up the
| config.
|
| The default should be local first, do as much as possible, and
| then _if the user /really/ wants to_, make the collated prompt
| send a very few set of tokens to openAI.
| hospitalJail wrote:
| ClosedAI has freaked me out with how much power they have, and
| how irresponsible they are with it.
|
| I'm so horrified that they are going to take away the ability
| to ask medical questions when the AMA comes knocking at their
| door.
| Der_Einzige wrote:
| Even GPT-3.5-turbo-16K isn't good enough for most retrieval
| augmented generation tasks.
|
| Locally ran LLMs are far worse.
|
| I don't like it either, but for now, if you want good RAG, you
| have to use GPT-4
| sheeshkebab wrote:
| Amen, local first should be the default for anything that sucks
| all my data.
|
| Although until these things can do my laundry none of them
| deserve any of my compute time either.
| persedes wrote:
| Gpt4all does exactly that. You can choose between local model
| or bring your own openai token.
| trolan wrote:
| The only OpenAI 'crap' being used here is to generate the
| embeddings. Right now, OpenAI has some of the best and cheapest
| embeddings possible, especially for personal projects.
|
| Once the vectors are created tho, you're completely off the
| cloud if you so choose.
|
| You can always swap out the embedding generator too, because
| LangChain abstracts that for your exact gripes.
|
| Everything else is already using huggingface here and can be
| swapped out for any other model besides GPT2 which supports the
| prompts.
| space_fountain wrote:
| Do you have citations on OpenAI embeddings being some of the
| cheapest and best? The signs I've seen points almost in the
| opposite direction?
| ben_w wrote:
| The only embeddings I currently see listed on
| https://openai.com/pricing are Ada v2, at $0.1/million
| tokens.
|
| Even if the alternative is free, how much do you value your
| time, how long will it take to set up an alternative, and
| how much use will you get out of it? If you're getting less
| than a million tokens and it takes half an hour longer to
| set up, you'd better be a student with zero literally
| income because that cost of time matches the UN abject
| poverty level. This is also why it's never been the year of
| linux on the desktop, and why most businesses still don't
| use Libre Office and GIMP.
|
| I can't speak for quality; even if I used that API
| directly, this whole area is changing too fast to really
| keep up.
| colobas wrote:
| Can you elucidate on what those signs are? Thanks in
| advance
| muggermuch wrote:
| As per the Massive Text Embedding Benchmark (MTEB)
| Leaderboard maintained by Huggingface, OpenAI's embedding
| models are _not_ the best.
|
| https://huggingface.co/spaces/mteb/leaderboard
|
| Of course, that's far from saying that they're the worst,
| or even headed that way. Just not the best (those would
| be a couple of fully opensource models, including those
| of the Instructor family, which we use at my workplace).
| AussieWog93 wrote:
| Honestly, for me, getting good quality results matters way more
| than keeping my searches private. And for that, nothing
| compares with GPT4.
| dmezzetti wrote:
| txtai makes it easy to use Hugging Face embeddings and Faiss,
| all local and configurable. https://github.com/neuml/txtai
|
| paperai is a sub-project focused on processing
| medical/scientific papers. https://github.com/neuml/paperai
|
| Disclaimer: I am the author of both
| jstummbillig wrote:
| What do you (or anyone else, feel free to chime in) do with
| other LLMs that makes them useable for anything that is not
| strictly tinkering?
|
| Here is my premise: We are past the wonder stage. I want to
| actually get stuff done efficiently. From what I have tested so
| far, the only model that allows me to do that halfway reliably
| is GPT-4.
|
| Am I incompetent or are we really just wishfully thinking in HN
| spirit that other LLMs are a lot better at being applied to
| actual tasks that require a certain level of quality,
| consistency and reliability?
| JimmyRuska wrote:
| It's difficult to compete. A small business might answer 10,000
| requests to their chat bot. The options are
|
| - Pay openai less than $50mo
|
| - Manage cloud gpus, hire ml engineers > $1000/mo
|
| - Buy a local 4090 and put it under someone's desk, $no
| reliability +$1500 fixed
|
| Any larger business will need scalability and you still can't
| compete with openai pricing.
|
| Maybe one of you startup inclined people can make an openllama
| startup that charges by request and allows for finetuning,
| vector storage
| srcthr wrote:
| People don't scale. This is personal. Only 3 is a good choice
| for people in a site with the name hacker something.
| zikohh wrote:
| Have you seen PrivateGPT. It's quite good and free.
| yoyopa wrote:
| what hardware do you need for that?
| drdaeman wrote:
| Consumer-grade, AFAIK it's GPT4All with LLaMA.
| EGreg wrote:
| Link?
| hoopsman wrote:
| Presumably https://github.com/imartinez/privateGPT
| mvkel wrote:
| It's not nearly usable. It's functional in that it spits out
| a response. Can that response be used for anything useful?
| No.
| [deleted]
| akira2501 wrote:
| > is that pretty much all of them require OpenAI.
|
| They're not here to release an actual product. They're here to
| release part of a CV proving they have "OpenAI" experience. I'm
| assuming this is the result of OpenAI not actually having any
| homegrown certification program of their own.
| gumby wrote:
| > OpenAI not actually having any homegrown certification
| program
|
| A bit off topic but where are certifications (e.g. Cisco,
| Microsoft) useful? I am sure they _are_ useful (both to
| candidates and companies) because people go to the effort to
| get these certs, and if they were useless everyone would have
| stopped long ago. I don 't assume people do it for ego
| satisfaction.
|
| But I've never worked anywhere where it has come up as an
| interview criterion (nobody has ever pointed it out when we
| are looking at a resume, for example). Is it a big business
| thing? Is it just an HR thing?
| n4te wrote:
| I've only ever seen it as a people who don't have a job
| thing.
| tw04 wrote:
| Mainly when applying for a corporate job where you have 0
| referrals. It's a guidepost that you at least have some
| idea what you're doing and are worth interviewing when
| people can't find someone who knows you and your previous
| work.
| AJRF wrote:
| Is there a company that makes a hosted version of something like
| this? I quite want a little AI that I can feed all my data to to
| ask questions to.
| luccasiau wrote:
| https://libraria.dev/ offers this and more as a service. It has
| added conveniences like integration with your google drive,
| youtube videos, and such
| egonschiele wrote:
| Depending on the size of your data, chiseleditor.com is a free
| option.
| einpoklum wrote:
| Don't build a personal ChatGPT, and don't let OpenAI, Microsoft
| and their business partners (and probably the US government) have
| a bunch of your personal and private information.
| WhackyIdeas wrote:
| So avoid all Microsoft products too?
| hi wrote:
| Keep your data private and don't leak it to third parties. Use
| something like privateGPT (32k stars). Not your keys, not your
| data.
|
| "Interact privately with your documents using the power of GPT,
| 100% privately, no data leaks"[0]
|
| [0] https://github.com/imartinez/privateGPT
| leach wrote:
| How does this run on an Intel Mac? I have a 6 core i9. Haven't
| been able to get an M series yet so Im wondering if it would be
| more worth it to run it in a cloud computing environment with a
| GPU.
| hi wrote:
| >Mac Running Intel When running a Mac with Intel hardware
| (not M1), you may run into clang: error: the clang compiler
| does not support '-march=native' during pip install.
|
| If so set your archflags during pip install. eg:
| ARCHFLAGS="-arch x86_64" pip3 install -r requirements.txt
|
| https://github.com/imartinez/privateGPT#mac-running-intel
| leach wrote:
| I'm curious about the response times though, i imagine they
| will be quite slow on an intel Mac
| 3rdrodeo wrote:
| [flagged]
| EGreg wrote:
| Oh, not this again. No, it's not a net negative in ALL forms.
| And if you really were concerned about downsides, AI has a
| ton more potential downsides than Web3 ever did, including
| human extinction, as many of its top proponents have come out
| and publicly said. Nothing even remotely close to that is the
| case for Web3 at all:
|
| https://www.nytimes.com/2023/05/30/technology/ai-threat-
| warn...
|
| https://www.theguardian.com/technology/2023/may/10/ai-
| poses-...
|
| https://www.theguardian.com/technology/2023/may/30/risk-
| of-e...
|
| https://www.theverge.com/2023/5/30/23742005/ai-risk-
| warning-...
| 3rdrodeo wrote:
| " many of its top proponents have come out and publicly
| said." You don't have to uncritically accept that, it's far
| more likely that they're just self aggrandizing in a "wow
| I'm so smart my inventions can destroy the world, better
| give me more money and write articles about me to make sure
| that doesn't happen".
| EGreg wrote:
| I see, so when it comes to the top experts in AI
| screaming "we made a mistake, please be careful" we
| should use nuance and actually conclude the opposite --
| that we should press ahead and they're wrong.
|
| But with Web3, we should just listen to a bunch of random
| no-name haters say "there are NO GOOD APPLICATIONS, trust
| us, period, stop talking about it", use no nuance or
| critical thinking of our own, and simply stop building on
| Web3.
|
| Do you happen to see the extreme double standard here
| you're employing, while trying to get people to see
| things your way?
| 3rdrodeo wrote:
| The crypto group had a lot of time and even more money to
| make a compelling product that took off and so far
| they've failed. We've watched fraud after fraud as
| they've shown themselves to just be ignorant and arrogant
| ideologues who don't understand how the "modern" finance
| system came to be, what the average user wants out of
| financial or social products, or just outright scammers.
| We can keep sinking money into a bottomless pit or we can
| move on and learn from their mistakes.
|
| I didn't say to dismiss any concerns out of hand, but the
| whole idea of "x-risk" or "human extinction" from ai is
| laughable and isn't taken seriously by most people. Again
| if you think critically about the whole idea of "human
| extinction" from any of the technology being talked about
| you _should_ see it as nonsense.
| peyton wrote:
| I dunno, on the crypto side stablecoins are pretty
| compelling for hassle-free cross-border transfers--
| there's $125bn in circulation, which to me means it's
| taken off.
|
| On the AI side, I mean for example it's not laughable to
| think anybody on the planet could just feed a bunch of
| synthetic biology papers to a model and start designing
| bioweapons. It's not hard to get your hands on secondhand
| lab equipment...
| unshavedyak wrote:
| Is this robust enough to feed all your emails and chat logs
| into it and have convos with it? Will it be able to extract
| context to figure out questions to recent logs, etc?
| SecurityNoob wrote:
| 100% private? Hmm. I think with the amount of paranoia that the
| folks in power have about local LLM's, I wouldn't be in the
| slightest surprised that the Windows telemetry will be
| reporting back what people are doing with them. And anyone who
| thinks otherwise is in my view just absolutely naive beyond
| hope.
| quickthrower2 wrote:
| The author has a demo of this here:
| https://www.swamisivananda.ai/
| syntaxing wrote:
| gpt4all has this truly locally. I recommend those with a decent
| GPU to give it a go.
| fbdab103 wrote:
| I assume this is the link: https://github.com/nomic-ai/gpt4all
| ?
| JimmyRuska wrote:
| Anyone know how milvus, quickwit, pinecone compares?
|
| I've been thinking about seeing if there's consulting
| opportunities for local businesses for LLMs, finetuning/vector
| search, chat bots. Also making tools to make it easier to drag
| and drop files and get personalized inference. Recently I saw
| this one pop into my linkedin feed, https://gpt-trainer.com/ .
| There's been a few others for documents I've found
|
| https://www.explainpaper.com/
|
| https://www.konjer.xyz/
|
| Nope nope, wouldn't want to compete with that on pricing. Local
| open source LLMs on a 3090 would also be a cool service, but
| wouldn't have any scalability.
|
| Are there any other finetuning or vector search context startups
| you've seen?
| gigel82 wrote:
| I don't get it, GPT-2 is (one of the few) open models from
| OpenAI, you can just run it locally, why would you use their API
| for this? https://github.com/openai/gpt-2
___________________________________________________________________
(page generated 2023-07-08 23:00 UTC)