I have 100 medical pdfs. Is there a chatgpt/pdf/crawling/scraping/searching software that can index everything and offer fast "proximity" searching? I have already tried the following (in post). reddit / selfhosted ---------------------------------------------------------------------- Acrobatic_Wealth7483 · 2023-12-04 20:14 Agent Ransack, Recoll, Foxit & Adobe DC, DocFetcher. My ideal piece of software would be like an offline search engine that is ran on the desktop. Or something like ChatGPT that uses only information from my pdfs to give me answers. ---------------------------------------------------------------------- emaiksiaime · 2023-12-04 20:17 PrivateGPT ---------------------------------------------------------------------- HTTP_404_NotFound · 2023-12-04 20:22 Don't know if it would work for you- But, Paperless-NGX works well for my documents. ---------------------------------------------------------------------- TestPilot1980 · 2023-12-04 20:27 Have you tried chromadb? Not sure I understand your question but are you trying to build a RAG? ---------------------------------------------------------------------- Splendor0806 · 2023-12-04 20:45 Paperless-ngx have this function ---------------------------------------------------------------------- Acrobatic_Wealth7483 · 2023-12-04 21:12 what is a rag? ---------------------------------------------------------------------- Pabsilon · 2023-12-04 21:18 Paperless-ngx, previously recommended, passes OCR through all your documents. That means you'll be able to control -f / search in the GUI etc.. You can also add your own tags to documents and it can learn how to tag new stuff you feed it. ---------------------------------------------------------------------- cornetto_catface · 2023-12-04 21:27 Not selfhosted and I haven't read the privacy policy but [askyourpdf](https://askyourpdf.com/) should tick some of your requirements! ---------------------------------------------------------------------- TestPilot1980 · 2023-12-04 21:28 It might what you are exactly looking for. It the process of chunking and indexing your pdf’s. So when you interact with a LLM, you send a relevant chunk of your pdf with your question so the LLM is able to answer better. ---------------------------------------------------------------------- jjasghar · 2023-12-05 01:57 Watch this: https://youtu.be/tcqEUSNCn8I?si=C5iLrDVshS-weEe6 With a little work you might get what you want. ---------------------------------------------------------------------- Dubban22 · 2023-12-05 02:12 Paperless or this https://www.reddit.com/r/selfhosted/s/6MJYNJhcPs ---------------------------------------------------------------------- Due_Royal_2220 · 2023-12-05 02:27 Another vote for Paperless-NGX. Mine (mostly technical docs / manuals): Total documents: 330 Total characters: 52,226,302 ---------------------------------------------------------------------- ElongatedElf2863 · 2023-12-05 02:44 https://calibre-ebook.com/about ---------------------------------------------------------------------- trz_0 · 2023-12-05 02:51 You can check r/LocalLLaMA but PDF is not the best format to do so. It has many underlying components (glyphs etc.) that would make the search unefficient. If you can convert these to EPUB or best OCR them first you can then ran a local LLM (similar to ChatGPT) to query them. ---------------------------------------------------------------------- srxz · 2023-12-05 03:06 I just did an rag with AWS to a client in one week with Claude v2 and opensearch, it's easy but costs a lot, if you don't mind to use some service there's one basically free with infinite data sources, you can basically upload everything and it will vectorize and offer similarity search with context chat. The service is called mendable AI it's pretty good I'm using it to embed all my self-hosted documentation ---------------------------------------------------------------------- ChocoDarkMatter · 2023-12-05 04:25 If your medical pdfs are relatively small in size, you can combine them with acrobat and then upload to gpt. I was able to get 14 books, about 200-400 pages each, in total about 3500 pages and it came out to about 160mb, which is well below the 512mb limit per file. You can upload 10 said files. That comes out to roughly 350-450 books. ---------------------------------------------------------------------- [deleted] · 2023-12-05 05:05 [removed] ---------------------------------------------------------------------- k2kuke · 2023-12-05 05:46 This is the correct answer: https://github.com/imartinez/privateGPT ---------------------------------------------------------------------- SeaNefariousness4835 · 2023-12-05 12:54 This is literally what elasticsearch was made for. Orig use case was to search super fast recipe collection. But it basically reads all the documents and each word it indexes allowing you to do fuzzy matching ---------------------------------------------------------------------- Zoenboen · 2023-12-05 13:38 Ollama. Comes with two scripts that require Python. [Ingest.py](https://Ingest.py) will create a database of your document, then using [PrivateGPT.py](https://PrivateGPT.py) you can then chat with the documents using LLAMA2. These examples are in the... examples directory. I've used in on a Mac Mini and it works well. I ingested my daily notes and my digital journal and then went back and asked it about the relationship with my wife. It correctly said that she has emotional issues and the marriage is ending. ---------------------------------------------------------------------- mayo551 · 2023-12-05 13:46 I used to use paperless-ng but now I just use samba with elasticsearch. You can index the content to elasticsearch with a script such as fscrawler. ---------------------------------------------------------------------- Acrobatic_Wealth7483 · 2023-12-05 14:12 Could I pay you set to up paperless NGX for me? ---------------------------------------------------------------------- Acrobatic_Wealth7483 · 2023-12-05 14:12 Could I pay you set to up paperless NGX for me? ---------------------------------------------------------------------- Acrobatic_Wealth7483 · 2023-12-05 14:12 Could I pay you set to up elasticssearch for me? ---------------------------------------------------------------------- Acrobatic_Wealth7483 · 2023-12-05 14:12 Could I pay you set to up elasticssearch for me? ---------------------------------------------------------------------- HTTP_404_NotFound · 2023-12-05 14:16 > Could I pay you set to up paperless NGX for me? Sorry, I don't do freelancing these days. But, the installation process is documented quite well- https://docs.paperless-ngx.com/setup/#docker_script ---------------------------------------------------------------------- mayo551 · 2023-12-05 14:17 I see you have already asked several people. I'm sure one of them will help you. ---------------------------------------------------------------------- vanalle · 2023-12-05 15:51 retrieval augmented generation ---------------------------------------------------------------------- kmisterk · 2023-12-05 17:15 ## Post Removed Hey, Acrobatic_Wealth7483! Thanks for your submission on selfhosted. Your post has been removed due to a violation of the purpose of the subreddit: **Self Hosting** ## Not Self-Hosted When it comes to posts regarding applications in this subreddit, they must feature a self-hosted tool, or a tool that *can be* self-hosted, or some kind of related information, help request, or otherwise related to a tool that is something that one can self-host. If you believe this post *is in fact* relevant to /r/selfhosted, feel free to [message the mods](https://www.reddit.com/message/compose/?to= /r/selfhosted&subject=Removed For Not Being Self- Hosted&message=[Removed Post](https://www.reddit.com/r/selfhosted/comments/18at3x0/-/\)) with your concerns.