[HN Gopher] Local Deep Research - ArXiv, wiki and other searches...
___________________________________________________________________
Local Deep Research - ArXiv, wiki and other searches included
Author : learningcircuit
Score : 147 points
Date : 2025-03-11 07:51 UTC (15 hours ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| learningcircuit wrote:
| Example output: https://github.com/LearningCircuit/local-deep-
| research/blob/...
| sinenomine wrote:
| You could be the first if you were to develop an eval
| (preferably automated with llm as judge) and compared local
| deep research with perplexity's, openai's and deepseek's
| implementations on high-information questions.
| learningcircuit wrote:
| How do they evaluate the quality of the report? It's one of
| the most important things for me.
| mentalgear wrote:
| Given a benchmark corpus, the evaluation criteria could be:
|
| - Facts extracted: the amount of relevant facts extracted
| from the corpus
|
| - Interpretations : based on the facts, % of correct
| interpretations made
|
| - Correct Predictions: based on the above, % of correct
| extrapolations / interpolations / predictions made
|
| The ground truth could be in JSON file per example. (If the
| solution you want to benchmark uses a graph db, you could
| compare these aspects with a LLM as judge.)
|
| ---
|
| The actual writing is more about formal/business/academic
| style, and I find less relevant for a benchmark.
|
| However I would find it crucial to run a "reverse RAG" over
| the generated report to ensure each claim has a source. [0]
|
| [0] https://venturebeat.com/ai/mayo-clinic-secret-weapon-
| against...
| mentalgear wrote:
| I applaud the effort for the local (lo-fi) space ! Yet, reading
| over the example linked in the docs (which does not seem cheery-
| picked, kudos for that!), my impression is that the document is a
| rather messy outcome [1].
|
| I think what's missing is one (or more) step in-between, possible
| a graph database (eg[2]), which the LLM can place all it's
| information in, see relevant interconnections, query to question
| itself, and then generate the final report.
|
| (maybe the final report could be an interactive HTML file that
| the user can ask questions, or edit themselves).
|
| There's also a similar open-deep research tool called onyx [2],
| with I think has better UI/UX albeit not local. Maybe the author
| could consider porting this to local instead of rolling and
| maintaining another deep-research tool themselves ?
|
| I'm saying this, not because I think it's not a good project, but
| because there are a ton of open deep-research projects which I'm
| afraid will just fizzle out, and would be better if people would
| join forces working on those aspects they care most about (e.g.
| local aspect, or RAG strategies, etc) .
|
| [1] https://github.com/LearningCircuit/local-deep-
| research/blob/...
|
| [2] "In-Browser Graph RAG with Kuzu-WASM and WebLLM"
| https://news.ycombinator.com/item?id=43321523
|
| [3] https://github.com/onyx-dot-app/onyx
| bilater wrote:
| I had my own spin on deep research which you might find this
| easier to navigate: https://github.com/btahir/open-deep-
| research
| TeMPOraL wrote:
| > _I think what 's missing is one (or more) step in-between,
| possible a graph database (eg[2]), which the LLM can place all
| it's information in, see relevant interconnections, query to
| question itself, and then generate the final report._
|
| Quickly, productize this (and call it DeepRAG, or DERP) before
| it explodes in late 2025 - you may just beat the market to it!
|
| See: https://news.ycombinator.com/item?id=43267539
| wahnfrieden wrote:
| Is anyone using (local) LLMs to directly search for (by scanning
| over) relevant materials from a corpus rather than relying on
| vector search?
| suprjami wrote:
| Generally this fails.
|
| Most LLMs lose the ability to track facts over about 20k words
| of content, the best can manage maybe 40k words.
|
| Look for "needle" benchmark tests, as in needle-in-haystack.
|
| Not to mention the memory requirements of such a huge context
| like 128k or 1M tokens. Only people with enterprise servers at
| home could run that locally.
| learningcircuit wrote:
| Very good answer. It is very hard with small LLM.
| alchemist1e9 wrote:
| Nice work!
|
| I've been thinking recently that a local collection of pre-
| processed for RAG using curated focused structured information
| might be a good complement to this dynamic searching approach.
|
| I see this used LangChain, might be worth checking into txtai.
|
| https://neuml.github.io/txtai/examples/
| bravura wrote:
| For web search, also consider the Kagi and Tavily APIs.
| learningcircuit wrote:
| Thank you I will add
| throwaway24681 wrote:
| Looks very cool. How does this compare to the RAG features
| provided by open-webui?
|
| There is web search and a way to embed documents, but so far it
| seems like the results are subpar as details are lost in
| embeddings. Is this much better?
| learningcircuit wrote:
| Give me a question and I can give you the output? So you can
| compare.
| throwaway24681 wrote:
| I tried it myself. It looks like this can do a lot more than
| open-webui's web search in terms of detail, which sounds
| useful, thanks for making it open source.
|
| It seems to have a weird behavior of specifying a date when I
| didn't ask for it, is this expected? Also, I feel like
| searching "questions" is not optimal for most search engines,
| and it should instead search in terms of keywords.
|
| Also, I wish there can be a more informative log at a
| slightly higher level - I don't need to see every request
| being made, but I do want to see a summary of what's
| happening at each step, like the prompt used, result, and the
| new search being done.
|
| On another note, for local models, reasoning models have
| significant advantages over non-reasoning models. Can they be
| used for this?
| learningcircuit wrote:
| Very good ideas I will try to include them.
|
| Thinking models... You can use them. In fact I started the
| project with them but not sure they help too much for this
| task. They definitely make it slower
| antonkar wrote:
| I think the guy who'll make the 3D game-like GUI for LLMs is the
| next Jobs/Gates/Musk and Nobel Prize Winner (I think it'll solve
| alignment by having millions of eyes on the internals of LLMs),
| because computers became popular only after the OS with a GUI
| appeared, current chatbots are a bit like a command line in
| comparison. I just started ASK HN to let people and me share
| their AI safety ideas, both crazy and not:
| https://news.ycombinator.com/item?id=43332593
| tecleandor wrote:
| You just posted the same comment three times in three different
| posts in 10 minutes. I'd say it would be nice to take it a bit
| slower...
| antonkar wrote:
| Yep, it's a bit different, I won't do it again. The problem
| is important, I wanted to hear other people's ideas, you can
| google "share AI safety ideas", I posted the same question in
| a bunch of places and it created some discussions
| jeffreyw128 wrote:
| This is cool!
|
| If you want to add embeddings over internet as a source, you
| should try out exa.ai. Includes: wikipedia, tens of thousands of
| news feeds, Github, 70M+ papers including all of arxiv, etc.
|
| disclaimer: I am one of the founders (:
| nhggfu wrote:
| looks siiiick. congrats + good luck
| learningcircuit wrote:
| I will add it. Its very easy to integrate new search engines.
| ein0p wrote:
| Is there some kind of a tool which would provide AI search
| experience _and mix in the contents from my bookmarks_ (that is,
| fetch/cache/index/RAG the contents of pages those bookmarks point
| to) when creating the report? Bookmarking is an useless dumpster
| fire right now. This could make it useful again.
|
| Currently the failure mode I see quite often in e.g. OpenAIs deep
| research is it sources its answer from an obviously low-authority
| source and provides a reference to that as if it's a scientific
| journal. The answer gets screwed up by that as well, because such
| sources rarely contain anything of value, and even if other
| sources are high quality, low quality source(s) mess everything
| up.
|
| Emphasizing the content I've already curated (via bookmarks)
| could significantly boost the SNR.
| learningcircuit wrote:
| If you have PDF collection you could include it into the local
| search and give it very high relevance?
___________________________________________________________________
(page generated 2025-03-11 23:01 UTC)