[HN Gopher] Self-Retrieval: Building an information retrieval sy...
___________________________________________________________________
Self-Retrieval: Building an information retrieval system with one
LLM
Author : PaulHoule
Score : 168 points
Date : 2024-03-09 01:46 UTC (21 hours ago)
(HTM) web link (arxiv.org)
(TXT) w3m dump (arxiv.org)
| skybrian wrote:
| This sounds like an interesting way to ensure accurate quotes:
|
| > To accurately generate the exact passages in the given corpus,
| we employ a trie-based constrained decoding algorithm (Chen et
| al., 2020; Cao et al., 2021; Lu et al., 2021) in which the
| generated tokens can be constrained in the dynamic vocabulary.
| Specifically, instead of generating a token from the entire
| target vocabulary at each step, we use a prefix tree (trie) to
| constraint the target vocabulary and ensure that the generated
| content is within the corpus. During the construction of trie, we
| remove stop words from the initial token to improve semantic
| representation of the trie.
| mmoskal wrote:
| This kind of low level strategies can be tried on many models
| using AICI, including the substring one
| https://github.com/microsoft/aici/blob/main/controllers/aici...
| skybrian wrote:
| I wonder if anyone has tried giving an LLM a tool that
| ensures that it copies a quote from a document accurately? It
| seems like it would be a pretty simple way to avoid some
| hallucinations.
| visarga wrote:
| Yes, it would be good for extractive QA
| worldsayshi wrote:
| Do I understand it correctly if the idea here is to insert
| some logic into the inference loop?
|
| I might not have a clear idea how the inference loop works
| but it sounds like a whole host of solutions could present
| themselves if it was easy to plug in various types of logic
| at inference.
| throwup238 wrote:
| Yes the inference loop provides a list of "logits" every
| loop which are possible predictions for the next token.
| LLM parameters like temperature and top_p configure the
| default logic that selects one of those logits but that
| logic can be replaced with anything you want. It just has
| to select the next token and feed it back into inference.
|
| Llama.cpp and its downstream users like ollama have long
| supported using BNF grammars to constrain the output that
| way.
| 3abiton wrote:
| How would this perform compared to RAG?
| potatoman22 wrote:
| I believe you'd use it with RAG. It would ensure the
| model properly sites the text you feed into it.
| threatripper wrote:
| Is that the same method that is used to generate valid JSON by
| disallowing any token that would cause a syntax error?
| swsieber wrote:
| This reminded me of that... but I also don't know enough to
| say 100% for sure.
| skybrian wrote:
| Yes, it's the same principle. When it's in "quote mode" then
| it only allows tokens that continue or end a known quote.
| mirekrusin wrote:
| Indeed.
|
| Use trie constraint for quotes.
|
| Use BNF for grammar languages (json, python etc).
|
| Projects like llama.cpp/ollama should make it
| automatic/dynamic, just rely on triple quote sections where you
| enter those constraint modes automatically.
|
| Ie. every time you enter into section starting with "```json"
| you automatically switch to JSON BNF.
|
| Every time you enter into "```json:Foo" you enter JSON BNF +
| JSON-SCHEMA for Foo object definition.
|
| "```python" for python grammar etc.
|
| "```quote:documentRef" you enter trie based constraints.
|
| "```llm:otherllm" you enter other llm.
|
| "```whatever:whatever" you enter whatever you want.
|
| If you want just json output you start output with "```json"
| and that's it.
|
| As its all inference time it could be plugin based, ie:
|
| 1. character based - given input (from the start of opening
| "```foo") it returns allowed next characters, or
|
| 2. token based - same as above but returns allowed native
| tokens (not sure how performance would behave here, would it be
| acceptable?)
|
| IMHO also very interesting area would be exploring stable AST
| representations for programming languages (a'la darklang I
| believe?) - where variable names are detached from AST itself,
| ie. differently named functions that otherwise have the same
| structure have precisely the same AST representation. This
| would dramatically reduce space to navigate around.
|
| ps. [0] in case somebody else is interested
|
| [0] https://github.com/ollama/ollama/issues/3019
| sroussey wrote:
| Why have that issue on ollama instead of llama.cpp?
| kastnerkyle wrote:
| This same technique, extended can work well for detecting
| plagiarism from the underlying corpus as well, by tracking a
| trie of "good" completions in the n-gram sense, and a longer
| trie of "no-good" completions. This technique was (to my
| knowledge) first shown in [0], and particularly [1] is a really
| interesting video discussing these topics around max-order
| grams even in a Markovian setting. I used this technique a bit
| in symbolic music generation and was quite pleased with the
| results, always planned to work it into whatever next models.
|
| I think there are a lot of methods from these older Markovian
| setups that can be employed in the outputs samplers of modern
| models, as well as the inclusion of structured searches and so
| on. Parts of deep learning have always focused on structured
| output search, but historically the LLM style generative
| setting has not employed these approaches (though I find beam
| search for generative settings needs tweaking, it usually works
| pretty well in smaller scale problems for me).
|
| [0] Avoiding Plagiarism in Markov Sequence Generation,
| Papadopoulos et. al.
| https://axon.cs.byu.edu/Dan/673/papers/papadopoulos.pdf
|
| [1] https://www.youtube.com/watch?v=zmDS5_w27bU
| reissbaker wrote:
| This is interesting, but unless I'm misreading the paper, it
| looks like they're training an LLM on the corpus. I can easily
| see why that would result in better performance than an off-the-
| shelf embeddings model, but... it won't work for a corpus that
| changes frequently, since you'll constantly have to retrain the
| LLM. That's sort of the point of RAG: how do you get the right
| information into an LLM as context, for data that changes so
| frequently that you can't directly train on it?
| jerpint wrote:
| I'm also not sure with this method if tje LLM can exactly cite
| its source, which is another great benefit of RAG
| vintermann wrote:
| There are some interesting corpora that I would like to search
| smartly, that shouldn't change too often. For instance "all
| Norwegian newspapers printed before 1980".
| HellsMaddy wrote:
| This was my takeaway as well. None of the other retrieval
| methods this paper benchmarks against are specifically trained
| on the corpus. I think it would be more fair to compare "Self-
| Retrieval" against models which have been fine-tuned on the
| corpus.
| tinco wrote:
| "Specifically, we treat each original sentence in the document as
| an index and the document itself as the object of the index,
| allowing the LLM to memorize documents and build indexes through
| self-supervised learning"
|
| This is clever, I haven't seen an effective way to train an LLM
| to search a document yet and I can imagine this being very
| effective. I suppose this relies on the over fitting you get when
| fine tuning on a very small dataset.
| nmca wrote:
| I like methods of this flavour, pioneered as I understand it by
| Fabio Petroni. Very elegant, particularly because you can change
| the distribution over substrings in O(params) time instead of
| O(index size).
|
| What's funny about them is that it's a fairly involved procedure
| that turns your language model into an actual stochastic parrot,
| both showing that such a model useful and demonstrating that the
| original parrot concept was rather ill conceived.
| swyx wrote:
| i need someone to explain this paper in direct language, because
| this paper keeps saying things like "internalizes the corpus to
| retrieve into a LLM via a natural language indexing
| architecture." but never ever explains what the heck
| "internalizing" means, and where the "natural language index"
| sits in the architecture. Figure 2 is the only thing somewhat
| resembling an arch diagram and just uses these vague undefined
| terms to describe it. am i missing something or just voicing the
| same confusion everyone else has?
| manojlds wrote:
| Try using Claude and ask questions about the paper
| beardedwizard wrote:
| I sometimes think there is a hidden language these papers use.
| To what end is not clear.
| intalentive wrote:
| "The aim of indexing is to internalize the corpus into the
| parameters of LLM."
|
| They train the LLM directly on the corpus so that the documents
| are embedded in its weights.
|
| There are some ESL issues in the paper but it's not too hard to
| understand.
| swyx wrote:
| > They train the LLM directly on the corpus so that the
| documents are embedded in its weights.
|
| they do not outright say that in the paper as far as i could
| tell. i only got it from reading hn comments. just very
| confused why they use a nonstandard term like "internalize"
| which just pisses me off because ML is hard enough without
| inventing your own terms
| anthonyjacobs wrote:
| This seems compute intensive for rag utilities or im missing
| something, i don't really see the application when you could get
| the same results with sparse embeddings(keywords) then re-
| ranking. It is promising-with context length issues, amnesia
| becomes a problem because the retrieved docs often bloats out the
| context window, this is the right direction for personal llms
| mark_l_watson wrote:
| This is a really good idea. Sort of like custom Silicon, but
| making an application specific LLM.
|
| Perhaps such a custom LLM will be available on HF, Ollama, etc.
| before long?
___________________________________________________________________
(page generated 2024-03-09 23:01 UTC)