[HN Gopher] Graph of Thoughts: Solving Elaborate Problems with L...
___________________________________________________________________
Graph of Thoughts: Solving Elaborate Problems with Large Language
Models
Author : jonbaer
Score : 201 points
Date : 2023-08-24 13:44 UTC (9 hours ago)
(HTM) web link (arxiv.org)
(TXT) w3m dump (arxiv.org)
| refulgentis wrote:
| Need more data.
|
| - Complex generalization with a simple unstated justification:
| last 'paper' like this was ToT, and a tree is a graph with
| constraints.
|
| - Framework is discussed cognitively, units of "thoughts"
| "scored". (AutoGPT redux, having the LLM eat its own output
| repetitively improves things but isn't a panacea)
|
| - Only sorting demonstrated "due to space constraints" -- unclear
| what that means, it seems much more likely it was self-enforced
| time constraints
|
| - Error rate is consistently 14%.
|
| - ~10x the cost for ~15% error rate in sorting instead of ~30%
|
| - GPT3.5
| creer wrote:
| I keep feeling that LLMs are one direction to address the thorny
| "common sense" issue of AI. Mountains of training text
| incorporate, probably, most common sense (and a lot of nonsense).
| It's beautiful to see so many ideas come out currently to make
| better use of the models. Including the fast progress made with
| image generation.
| marcopicentini wrote:
| What are other use cases that could be made only by LLM ?
|
| Number sorting is faster using code.
| creer wrote:
| I don't think efficiency is important at this point. Finding
| that it's possible "this way" opens the door for more work and
| more applications. (Which doesn't prevent others to already
| work on efficiency.)
| empath-nirvana wrote:
| The point of using number sorting for this paper is that its
|
| A) difficult to impossible for an LLM to do in a single pass B)
| easy to verify the correctness.
|
| In general, the point isn't finding things that only an LLM can
| do, but find things that LLMs can do with decent results at
| lower cost than getting a human to do it.
| jbay808 wrote:
| It is only difficult for a LLM to sort a list of numbers if
| the list is longer than half of the context window. (Source:
| I tested this myself[1]). The sorts are not error-free
| _every_ time, but with sufficient training they become error-
| free the vast majority of the time, even for long lists. This
| is not especially surprising because transformers are capable
| of directly representing sorting programs.[2]
|
| [1] https://jbconsulting.substack.com/p/its-not-just-
| statistics-...
|
| [2] https://arxiv.org/abs/2106.06981
| [deleted]
| firewolf34 wrote:
| The more I read about ML, the more I begin to believe that -
| psychologically speaking - hierarchy (esp. graph structures,
| trees) are absolutely core to advanced information processing in
| general.
| brutusborn wrote:
| I won't pretend to understand it, but this reminds me of the
| idea of markov blankets when using the free energy principle to
| model congition. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7
| 955287/#:~:tex....
| macawfish wrote:
| This article is about how non-hierarchical graphs, those with
| cycles, are performing better than trees or chains.
| theptip wrote:
| I suspect that you'll still find strong hierarchy in an
| optimized/well-performing graph of thought. The human brain,
| for example, also has recurrence, but it's limited.
|
| It seems pretty intuitive that you'd get a "task / subtask"
| split for example, with feedback from the latter, but
| semantic content largely flowing from the former to the
| latter.
| visarga wrote:
| We got symbolic AI sneaked into the connectionist model by
| making a graph of thoughts. A graph can explicitly implement
| any algorithm or data structure.
|
| They could make it more efficient by implementing a kind of
| "hard attention". Each token should have access to a sparse
| subset of the whole input, so it would be like a node in a
| graph only having access to a few neighbours. Could solve the
| very large context issue. This can also be parallelised,
| running all thought nodes in parallel, of course each with a
| sparse view of the whole input making it much faster.
|
| For example when reading a long book, the model would spawn
| nodes for each person, location or event of interest, and they
| would track the source text as the action develops. A mental
| map of the book. That would surely help a model deal with many
| moving pieces of information.
|
| Or when solving a problem, the model could spawn a node to work
| on a subproblem, parametrised by the parent node with the right
| inputs. Then the node would report back with the answer and the
| parent continues. This would work with recursive calls.
|
| The new cpu is the LLM and the clock tick is 1 token.
| barrenko wrote:
| Could you expand on "A graph can explicitly implement any
| algorithm or data structure."?
| gryn wrote:
| you can use graph transformation to perform general
| computation.
|
| https://en.wikipedia.org/wiki/Graph_rewriting
|
| probably not was GP meant, but something along those lines.
| visarga wrote:
| You could create a node for each execution step, or data
| field.
| Vox_Leone wrote:
| And thus you could use UML to formalize prompting. An
| activity diagram can be viewed as a chain of thought.
| Fulfilling the UML promise.
| zalyalov wrote:
| Weird that they claim to use arbitrary graphs, while in reality
| it's a weird subclass of DAGs with one-vertex loops kind of
| allowed, except they don't really make sense to be represented as
| loops.
| Cloudly wrote:
| https://github.com/spcl/graph-of-thoughts Code for the paper too
| knexer wrote:
| This is a really natural extension of CoT. I was experimenting
| for a month or two with a similar concept in a hobby project this
| past spring: https://github.com/knexer/llmtaskgraph . I'm really
| excited to see more people exploring in this direction!
|
| I was focusing more on an engineering perspective; modeling a
| complex LLM-and-code process as a dependency graph makes it easy
| to:
|
| - add tracing to continuously measure and monitor even post-
| deployment
|
| - perform reproducible experiments, a la time-rewinding debugging
|
| - speed up iteration on prompts by caching the parts of the
| program you aren't working on right now
|
| My test case was using GPT4 to implement the operators in a
| genetic algorithm, which tbh is a fascinating concept of its own.
| I drifted away after a while (curse that ADHD) but had a great
| time with the project in the meantime.
| brutusborn wrote:
| This is fantastic. I'd love to see a system that uses an LLM to
| generate knowledge graphs from academic papers to make them
| machine readable.
|
| Some kind of prompt like "does paper P contain idea A and does it
| suggest that A is true." Then you could automatically categorise
| citations by whether they agree/disagree with the cited paper.
|
| Sometimes I see papers with 2,000 citations and I wonder: how
| many of those are dis/agreeing with the paper.
| nvm0n2 wrote:
| This has already been studied. Negative citations are
| vanishingly rare. So virtually all of them will be either
| neutral or positive.
| photonthug wrote:
| Applications to case law might be interesting since
| establishing precedent is somewhat more nonbinary
| jll29 wrote:
| > Sometimes I see papers with 2,000 citations and I wonder:
| how many of those are dis/agreeing with the paper.
|
| One example of an author that is very influential, despite
| causing a lot of disagreement (even in more than one
| discipline) is Noam Chomsky, who is also the most cited
| person alive, and the second most cited person in recorded
| history after Aristotle. His views about generative grammar
| are in part revolutionary, in part plain wrong; your
| assessment of his views about the Palestine conflict and U.S.
| foreign politics will largely depend on your political
| leanings; and his contribution to formal language theory is
| fundamental regardless of your leanings (Chomsky hierarchy;
| Chomsky Normal Form).
|
| > This has already been studied. Negative citations are
| vanishingly rare. So virtually all of them will be either
| neutral or positive. Might be a difference between
| science/engineering (where true) and humanities (where a
| larger amount is negative).
| anentropic wrote:
| There are already models which are specialised to this task,
| e.g. https://huggingface.co/Babelscape/rebel-large (if I
| understood you correctly)
|
| Though with LLM and sufficient context length you could
| probably just use that prompt directly on the academic paper
| without ever generating a knowledge graph
| throwaway290 wrote:
| By that logic why don't you just directly ask the LLM on
| whether a citation agrees or not, you are already trusting it
| to be correct with that graph in the first place...
| 3abiton wrote:
| I already feel synthesis is becoming a useless skill for
| humans.
| brutusborn wrote:
| You are correct, I wrote that in a rush and mixed up examples
| in my head.
|
| I don't think you need to trust the LLMs for this kind of
| thing to be very useful. The LLM could generate the KG with
| every node labelled as "autogenerated." When you use the
| graph for research, you are still going to read the papers
| you are interested in so you can then update the relevant
| citation node with the label "human checked."
|
| If a research group uses the same graph over time, the nodes
| will gradually become "trustworthy" (ie verified by humans).
| Maybe even get reviewers to update a papers graph during
| review and publish that for other groups to add to their
| graphs.
| visarga wrote:
| That's a great idea. It would be easier to know if anyone
| done what you want, get a better overview of the current
| knowledge.
| jstx1 wrote:
| How well do these papers replicate? In some of experiments with
| GPT-4 I've seen chain-of-thought style prompting make answers
| noticeably worse than plainly asking a question once.
| rmbyrro wrote:
| Really? Do you have any examples to share? I'd be surprised to
| see that in action.
| sorokod wrote:
| For the kind of synergy I demand, only the hypergraph of thought
| ( HoT ) will do.
| groceryheist wrote:
| Intractable. Only the simplical complex of thought provides the
| optimal balance of expressiveness and constraint.
| sorokod wrote:
| Well ok, maybe only for distilling the essence of whole
| networks of thoughts
___________________________________________________________________
(page generated 2023-08-24 23:00 UTC)