[HN Gopher] Graph Language Models
       ___________________________________________________________________
        
       Graph Language Models
        
       Author : Anon84
       Score  : 100 points
       Date   : 2024-09-03 06:49 UTC (16 hours ago)
        
 (HTM) web link (aclanthology.org)
 (TXT) w3m dump (aclanthology.org)
        
       | fisian wrote:
       | In case you haven't heard about Graph NNs yet, pyyorch geometric
       | is a library to build GNNs and has some info how they work [1].
       | 
       | [1]: https://pytorch-geometric.readthedocs.io/
        
       | kelsey98765431 wrote:
       | Graphing is not only the future, it's the present that is
       | happening behind closed doors that you don't get to hear about.
       | The things that can be done already are insane and the
       | capabilities are growing quickly. Agentic graph based systems are
       | the future for ASI.
        
         | freilanzer wrote:
         | And how do you hear about the things happening behind closed
         | doors? Source: just trust me.
        
         | awestroke wrote:
         | The future is looking bright indeed for Addiction Severity
         | Index
        
         | esafak wrote:
         | https://thegradient.pub/transformers-are-graph-neural-networ...
        
         | OutOfHere wrote:
         | Anyone using the word "agentic" has jumped on the loony train.
         | These people use it to discriminate against AIs that they
         | personally dislike, claiming it's not sufficiently "agentic".
         | In reality there is a wide spectrum of how a system uses AI,
         | and there is no hard line separating agentic from non-agentic.
        
       | VHRanger wrote:
       | I've talked about this before on my blog:
       | https://singlelunch.com/2020/12/28/why-im-lukewarm-on-graph-...
       | 
       | Basically, language models are already graph neural networks. To
       | understand why, go back to Word2Vec/GLoVe: word embedding
       | distance represent co-occurrence frequency of words in a
       | sentence.
       | 
       | Note how this is the same as a graph embedding problem: words are
       | nodes, and the edge weight is co-occurence frequency. You embed
       | the graph nodes. In fact, this is stated in formal math in the
       | GLoVe paper.
       | 
       | The LLM architecture is basically doing the same thing, except
       | the graph is conditional occurence based on the previous
       | contextual words.
       | 
       | This setup makes for a graph with a truly astronomic number of
       | nodes (word|context) and edges. This huge graph exists only in
       | the land of abstract math, but it also shows why LLMs require so
       | many parameters to perform well.
       | 
       | In any case, 4 years on, I'm still pretty lukewarm on the current
       | gen of graph neural network architectures.
       | 
       | Case in point: the OP paper is pretty much the classic ML paper
       | mill setup of "take some existing algorithm, add some stuff over
       | it, spend a ton hyperparameter searching on your algo and show it
       | beats some 2 year old baseline".
        
       | intalentive wrote:
       | Yes transformers are a class of graph NN but there can still be
       | value in dealing directly with both text and knowledge graphs
       | (for RAG-style applications, presumably) because the graph
       | enforces additional constraints.
       | 
       | It is like the difference between concrete and abstract syntax.
       | LLMs frequently generate code that won't compile, since they
       | predict tokens not AST nodes. They are underconstrained for the
       | task.
       | 
       | How to address? You can train a single model to handle both, as
       | the authors did, or you can manually enforce constraints while
       | decoding.
        
       ___________________________________________________________________
       (page generated 2024-09-03 23:01 UTC)