https://aclanthology.org/2024.acl-long.245/ ACL Logo ACL Anthology * News(current) * FAQ(current) * Corrections(current) * Submissions(current) * Github [ ] Graph Language Models Moritz Plenz, Anette Frank --------------------------------------------------------------------- Abstract While Language Models (LMs) are the workhorses of NLP, their interplay with structured knowledge graphs (KGs) is still actively researched. Current methods for encoding such graphs typically either (i) linearize them for embedding with LMs - which underutilize structural information, or (ii) use Graph Neural Networks (GNNs) to preserve the graph structure - but GNNs cannot represent text features as well as pretrained LMs. In our work we introduce a novel LM type, the Graph Language Model (GLM), that integrates the strengths of both approaches and mitigates their weaknesses. The GLM parameters are initialized from a pretrained LM to enhance understanding of individual graph concepts and triplets. Simultaneously, we design the GLM's architecture to incorporate graph biases, thereby promoting effective knowledge distribution within the graph. This enables GLMs to process graphs, texts, and interleaved inputs of both. Empirical evaluations on relation classification tasks show that GLM embeddings surpass both LM- and GNN-based baselines in supervised and zero-shot setting, demonstrating their versatility. Anthology ID: 2024.acl-long.245 Volume: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) Month: August Year: 2024 Address: Bangkok, Thailand Editors: Lun-Wei Ku, Andre Martins, Vivek Srikumar Venue: ACL SIG: Publisher: Association for Computational Linguistics Note: Pages: 4477-4494 Language: URL: https://aclanthology.org/2024.acl-long.245 DOI: Bibkey: plenz-frank-2024-graph Cite (ACL): Moritz Plenz and Anette Frank. 2024. Graph Language Models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4477-4494, Bangkok, Thailand. Association for Computational Linguistics. Cite (Informal): Graph Language Models (Plenz & Frank, ACL 2024) Copy Citation: BibTeX Markdown MODS XML Endnote More options... PDF: https://aclanthology.org/2024.acl-long.245.pdf PDF Cite Search --------------------------------------------------------------------- Export citation x * BibTeX * MODS XML * Endnote * Preformatted @inproceedings{plenz-frank-2024-graph, title = "Graph Language Models", author = "Plenz, Moritz and Frank, Anette", editor = "Ku, Lun-Wei and Martins, Andre and Srikumar, Vivek", booktitle = "Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)", month = aug, year = "2024", address = "Bangkok, Thailand", publisher = "Association for Computational Linguistics", url = "https://aclanthology.org/2024.acl-long.245", pages = "4477--4494", abstract = "While Language Models (LMs) are the workhorses of NLP, their interplay with structured knowledge graphs (KGs) is still actively researched. Current methods for encoding such graphs typically either (i) linearize them for embedding with LMs {--} which underutilize structural information, or (ii) use Graph Neural Networks (GNNs) to preserve the graph structure {--} but GNNs cannot represent text features as well as pretrained LMs. In our work we introduce a novel LM type, the Graph Language Model (GLM), that integrates the strengths of both approaches and mitigates their weaknesses. The GLM parameters are initialized from a pretrained LM to enhance understanding of individual graph concepts and triplets. Simultaneously, we design the GLM{'}s architecture to incorporate graph biases, thereby promoting effective knowledge distribution within the graph. This enables GLMs to process graphs, texts, and interleaved inputs of both. Empirical evaluations on relation classification tasks show that GLM e mbeddings surpass both LM- and GNN-based baselines in supervised and zero-shot setting, demonstrating their versatility.", } Download as File Copy to Clipboard Graph Language Models Moritz Plenz author Anette Frank author 2024-08 text Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) Lun-Wei Ku editor Andre Martins editor Vivek Srikumar editor Association for Computational Linguistics Bangkok, Thailand conference publication While Language Models (LMs) are the workhorses of NLP, their interplay with structured knowledge graphs (KGs) is still actively researched. Current methods for encoding such graphs typically either (i) linearize them for embedding with LMs - which underutilize structural information, or (ii) use Graph Neural Networks (GNNs) to preserve the graph structure - but GNNs cannot represent text features as well as pretrained LMs. In our work we introduce a novel LM type, the Graph Language Model (GLM), that integrates the strengths of both approaches and mitigates their weaknesses. The GLM parameters are initialized from a pretrained LM to enhance understanding of individual graph concepts and triplets. Simultaneously, we design the GLM's architecture to incorporate graph biases, thereby promoting effective knowledge distribution within the graph. This enables GLMs to process graphs, texts, and interleaved inputs of both. Empirical evaluations on relation classification tasks show that GLM embeddings surpass both LM- and GNN-based baselines in supervised and zero-shot setting, demonstrating their versatility. plenz-frank-2024-graph https://aclanthology.org/2024.acl-long.245 2024-08 4477 4494 Download as File Copy to Clipboard %0 Conference Proceedings %T Graph Language Models %A Plenz, Moritz %A Frank, Anette %Y Ku, Lun-Wei %Y Martins, Andre %Y Srikumar, Vivek %S Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) %D 2024 %8 August %I Association for Computational Linguistics %C Bangkok, Thailand %F plenz-frank-2024-graph %X While Language Models (LMs) are the workhorses of NLP, their interplay with structured knowledge graphs (KGs) is still actively researched. Current methods for encoding such graphs typically either (i) linearize them for embedding with LMs - which underutilize structural information, or (ii) use Graph Neural Networks (GNNs) to preserve the graph structure - but GNNs cannot represent text features as well as pretrained LMs. In our work we introduce a novel LM type, the Graph Language Model (GLM), that integrates the strengths of both approaches and mitigates their weaknesses. The GLM parameters are initialized from a pretrained LM to enhance understanding of individual graph concepts and triplets. Simultaneously, we design the GLM's architecture to incorporate graph biases, thereby promoting effective knowledge distribution within the graph. This enables GLMs to process graphs, texts, and interleaved inputs of both. Empirical evaluations on relation classification tasks show that GLM embeddings surpass bot h LM- and GNN-based baselines in supervised and zero-shot setting, demonstrating their versatility. %U https://aclanthology.org/2024.acl-long.245 %P 4477-4494 Download as File Copy to Clipboard Markdown (Informal) [Graph Language Models](https://aclanthology.org/2024.acl-long.245) (Plenz & Frank, ACL 2024) * Graph Language Models (Plenz & Frank, ACL 2024) ACL * Moritz Plenz and Anette Frank. 2024. Graph Language Models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4477-4494, Bangkok, Thailand. Association for Computational Linguistics. Copy Markdown to Clipboard Copy ACL to Clipboard Creative Commons License ACL materials are Copyright (c) 1963-2024 ACL; other materials are copyrighted by their respective copyright holders. Materials prior to 2016 here are licensed under the Creative Commons Attribution-NonCommercial-ShareAlike 3.0 International License. Permission is granted to make copies for the purposes of teaching and research. Materials published in or after 2016 are licensed on a Creative Commons Attribution 4.0 International License. The ACL Anthology is managed and built by the ACL Anthology team of volunteers. Site last built on 02 September 2024 at 14:51 UTC with commit c936af1 .