https://arxiv.org/abs/2308.08742

Skip to main content
Cornell University
We gratefully acknowledge support from the Simons Foundation, member
institutions, and all contributors. Donate
 
arxiv logo > cs > arXiv:2308.08742
[                    ]

Help | Advanced Search

[All fields        ]
Search
arXiv logo
Cornell University Logo
[                    ] GO
quick links

  * Login
  * Help Pages
  * About

Computer Science > Computation and Language

arXiv:2308.08742 (cs)
[Submitted on 17 Aug 2023 (v1), last revised 22 Aug 2023 (this
version, v2)]

Title:PMET: Precise Model Editing in a Transformer

Authors:Xiaopeng Li, Shasha Li, Shezheng Song, Jing Yang, Jun Ma, Jie
Yu
Download a PDF of the paper titled PMET: Precise Model Editing in a
Transformer, by Xiaopeng Li and 5 other authors
Download PDF

    Abstract: Model editing techniques modify a minor proportion of
    knowledge in Large Language Models (LLMs) at a relatively low
    cost, which have demonstrated notable success. Existing methods
    assume Transformer Layer (TL) hidden states are values of
    key-value memories of the Feed-Forward Network (FFN). They
    usually optimize the TL hidden states to memorize target
    knowledge and use it to update the weights of the FFN in LLMs.
    However, the information flow of TL hidden states comes from
    three parts: Multi-Head Self-Attention (MHSA), FFN, and residual
    connections. Existing methods neglect the fact that the TL hidden
    states contains information not specifically required for FFN.
    Consequently, the performance of model editing decreases. To
    achieve more precise model editing, we analyze hidden states of
    MHSA and FFN, finding that MHSA encodes certain general knowledge
    extraction patterns. This implies that MHSA weights do not
    require updating when new knowledge is introduced. Based on above
    findings, we introduce PMET, which simultaneously optimizes
    Transformer Component (TC, namely MHSA and FFN) hidden states,
    while only using the optimized TC hidden states of FFN to
    precisely update FFN weights. Our experiments demonstrate that
    PMET exhibits state-of-the-art performance on both the
    COUNTERFACT and zsRE datasets. Our ablation experiments
    substantiate the effectiveness of our enhancements, further
    reinforcing the finding that the MHSA encodes certain general
    knowledge extraction patterns and indicating its storage of a
    small amount of factual knowledge. Our code is available at this
    https URL.

Comments: Preprint. Under review
Subjects: Computation and Language (cs.CL); Artificial Intelligence
          (cs.AI); Machine Learning (cs.LG)
Cite as:  arXiv:2308.08742 [cs.CL]
          (or arXiv:2308.08742v2 [cs.CL] for this version)
          https://doi.org/10.48550/arXiv.2308.08742
          Focus to learn more
          arXiv-issued DOI via DataCite

Submission history

From: Xiaopeng Li [view email]
[v1] Thu, 17 Aug 2023 02:33:43 UTC (512 KB)
[v2] Tue, 22 Aug 2023 03:19:16 UTC (521 KB)
Full-text links:

Download:

  * Download a PDF of the paper titled PMET: Precise Model Editing in
    a Transformer, by Xiaopeng Li and 5 other authors
    PDF
  * Other formats

(license)
Current browse context:
cs.CL
< prev   |   next >
new | recent | 2308
Change to browse by:
cs
cs.AI
cs.LG

References & Citations

  * NASA ADS
  * Google Scholar
  * Semantic Scholar

a export BibTeX citation Loading...

BibTeX formatted citation

x
[loading...          ]
Data provided by:

Bookmark

BibSonomy logo Reddit logo
(*) Bibliographic Tools

Bibliographic and Citation Tools

[ ] Bibliographic Explorer Toggle
Bibliographic Explorer (What is the Explorer?)
[ ] Litmaps Toggle
Litmaps (What is Litmaps?)
[ ] scite.ai Toggle
scite Smart Citations (What are Smart Citations?)
( ) Code, Data, Media

Code, Data and Media Associated with this Article

[ ] Links to Code Toggle
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
[ ] DagsHub Toggle
DagsHub (What is DagsHub?)
[ ] Links to Code Toggle
Papers with Code (What is Papers with Code?)
[ ] ScienceCast Toggle
ScienceCast (What is ScienceCast?)
( ) Demos

Demos

[ ] Replicate Toggle
Replicate (What is Replicate?)
[ ] Spaces Toggle
Hugging Face Spaces (What is Spaces?)
( ) Related Papers

Recommenders and Search Tools

[ ] Link to Influence Flower
Influence Flower (What are Influence Flowers?)
[ ] Connected Papers Toggle
Connected Papers (What is Connected Papers?)
[ ] Core recommender toggle
CORE Recommender (What is CORE?)

  * Author
  * Venue
  * Institution
  * Topic

( ) About arXivLabs

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and
share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have
embraced and accepted our values of openness, community, excellence,
and user data privacy. arXiv is committed to these values and only
works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community?
Learn more about arXivLabs.

Which authors of this paper are endorsers? | Disable MathJax (What is
MathJax?)

  * About
  * Help

  * Click here to contact arXiv Contact
  * Click here to subscribe Subscribe

  * Copyright
  * Privacy Policy

  * Web Accessibility Assistance
  * arXiv Operational Status
    Get status notifications via email or slack