https://news.mit.edu/2024/large-language-models-use-surprisingly-simple-mechanism-retrieve-stored-knowledge-0325 Skip to content | Massachusetts Institute of Technology MIT Top Menu| * Education * Research * Innovation * Admissions + Aid * Campus Life * News * Alumni * About MIT * More | Search MIT Search websites, locations, and people [ ] See More Results Suggestions or feedback? MIT News | Massachusetts Institute of Technology Subscribe to MIT News newsletter Browse Enter keywords to search for news articles: [ ] Submit Browse By Topics View All - Explore: * Machine learning * Social justice * Startups * Black holes * Classes and programs Departments View All - Explore: * Aeronautics and Astronautics * Brain and Cognitive Sciences * Architecture * Political Science * Mechanical Engineering Centers, Labs, & Programs View All - Explore: * Abdul Latif Jameel Poverty Action Lab (J-PAL) * Picower Institute for Learning and Memory * Media Lab * Lincoln Laboratory Schools * School of Architecture + Planning * School of Engineering * School of Humanities, Arts, and Social Sciences * Sloan School of Management * School of Science * MIT Schwarzman College of Computing View all news coverage of MIT in the media - Listen to audio content from MIT News - Subscribe to MIT newsletter - Close Breadcrumb 1. MIT News 2. Large language models use a surprisingly simple mechanism to retrieve some stored knowledge Large language models use a surprisingly simple mechanism to retrieve some stored knowledge Researchers demonstrate a technique that can be used to probe a model to see what it knows about new subjects. Adam Zewe | MIT News Publication Date: March 25, 2024 Press Inquiries Press Contact: Abby Abazorius Email: abbya@mit.edu Phone: 617-253-2709 MIT News Office Close Illustration of a blue robot-man absorbing and generating info. On left are research and graph icons going into his brain. On right are speech bubble icons, as if in conversation. Caption: Researchers from MIT and elsewhere found that complex large language machine-learning models use a simple mechanism to retrieve stored knowledge when they respond to a user prompt. The researchers can leverage these simple mechanisms to see what the model knows about different subjects, and also possibly correct false information that it has stored. Credits: Image: iStock Previous image Next image Large language models, such as those that power popular artificial intelligence chatbots like ChatGPT, are incredibly complex. Even though these models are being used as tools in many areas, such as customer support, code generation, and language translation, scientists still don't fully grasp how they work. In an effort to better understand what is going on under the hood, researchers at MIT and elsewhere studied the mechanisms at work when these enormous machine-learning models retrieve stored knowledge. They found a surprising result: Large language models (LLMs) often use a very simple linear function to recover and decode stored facts. Moreover, the model uses the same decoding function for similar types of facts. Linear functions, equations with only two variables and no exponents, capture the straightforward, straight-line relationship between two variables. The researchers showed that, by identifying linear functions for different facts, they can probe the model to see what it knows about new subjects, and where within the model that knowledge is stored. Using a technique they developed to estimate these simple functions, the researchers found that even when a model answers a prompt incorrectly, it has often stored the correct information. In the future, scientists could use such an approach to find and correct falsehoods inside the model, which could reduce a model's tendency to sometimes give incorrect or nonsensical answers. "Even though these models are really complicated, nonlinear functions that are trained on lots of data and are very hard to understand, there are sometimes really simple mechanisms working inside them. This is one instance of that," says Evan Hernandez, an electrical engineering and computer science (EECS) graduate student and co-lead author of a paper detailing these findings. Hernandez wrote the paper with co-lead author Arnab Sharma, a computer science graduate student at Northeastern University; his advisor, Jacob Andreas, an associate professor in EECS and a member of the Computer Science and Artificial Intelligence Laboratory (CSAIL); senior author David Bau, an assistant professor of computer science at Northeastern; and others at MIT, Harvard University, and the Israeli Institute of Technology. The research will be presented at the International Conference on Learning Representations. Finding facts Most large language models, also called transformer models, are neural networks. Loosely based on the human brain, neural networks contain billions of interconnected nodes, or neurons, that are grouped into many layers, and which encode and process data. Much of the knowledge stored in a transformer can be represented as relations that connect subjects and objects. For instance, "Miles Davis plays the trumpet" is a relation that connects the subject, Miles Davis, to the object, trumpet. As a transformer gains more knowledge, it stores additional facts about a certain subject across multiple layers. If a user asks about that subject, the model must decode the most relevant fact to respond to the query. If someone prompts a transformer by saying "Miles Davis plays the. . ." the model should respond with "trumpet" and not "Illinois" (the state where Miles Davis was born). "Somewhere in the network's computation, there has to be a mechanism that goes and looks for the fact that Miles Davis plays the trumpet, and then pulls that information out and helps generate the next word. We wanted to understand what that mechanism was," Hernandez says. The researchers set up a series of experiments to probe LLMs, and found that, even though they are extremely complex, the models decode relational information using a simple linear function. Each function is specific to the type of fact being retrieved. For example, the transformer would use one decoding function any time it wants to output the instrument a person plays and a different function each time it wants to output the state where a person was born. The researchers developed a method to estimate these simple functions, and then computed functions for 47 different relations, such as "capital city of a country" and "lead singer of a band." While there could be an infinite number of possible relations, the researchers chose to study this specific subset because they are representative of the kinds of facts that can be written in this way. They tested each function by changing the subject to see if it could recover the correct object information. For instance, the function for "capital city of a country" should retrieve Oslo if the subject is Norway and London if the subject is England. Functions retrieved the correct information more than 60 percent of the time, showing that some information in a transformer is encoded and retrieved in this way. "But not everything is linearly encoded. For some facts, even though the model knows them and will predict text that is consistent with these facts, we can't find linear functions for them. This suggests that the model is doing something more intricate to store that information," he says. Visualizing a model's knowledge They also used the functions to determine what a model believes is true about different subjects. In one experiment, they started with the prompt "Bill Bradley was a" and used the decoding functions for "plays sports" and "attended university" to see if the model knows that Sen. Bradley was a basketball player who attended Princeton. "We can show that, even though the model may choose to focus on different information when it produces text, it does encode all that information," Hernandez says. They used this probing technique to produce what they call an "attribute lens," a grid that visualizes where specific information about a particular relation is stored within the transformer's many layers. Attribute lenses can be generated automatically, providing a streamlined method to help researchers understand more about a model. This visualization tool could enable scientists and engineers to correct stored knowledge and help prevent an AI chatbot from giving false information. In the future, Hernandez and his collaborators want to better understand what happens in cases where facts are not stored linearly. They would also like to run experiments with larger models, as well as study the precision of linear decoding functions. "This is an exciting work that reveals a missing piece in our understanding of how large language models recall factual knowledge during inference. Previous work showed that LLMs build information-rich representations of given subjects, from which specific attributes are being extracted during inference. This work shows that the complex nonlinear computation of LLMs for attribute extraction can be well-approximated with a simple linear function," says Mor Geva Pipek, an assistant professor in the School of Computer Science at Tel Aviv University, who was not involved with this work. This research was supported, in part, by Open Philanthropy, the Israeli Science Foundation, and an Azrieli Foundation Early Career Faculty Fellowship. Share this news article on: * X * Facebook * LinkedIn * Reddit * Print Paper Paper: "Linearity of Relation Decoding in Transformer Language Models" Related Links * Evan Hernandez * Jacob Andreas * Language and Intelligence Group * Computer Science and Artificial Intelligence Laboratory * Department of Electrical Engineering and Computer Science * School of Engineering * MIT Schwarzman College of Computing Related Topics * Research * Computer science and technology * Artificial intelligence * Machine learning * Algorithms * Human-computer interaction * Computer Science and Artificial Intelligence Laboratory (CSAIL) * Electrical Engineering & Computer Science (eecs) * School of Engineering * MIT Schwarzman College of Computing Related Articles example of image system can understand Demystifying machine-learning systems Digital illustration of a white robot with a magnifying glass, looking at a circuit-style display of a battery with a brain icon. The room resembles a lab with a white table, and there are two tech-themed displays on the wall showing abstract neural structures in glowing turquoise. A wire connects the robot's magnifying glass to the larger display. AI agents help explain other AI systems Jacob Andreas leans forward with his arms resting on the table, speaking to the photographer. Outdated computer hardware is on either side of him. 3 Questions: Jacob Andreas on large language models A blue neural network is in a dark void. A green spotlight shines down on the network and reveals a hidden layer underneath. The green light shows a new, white neural network below. Solving a machine-learning mystery Previous item Next item More MIT News Three close up photos of speakers at a conference: Julie Shah, Ben Armstrong, and Kate Kellogg MIT launches Working Group on Generative AI and the Work of the Future New initiative is convening leading companies and nonprofits with support from Google's Community Grants Fund. Read full story - Two men in hardhats and safety vests, seen from behind, inspect a forest of electrical pylons and wires on a cloudless day Atmospheric observations in China show rise in emissions of a potent greenhouse gas Global warming potential of sulfur hexafluoride (SF6) is more than 24,000 times that of carbon dioxide. Read full story - A view of the steps and columns of 77 Mass Ave, as seen through The Alchemist Sculpture. Glimpses of the numbers and mathematical symbols are seen around the image. Second round of seed grants awarded to MIT scholars studying the impact and applications of generative AI The 16 finalists -- representing every school at MIT -- will explore generative AI's impact on privacy, art, drug discovery, aging, and more. Read full story - A view from behind of about 15 people dressed head-to-toe in white cleanroom suits, facing another gowned-up individual gesturing as they speak. VIAVI Solutions joins MIT.nano Consortium International technology company becomes sustaining member of industry group. Read full story - Four teenagers sit at a classroom table and write. Is it the school, or the students? Study shows perceptions of "good" schools are heavily dependent on the preparation of the students entering them. Read full story - A young Black woman wearing a brightly patterned top and braids smiles over a set table at a restaurant. Student spotlight: Victory Yinka-Banjo The junior, who is majoring in computer science and molecular biology, wants to "make it a norm to lift others as I continue to climb." Read full story - * More news on MIT News homepage - More about MIT News at Massachusetts Institute of Technology This website is managed by the MIT News Office, part of the Institute Office of Communications. News by Schools/College: * School of Architecture and Planning * School of Engineering * School of Humanities, Arts, and Social Sciences * MIT Sloan School of Management * School of Science * MIT Schwarzman College of Computing Resources: * About the MIT News Office * MIT News Press Center * Terms of Use * Press Inquiries * Filming Guidelines * RSS Feeds Tools: * Subscribe to MIT Daily/Weekly * Subscribe to press releases * Submit campus news * Guidelines for campus news contributors * Guidelines on generative AI Massachusetts Institute of Technology MIT Top Level Links: * Education * Research * Innovation * Admissions + Aid * Campus Life * News * Alumni * About MIT * Join us in building a better world. Massachusetts Institute of Technology 77 Massachusetts Avenue, Cambridge, MA, USA Recommended Links: * Visit * Map (opens in new window) * Events (opens in new window) * People (opens in new window) * Careers (opens in new window) * Contact * Privacy * Accessibility * + Social Media Hub + MIT on X + MIT on Facebook + MIT on YouTube + MIT on Instagram