https://www.nature.com/articles/d41586-023-04043-w Skip to main content Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript. Advertisement Advertisement Nature * View all journals * Search * Log in * Explore content * About the journal * Publish with us * Subscribe * Sign up for alerts * RSS feed 1. nature 2. news 3. article * NEWS * 14 December 2023 DeepMind AI outdoes human mathematicians on unsolved problem Large language model improves on efforts to solve combinatorics problems inspired by the card game Set. By * Davide Castelvecchi 1. Davide Castelvecchi View author publications You can also search for this author in PubMed Google Scholar * Twitter * Facebook * Email A player holds a hand of Set game cards over a green table. In the game Set, players must identify combinations of cards based on the shape, colour, shading and number of symbols.Credit: Valery Voennyy/Alamy The card game Set has long inspired mathematicians to create interesting problems. Now, a technique based on large language models (LLMs) is showing that artificial intelligence (AI) can help mathematicians to generate new solutions. The AI system, called FunSearch, made progress on Set-inspired problems in combinatorics, a field of mathematics that studies how to count the possible arrangements of sets containing finitely many objects. But its inventors say that the method, described in Nature on 14 December^1, could be applied to a variety of questions in maths and computer science. "This is the first time anyone has shown that an LLM-based system can go beyond what was known by mathematicians and computer scientists," says Pushmeet Kohli, a computer scientist who heads the AI for Science team at Google Deepmind in London. "It's not just novel, it's more effective than anything else that exists today." This is in contrast to previous experiments, in which researchers have used large language models to solve maths problems with known solutions, says Kohli. Mathematical chatbot FunSearch automatically creates requests for a specially trained LLM, asking it to write short computer programs that can generate solutions to a particular mathematical problem. The system then checks quickly to see whether those solutions are better than known ones. If not, it provides feedback to the LLM so that it can improve at the next round. "The way we use the LLM is as a creativity engine," says DeepMind computer scientist Bernardino Romera-Paredes. Not all programs that the LLM generates are useful, and some are so incorrect that they wouldn't even be able to run, he says. But another program can quickly toss the incorrect ones away and test the output of the correct ones. [d41586-023] DeepMind AI invents faster algorithms to solve tough maths puzzles The team tested FunSearch on the 'cap set problem'. This evolved out of the game Set, which was invented in the 1970s by geneticist Marsha Falco. The Set deck contains 81 cards. Each card displays one, two or three symbols that are identical in colour, shape and shading -- and, for each of these features, there are three possible options. Together, these possibilities add up to 3x3x3x3 = 81. Players have to turn over the cards and spot special combinations of three cards called sets. Mathematicians have shown that players are guaranteed to find a set if the number of upturned cards is at least 21. They have also found solutions for more-complex versions of the game, in which abstract versions of the cards have five or more properties. But some mysteries remain. For example, if there are n properties, where n is any whole number, then there are 3n possible cards -- but the minimum number of cards that must be revealed to guarantee a solution is unknown. This problem can be expressed in terms of discrete geometry. There, it is equivalent to finding certain arrangements of three points in an n-dimensional space. Mathematicians have been able to put bounds on the possible general solution -- given n, they have found that the required number of 'cards on the table' must be greater than that given by a certain formula, but smaller than that given by another. Human-machine collaboration FunSearch was able to improve on the lower bound for n = 8, by generating sets of cards that satisfy all the requirements. "We don't prove that we cannot improve over that, but we do get a construction that goes beyond what was known before," says DeepMind computer scientist Alhussein Fawzi. One important feature of FunSearch is that people can see the successful programs created by the LLM and learn from them, says co-author Jordan Ellenberg, a mathematician at the University of Wisconsin-Madison. This sets the technique apart from other applications, in which the AI is a black box. "What's most exciting to me is modelling new modes of human-machine collaboration," Ellenberg adds. "I don't look to use these as a replacement for human mathematicians, but as a force multiplier." doi: https://doi.org/10.1038/d41586-023-04043-w References 1. Romera-Paredes, B. et al. Nature https://doi.org/10.1038/ s41586-023-06924-6 (2023). Article Google Scholar Download references Reprints and Permissions Related Articles * [d41586-023] DeepMind AI invents faster algorithms to solve tough maths puzzles * DeepMind's AI helps untangle the mathematics of knots * How will AI change mathematics? Rise of chatbots highlights discussion Subjects * Computer science * Machine learning * Mathematics and computing Latest on: Computer science Mathematical discoveries from program search with large language models Article 14 DEC 23 ChatGPT and science: the AI system was a force in 2023 -- for good and bad ChatGPT and science: the AI system was a force in 2023 -- for good and bad News Feature 13 DEC 23 OpenAI's chief scientist helped to create ChatGPT -- while worrying about AI safety OpenAI's chief scientist helped to create ChatGPT -- while worrying about AI safety News Feature 13 DEC 23 Machine learning Nature's 10: ten people (and one non-human) who helped shape science in 2023 Nature's 10: ten people (and one non-human) who helped shape science in 2023 News Feature 13 DEC 23 ChatGPT and science: the AI system was a force in 2023 -- for good and bad ChatGPT and science: the AI system was a force in 2023 -- for good and bad News Feature 13 DEC 23 OpenAI's chief scientist helped to create ChatGPT -- while worrying about AI safety OpenAI's chief scientist helped to create ChatGPT -- while worrying about AI safety News Feature 13 DEC 23 Mathematics and computing Mathematical discoveries from program search with large language models Article 14 DEC 23 ChatGPT and science: the AI system was a force in 2023 -- for good and bad ChatGPT and science: the AI system was a force in 2023 -- for good and bad News Feature 13 DEC 23 OpenAI's chief scientist helped to create ChatGPT -- while worrying about AI safety OpenAI's chief scientist helped to create ChatGPT -- while worrying about AI safety News Feature 13 DEC 23 Nature Careers Jobs * Postdoctoral / Research Scientist positions in Molecular Immunology / Cancer Immunology Dallas, Texas (US) The University of Texas Southwestern Medical Center (UT Southwestern Medical Center) [] * Eight Professorship Calls - Geoscience, Green Transition and Climate Research The Geological Survey of Denmark and Greenland (GEUS) is an independent research institution under the Danish Ministry of Climate, Energy and Utili... Copenhagen, Hovedstaden (DK) GEUS [] * W3 Research Professor for Media-Supported Learning in Informal and Extracurricular Learning Venues The Psychology Department of the Faculty of Science at University of Tubingen and the independent Leibniz-Institut fur Wissensmedien (IWM) are seek... Tubingen, Baden-Wurttemberg (DE) Leibniz-Institut fur Wissensmedien (IWM) [] * Assistant Professor (Research) The Department of Surgery, Division of Urology of McGill University is inviting applicants for a position at the rank of Assistant Professor. Montreal, Quebec (CA) McGill University - Faculty of Medicine and Health Sciences - Department of Surgery [] * Diabetes Faculty The College of Health Sciences invites applications for faculty positions in the Diabetes Program. Qatar (QA) University of Doha for Science and Technology [] You have full access to this article via your institution. Download PDF Related Articles * [d41586-023] DeepMind AI invents faster algorithms to solve tough maths puzzles * DeepMind's AI helps untangle the mathematics of knots * How will AI change mathematics? Rise of chatbots highlights discussion Subjects * Computer science * Machine learning * Mathematics and computing Advertisement Sign up to Nature Briefing An essential round-up of science news, opinion and analysis, delivered to your inbox every weekday. Email address [ ] [ ] Yes! Sign me up to receive the daily Nature Briefing email. I agree my information will be processed in accordance with the Nature and Springer Nature Limited Privacy Policy. Sign up * Close Nature Briefing Sign up for the Nature Briefing newsletter -- what matters in science, free to your inbox daily. Email address [ ] Sign up [ ] I agree my information will be processed in accordance with the Nature and Springer Nature Limited Privacy Policy. Close Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing Explore content * Research articles * News * Opinion * Research Analysis * Careers * Books & Culture * Podcasts * Videos * Current issue * Browse issues * Collections * Subjects * Follow us on Facebook * Follow us on Twitter * Subscribe * Sign up for alerts * RSS feed About the journal * Journal Staff * About the Editors * Journal Information * Our publishing models * Editorial Values Statement * Journal Metrics * Awards * Contact * Editorial policies * History of Nature * Send a news tip Publish with us * For Authors * For Referees * Language editing services * Submit manuscript Search Search articles by subject, keyword or author [ ] Show results from [All journals] Search Advanced search Quick links * Explore articles by subject * Find a job * Guide to authors * Editorial policies Nature (Nature) ISSN 1476-4687 (online) ISSN 0028-0836 (print) nature.com sitemap About Nature Portfolio * About us * Press releases * Press office * Contact us Discover content * Journals A-Z * Articles by subject * Protocol Exchange * Nature Index Publishing policies * Nature portfolio policies * Open access Author & Researcher services * Reprints & permissions * Research data * Language editing * Scientific editing * Nature Masterclasses * Live Expert Trainer-led workshops * Research Solutions Libraries & institutions * Librarian service & tools * Librarian portal * Open research * Recommend to library Advertising & partnerships * Advertising * Partnerships & Services * Media kits * Branded content Professional development * Nature Careers * Nature Conferences Regional websites * Nature Africa * Nature China * Nature India * Nature Italy * Nature Japan * Nature Korea * Nature Middle East * Privacy Policy * Use of cookies * Your privacy choices/Manage cookies * Legal notice * Accessibility statement * Terms & Conditions * Your US state privacy rights Springer Nature (c) 2023 Springer Nature Limited