https://www.wired.com/story/openai-ceo-sam-altman-the-age-of-giant-ai-models-is-already-over/ Skip to main content Open Navigation Menu To revist this article, visit My Profile, then View saved stories. Close Alert WIRED OpenAI's CEO Says the Age of Giant AI Models Is Already Over * Backchannel * Business * Culture * Gear * Ideas * Science * Security More To revist this article, visit My Profile, then View saved stories. Close Alert Sign In Search * Backchannel * Business * Culture * Gear * Ideas * Science * Security * Podcasts * Video * Artificial Intelligence * Climate * Games * Newsletters * Magazine * Events * Wired Insider * Jobs * Coupons Sam Altman Photograph: JASON REDMOND/Getty Images Will Knight Business Apr 17, 2023 7:00 AM OpenAI's CEO Says the Age of Giant AI Models Is Already Over Sam Altman says the research strategy that birthed ChatGPT is played out and future strides in artificial intelligence will require new ideas. * * * * * * * * The stunning capabilities of ChatGPT, the chatbot from startup OpenAI, has triggered a surge of new interest and investment in artificial intelligence. But late last week, OpenAI's CEO warned that the research strategy that birthed the bot is played out. It's unclear exactly where future advances will come from. OpenAI has delivered a series of impressive advances in AI that works with language in recent years by taking existing machine-learning algorithms and scaling them up to previously unimagined size. GPT-4, the latest of those projects, was likely trained using trillions of words of text and many thousands of powerful computer chips. The process cost over $100 million. But the company's CEO, Sam Altman, says further progress will not come from making models bigger. "I think we're at the end of the era where it's going to be these, like, giant, giant models," he told an audience at an event held at MIT late last week. "We'll make them better in other ways." Altman's declaration suggests an unexpected twist in the race to develop and deploy new AI algorithms. Since OpenAI launched ChatGPT in November, Microsoft has used the underlying technology to add a chatbot to its Bing search engine, and Google has launched a rival chatbot called Bard. Many people have rushed to experiment with using the new breed of chatbot to help with work or personal tasks. Meanwhile, numerous well-funded startups, including Anthropic, AI21, Cohere, and Character.AI, are throwing enormous resources into building ever larger algorithms in an effort to catch up with OpenAI's technology. The initial version of ChatGPT was based on a slightly upgraded version of GPT-3, but users can now also access a version powered by the more capable GPT-4. Altman's statement suggests that GPT-4 could be the last major advance to emerge from OpenAI's strategy of making the models bigger and feeding them more data. He did not say what kind of research strategies or techniques might take its place. In the paper describing GPT-4, OpenAI says its estimates suggest diminishing returns on scaling up model size. Altman said there are also physical limits to how many data centers the company can build and how quickly it can build them. Nick Frosst, a cofounder at Cohere who previously worked on AI at Google, says Altman's feeling that going bigger will not work indefinitely rings true. He, too, believes that progress on transformers, the type of machine learning model at the heart of GPT-4 and its rivals, lies beyond scaling. "There are lots of ways of making transformers way, way better and more useful, and lots of them don't involve adding parameters to the model," he says. Frosst says that new AI model designs, or architectures, and further tuning based on human feedback are promising directions that many researchers are already exploring. Each version of OpenAI's influential family of language algorithms consists of an artificial neural network, software loosely inspired by the way neurons work together, which is trained to predict the words that should follow a given string of text. Most Popular * Ford F-150 Lightning pickup truck on stage at an event with an American flag in the background Business The US Wants to Close an 'SUV Loophole' That Supersized Cars Aarian Marshall * The Minister of Infrastructure Matteo Salvini standing next to a model of the planned suspension bridge over the Strait of Messina Business The World's Longest Suspension Bridge Is History in the Making Jacopo Prisco * Steven Yeun screaming from a car window as Danny in Beef Culture The 45 Best Shows on Netflix Right Now WIRED Staff * Harry Potter books against a white background Culture Who Needs the New Harry Potter Series? Angela Watercutter * The first of these language models, GPT-2, was announced in 2019. In its largest form, it had 1.5 billion parameters, a measure of the number of adjustable connections between its crude artificial neurons. At the time, that was extremely large compared to previous systems, thanks in part to OpenAI researchers finding that scaling up made the model more coherent. And the company made GPT-2's successor, GPT-3, announced in 2020, still bigger, with a whopping 175 billion parameters. That system's broad abilities to generate poems, emails, and other text helped convince other companies and research institutions to push their own AI models to similar and even greater size. After ChatGPT debuted in November, meme makers and tech pundits speculated that GPT-4, when it arrived, would be a model of vertigo-inducing size and complexity. Yet when OpenAI finally announced the new artificial intelligence model, the company didn't disclose how big it is--perhaps because size is no longer all that matters. At the MIT event, Altman was asked if training GPT-4 cost $100 million; he replied, "It's more than that." Although OpenAI is keeping GPT-4's size and inner workings secret, it is likely that some of its intelligence already comes from looking beyond just scale. On possibility is that it used a method called reinforcement learning with human feedback, which was used to enhance ChatGPT. It involves having humans judge the quality of the model's answers to steer it towards providing responses more likely to be judged as high quality. The remarkable capabilities of GPT-4 have stunned some experts and sparked debate over the potential for AI to transform the economy but also spread disinformation and eliminate jobs. Some AI experts, tech entrepreneurs including Elon Musk, and scientists recently wrote an open letter calling for a six-month pause on the development of anything more powerful than GPT-4. At MIT last week, Altman confirmed that his company is not currently developing GPT-5. "An earlier version of the letter claimed OpenAI is training GPT-5 right now," he said. "We are not, and won't for some time." Get More From WIRED * Get the best stories from WIRED's iconic archive in your inbox * A tiny blog took on Big Surveillance in China--and won * In the war on bacteria, it's time to call in the phages * Robotaxis are going to sound weird * The magic and minstrelsy of generative AI * Artificial wombs will change abortion rights forever * [?] Embrace the new season with the Gear team's best picks for best tents, umbrellas, and robot vacuums [undefined] Will Knight is a senior writer for WIRED, covering artificial intelligence. He was previously a senior editor at MIT Technology Review, where he wrote about fundamental advances in AI and China's AI boom. Before that, he was an editor and writer at New Scientist. He studied anthropology and journalism in... Read more Senior Writer * Topicsartificial intelligenceneural networksdeep learningmachine learningChatGPTOpenAIGooglechatbots More from WIRED [undefined] Amazon Is Joining the Generative AI Race The ecommerce giant doesn't have a ChatGPT rival, but it wants to sell you the tools you need to build one. Will Knight Close-up of the red and yellow lights of a traffic stoplight at night In Sudden Alarm, Tech Doyens Call for a Pause on ChatGPT Tech luminaries, renowned scientists, and Elon Musk warn of an "out-of-control race" to develop and deploy ever-more-powerful AI systems. Will Knight Red Colored Duct Tape Stripes on White Background Direct Above View China's ChatGPT Rival Needs to Watch Its Words Search giant Baidu's Ernie Bot met online jeers and also faces the challenge of operating on a firewalled internet ruled by government censorship. Will Knight A nearly completed white jigsaw puzzle with one remaining piece laying on top; web plug-in concept Now That ChatGPT Is Plugged In, Things Could Get Weird Letting the chatbot interact with the live internet will make it more useful--and more problematic, too. Will Knight Green plastic toy soldiers on a pink background Let the AI Coding Wars Begin! The way artificial intelligence can rewrite software will have huge implications for the tech industry--and everyone else, too. Will Knight Red, blue, yellow, and green speech bubbles standing up in a circle Google Rolls Out Its Bard Chatbot to Battle ChatGPT A new bot has entered the chat. But Google warns that, like its competitor, it will sometimes "hallucinate." Will Knight Illustration of a jail window in the shape of a chat bubble, with the bars bent The Hacking of ChatGPT Is Just Getting Started Security researchers are jailbreaking large language models to get around safety rules. Things could get much worse. Matt Burgess An upright magnifying glass with a white bar stands over a bunch of fallen smaller bars and magnifying glasses. ChatGPT Opened a New Era in Search. Microsoft Could Ruin It Startups say Microsoft and its Bing chatbot--not just Google--are stifling competition when it comes to creating better search engines. Paresh Dave WIRED WIRED is where tomorrow is realized. It is the essential source of information and ideas that make sense of a world in constant transformation. The WIRED conversation illuminates how technology is changing every aspect of our lives--from culture to business, science to design. The breakthroughs and innovations that we uncover lead to new ways of thinking, new connections, and new industries. * * * * * * More From WIRED * Subscribe * Newsletters * FAQ * Wired Staff * Press Center * Coupons * Editorial Standards * Black Friday * Archive Contact * Advertise * Contact Us * Customer Care * Jobs * RSS * Accessibility Help * Conde Nast Store * Conde Nast Spotlight * Do Not Sell My Personal Info (c) 2023 Conde Nast. All rights reserved. Use of this site constitutes acceptance of our User Agreement and Privacy Policy and Cookie Statement and Your California Privacy Rights. WIRED may earn a portion of sales from products that are purchased through our site as part of our Affiliate Partnerships with retailers. The material on this site may not be reproduced, distributed, transmitted, cached or otherwise used, except with the prior written permission of Conde Nast. Ad Choices Select international siteUnited States * UK * Italia * Japon