https://www.independent.co.uk/tech/artificial-intelligence-deepmind-ai-catastrophe-b2166954.html Jump to content Support us Contribute Subscribe Login Support us Contribute Subscribe Login * News + UK + World + Europe * Independent TV * Politics * Climate + Climate Voices + Climate News + Climate Life + Electric vehicles + SGI * Voices + Mary Dejevsky * Indy100 * Sport + US sports + Soccer + Tennis + Boxing + Formula 1 + UFC * Culture + Film + TV + Music + Books + Crosswords & Puzzles * Travel * indy/life + Fashion & Beauty + Tech + Food & Drink + Health & Fitness + Love & relationships + Electric vehicles * IndyBest * My Independent + Most commented + Ask Me Anything + Virtual Events + Independent Newsletters + Competitions * UK Edition * US Edition * Asia Edition * Edicion en Espanol Thank you for registering Please refresh the page or navigate to another page on the site to be automatically logged inPlease refresh your browser to be logged in Tech 'Existential catastrophe' caused by AI is likely unavoidable, DeepMind researcher warns 'We should be progressing slowly - if at all - toward the goal of more powerful AI,' new paper warns Anthony Cuthbertson Wednesday 14 September 2022 14:35 Comments Article bookmarked Find your bookmarks in your Independent Premium section, under my profile [ ]Don't show me this message again

Google's DeepMind in one of the leaders in artificial intelligence research

Google's DeepMind in one of the leaders in artificial intelligence research (DeepMind) For free real time breaking news alerts sent straight to your inbox sign up to our breaking news emails Sign up to our free breaking news emails [ ] Please enter a valid email address Please enter a valid email address SIGN UP [ ]I would like to be emailed about offers, events and updates from The Independent. Read our privacy notice Thanks for signing up to the Breaking News email {{ #verifyErrors }}{{ message }}{{ /verifyErrors }}{{ ^verifyErrors }}Something went wrong. Please try again later{{ /verifyErrors }} Researchers from the University of Oxford and Google's artificial intelligence division DeepMind have claimed that there is a high probability of advanced forms of AI becoming "existentially dangerous to life on Earth". In a recent article in the peer-reviewed journal AI Magazine, the researchers warned that there would be "catastrophic consequences" if the development of certain AI agents continues. Leading philosphers like Oxford University's Nick Bostrom have previously spoken of the threat posed by advanced forms of artificial intelligence, though one of authors of the new paper claimed such warnings did not go far enough. "Bostrom, [computer scientist Stuart] Russell, and others have argued that advanced AI poses a threat to humanity," Michael Cohen wrote in a Twitter thread accompanying the article. "Under the conditions we have identified, our conclusion is much stronger than that of any previous publication - an existential catastrophe is not just possible, but likely." Recommended * DeepMind's AI can now play football after learning how to work as a team The paper proposes a scenario whereby an AI agent figures out a strategy to cheat in order to receive a reward that it is pre-programmed to seek. In order to maximize its reward potential, it requires as much energy as is possible to obtain. The thought experiment sees humanity ultimately competing against the AI for energy resources. "Winning the competition of 'getting to use the last bit of available energy' while playing against something much smarter than us would probably be very hard," Mr Cohen wrote. "Losing would be fatal. "These possibilities, however theoretical, mean we should be progressing slowly - if at all - toward the goal of more powerful AI." DeepMind has already proposed a safeguard against such an eventuality, dubbing it "the big red button". In a 2016 paper titled 'Safely Interruptible Agents', the AI firm outlined a framework for preventing advanced machines from ignoring turn-off commands and becoming an out-of-control rogue agent. Recommended * Kenan Thompson reunites with his 'Good Burger' co-star Kel Mitchell at the Emmys * This app can detect Covid from just your voice 'more accurately than lateral flow' * DeepMind's AI can now play football after learning how to work as a team Professor Bostrom previously described DeepMind - whose AI accomplishments include beating human champions at the boardgame Go and manipulating nuclear fusion - as the closest to creating human-level artificial intelligence. The Sweidish philospher also said it would be a "great tragedy" if AI development did not continue, as it holds the potential to cure diseases and advance civilisation at an otherwise impossible rate. Register for free to continue reading Registration is a free and easy way to support our truly independent journalism By registering, you will also enjoy limited access to Premium articles, exclusive newsletters, commenting, and virtual events with our leading journalists [ ]Email Please enter a valid email Please enter a valid email [ ]Password Must be at least 6 characters, include an upper and lower case character and a number Must be at least 6 characters, include an upper and lower case character and a number Must be at least 6 characters, include an upper and lower case character and a number [ ]First name Please enter your first name Special characters aren't allowed Please enter a name between 1 and 40 characters [ ]Last name Please enter your last name Special characters aren't allowed Please enter a name between 1 and 40 characters [Select your year of birth] You must be over 18 years old to register You must be over 18 years old to register Year of birth [ ]I would like to be emailed about offers, events and updates from The Independent. Read our Privacy notice Opt-out-policy You can opt-out at any time by signing in to your account to manage your preferences. Each email has a link to unsubscribe. [{{userRef}} ] {{#verifyErrors}} {{message}} {{/verifyErrors}} {{^verifyErrors}} {{message}} {{/verifyErrors}} Create my account By clicking 'Create my account' you confirm that your data has been entered correctly and you have read and agree to our Terms of use, Cookie policy and Privacy notice. This site is protected by reCAPTCHA and the Google Privacy policy and Terms of service apply. RegisterI'll try later Already have an account? sign in By clicking 'Register' you confirm that your data has been entered correctly and you have read and agree to our Terms of use, Cookie policy and Privacy notice. This site is protected by reCAPTCHA and the Google Privacy policy and Terms of service apply. Register for free to continue reading Registration is a free and easy way to support our truly independent journalism By registering, you will also enjoy limited access to Premium articles, exclusive newsletters, commenting, and virtual events with our leading journalists register [ ]Email Please enter a valid email Please enter a valid email [ ]Password Must be at least 6 characters, include an upper and lower case character and a number Must be at least 6 characters, include an upper and lower case character and a number Must be at least 6 characters, include an upper and lower case character and a number [ ]First name Please enter your first name Special characters aren't allowed Please enter a name between 1 and 40 characters [ ]Last name Please enter your last name Special characters aren't allowed Please enter a name between 1 and 40 characters [Select your year of birth] You must be over 18 years old to register You must be over 18 years old to register Year of birth [ ]I would like to be emailed about offers, events and updates from The Independent. Read our Privacy notice Opt-out-policy You can opt-out at any time by signing in to your account to manage your preferences. Each email has a link to unsubscribe. [{{userRef}} ] {{#verifyErrors}} {{message}} {{/verifyErrors}} {{^verifyErrors}} {{message}} {{/verifyErrors}} Create my account By clicking 'Create my account' you confirm that your data has been entered correctly and you have read and agree to our Terms of use, Cookie policy and Privacy notice. This site is protected by reCAPTCHA and the Google Privacy policy and Terms of service apply. RegisterI'll try later Already have an account? sign in By clicking 'Register' you confirm that your data has been entered correctly and you have read and agree to our Terms of use, Cookie policy and Privacy notice. This site is protected by reCAPTCHA and the Google Privacy policy and Terms of service apply. More about artificial intelligenceDeepMind Join our new commenting forum Join thought-provoking conversations, follow other Independent readers and see their replies Comments 1/1An AI 'catastrophe' is likely unavoidable, DeepMind researchers warn An AI 'catastrophe' is likely unavoidable, DeepMind researchers warn Google's DeepMind in one of the leaders in artificial intelligence research DeepMind Subscribe to Independent Premium to bookmark this article Want to bookmark your favourite articles and stories to read or reference later? Start your Independent Premium subscription today. Subscribe Already subscribed? Log in Most popular Popular videos Sponsored Features Voucher Codes Very Discount Code 20% off selected fashion & sportswear over PS75 - Very discount code Booking Com Discounts Getaway Deals up to 15% off with Booking.com in September Aliexpress Promo Code Receive $5 off a $20+ order with this AliExpress discount code Samsung Discount Codes 25% off all orders & free next day delivery - Samsung student discount Virgin Mobile Promo Codes Get PS50 cash with friend referrals at Virgin Mobile Indy / Compare Compare UK Broadband DealsCompare broadband packages side by side to find the best deal for you Cheap Broadband Deals in 2022Compare cheap broadband deals from providers with fastest speed in your area Fibre Broadband Deals September 2022All you need to know about fibre broadband Best Apple iPhone Deals in the UK September 2022Compare iPhone contract deals and get the best offer this September Compare Mobile Phone DealsCompare the best mobile phone deals from the top networks and brands * Get in touch + Contact us + Jobs + * Our Products + Subscribe + Register + Newsletters + Donate + Install our app + Archive * Other publications + International editions + Independent en Espanol + Independent Arabia + Independent Turkish + Independent Persian + Independent Urdu + Evening Standard + Novaya Gazeta * Extras + Puzzles + All topics + Voucher codes + Compare + Independent Advertising + Syndication + Working at The Independent * Legal + Code of conduct and complaints + Contributors + Cookie policy + Donations Terms & Conditions + Privacy notice + User policies + Modern Slavery Act Thank you for registering Please refresh the page or navigate to another page on the site to be automatically logged inPlease refresh your browser to be logged in Close US EditionChange UK Edition Asia Edition Edicion en Espanol {{#items}} {{#stampSmall}}{{/stampSmall}}{{#stampClimate}}{{/stampClimate}}{{# stampPremium}}{{/stampPremium}}{{title}}{{#desc}}{{desc}}{{/desc}} {{#children}} * {{title}} {{/children}} {{/items}} * Indy100 * Crosswords & Puzzles * Most Commented * Newsletters * Ask Me Anything * Virtual Events * Vouchers * Compare Log in [ ]Email address [ ]Password Email and password don't match Submit Forgotten your password? New to The Independent? Register Or if you would prefer: SIGN IN WITH GOOGLE Want an ad-free experience? View offers This site is protected by reCAPTCHA and the Google Privacy notice and Terms of service apply. Hi {{indy.fullName}} * My Independent Premium * Account details * Help centre * Logout