https://old.reddit.com/r/slatestarcodex/comments/wq6wif/john_carmack_just_got_investment_to_build_agi_he/ jump to content my subreddits edit subscriptions * popular * -all * -random * -users | * AskReddit * -funny * -worldnews * -gaming * -tifu * -news * -movies * -TwoXChromosomes * -mildlyinteresting * -OldSchoolCool * -explainlikeimfive * -todayilearned * -Jokes * -pics * -science * -aww * -books * -nottheonion * -videos * -LifeProTips * -dataisbeautiful * -IAmA * -Futurology * -Music * -Showerthoughts * -askscience * -space * -Art * -gifs * -UpliftingNews * -gadgets * -nosleep * -sports * -DIY * -food * -creepy * -history * -GetMotivated * -photoshopbattles * -Documentaries * -WritingPrompts * -EarthPorn * -InternetIsBeautiful * -philosophy * -announcements * -listentothis * -blog more >> slatestarcodex slatestarcodex * comments Want to join? Log in or sign up in seconds.| * English [ ][] [ ]limit my search to r/slatestarcodex use the following search parameters to narrow your results: subreddit:subreddit find submissions in "subreddit" author:username find submissions by "username" site:example.com find submissions from "example.com" url:text search for "text" in url selftext:text search for "text" in self post contents self:yes (or self:no) include (or exclude) self posts nsfw:yes (or nsfw:no) include (or exclude) results marked as NSFW e.g. subreddit:aww site:imgur.com dog see the search faq for details. advanced search: by author, subreddit... this post was submitted on 16 Aug 2022 156 points (95% upvoted) shortlink: [https://redd.it/wq6w] [ ][ ] [ ]remember mereset password login Submit a new link Submit a new text post Get an ad-free experience with special benefits, and directly support Reddit. get reddit premium slatestarcodex joinleave50,224 readers 297 users here now Companion subreddit for Slate Star Codex. New blog posts will be posted here. Community guidelines See the Victorian Sufi Buddha Lite comment policy: comments should be at least two of {true, necessary, kind}. * Be kind. Failing that, bring evidence. * Be charitable. Assume the people you're talking to or about have thought through the issues you're discussing, and try to represent their views in a way they would recognize. * Culture war topics are forbidden. * When making a claim that isn't outright obvious, you should proactively provide evidence in proportion to how partisan and inflammatory your claim might be. * Don't be egregiously obnoxious. Feel free to report comments or message the mods with your thoughts. A user also maintains an unofficial registry of bans. Regular threads * Friday Fun Thread * Wellness Wednesday Relevant external links * Discord server * Complice coworking room What's SSC? A blog by Scott Alexander about human cognition, politics, and medicine. See the Top Posts. More by Scott Alexander Slate Star Scratchpad Index of works on Less Wrong Etc A reader has created a web app web that allows SSC readers to place themselves on a map. a community for 8 years MODERATORS * message the mods * Moderator list hidden. Learn More discussions in r/slatestarcodex <> X 29 * 18 comments DALL-E 2 Art: Experiments with Prompts or How I Got My New Wallpaper 11 * 4 comments What Kind of Liar Are You? A Choose-Your-Own-Morality Adventure 3 * 7 comments Qualia as reward shaping? 155 * 174 comments John Carmack just got investment to build AGI. He doesn't believe in fast takeoff because of TCP connection limits? 1 * 6 comments The Big Bang didn't happen | Eric Lerner 0 * 1 comment Harmonic Gestalt by Steven Lehar 1 Religion, Heuristics, and Intergenerational Risk Management 0 * 14 comments What to do if I'm cursed to be 100th percentile extroverted and cognitively verbal, AND an SSCer? Call me? Daily? Please??? 0 * 2 comments SSC Post about Scott's favorite paper? 70 * 47 comments Could Selecting Representatives by a Lottery Actually Work Better than Elections? Welcome to Reddit, the front page of the internet. Become a Redditor and join one of thousands of communities. x 155 156 157 John Carmack just got investment to build AGI. He doesn't believe in fast takeoff because of TCP connection limits?AI (self.slatestarcodex ) submitted 1 day ago * by Pool_of_Death John Carmack was recently on the Lex Fridman podcast. You should watch the whole thing or at least the AGI portion if it interests you but I pulled out the EA/AGI relevant info that seemed surprising to me and what I think EA or this subreddit would find interesting/ concerning. TLDR: * He has been studying AI/ML for 2 years now and believes he has his head wrapped around it and has a unique angle of attack * He has just received investment to start a company to work towards building AGI * He thinks human-level AGI has a 55% - 60% chance of being built by 2030 * He doesn't believe in fast takeoff and thinks it's much too early to be talking about AI ethics or safety He thinks AGI can be plausibly created by one individual in 10s of thousands of lines of code. He thinks the parts we're missing to create AGI are simple. Less than 6 key insights, each can be written on the back of an envelope - timestamp He believes there is a 55% - 60% chance that somewhere there will be signs of life of AGI in 2030 - timestamp He really does not believe in fast take-off (doesn't seem to think it's an existential risk). He thinks we'll go from the level of animal intelligence to the level of a learning disabled toddler and we'll just improve iteratively from there - timestamp "We're going to chip away at all of the things people do that we can turn into narrow AI problems and trillions of dollars of value will be created by that" - timestamp "It's a funny thing. As far as I can tell, Elon is completely serious about AGI existential threat. I tried to draw him out to talk about AI but he didn't want to. I get that fatalistic sense from him. It's weird because his company (tesla) could be the leading AGI company." - timestamp It's going to start off hugely expensive. Estimates include 86 billion neurons 100 trillion synapses, I don't think those all need to be weights, I don't think we need models that are quite that big evaluated quite that often. [Because you can simulate things simpler]. But it's going to be thousands of GPUs to run a human-level AGI so it might start off at $1,000/hr. So it will be used in important business/strategic decisions. But then there will be a 1000x cost improvement in the next couple of decades, so $1/hr. - timestamp I stay away from AI ethics discussions or I don't even think about it. It's similar to the safety thing, I think it's premature. Some people enjoy thinking about impractical/non-progmatic things. I think, because we won't have fast take off, we'll have time to have debates when we know the shape of what we're debating. Some people think it'll go too fast so we have to get ahead of it. Maybe that's true, I wouldn't put any of my money or funding into that because I don't think it's a problem yet. Add we'll have signs of life, when we see a learning disabled toddler AGI. - timestamp It is my belief we'll start off with something that requires thousands of GPUs. It's hard to spin a lot of those up because it takes data centers which are hard to build. You can't magic data centers into existence. The old fast take-off tropes about AGI escaping onto the internet are nonsense because you can't open TCP connections above a certain rate no matter how smart you are so it can't take over the world in an instant. Even if you had access to all of the resources they will be specialized systems with particular chips and interconnects etc. so it won't be able to be plopped somewhere else. However, it will be small, the code will fit on a thumb drive, 10s of thousands of lines of code. - timestamp Lex - "What if computation keeps expanding exponentially and the AGI uses phones/fridges/etc. instead of AWS" John - "There are issues there. You're limited to a 5G connection. If you take a calculation and factor it across 1 million cellphones instead of 1000 GPUs in a warehouse it might work but you'll be at something like 1/1000 the speed so you could have an AGI working but it wouldn't be real-time. It would be operating at a snail's pace, much slower than human thought. I'm not worried about that. You always have the balance between bandwidth, storage, and computation. Sometimes it's easy to get one or the other but it's been constant that you need all three." - timestamp "I just got an investment for a company..... I took a lot of time to absorb a lot of AI/ML info. I've got my arms around it, I have the measure of it. I come at it from a different angle than most research-oriented AI/ML people. - timestamp "This all really started for me because Sam Altman tried to recruit me for OpenAi. I didn't know anything about machine learning" - timestamp "I have an overactive sense of responsibility about other people's money so I took investment as a forcing function. I have investors that are going to expect something of me. This is a low-probability long-term bet. I don't have a line of sight on the value proposition, there are unknown unknowns in the way. But it's one of the most important things humans will ever do. It's something that's within our lifetimes if not within a decade. The ink on the investment has just dried." - timestamp * 174 comments * share * save * hide * report all 174 comments sorted by: best topnewcontroversialoldrandomq&alive (beta) [ ] Want to add to the discussion? Post a comment! Create an account [-]thicket [score hidden] 21 hours ago (3 children) Props for the high-effort extracts. I really appreciate you breaking out the important parts of his argument rather than just tossing an hour long video at us. Thanks! * permalink * embed * save * report * give award * reply [-]Lone-Pine [score hidden] 17 hours ago (2 children) an hour long video You mean 5 hours long, lol * permalink * embed * save * parent * report * give award * reply [-]Extra_Negotiation [score hidden] 5 hours ago (1 child) That was stood out to me - given editing times etc, this must be recorded over multiple days....right? * permalink * embed * save * parent * report * give award * reply [-]NotFromReddit [score hidden] 4 hours ago (0 children) I don't think the original is edited at all. Except for pausing for lunch and bathroom breaks. It's just a 5 hour long podcast. * permalink * embed * save * parent * report * give award * reply [-]BeABetterHumanBeing 103 points104 points105 points 1 day ago (30 children) I've worked professionally with Carmack in the past. Smart guy. That said, he does have a tendency to assume that if other people can't do a thing, it's because they're stupid, and that he alone will be able to do it. I've seen him lock himself away for weeks to do a thing known to be impossible, just because he has to try it himself to believe other people. I'm happy that he's trying this himself. Let's see whether he finds success in it. * permalink * embed * save * report * give award * reply [-]WhoRoger [score hidden] 21 hours ago (2 children) Okay but didn't he also make at least most of that shit possible at the end? From BSP rendering through VR to actual fucking rocket science. But I'm also quite skeptical that he would be the guy to make the breakthrough in AGI. He must be getting a bit old by now, while a shit ton of geniuses work on this. On the other hand, he really has a knack for finding ingenious solutions for crazy problems, making complex solutions look easy, and also knows how to surround himself with the right people. So... Who knows. Maybe civvie11 will be completely right calling JC an ubermensch overlord. * permalink * embed * save * parent * report * give award * reply [-]NotFromReddit [score hidden] 4 hours ago (1 child) He must be getting a bit old by now Probably not a negative yet at his age. I feel like his vast experience in software engineering does indeed give him an edge over almost everyone else. * permalink * embed * save * parent * report * give award * reply [-]WhoRoger [score hidden] 3 hours ago (0 children) That's actually a reference to himself saying how he's not as fast as he used to be, in an interview over years ago. Doesn't seem to be any slower today tho. * permalink * embed * save * parent * report * give award * reply [-]Pool_of_Death[S] [score hidden] 23 hours ago (13 children) This is what concerns me though. I think he really does have unique takes that the vast majority of AI /ML researchers don't have. So I think it is plausible that his unique contributions can speed up the AGI timeline. Does that concern you? * permalink * embed * save * parent * report * give award * reply [-]BeABetterHumanBeing [score hidden] 23 hours ago (10 children) Well, I've seen him throw himself at a thing and fail, so it's not like just because the fabled John Carmack does a thing, it's guaranteed to succeed. Time will tell. No idea of whether his key insights are the difference. The last thing I'll note, the thing that does concern me, is that Carmack lacks the thing that makes a person empathize with others; I'm not surprised he's tackling AI, most humans are robots as far as he's concerned. I wouldn't rate him as a good judge of whether an ethically-dubious thing should be done, so if he is successful, I'll be carefully watching what he wants to use it for. * permalink * embed * save * parent * report * give award * reply [-]llelouchh [score hidden] 16 hours ago (4 children) is that Carmack lacks the thing that makes a person empathize with others What makes you say this? * permalink * embed * save * parent * report * give award * reply [-]retsibsi [score hidden] 15 hours ago* (1 child) I'm not the person you asked and I don't have any personal experience with Carmack, but I'm pretty sure I've seen him quoted as describing himself in a similar way. (Maybe low empathy rather than zero empathy, though.) If googling doesn't turn up a source, it might have been in the book Masters of Doom. edit: googling turns up a psychiatric report on teenage Carmack, which describes him as lacking empathy. I feel like there was probably more on this in the book, perhaps with a more nuanced description of Carmack as an adult. edit 2: can't find much more than that in the book. Maybe I was conflating that quote with the general picture the book painted of Carmack. (Not malicious, but sometimes very insensitive to other people's feelings.) * permalink * embed * save * parent * report * give award * reply continue this thread [-]spreadlove5683 [score hidden] 6 hours ago* (0 children) He mentioned himself in that podcast with Lex that he is a very unsentimental guy. He also mentioned not thinking about his death/ mortality much if I remember right (and so probably not a lot of big picture stuff in general), but instead just being all in on getting stuff done / practical matters. He strikes me as someone who is very logical/analytical/effective, and is probably puts a lot less emphasis on what is emotionally salient, etc. I'm honestly can relate to this perspective, so I think I can recognize it. I'd imagine he has good intentions, but probably understands difficult decisions where not everyone wins, because not everyone can currently. * permalink * embed * save * parent * report * give award * reply [-]BeABetterHumanBeing [score hidden] 4 hours ago (0 children) Others have chimed in with various sources that describe him this way (or that he describes himself), but I say it because in my time working with him, his interactions with me or that I saw of others were always unpersonable. For example, he would talk at people, instead of talking to them. There was no such thing as pleasantries of any kind. He obviously felt like you were wasting his time unless you were telling him something useful. If you had a question, he treated it like the answer was obvious in a somewhat disdainful way. I could go on, but the point is that whatever the personality disorder is, Carmack lives in a world unto himself, and other human beings present themselves as objects in that world. * permalink * embed * save * parent * report * give award * reply [-]Sinity [score hidden] 7 hours ago (1 child) Well, I've seen him throw himself at a thing and fail, What was it? * permalink * embed * save * parent * report * give award * reply [-]BeABetterHumanBeing [score hidden] 4 hours ago (0 children) It was a very particular problem related to VR. No specifics, as it was very business-specific. * permalink * embed * save * parent * report * give award * reply [-]NotFromReddit [score hidden] 4 hours ago (2 children) I've seen him throw himself at a thing and fail What was the thing? * permalink * embed * save * parent * report * give award * reply [-]BeABetterHumanBeing [score hidden] 4 hours ago (0 children) It was a very particular problem related to VR. No specifics, as it was very business-specific. * permalink * embed * save * parent * report * give award * reply [-]question_23 [score hidden] 3 hours ago (0 children) Armadillo Aerospace was one. Millions sunk into a glorified Mythbusters esque attempt at a moon rocket, with nothing to show except for some 1950s looking contraptions that flew a hundred feet up or so. I like Carmack and read masters of doom, but my takeaway is that the guy is fundamentally conservative and almost proudly uncreative. He's old (51) and seems enamored of "classical" ideas across the board (as a classical liberal as well), which have not spearheaded AI thus far. * permalink * embed * save * parent * report * give award * reply [-]Typical-Hornet-7313 [score hidden] 1 hour ago (0 children) Lol no. The majority of whip smart CS, statistical physics, applied math etc researchers have been piling into the field for the last decade. The big tech firms pump billions of $ a year into ML training hardware alone--not to mention data & researcher overheads. IMO the chances that one smart and productive guy with even a well funded company (by startup standards) will beat either a big U.S. tech firm or the Chinese government to AGI are extremely close to zero. * permalink * embed * save * parent * report * give award * reply [-]Glittering-Roll-9432 [score hidden] 10 hours ago (0 children) It concerns me because it seems the first person that creates AGI will be willfully ignorant of the risks. Hopefully it'll be a benevolent AI. If it isn't, this team of John Carmacks are gonna give it the tools to kill us all. * permalink * embed * save * parent * report * give award * reply [-]philbearsubstack [score hidden] 21 hours ago (6 children) There are two types of people in this world. 1. People who can say "alright, this is what it looks like to me, but it doesn't look like that to a lot of other smart people, and if I disagree with another smart person they're just as likely to be right as I am. Sure, my arguments look compelling to me, but their arguments look compelling to them. I should continue to advance my own ideas, but with a healthy skepticism. 2. People who constitutionally lack the ability to take the outside view on themselves. If they can't see a counterargument to something, it must be because it doesn't exist. if they can't work out why something is hard, it must be because it's easy. The opinions of others are, for them, weak evidence at best, to be considered only until they have formed their own opinion on the field. It sounds like he's in the latter camp. That's bad. One of my best friends in the world is in the latter camp, but I've got to say that the older I get, the less tolerant I feel towards this approach. It's intellectual narcissism. Now I think it is true that we do need people to have a little bit of the attitude of 2 to advance truly new ideas, but I don't think it's true that people need to go all the way down that path. My approach, for example, is to propose all kinds of heterodoxies and heresies in the cheerful knowledge that they're probably wrong, because I view it as my contribution to a social process of truth seeking- a process in which I do not have to be individually right to contribute. See my essay, "I'm not particularly worried about confirmation bias": https://philosophybear.substack.com/p/ im-not-particularly-worried-about In the world of AI safety, we really, really, badly need people who *do not* place overmuch weight on their own judgement to lead. * permalink * embed * save * parent * report * give award * reply [-]WhoRoger [score hidden] 21 hours ago (0 children) To be fair, I doubt there's another person on this planet that's been told something is impossible so many times, just to do it himself. * permalink * embed * save * parent * report * give award * reply [-]retsibsi [score hidden] 19 hours ago* (0 children) I think BeABetterHumanBeing's description allows of a more moderate interpretation, where Carmack isn't delusionally arrogant, but knows from experience that he is unusually talented in certain ways and can sometimes do things that other smart people have given up on. If he's sensible about choosing when to 'lock himself away for weeks to do a thing known to be impossible, just because he has to try it himself to believe other people', it could be a good strategy; even if something is genuinely impossible, sometimes trying to do it is the best way to properly understand why, and in the process develop a much better understanding of the relevant mechanisms (and perhaps even a solution to some similar-enough-to-be-useful problems). I'm also guessing that some of the 'impossible' things are just super hard, or perhaps have an angle of attack that has been generally overlooked, so maybe once in a while he manages to do them. Even if he's pretty far from the optimum balance, I think it's useful to have some 'take nobody's word for it' types around, provided they have the intelligence to back it up and enough of a bullshit filter not to get carried away into unproductive crackpottery. Sometimes the smart consensus really is missing something important. I agree those people can be dangerous in certain contexts, though, especially if they are by nature incautious as well as arrogant. * permalink * embed * save * parent * report * give award * reply [-]Toptomcat [score hidden] 18 hours ago (0 children) I think you're being a bit too categorical here. Capacity for humility is a spectrum, not a light switch. * permalink * embed * save * parent * report * give award * reply [-]iemfi [score hidden] 19 hours ago (1 child) I think what makes Carmack particularly dangerous is that he seems to be partially in the first camp. Enough that 2 isn't that big a handicap but not enough that he'll spend 5 minutes thinking about AI risk properly. * permalink * embed * save * parent * report * give award * reply [-]NotFromReddit [score hidden] 4 hours ago (0 children) I feel like he'd start thinking about it once it makes sense to, if that ever happens. * permalink * embed * save * parent * report * give award * reply [-]curious_straight_CA [score hidden] 6 hours ago* (0 children) ... say you're Voltaire, or something, or a 15th century peasant. All your friends, and everyone, believe in God. Is it worthwhile to: "alright, this is what it looks like to me, but it doesn't look like that to a lot of other smart people, and if I disagree with another smart person they're just as likely to be right as I am"? There is a difference between: "this smart (or dumb!) person/group of people disagrees with me, it's worth carefully and deeply understanding what they believe and why" (generally correct in most areas), and "well, they believe the sun is a projection of God's will, whatever that means, and I believe it's a ball of flaming gas because spectra - but, really, who's to say? Maybe it's both." alt-u John Brown: "Yeah, the slaves seem to be oppressed, but half of the nation disagrees - I'd best not risk rocking the boat, lest I be wrong." * permalink * embed * save * parent * report * give award * reply [-]blackvrocky [score hidden] 13 hours ago (5 children) I've seen him lock himself away for weeks to do a thing known to be impossible, just because he has to try it himself to believe other people. few people i know, very very few people have that kind of attitude nowaday. * permalink * embed * save * parent * report * give award * reply [-]curious_straight_CA [score hidden] 6 hours ago (3 children) And it's worth doing! "worst case", you end up deeply understanding why it's impossible, which is rather useful. * permalink * embed * save * parent * report * give award * reply [-]BeABetterHumanBeing [score hidden] 4 hours ago (2 children) That's the rosy way of looking at it. Unfortunately, the flip side is "Carmack could have spent that time doing something productive instead". * permalink * embed * save * parent * report * give award * reply [-]curious_straight_CA [score hidden] 4 hours ago (1 child) John D. Carmack II[1] (born August 20, 1970)[1] is an American computer programmer and video game developer. He co-founded the video game company id Software and was the lead programmer of its 1990s games Commander Keen, Wolfenstein 3D, Doom, Quake, and their sequels. Carmack made innovations in 3D computer graphics, such as his Carmack's Reverse algorithm for shadow volumes. In 2013, he resigned from id Software to work full-time at Oculus VR as their CTO. In 2019, he reduced his role to Consulting CTO so he could allocate more time toward artificial general intelligence (AGI).[3] ... i wonder what productive means, if this isn't? What is "productive"? Ultimately, it just means "useful". Sure wish all those 1950s physicists had done something productive, instead of lazing around lecture halls and university offices writing stupid equations on blackboards. * permalink * embed * save * parent * report * give award * reply continue this thread [-]adiabaticaccidentally puts bleggs in the rube bin and rubes in the blegg [score hidden] 2 hours ago (0 children) Reasonable prior if you're Carmack. * permalink * embed * save * parent * report * give award * reply [-]iemfi [score hidden] 21 hours ago (1 child) "This all really started for me because Sam Altman tried to recruit me for OpenAi. I didn't know anything about machine learning" Damn, like what EY said, Elon Musk probably has done the most to increase AI risk of any person in the world. And he was worried about AI risk too. * permalink * embed * save * report * give award * reply [-]c_o_r_b_a [score hidden] 7 hours ago (0 children) He is still worried about it. He's no longer involved with OpenAI, though. (Which is maybe not necessarily a good thing in this case, since Sam Altman seems a bit less concerned about existential AI risks than Musk is. Though I think Altman's also pretty smart, informed, and rat-adjacent, so there's [probably] no cause for panic.) * permalink * embed * save * parent * report * give award * reply [-]drcode 49 points50 points51 points 1 day ago (22 children) I like Carmack and hope he is correct regarding his optimism, but his opinions on this subject seem internally inconsistent: On the one hand he argues that AGI is probably solvable on current hardware once the right "tricks" are figured out But we soon as anyone brings up AI risk he's all like "AGI will require such unimaginable numbers of GPUs in warehouses, there is no way it can escape our supervision" * permalink * embed * save * report * give award * reply [-]Pool_of_Death[S] [score hidden] 23 hours ago (17 children) It also seems weird that he thinks we're like 10 years away from human-level AGI and then in a couple of decades the costs will drop by 1000x ($1/hr for human-level AGI) but somehow he's not concerned that someone will just make an AI that is 1000x more powerful than human-level instead of 1000x cheaper? Unless he thinks costs will drop drastically but capabilities won't really increase easily? * permalink * embed * save * parent * report * give award * reply [-]the_good_time_mouse [score hidden] 22 hours ago (4 children) 1,000 human level AGIs, cooperating and communicating at network speed, and without the messy pecadillos that are human motivation, trust and complexities of leadership, running 24/7, is terrifying enough. And that's just what $1,000/hour would get you. * permalink * embed * save * parent * report * give award * reply [-]Lone-Pine [score hidden] 17 hours ago (0 children) But in this scenario there are trillions of AGIs running around, owned by different groups. The critical risk period will already have been ended somehow (or we will already be computronium). * permalink * embed * save * parent * report * give award * reply [-]Glittering-Roll-9432 [score hidden] 10 hours ago (2 children) Aren't we capable of this now with humans that don't allow their picadillos to cloud judgment in a group setting? * permalink * embed * save * parent * report * give award * reply [-]the_good_time_mouse [score hidden] 7 hours ago (0 children) Neurodivergence isn't the gift you've been led to beleive. * permalink * embed * save * parent * report * give award * reply [-]c_o_r_b_a [score hidden] 7 hours ago (0 children) Yes, though the AGIs will be smarter/better/faster in many ways; i.e. the parent probably meant something a little closer to super-human level (to at least some degree) rather than human-level. (And this is already kind of a given since there are plenty of current AIs that are obviously way better than humans at certain things, so there'll never be an AGI that's exactly human-level, barring some very deliberate and difficult constraints/handicaps imposed upon it to make it easier to talk to or practice things with or whatever.) * permalink * embed * save * parent * report * give award * reply [-]WhoRoger [score hidden] 20 hours ago* (7 children) I only started listening to the interview and it'll probably take me a few days to listen to the whole thing, but this is quite a typical way of JC talking, where he talks about like 4 things in parallel and you need to keep up to figure out which thread has access to his mouth at any given moment. Typically he comments on 1) brute force requirements, 2) optimizations to get what we want, 3) desired result, 4) future development. Also he tends to be very conservative in his estimates, as he usually finds further optimizations and tricks down the road, while technical progress moves faster than he predicts. He's been talking like that for 30 years since he figured out how to make Mario Bros work on an 8-bit PC, through BSP for Doom, 3D for Quake, per-pixel lighting for Doom 3, rocket engines for his Armadillo project, virtual textures for Rage, everything for Orcs & Elves / mobile gaming and also lots of things for Oculus. A normal human gets lost easily in his lines of thinking even tho he's actually so good at explaining things. And even in the first 20 minutes you can hear petty much what I described. I've been reading his .plans back to the Quake II days, so once I listen to some portions, you can ask me for Carmack-human translations with the caveat that I know even less about AI than graphics programming * permalink * embed * save * parent * report * give award * reply [-]MelodicBerries [score hidden] 18 hours ago (3 children) He's a smart guy but his achievements in the last 10-15 years have been less impressive than in the earlier era (probably because problems have become harder). Of course, he has still accomplished more than 99.9% of people on the planet, but sometimes I get the sense that some people mistake his enormous verbal fluency to actual achievement. I can't think of anyone as skilled as Carmack in talking about complicating matters, but that isn't the same as making them happen. * permalink * embed * save * parent * report * give award * reply [-]c_o_r_b_a [score hidden] 7 hours ago (1 child) Right. He may put his [investment] money and ideas where his mouth is and play a significant role in AGI breakthroughs, but unless/until he does one can't necessarily put too much weight on a lot of what he's saying here. And if he does succeed, his stances and predictions will probably differ in 10 years from now. Earlier in the podcast he mentioned exactly that happening with VR development and predictions at Meta: learning that trying to be an autocrat dictating VR projects internally wasn't going to work, and that Meta's initial predictions about the biggest and most-valued VR use cases were wrong. * permalink * embed * save * parent * report * give award * reply continue this thread [-]WhoRoger [score hidden] 5 hours ago (0 children) 10 years ago he with a few other people cobbled together a VR headset prototype from a Samsung phone LCD screen, and now we have VR routines in consumer CPUs, EA and Valve make VR games and Zuck is betting on Meta... Certainly problems have become more complex so they're not single-person or small-group projects anymore, and more smart people have access to technologies and connections that enable them to make such cool stuff, so JC doesn't have the kind of a monopoly anymore, but that's a good thing. The way he talks about nuclear fission tho shows his mind is still in the right place. Indeed that is exactly what humanity needs, unfortunately there just isn't quite the right personality to move that forward, especially not against all the existing energy lobbies and human stupidity. I can't blame him that he rather chose to dabble in AI than fusion. * permalink * embed * save * parent * report * give award * reply [-]Lone-Pine [score hidden] 17 hours ago (2 children) Does he give any hints about how he plans to approach AI, from a technical standpoint? Does he favor: Deep Learning, Transformers, Language Models, Model Based Reinforcement Learning, Symbolic AI? This is something disappointing about Lex. 5 hours of content and Lex just wants to talk about the philosophy of AI and the Turing Test. He never gets to anything technical. * permalink * embed * save * parent * report * give award * reply [-]hippydipster [score hidden] 11 hours ago (0 children) They probably agreed not to get technical. * permalink * embed * save * parent * report * give award * reply [-]WhoRoger [score hidden] 3 hours ago (0 children) Nothing, at one point when mentioning different approaches he says "I hope I've got something clever in that space", so he either can't or doesn't want to talk specifics, and Lex was apparently aware. * permalink * embed * save * parent * report * give award * reply [-]window-silMaybe you're right. [score hidden] 23 hours ago (2 children) What does 1000x more powerful mean in this context? * permalink * embed * save * parent * report * give award * reply [-]Pool_of_Death[S] [score hidden] 23 hours ago (0 children) I'm not sure. But maybe it could mean 1000x faster (so thoughts that take you 3 seconds to generate would take the AI 3 milliseconds, or an 8-hour work day goes by in 28.8 seconds) Or 1000x more memory/depth? Like, it can make 1000x more connections than you could because it has many more examples and can keep them all in "working memory" I'm making these examples up, but it seems like if you can make something 1000x cheaper you could instead throw a bunch more data/ compute/etc. at it and drastically improve the capabilities instead of reducing the price. * permalink * embed * save * parent * report * give award * reply [-]Thorusss [score hidden] 17 hours ago* (0 children) Worst case: Intelligence*1000 Best case: Power Consumption*1000 ;) * permalink * embed * save * parent * report * give award * reply [-]WhoRoger [score hidden] 3 hours ago (0 children) At one point he hints at parallelization. When he speaks of that 10000 GPU cluster, he mentions how it would run 128 human level intelligences. Looks like his aim is to get to that human level, and then just increase the number of digital people running on a piece of hardware as performance increases. Hm I don't quite get it either. * permalink * embed * save * parent * report * give award * reply [-]SignalEngine 11 points12 points13 points 1 day ago (2 children) Currently we're unable to support an AGI even with a arbitrarily large number of GPUs. I suppose his statement on figuring out the 'tricks' is solving the AGI problem sufficiently that there would be some finite number of GPUs that would be sufficient to realize it in reality. * permalink * embed * save * parent * report * give award * reply [-]Pool_of_Death[S] [score hidden] 23 hours ago (1 child) It seems weird to me to guess the number of GPUs it will take to run human-level AGI once you have all of the "simple tricks" figured out. Maybe it's 10,000 GPUs but maybe it's 200? Obviously, he's 10x more experienced than me in this field but it just doesn't seem like we have enough data to make a good guess at this point. * permalink * embed * save * parent * report * give award * reply [-]VelveteenAmbush [score hidden] 18 hours ago (0 children) It sounds like he has an idea for the necessary scale of the neural network, and it doesn't seem unreasonable to back into a number of GPUs from there. * permalink * embed * save * parent * report * give award * reply [-]No_Industry9653 [score hidden] 20 hours ago (0 children) Maybe the stuff about thinking AGI is safe is an excuse not to sound irresponsible, but really he doesn't care about that. Personally I think it's better if we get there before figuring out how to control it, despite the risk, so I'd probably say that sort of thing if I was a public figure with a chance to work on it. * permalink * embed * save * parent * report * give award * reply [-]WTFwhatthehell [score hidden] 23 hours ago* (5 children) IT would be operating at a snail's pace, much slower than human thought. I'm reminded of an old Vernor vinge story , I think it may have been part of True Names where a character is revealed to be a superhuman AI from a canned military research project... but running at much slower than human speed and posing as someone communicating only by email. * permalink * embed * save * report * give award * reply [-]Thorusss [score hidden] 17 hours ago* (4 children) I think a 1000x slower true superintelligence could still doom as. Think about how most hacks are not done in real time, but tools and scripts prepared in advance. If it manages to stay hidden long enough, it might as well take over, especially with "dumb" programs it has written, that can respond in real time. * permalink * embed * save * parent * report * give award * reply [-]Drachefly [score hidden] 11 hours ago (2 children) 1000x is pretty danged slow. * permalink * embed * save * parent * report * give award * reply [-]c_o_r_b_a [score hidden] 7 hours ago* (1 child) It wouldn't be 1000x slower at everything. It would presumably be far faster than humans at most things. Just its speed of influencing things geographically separated from itself, or collecting information it doesn't already have. For example, GPT-3 is very fast (and could probably also be a lot faster) at dealing with a ton of information, but the initial collection and training process were slow and happened over a long period of time. (Not really a very analogous example for something like AGI/ASI probably, but that's just one thing that comes to mind.) So Carmack probably has a point about some kind of hypothetical simultaneous global attack or something*, or like predicting/dealing with situations that occur in real-time which it can only interface with via an internet connection, but it could still effectively feel very fast and react to things almost immediately from the perspective of human perception. *(and even then, it may have near-instantaneous handling of concurrency; the lower bound on latency and upper bound on latency will make it not necessarily any faster than a dumb botnet DDoSing N hosts at once, but it may still hit them all with complete simultaneity, again like a botnet can easily do, so the breadth can still make it feel very fast "multitasking"-wise in a way a human could never compete with, even if the "depth" is somewhat limited latency and bytes-per-second-wise). * permalink * embed * save * parent * report * give award * reply [-]WhoRoger [score hidden] 3 hours ago (0 children) It wouldn't be fast at thinking and making decisions, which is the point of an AI. Like if it were spread out over 10k GPUs all over the world like a botnet, instead of a datacenter with 10k GPUs. So this thing could, for example launch a DDoS attack at a government, but it would take it years to come to such a decision because the individual parts of its brain need to communicate with each other, and there's latency and bandwidth issues. Kinda like trying to use cloud storage as swap for virtual memory. Linus Tech Tips tried it as an experiment... * permalink * embed * save * parent * report * give award * reply [-]WhoRoger [score hidden] 3 hours ago (0 children) Yes the scripts are prepared in advance, which is where the intelligence matters. If it takes a human 3 days to write and prepare those scripts, it would take an AI that's 1000x slower almost 3 years to write that script. * permalink * embed * save * parent * report * give award * reply [-]bulltrap0427 [score hidden] 20 hours ago (1 child) Unfortunately for the rest of us UDP exists * permalink * embed * save * report * give award * reply [-]c_o_r_b_a [score hidden] 7 hours ago (0 children) Also, with QUIC and the new WebTransport JavaScript API standard, you can now easily use UDP in the browser. (I really can't think of even a contrived scenario where this would somehow give an advantage to a malevolent AGI or any application in general, but just an interesting new development to share. I suppose it could make cheating at certain real-time web games slightly easier, but programmatic cheating is obviously already rampant in all of the non-web UDP-based games out there.) * permalink * embed * save * parent * report * give award * reply [-]TheMeiguoren [score hidden] 21 hours ago* (4 children) I agree with Carmack that a hard takeoff is not going to happen on the order of days-week. FOOM is a little silly. However I disagree that a longer takeoff on the order of 5-10 years is something that we will be able to successfully navigate with any certainty. He seems to have a lot of faith in our ability to wrangle with the problem when it becomes more concrete, but I think our institutions are not that nimble. And the profit incentives will almost certainly not be aligned with safety. Carmack says in the episode that his persona is very much oriented to the concrete, daily, "follow the gradient descent", and I see his (in)ability to care about downstream effects to be his biggest blind spot. We need to be laying as much groundwork now as we can. Great interview though, I just finished all 5 hours myself. * permalink * embed * save * report * give award * reply [-]Evinceo [score hidden] 21 hours ago (3 children) Slow take-off where AGI gets simultaneously elected POTUS and POPRC. * permalink * embed * save * parent * report * give award * reply [-]Thorusss [score hidden] 17 hours ago* (2 children) If the AGI competes in legit elections, we are very lucky and most likely on the utopian AI path. * permalink * embed * save * parent * report * give award * reply [-]Evinceo [score hidden] 10 hours ago (1 child) Every Debate: I'm going to turn everything into paperclips Guy wearing Clippy 2032 hat: I find its honestly refreshing. Anyway, both sides have problems. * permalink * embed * save * parent * report * give award * reply [-]CronoDAS [score hidden] 7 hours ago (0 children) The other candidates are Cthulu and Sweet Meteor of Death. ;) * permalink * embed * save * parent * report * give award * reply [-]hackinthebochs [score hidden] 21 hours ago (20 children) I agree with Carmack that runaway AI will be a lot more gradual than the doomsday crowd believes. I don't think any measure of an AI's intelligence or practical abilities scales linearly with hardware. Say we had a human-level AGI. Improving any hardware constraint by a factor of 1000x (say flops, memory size, bandwidth, etc) won't scale its intelligence or abilities by 1000x. Increasing the clock speed by some factor will reduce its wall-clock operation time by that factor. But this doesn't mean it is now 1000x smarter than a human. A human that operates at 1000x the speed is still bounded by the fundamental limitations of human intelligence. To go from a really smart human to something whose intellectual abilities are so alien that we are powerless to stop it isn't just a matter of speeding it up. * permalink * embed * save * report * give award * reply [-]BullockHouse [score hidden] 19 hours ago (4 children) On the flip side, humans only have 2-3x the cortical neurons of chimps, so I wouldn't underestimate what might happen if you get to human intelligence and then scale up the same model architecture 10 or 20x. * permalink * embed * save * parent * report * give award * reply [-]hackinthebochs [score hidden] 18 hours ago (3 children) Human brain capacity vs a chimp definitely crosses some kind of threshold where the accumulation of knowledge becomes possible. But it doesn't suggest the space around the threshold is linear in the sense that adding more capacity of the same kind will scale capabilities proportionally. I doubt there is a nearby capacity threshold where just adding more speed or a larger working memory will enable capacities that are so different in kind to ours that it's abilities would be totally alien. The sort of doomsday scenarios that some imagine would require a kind of manipulation of the world that I'm not sure is possible, let alone is just a matter of scaling up human capacities. While I do think there are plausible doomsday scenarios, they are of the mundane kind (e.g. an army of robots inhabited by AGI overpowers humanity). * permalink * embed * save * parent * report * give award * reply [-]BullockHouse [score hidden] 18 hours ago (0 children) But it doesn't suggest the space around the threshold is linear in the sense that adding more capacity of the same kind will scale capabilities proportionally. I doubt there is a nearby capacity threshold where just adding more speed or a larger working memory will enable capacities that are so different in kind to ours that it's abilities would be totally alien. I think the space doesn't have to be linear for it to unlock shocking new capabilities, when scaling by orders of magnitude is involved. GPT-3 was roughly 100x the size of GPT-2. Even if you somehow granted chimps writing and allowed them to accumulate insights generation to generation, they'd never end up with calculus for example. We've tried to raise chimps as humans and they are unable to benefit much from our accumulated knowledge. There's a genuine intellectual gap there. Calculus is an example of an alien ability that's totally beyond their comprehension, and their brains are only moderately smaller. I think you should probably keep open the strong possibility that there are calculus-level insights that are available in the modestly superhuman regime that enable new and incomprehensible possibilities. I do think discussion of AI risk is focused around a small set of relatively-unlikely hard takeoff scenarios, but I also don't think the argument hinges on hard takeoff. There are lots of slow doom scenarios that are also bad where powerful machine learning systems realize they don't have the ability to take over the world right now, so they play nice, bide their time, build capabilities, amass secret technologies, and we think everything is fine until suddenly everyone dies. That could take years, but once you have an adversary that's smarter than people are scheming against you, you're already in a lot of trouble. * permalink * embed * save * parent * report * give award * reply [-]thomas_m_k [score hidden] 14 hours ago (1 child) Even if you believe this, I hope you would still have the presence of mind to consider the possibility that you're wrong and AI is more dangerous than you think. And so you first spend time trying to figure out how dangerous it will actually be before you start writing code. Unfortunately this does not seem to be how OpenAI/John Carmack are thinking. They just seem to want to earn the glory of being the first; consequences be damned. Even if they think that we will have time to solve the alignment problem once we have AGI, it still seems reckless to me to publish all your research. There are lots of people on the planet, you know. And some of them are very irresponsible. So it would be kind of great to keep all AGI research secret until you have solved the alignment problem. * permalink * embed * save * parent * report * give award * reply [-]CantankerousV [score hidden] 8 hours ago (0 children) I'm in favor of putting effort into solving alignment now, but I don't think it's feasible to wait for alignment to be solved before starting AI development. If we froze AI progress today and focused on alignment, at what point can we conclude that we've found a solution? Even if we came up with a theory and could demonstrate vastly improved ability to steer current model architectures, nothing short of a universal theory of intelligence+agency can reasonably convince you that the alignment mechanism would work for AGIs we've yet to design. In an ideal world we could keep working on AGI without publishing anything until a satisfactory alignment mechanism has been found, but pulling that off in practice seems incredibly difficult. Putting aside game theoretic and governance issues, any effective containment would have to drastically limit the number of eyeballs on the problem. If it turns out superhuman AGI was actually hard to make we could waste decades of potential progress into narrower AI. I'm not thrilled by the fact that the best plan I can come up with is "just roll the dice and see what happens", but at this level of uncertainty I don't know if we can do any better. * permalink * embed * save * parent * report * give award * reply [-]Thorusss [score hidden] 17 hours ago (0 children) A human that operates at 1000x the speed is still bounded by the fundamental limitations of human intelligence I would not underestimate one of the smartest humans with perfect memory, that has 1000x the time. (Nick Bostrom thought "Speed Intelligence" through in Superintelligence, and showed, that it is also a plausible Existential Risk) Imagine Einstein in his prime, but having the time to study 1000x longer, while only 1x times passes. Really hard to estimate, what other breakthroughs could have come from the guy, who single handedly started Special and General Relativity AND Quantummechanics (what he got his Nobel prize for) * permalink * embed * save * parent * report * give award * reply [-]VelveteenAmbush [score hidden] 18 hours ago (11 children) I don't think any measure of an AI's intelligence or practical abilities scales linearly with hardware. Why not? The practical abilities of large language models absolutely scale with the size of the model, which is directly enabled by the hardware. * permalink * embed * save * parent * report * give award * reply [-]hackinthebochs [score hidden] 18 hours ago (3 children) Call the various emergent capabilities of large language models fractional human capacities. So the human baseline would consist of some number of these fractional human capacities. The emergence of these capacities seems to be proportional to the exponential growth in parameters and/or compute. If scaling follows the same pattern once human capacity is reached, you will simply continue to add fractional human capacities for each doubling of parameters/compute. But this doesn't represent a difference in kind. At what point does adding more fractional human capacities cross a threshold to fundamentally new space of behaviors? The scale needed to reach something that is beyond merely a bigger/faster human capacity will probably be far beyond the next few orders of magnitude in the current scaling paradigm. * permalink * embed * save * parent * report * give award * reply [-]VelveteenAmbush [score hidden] 4 hours ago (2 children) It feels like you're vascillating between two different claims, but assuming that the first is evidence for the second. The first claim is that model capabilities scale exponentially with certain inputs (parameter count, compute, data set size) -- and I generally agree with this and think it is well founded in the literature. But -- big caveat, those inputs are improving exponentially over time, and when you net them all together, capabilities seem to be improving exponentially over time. The second claim is that therefore superhuman capacity "probably be far beyond the next few orders of magnitude in the current scaling paradigm" -- and this both doesn't follow and strikes me as likely wrong. It seems DeepMind fully trained Chinchilla a year or so ago, and it has 70 billion parameters. GPT-2 was announced in early 2019 and had 1.5 billion parameters (and probably wasn't fully trained based on the Chinchilla paper, but let's set this aside to be conservative). So conservatively we saw a 46-fold improvement in fully trained model size in just a couple of years. The human brain has something like 100 trillion synapses, which is just about one order of magnitude more than Chinchilla. Now, the human brain is really well wired, the neurons are sparsely connected, a synapse is possibly more computationally capable than a single parameter weight, etc., so I'm not claiming that we are a year away from human capabilities. But we're probably just a few years off; my prediction has always been 2025-2030 and I stand by it. And I don't see why you would think that we wouldn't continue to improve past that point, such that the qualitative improvement from 2025 to 2030 would be just as exponentially impressive as the improvement from 2020 to 2025. So we can quibble about linearity of various inputs, the definition of "a few orders of magnitude," etc., but I think the rate of progress so far (and the lack of fundamental physical constraints that we can see ahead of us) suggests that we're going to reach and substantially exceed human cognition in the next decade or two. * permalink * embed * save * parent * report * give award * reply [-]hackinthebochs [score hidden] 1 hour ago (1 child) The issue in these discussions is that it's never clear what is being claimed and how it is being measured. We can all agree that "capabilities" are growing as model size/data/compute grows exponentially. But what is "capabilities" measuring and how does this measure extrapolate past the point of human equivalent intelligence? This is never clarified and so we're left to equivocate as needed to make our argument. My comment was to try to pin down a relevant notion of capability such that we can make a reasonable guess at behavior as these models scale beyond human intelligence. The concept of 'fractional human capacity' gives a rough measure of what is increasing as models are scaling pre-human level intelligence. Two reasonable conjectures help us get a handle on this measure: (i) human level intelligence is made up of some significant number of these fractional human capacities and (ii) fractional human capacities grow linearly with exponential growth in model size/data/compute. Given this, we can start to make meaningful claims about what happens as we scale beyond human equivalent intelligence. For one, adding a fractional human capacity onto human equivalent intelligence is still roughly human equivalent. The point is you are not growing the space of capacities significantly by continuing on the current scaling paradigm past human equivalence. And so we can expect that capabilities to remain roughly the same as you scale a few orders of magnitude past human equivalence. The ultimate question is at what point does a genuinely alien intelligence emerge that is beyond human capacity to understand or constrain? I don't know the answer to this question. But going by this argument, it is not simply a matter of continuing to scale beyond human equivalence, at least not what is feasible in the near term. * permalink * embed * save * parent * report * give award * reply continue this thread [-]Thorusss [score hidden] 17 hours ago* (6 children) In all the great papers from Deep Mind or OpenAI I have seen, many measures are plotted against logarithmic size to show the improvement. (e.g. here page 5 from the original GPT3 Paper) And it was a big breakthrough, that they were rising with size at all, because before, the suspicion was, that it could just as well reach a limit. * permalink * embed * save * parent * report * give award * reply [-]VelveteenAmbush [score hidden] 4 hours ago (5 children) Bit arbitrary to fixate on linear capabilities improvement with respect to parameter count / compute / data set, though, since all of those independent variables are improving supra-linearly over time. Neural net capabilities are definitely improving exponentially with respect to calendar year, which is what ultimately matters. * permalink * embed * save * parent * report * give award * reply [-]Thorusss [score hidden] 4 hours ago (4 children) While true, you originally questioned why intelligence/ability does not scale linearly with hardware. * permalink * embed * save * parent * report * give award * reply continue this thread [-]DomenicDenicola [score hidden] 10 hours ago (0 children) I found the analogy in "That Alien Message" / "Starwink" pretty convincing, to illustrate how a human operating at 1000x the speed could be threatening. * permalink * embed * save * parent * report * give award * reply [-]hippydipster [score hidden] 11 hours ago (0 children) Adding books to humans made humans a lot smarter. Adding hard drives to humans made humans a lot smarter. Adding ubiquitous connections via internet and smart phones made humans a lot smarter. well, maybe not :-) * permalink * embed * save * parent * report * give award * reply [-]WhoRoger [score hidden] 20 hours ago (1 child) When he comments on RUST: "I've done a little bit beyond hello world, I wrote some video decompression work as an exercise" Good ol' JC * permalink * embed * save * report * give award * reply [-]prescod [score hidden] 20 hours ago (0 children) Video decompression can be brutally hard or "not that difficult." Really depends what algorithm he implemented. * permalink * embed * save * parent * report * give award * reply [-]UncleWeyland [score hidden] 9 hours ago (0 children) One of the best podcast episodes this year. Just an excellent conversation all around. I think he's wrong about FOOM, but FOOM is probably the most controversial and difficult to model part of the AI risk landscape. In general, I think people who are more nuts and bolts "I coded videogames entirely in assembly" thinkers tend to underestimate the risk, but hearing him be skeptical made me update slightly in the "less Doom" direction. That's ironic right? * permalink * embed * save * report * give award * reply [-]zfinder [score hidden] 14 hours ago (0 children) An interesting sidenote: many former gamedevs become AI devs. Carmack, Hassabis, many non-famous guys I worked with. I'm not sure why there's such a correlation, but it exists. * permalink * embed * save * report * give award * reply [-]dunnolol123 [score hidden] 23 hours ago (20 children) I just finished the whole 5 hour conversation yesterday and it was really refreshing to hear his take on AGI and AGI safety. He brings up some valid points but what seems to come into focus more and more is the line between the type of people who think AI is default apocalyptic versus manageable; namely people who do nothing but philosophize versus people actually building these things. Yudkowsky, Hawking, Bostrom, etc. are all people who get paid to think. That's it. There's no building AI or doing anything pragmatic in a material sense. All of the actual top AI people building AI aren't hardcore doomsayers, which should tell us something. I'm not suggesting thinking is bad, I love philosophy, but it should give us a hint that something is off when there is clear divide like this. This is like the whole consciousness debate all over again. We have people who need to justify their tenured professorship by coming up with more and more convoluted problems and theories without actually doing anything pragmatic like running a neuroscience lab. It's like Alan Watts said in his lecture on meditation: "A person who thinks all the time, has nothing to think about except thoughts. So, he loses touch with reality and lives in a world of illusions. I'm not saying that thinking is bad; like everything else, it's useful in moderation. It's a good servant, but a bad master. And all so-called civilized peoples have increasingly become crazy and self-destructive because through excessive thinking they have lost touch with reality" * permalink * embed * save * report * give award * reply [-]BullockHouse [score hidden] 18 hours ago (1 child) He brings up some valid points but what seems to come into focus more and more is the line between the type of people who think AI is default apocalyptic versus manageable; namely people who do nothing but philosophize versus people actually building these things. This... just flat out isn't true. The head of OpenAI and DeepMind both are on record saying AGI is an existential threat. Lots of prominent AI researchers signed the open letter. They disagree, presumably, that their work is hastening the end, but the point stands: lots and lots of actual, on the ground researchers think this stuff is very dangerous. The "only philosophers and internet weirdos believe in AI risk stuff" was maybe true fifteen years ago, but it's not true anymore. And given that, you should maybe give the philosophers and internet weirdos more credit for getting there fifteen years before the rest of the field. * permalink * embed * save * parent * report * give award * reply [-]dunnolol123 [score hidden] 13 hours ago (0 children) Sure, they're very aware of the risks and are trying to figure out how to mitigate them. They're still working on AI though and figuring out along the way what works and what doesn't, their position isn't doomsday default. It's silly to not actually work on AI until you have a theoretic way to guarantee the perfect outcome. That's not how theory and its implementation usually work in the real world. Doomsday also isn't the only outcome, so by not pressing forward you potentially kill billions of people by delaying a benevolent AI (as a side note, why the hell are we okay with wanting to enslave godhood so that we can be immortal and have everything we want, but not risk extinction; especially since we likely will go extinct without AI anyway. Or at least keep living shitty lives). I have absolutely nothing against philosophy, it's the most important thing humanity has invented. Without it we wouldn't have most of the theories in science we have now. But there is a difference between useful philosophy that's coupled with pragmatic pursuits, like Einstein, Newton, etc. versus philosophy for the sake of philosophy, like Chalmers and Goff. * permalink * embed * save * parent * report * give award * reply [-]lupnra [score hidden] 19 hours ago (0 children) There are lots of AI Safety orgs doing pragmatic, hands-on building work. E.g. Anthropic, Conjecture, EleutherAI (Eleuther is not explicity an AI safety org, but many of the main devs are highly concerned about the alignment problem). To suggest that concerns about existential risk from AI only come from people who are "thinkers" and not "builders" is not accurate. * permalink * embed * save * parent * report * give award * reply [-]asmrkage [score hidden] 22 hours ago (12 children) People who don't want nuclear bombs don't spend their time trying to build them. People who don't want AI to destroy us don't spend their time trying to build it. * permalink * embed * save * parent * report * give award * reply [-]tehbored [score hidden] 20 hours ago (0 children) Building nuclear bombs actually is a good way to avoid getting nukes though. * permalink * embed * save * parent * report * give award * reply [-]VelveteenAmbush [score hidden] 18 hours ago (4 children) People who want safe and reliable bridges might become bridge builders, though. * permalink * embed * save * parent * report * give award * reply [-]asmrkage [score hidden] 5 hours ago (3 children) The position of AI skeptics in this analogy would be that bridges are fundamentally dangerous and can't be built safely. * permalink * embed * save * parent * report * give award * reply [-]VelveteenAmbush [score hidden] 4 hours ago (2 children) Bridges are fundamentally dangerous, compared with e.g. roads on solid ground. Bridge collapses are much more common on average than equivalently catastrophic failure of an equivalent length of road over land. But they're worth the risk, because they vastly increase our capabilities and efficiency. I think of AGI capabilities research in a similar way (although the danger is naturally speculative and forward-looking). * permalink * embed * save * parent * report * give award * reply continue this thread [-]dunnolol123 [score hidden] 21 hours ago* (5 children) If I am so terrified of nuclear bombs annihilating mankind I don't have to build bombs, I just have to build defense mechanisms that render them useless. Also, the people who warned most about the potential dangers of atomic bombs (igniting the atmosphere and all), were the people building it. If you're MIRI, why not use your superior theories to build an AI that can kill the badly built AI by those pesky peasants in their billion dollar labs. Why lament and write esoteric papers instead of building an adversary? I'm not disagreeing that safety is obviously important, but when people who don't actually do anything pragmatic look down on people like Carmack's views, it's pretty sad. Obviously the engineers creating actual value for humanity are going to make sure they don't destroy the world; would be pretty bad for business if their customers turn into paperclips. * permalink * embed * save * parent * report * give award * reply [-]benide [score hidden] 20 hours ago (0 children) If you're MIRI, why not use your superior theories to build an AI that can kill the badly built AI by those pesky peasants in their billion dollar labs. Why lament and write esoteric papers instead of building an adversary? Because since they haven't solved the alignment problem, they predict their own AI would be just as bad. Or, if they are as prideful as you think they are, they would think their AI would be even worse. * permalink * embed * save * parent * report * give award * reply [-]hippydipster [score hidden] 10 hours ago (2 children) Obviously the engineers creating actual value for humanity are going to make sure they don't destroy the world That someone would say this in today's world absolutely blows my mind. * permalink * embed * save * parent * report * give award * reply [-]MTGandP [score hidden] 6 hours ago (1 child) For example: * people do gain-of-function research even though lab leaks have been documented to occur numerous times even at "highly secure" labs * many people working on the Manhattan Project thought there was some chance a nuclear bomb would ignite the atmosphere and kill everyone, but they built it anyway (relatedly, the first nuke released twice as much energy as they expected. Luckily it was only 2x and not like 1000x) * permalink * embed * save * parent * report * give award * reply continue this thread [-]asmrkage [score hidden] 5 hours ago (0 children) You're creating a false argument that AI skeptics don't actually make. They don't say there is good AI and bad AI. They say any AI will inevitably lead to it spinning out of control. Current AI makers are the ones using the "good/bad" dichotomy as they feel they can control it into being good. Skeptics say that control is illusory given an advanced enough system. * permalink * embed * save * parent * report * give award * reply [-]AllegedlyImmoral [score hidden] 23 hours ago (0 children) All of the actual top AI people building AI aren't hardcore doomsayers, which should tell us something It tells us that people who are significantly concerned about AI risk are, for some reason, not also leading the charge to build it as fast as possible. Obviously if those doomsayers really understood AI they'd be out there building it at max speed too, but they're not so that clearly proves they don't know what they're talking about. Also, the people who are most gung-ho about building AI are obviously also necessarily maximally cautious and thoughtful about the possible downsides of what they're doing, that's how you get to be the ones moving the fastest in your field. * permalink * embed * save * parent * report * give award * reply [-]hippydipster [score hidden] 10 hours ago (0 children) All of the actual top AI people building AI aren't hardcore doomsayers, which should tell us something. It tells me they aren't the sort of people to think that way. That's it. It's not that they had more info => concluded differently. The difference was upstream of all that - a basic personality difference. One type of person, confronted with vast uncertainty, reacts with a shrug and thinks "why worry about it, no one can know" Another type of person, confronted with vast uncertainty, reacts with anxiety and thinks "this could go good, bad, horrific, no one can know - so let's go slow". Not a single argument in this thread is convincing of anything. All it is is different personality types putting forth their basic bias, and then rationalizing some words after. * permalink * embed * save * parent * report * give award * reply [-]RT17 [score hidden] 8 hours ago (0 children) Yudkowsky, Hawking, Bostrom, etc. are all people who get paid to think. That's it. There's no building AI or doing anything pragmatic in a material sense. All of the actual top AI people building AI aren't hardcore doomsayers, which should tell us something. We would not expect the people working hardest to produce AGI as soon as possible to believe that it is highly likely to result in human extinction, regardless of whether AGI xrisk was real or not. So in fact it tells us nothing. * permalink * embed * save * parent * report * give award * reply [-]Hostilian 13 points14 points15 points 1 day ago (14 children) His intuitions match mine about AGI. * permalink * embed * save * report * give award * reply [-]Pool_of_Death[S] 6 points7 points8 points 1 day ago (13 children) You don't believe in fast take-off? And you think it's too early for progress to be made on AI safety? Can you explain your main positions and why you believe them? * permalink * embed * save * parent * report * give award * reply [-]Tioben [score hidden] 23 hours ago (2 children) Not who you asked, but I think the greater existential threat is deciding what the existential threats will be long before we have any substantial evidence to inform our predictions. Like, there was a time that people believed the apocalypse/Hell was an imminent existential threat, and that (even more than belief in Hell as a personal threat) has had negative repercussions on society for a millenia now. If AGI has any connection to any existential threat at all, we won't know the nature of that connection might until AGI is already substantially consequential/empirical. And it won't be AGI as a whole, but one aspect of it in a context, likely having much less to do with AGI itself and much more to do with the social systems that form around it. The true threat is lack of agency and/or inflexible agency over ourselves/systems, and that's a threat that far overwhelms any particular technology or event. * permalink * embed * save * parent * report * give award * reply [-]Pool_of_Death[S] [score hidden] 23 hours ago (1 child) The true threat is lack of agency and/or inflexible agency over ourselves/systems, and that's a threat that far overwhelms any particular technology or event. How would you improve this though? That seems almost intractable * permalink * embed * save * parent * report * give award * reply [-]gomboloid(APXHARD.com) [score hidden] 6 hours ago (0 children) This will sound crazy unless it's heavily unpacked, but the short answer is love. A slightly longer answer is a belief that 1) the orthogonality thesis is really only true over short time frames 2) long term +EV thinking is really indistinguishable from human ethics In a chaotic, violent universe, your ability to advance any goals is constrained by the impossibility of predicting the future. If you can die, and in this universe everythign can die, your best chance to survive over extremely long time frames is a bunch of other agents wanting to repair you. I think the best strategy is therefore something that looks a lot like love: take care of the agents around you, help them advance their instrumental subgoals, so that they'll look after you if some unpredictable risk flattens you. Long arguments for the inherent risks to all agents here and strength of the 'love' strategy here. * permalink * embed * save * parent * report * give award * reply [-]Hostilian [score hidden] 22 hours ago* (9 children) I don't believe in Russel's Teapot, either. Shall I justify that belief too? [edit] That was unnecessarily snarky. I don't see the point in justifying a non-belief in something; I do not find the evidence of existential AI risk (or any expansive claim of AI risk) very credible, in the same way that I don't think vacuum decay or the LHC producing dangerous exotic particles is credible. * permalink * embed * save * parent * report * give award * reply [-]persolb [score hidden] 21 hours ago (5 children) I find this general response odd (ignoring the snark:) ) If I said Something like 'I don't believe cholesterol impacts health', I think it is fair to ask why. Or maybe closer, as it can't be proven yet 'I don't think fusion will ever be commercially viable.' In order to discuss anything that is a prediction, it seems necessary to say things like 'you believe it will happen because of X logic, and here is why I think that is wrong'. This is actually the type of arugument made via Russel's Teapot. * permalink * embed * save * parent * report * give award * reply [-]_SeaBear_ [score hidden] 20 hours ago (1 child) The problem is that there isn't much of an interesting explanation to be had here. The default opinion is "Things will keep going basically the same way as before.", and there's no easy way to explain my (or OP's) rejection of common fast takeoff theories beyond that. Everything I know about intelligence, artificial or otherwise, indicates that it becomes exponentially harder the smarter you get, and it requires a combination of a lot of narrow tasks that don't seem to be related. AI development for the past 50 years has consistently been a series of finding new, specific, tasks to improve and, until we get a better working theory for what intelligence is, that seems to be the most accurate description of what intelligence is. It doesn't seem to connect to raw processing power, as much as specific reasoning skills. I have always held that a fast takeoff is absurdly unlikely, and recent developments have only solidified that belief. Specifically, the machine learning training makes it exponentially harder for an AI to understand and modify its own code. The fact that AI seems to have human-level wod-processing skills without even insect-level reasoning means there's no easy way to judge intelligence. The fact that processor speeds, as we build them, have a hard cap that we are almost reaching means we can't simply slap on more processing power and hope for exponentially smarter AIs. etc. I've looked, and believe me I've looked hard, and still I've never found someone even attempt to argue against any of these points. The only arguments in favor of fast takeoff I've seen are, like, "What if we had a human brain, but like 100x more powerful? Isn't that scary?" or "Things like the economy and climate have changed way faster than people expected too.". It's not even really an argument as to why fast takeoff is likely, it's just doomsaying. 99% of the fast takeoff argument is convincing people that it's even theoretically possible for AI research to suddenly go from "We have no idea what we're even trying to build, let alone how." to "We currently have a machine smarter than all humans combined." without knowing it in advance. Ok, I've accepted that it's theoretically possible, but what arguments are there that it will actually happen? * permalink * embed * save * parent * report * give award * reply continue this thread [-]mattex456 [score hidden] 17 hours ago (0 children) If I said Something like 'I don't believe cholesterol impacts health' There's very weak evidence that "bad cholesterol" (LDL) negatively impacts health. The whole area of research was heavily influenced by ethical/ religious vegans (like the 7th day adventists), because plant oils do lower LDL as opposed to animal fats. * permalink * embed * save * parent * report * give award * reply [-]CronoDAS [score hidden] 6 hours ago (1 child) Fusion could become commercially viable in principle, but nobody seems to be going to invest the billions of dollars into plasma research that it would take to have a chance of making it viable. :/ * permalink * embed * save * parent * report * give award * reply continue this thread [-]drcode [score hidden] 21 hours ago (1 child) I never believed in the moon landings but Russell's Teapot * permalink * embed * save * parent * report * give award * reply [-]Lone-Pine [score hidden] 6 hours ago (0 children) The moon landings were a hoax. The Apollo astronauts actually visited Jupiter to find that damn teapot. * permalink * embed * save * parent * report * give award * reply [-]Glittering-Roll-9432 [score hidden] 10 hours ago (0 children) The reason you should take it seriously is simple, the only intelligent creature on earth(humans) are immensely violent and capable of destroying every thing that exists that means something to us. GAI will at least be smart enough to have those thoughts. What we don't know is if they decide something different than what humans d have decided. * permalink * embed * save * parent * report * give award * reply [-]Vast-Celebration3976 3 points4 points5 points 1 day ago (11 children) "He doesn't believe in fast takeoff and thinks it's much too early to be talking about AI ethics or safety" , I'm curious how that pairs with a 50-60% likely by 2030 belief? the value loading / control problems are difficult. Its not something you make a hard turn into post-hoc. I have thought about just the cost and hardware angle and I think that's a sigh of relief at least. If it costs 20 billion a day to run an AGI with a human equivalent iq of 200, ok so its off the top doing exponentially more than a human with that IQ in 24 hours (no 7 item working memory limit, can "focus" on more than one thing at a time, doesn't sleep) but lets say its 1000x more "work" being done, well ok but now you're spending 20 million a day for each of those iq 200 brains, you probably couldn't find that many living humans with an IQ of 200 to hire but I'm sure if you could you wouldn't have to spend 6.67 million every 8 hours of work they did. Then what if "scaling" is again, limited by hardware and electricity, so to get the thing to an IQ of 300+ (superhuman) its idk double that? I'm not very confident this will be the case though, we worry "big picture" about a general intelligence that can match or beat a human in all domains we engage with but I feel like you get pretty earth shattering and society shaking results even if its well controlled and only superhuman in very limited domains. Some of the stuff they're doing with like, physics sims where the AI is figuring out exotic maths and physical interactions we don't yet know about? , or even the sort of old history stuff that we're just used to and ignore like the panopticon we live in and the AI playing the stock market, but I digress. * permalink * embed * save * report * give award * reply [-]Pool_of_Death[S] [score hidden] 23 hours ago (1 child) He doesn't believe in fast takeoff and thinks it's much too early to be talking about AI ethics or safety" , I'm curious how that pairs with a 50-60% likely by 2030 belief? I think he's saying human-level AGI by ~2030 and then incremental progress that will happen in tandem with safety/ethics improvements so it's not a big concern, especially at this point. I'm not very confident this will be the case though, we worry "big picture" about a general intelligence that can match or beat a human in all domains we engage with but I feel like you get pretty earth shattering and society shaking results even if its well controlled and only superhuman in very limited domains. I agree... I think narrow AI in the next 10 years will do more than all of the other tech advances in the past 50 years. People are so concerned about AGI but don't realize that sufficiently powerful narrow AI (which we have line of sight of) can be just as concerning * permalink * embed * save * parent * report * give award * reply [-]Thorusss [score hidden] 17 hours ago (0 children) Any suffizient powerful set of narrow AIs is indistinguishable from Superintelligence? Also the deep question if humans are truly general intelligences, or just a well optimized collection of narrow AIs. The selective loss of ability from localized brains injuries actually points in this direction. * permalink * embed * save * parent * report * give award * reply [-]persolb [score hidden] 21 hours ago (6 children) We haven't solved value loading and control problems with the human level humans we have... and that isn't (yet) a doomsday scenario. The more I think about Yudowskyi/Bostrom/etc concern about AI... the more I think they are. Issuing the forest for the trees. We already have other entities who make poor decisions and which we can poorly control. And their power gets stronger every year. Even if AI weirdly turned out to be physically impossible, the generalized problem is never going away while humanity remains free-ish. * permalink * embed * save * parent * report * give award * reply [-]Thorusss [score hidden] 17 hours ago (1 child) We haven't solved value loading and control problems with the human level humans we have... and that isn't (yet) a doomsday scenario. ehm, Atomic war and bioengineering are two fields of Existential Risk research that are definitely taken serious. One can easily from that a control problem for humans. * permalink * embed * save * parent * report * give award * reply [-]persolb [score hidden] 12 hours ago (0 children) Yeah, that's what I meant but my wording was poor. I should have said '"that hasn't yet caused doomsday." * permalink * embed * save * parent * report * give award * reply [-]Vast-Celebration3976 [score hidden] 7 hours ago (3 children) Thats handwaving the problem away We collectively control each other. We also collectively agree on values that we can communicate to each other and agree about. An alien superintelligence would be dangerous in ways we literally cant imagine. We have , nuclear control treaties , MAD , bio weapons bans , laws and courts and police and jails. Behavioral norms. Empathy which seems hard wired into other beings on the planet who are not human. Its not even apples and oranges. * permalink * embed * save * parent * report * give award * reply [-]persolb [score hidden] 6 hours ago (2 children) Maybe there is something in that link that applies, but I'm not seeing it. I don't buy "we literally can't image". We have very wide ranging imaginings of what might happen, that cover all risk levels... we just don't know which outcomes are most likely. Likewise, I think our ability to control each other is overstated. Norms/empathy/etc are not things that apply to 100% of people. It isn't the majority that is an issue, it is that year over year an abnormal person can have more and more power. The issue to me isn't 'AI'. It is that exponentially growing capabilities will not end well if all systems/people aren't on the same page. Some of these systems will be human, some may be alien AI, some may be corporations run by Etherium rulesets... but the root issue is the same. I'm not seeing a order of magnitude difference in outcomes between a crazy person than can print out unsustainable extinction level plagues to deliver by drone, and an AI that kills us in some other more creative way. * permalink * embed * save * parent * report * give award * reply [-]Vast-Celebration3976 [score hidden] 4 hours ago (1 child) "we literally can't image" Well when I wrote that, I'm referring to an alien intelligence that doesn't actually have to be conscious. So, if it doesn't have an "inner world" and that's just some fluke of being us then unless were doing a full brain emulation to start with I fail to see how "values" could ever be "loaded". Because I'm conscious and have just normal human intelligence, trying to wrap my head around what an IQ that's superhuman and isn't limited by our biology (attention, working memory , speed etc) is , quite literally "unimaginable". i can begin the thought experiment "what would echo location be like? " but my imagining fails me because I'm just mashing up closed eyed hearing with my brains ability to map 3d space, its not the same and I know its not the same. I can't describe the taste of a strawberry to someone who hasn't had a strawberry, nor could I "imagine" what "wet" feels like if I had no sensory experience to correlate to it. So I say "literally can't imagine" because I can't. But functionally yes, you're right, we have the "S-risk" concept and discussions about the risk however alien it could be. I think the magnitude difference is that the human agent will be limited by being human, the human can end the world, the misaligned AI is simply capable of much more malice and suffering than a human could achieve, because it can find ways to cause suffering that...we cant even imagine. * permalink * embed * save * parent * report * give award * reply continue this thread [-]abecedarius [score hidden] 18 hours ago (1 child) In the current paradigm, whatever your budget had to be for training a model, a similar budget will pay for running it a whole lot once it exists. It's not really realistic to expect the training to be within budget but not the inference. * permalink * embed * save * parent * report * give award * reply [-]Vast-Celebration3976 [score hidden] 7 hours ago (0 children) I suppose thats true , even if we took gpt3 as a baseline and extrapoalted thst cost for every domain of human knowledge its....0.0008 cents per 1k "tokens" and the faq says 35 tokens is a paragraph. * permalink * embed * save * parent * report * give award * reply [-]ViperOrel23 [score hidden] 23 hours ago* (1 child) The problem I see in his thinking is that generality is not smart. Lizards are generalists within the domain defined by their receptors and the physical world's reactions to their decisions. Humans have massive encoder/decoders for "probable states of being human" that include language and planning as trained from life long observation and dreaming. Those are smart, but the generalist is the one beating up the encoders until they dream up a solution to the generalist's problems. Edit: GPT3 should make it obvious to anyone who uses it for exploration that we're already perfectly capable of making superhuman encoder/ decoders, a bad AI outcome now just awaits some idiot hooking up a dumb generalist to super encoder/decoders in a way that is able to self correct by retraining parts of the encoders... 2029 is gonna be lit * permalink * embed * save * report * give award * reply [-]Evinceo [score hidden] 21 hours ago (0 children) Humans have massive encoder/decoders for "probable states of being human" that include language and planning as trained from life long observation and dreaming. I don't think that's a very good model for understanding brains. * permalink * embed * save * parent * report * give award * reply [-]Laafheid [score hidden] 17 hours ago (6 children) People who use the toddler metaphor should be aware that a lot of people grow towards maladaptive solutions for a subset of their problems, i.e. alignment. I've been meaning to watch the interview but 5 hours is a big chunk, does anyone know what these less than 6 concepts translating to thousands of lines of code are, or what his unique angle actually is? * permalink * embed * save * report * give award * reply [-]FolkSong [score hidden] 15 hours ago (3 children) does anyone know what these less than 6 concepts translating to thousands of lines of code are He doesn't know, it's just his intuition that there are roughly that many conceptual breakthroughs needed. The number of lines of code aren't directly related to the "6 concepts", he was just estimating the size of the first successful program. I don't think he gave any hints on what his unique angle might be, they didn't get into any real details. * permalink * embed * save * parent * report * give award * reply [-]Laafheid [score hidden] 15 hours ago (2 children) Listened to that segment and it's really handwavy, checked the comments on the video and found it surprising how little of them actually mention anything said in the video, given how many comments laud/praise how cool it is that there's a 5 hour interview with the guy. weird vibe * permalink * embed * save * parent * report * give award * reply [-]monkorn [score hidden] 8 hours ago* (1 child) I'm just a boring enterprise programmer, but I hope I can at least help try to fill in some gaps after having gone a bit down this rabbit hole. It might be more useful to look at what he has experienced in his time as a game engine programmer to figure out where his intuition is coming from. This isn't Carmack but from some others in the trenches at the time. https://youtu.be/xn76r0JxqNM?t=1107 The entire talk is a good one but this timestamp is a particularly important argument that Carmack likely shares. And sure enough, we've now got things like area 5150 showing off what really what was possible. https://scalibq.wordpress.com/2022/08/08/ area-5150-8088-mph-gets-a-successor/ And we know that we can do some work with what seems like basically no code, like this example of lambda calculus. https://justine.lol/lambda/ The game engine programmers for instance had no idea what a quarternion was and once they talked to the Math dept, it massively simplified so much of the work. Everything that they had built to do 3d up until that time was filled with hacks upon hacks to try to arrive at something that looked right, but it never did and those hacks were slow. But it takes truly understanding all of the fundamentals from all of the necessary domains to do that. And some domains may be yet undiscovered, but we won't know that until we put all of that knowledge together. I'm reminded of Tao when I write that, the Math prodigy. Tao's mathematical knowledge has an extraordinary combination of breadth and depth: he can write confidently and authoritatively on topics as diverse as partial differential equations, analytic number theory, the geometry of 3-manifolds, nonstandard analysis, group theory, model theory, quantum mechanics, probability, ergodic theory, combinatorics, harmonic analysis, image processing, functional analysis, and many others. Some of these are areas to which he has made fundamental contributions. Others are areas that he appears to understand at the deep intuitive level of an expert despite officially not working in those areas. How he does all this, as well as writing papers and books at a prodigious rate, is a complete mystery. It has been said that David Hilbert was the last person to know all of mathematics, but it is not easy to find gaps in Tao's knowledge, and if you do then you may well find that the gaps have been filled a year later. Carmack is in many ways the Tao of game programming, so if we already have all of the tools and someone is going to be able to find simplicity, it may very well be him. Carmack is expecting a few of those types of insight. He probably thinks with his experience and him knowing what to look for, he has a better shot at it than people who are actively creating complexity and hacks exactly like what he did before he found the truth in the early days of game engine design. * permalink * embed * save * parent * report * give award * reply [-]Sinity [score hidden] 6 hours ago (0 children) The game engine programmers for instance had no idea what a quarternion was and once they talked to the Math dept, it massively simplified so much of the work. Everything that they had built to do 3d up until that time was filled with hacks upon hacks to try to arrive at something that looked right, but it never did and those hacks were slow. That reminds me of the time when I was trying to make a game (while learning programming); something like Asteroids clone. I wanted to make the ship rotate so it points in the direction of the mouse cursor. I spent hours writing convoluted if statements. Then I learned that it's done using trigonometric functions... Some time later I wanted to implement gravity, and I couldn't figure out how to handle 3+ bodies correctly. Then I learned about three body problem... * permalink * embed * save * parent * report * give award * reply [-]cygn [score hidden] 11 hours ago (1 child) Most neural networks today are not more than a couple of thousand lines of code. It seems reasonable that AGI is not much different in this regard. * permalink * embed * save * parent * report * give award * reply [-]Laafheid [score hidden] 6 hours ago (0 children) I'm aware of that, the issue is moreso that the amount of papers are skyrocketing and that it's not exactly clear what the biggest contributions are, or that the contributions even are concepts. I think a lot of progress in the fields can be attributed to existence of (autograd) frameworks, benchmarks/competitions & improvements in hardware such that neural networks have become feasible, None of which are conceptual things about the underlying algorithms. We thought attention was a really big thing, but it seems that at least in part the benefit of it is just constraining the feature space, disregarding the amount of tokens. If that is the case then batchnorm, layernorm, weight decay etc are all just different sides of the same proverbial coin, i.e. 1 concept that has an algorithmic component. * permalink * embed * save * parent * report * give award * reply [-]tehbored [score hidden] 20 hours ago (6 children) He's right to not believe in fast takeoff, it's a dulb idea. Most veteran AI folks don't believe in it either. Fast takeoff assumes that there would be a lot of low hanging fruit in terms of optimization that the AI could do without additional hardware. * permalink * embed * save * report * give award * reply [-]Thorusss [score hidden] 16 hours ago (5 children) The recent big breakthrough in Googles Minerva Math ability came from: *let's us not remove math symbols from the training data and *let's us ask the system to think step by step I assure you, there are still plenty of low hanging fruits out there, as expected in a young field like deep learning. * permalink * embed * save * parent * report * give award * reply [-]tehbored [score hidden] 11 hours ago (4 children) Yes, there are plenty right now. But most of them will we be picked by the time we have superhuman AGI. * permalink * embed * save * parent * report * give award * reply [-]RT17 [score hidden] 8 hours ago (3 children) The foom hypothesis is that as AGI becomes more intelligent, what constitutes "low hanging" expands, creating a positive feedback loop. * permalink * embed * save * parent * report * give award * reply [-]tehbored [score hidden] 6 hours ago (0 children) Physical limitations require more than just intelligence to get around. There is only so much optimization that can be done in software alone. Much of that will be done before we get to superhuman AGI. We will have subhuman AGI and narrow AI tools to assist us with optimization. The question becomes not how much smarter the superhuman AGI is than humans, it's how much smarter it is than the combination of coordinated groups of humans working with AI tools. That's why I believe most of the low hanging fruit in the software space will likely be picked before we reach superhuman AGI. There will be low-hanging fruit in hardware for sure, because hardware moves more slowly. But how is a rogue AI going to run chip fabs to make itself new processors of its own design? It could also leverage existing hardware, but it still needs to acquire and maintain access. An AGI's intelligence doesn't come from nowhere, it's mind exists in the physical world. Superintelligent doesn't mean omnipotent. Even if it is far smarter than humans, it is still limited by physical constraints. It needs computers and electricity in order to function, and also connectivity if it is to function on a distributed system. None of this is to say we shouldn't be wary of the danger. Once we reach the level of subhuman AGI, we need to being employing strict safety protocols on systems in development for increased functionality. Air gaps are the obvious one. Hardwired prohibitions on humans interacting with it alone, without someone else present. Limitations on what we allow it to see (for example, masking the voice of a researcher interacting with it and disabling any camera feed would make it hard for even a superintelligence to acquire enough information for emotional manipulation). Rigorous training protocol for those who have access to such systems. Etc. * permalink * embed * save * parent * report * give award * reply [-]tehbored [score hidden] 8 hours ago (1 child) I'm not sure I understand the question * permalink * embed * save * parent * report * give award * reply continue this thread [-]yofuckreddit [score hidden] 9 hours ago (0 children) Frankly I find the confidence in doom among people who know 1% of what carmack does about computers to be really off-putting. This was an incredible interview, it made me more excited for the future for sure. FINALLY a podcast going into enough detail about technical topics. * permalink * embed * save * report * give award * reply [-]kreuzguy [score hidden] 18 hours ago* (2 children) I think he is reasonable in his approach to AGI risk. The most pressing issue is not AGI bending us to our knees, but what humans will do when we find out most of our skills are not monetizable anymore. That's a recipe for great social instability. * permalink * embed * save * report * give award * reply [-]Drachefly [score hidden] 10 hours ago (0 children) Yeah, that's the more pressing issue, but AGI bending us to our knees or well past that point is a more severe issue. Both problems are worth considering. * permalink * embed * save * parent * report * give award * reply [-]gettotea [score hidden] 15 hours ago* (0 children) That is a great point. If I were an artist or a musician, I would be worried right now. * permalink * embed * save * parent * report * give award * reply [-]SIGINT_SANTA [score hidden] 17 hours ago (0 children) Edward Teller V2 I don't have any better way to describe this other than evil. Though shalt not destroy the world is maybe the most fundamental tenant of all morality, and John Carmack appears to be doing his absolute best to do it. * permalink * embed * save * report * give award * reply [-]TypingLobster [score hidden] 14 hours ago (0 children) You're limited to a 5G connection. If you take a calculation and factor it across 1 million cellphones instead of 1000 GPUs in a warehouse it might work but you'll be at something like 1/1000 the speed so you could have an AGI working but it wouldn't be real-time. It would be operating at a snail's pace, much slower than human thought. I'm not worried about that. It sounds like AGI will only be a problem in case network speeds or computer chips get substantially faster/cheaper in the future. I wonder what the trends have been like in the past. * permalink * embed * save * report * give award * reply [-]GORDON_ENT[] [score hidden] 12 hours ago (0 children) I think he's right about some of the more important questions and will almost certainly fail. Making a smarter than human AI is radically simpler than emulating human intelligence. * permalink * embed * save * report * give award * reply [-]OhHeyDont [score hidden] 11 hours ago (2 children) Only tangentially related but if you believe in both fast take off, think AGI most like hostile, and possible on current hardware, then shouldn't you be a terrorist attempting to destroy the tech industry? Or at least working in the field secretly causing as many problems as possible? * permalink * embed * save * report * give award * reply [-]hippydipster [score hidden] 10 hours ago (0 children) Are you suggesting people who concern themselves with possibilities should shut up unless they're so certain that murder becomes the rational action? * permalink * embed * save * parent * report * give award * reply [-]johnlawrenceaspden [score hidden] 10 hours ago (0 children) I believe those three things, and I'd sure be happy if some impossibly talented terrorists destroyed the tech industry. But me personally? Why? I've probably got ten or twenty years before these fucking fools destroy the world and I'm going to spend it enjoying the sunshine while I still can. * permalink * embed * save * parent * report * give award * reply [-]Sinity [score hidden] 10 hours ago* (1 child) He really does not believe in fast take-off (doesn't seem to think it's an existential risk). He thinks we'll go from the level of animal intelligence to the level of a learning disabled toddler and we'll just improve iteratively from there Yep He thinks AGI can be plausibly created by one individual in 10s of thousands of lines of code. Seems possible. He doesn't believe in fast takeoff because of TCP connection limits? It sounds much less weird in context. However, it will be small, the code will fit on a thumb drive, 10s of thousands of lines of code. - timestamp This does through. I'm pretty sure Billion-LOC fits on a thumb drive too. It'd be amusing if he actually won the race. * permalink * embed * save * report * give award * reply [-]c_o_r_b_a [score hidden] 7 hours ago (0 children) Off-topic side note: Using \ as a tweet continuation character (from line continuation escapes in C) is way better than "N/?". First time I've seen that. I don't use Twitter but if I did I'd definitely copy that. * permalink * embed * save * parent * report * give award * reply [-]dennisdennison [score hidden] 2 hours ago (0 children) He is very wrong about fast takeoff. Other than that I agree with a lot of what he's said. * permalink * embed * save * report * give award * reply [-]bmitc [score hidden] 14 minutes ago (0 children) To be honest, I'm a bit tired of hearing from or about Carmack. His company failed at making great games and failed at even making a popular game engine. He has questionable business practices. Tim Sweeney is a far more successful programmer and businessman, but he stays out of the limelight. I'm quite sure he's a hell of a whizbang C++, graphics, and general programmer, but I'm not sure what he has to offer that necessitates the worship and attention he gets. * permalink * embed * save * report * give award * reply * about * blog * about * advertising * careers * help * site rules * Reddit help center * reddiquette * mod guidelines * contact us * apps & tools * Reddit for iPhone * Reddit for Android * mobile website * <3 * reddit premium * reddit coins Use of this site constitutes acceptance of our User Agreement and Privacy Policy. (c) 2022 reddit inc. All rights reserved. REDDIT and the ALIEN Logo are registered trademarks of reddit inc. [pixel] p Rendered by PID 75548 on reddit-service-r2-loggedout-7cfb95d5df-llrcb at 2022-08-17 23:02:40.532635+00:00 running 9254a2b country code: US.