[HN Gopher] John Carmack talk at Upper Bound 2025
___________________________________________________________________
John Carmack talk at Upper Bound 2025
Author : tosh
Score : 462 points
Date : 2025-05-23 05:14 UTC (17 hours ago)
(HTM) web link (twitter.com)
(TXT) w3m dump (twitter.com)
| kamranjon wrote:
| I was really excited when I heard Carmack was focusing on AI and
| am really looking forward to watching this when the video is up -
| but just from looking at the slides it seems like he tried to
| build a system that can play the Atari? Seems like a fun project,
| but curious what will come out of it or if there is an associated
| paper being released.
| johnb231 wrote:
| Atari games are widely used in Reinforcement Learning (RL)
| research as a standard benchmark.
|
| https://github.com/Farama-Foundation/Arcade-Learning-Environ...
|
| The goal is to develop algorithms that generalize to other
| tasks.
| sigmoid10 wrote:
| They _were_ highly used. OpenAI even included them in their
| RL Gym library back in the old days when they were still
| doing open research. But if you look at this leaderboard from
| 7 (yes, seven!) years ago [1], most of them were already
| solved way beyond human capabilities. But we didn 't get a
| really useful general purpose algorithm out of it. As an AI
| researcher, I always considered Atari a fun academic
| exercise, but nothing more. Similar to how recognising
| characters using convnets was cool in the nineties and early
| 00s, but didn't give us general purpose image understanding.
| Only modern GPUs and massive training datasets did. Nowadays
| most cutting-edge RL game research focuses on much more
| advanced games like Minecraft which is thought to be better
| suited. But I'm pretty sure it's still not enough. Even role-
| playing GTA VI won't be. We probably need a pretty advanced
| physical simulation of the real world before we can get
| agents to handle the real world. But that means solving the
| problem of generating such an environment first, because you
| can't train on the _actual_ real world due to the sample
| inefficiency of all current algorithms. Nvidia is doing some
| really interesting research in this direction by combining
| physics simulation and image generation models to simulate an
| environment, while getting accuracy and diversity at the same
| time into training data. But it still feels like some key
| ingredient is missing.
|
| [1]https://github.com/cshenton/atari-leaderboard
| mschuster91 wrote:
| > But it still feels like some key ingredient is missing.
|
| Continuous training is the key ingredient. Humans can use
| existing knowledge and apply it to new scenarios, and so
| can most AI. But AI cannot _permanently remember_ the
| result of its actions in the real world, and so its body of
| knowledge cannot expand.
|
| Take a toddler and an oven. The toddler has no concept of
| what an oven is other than maybe that it smells nice. The
| toddler will touch the oven, notice that it experiences
| pain (because the oven is hot) and learn that oven =
| danger. Place a current AI in a droid toddler body? It will
| never learn and keep touching the oven as soon as the
| information of "oven = danger" is out of the context
| window.
|
| For some cases this inability to learn is actually
| desirable. You don't want anyone and everyone to be able to
| train ChatGPT unsupervised, otherwise you get 4chan
| flooding it with offensive crap like they did to Tay [1],
| but for AI that physically interacts with the meatspace,
| constant evaluation and learning is all but mandatory if it
| is to safely interact with its surroundings. "Dumb" robots
| run regular calibration cycles for their limbs to make sure
| they are still aligned to compensate for random deviations,
| and so will AI robots.
|
| [1] https://en.wikipedia.org/wiki/Tay_(chatbot)
| sigmoid10 wrote:
| This kind of context management is not that hard, even
| when building LLMs. Especially when you have huge windows
| like we do today. Look at how ChatGPT can remember things
| permanently after you said them once using a function
| call to edit the permanent memory section inside the
| context. You can also see that in Anthropic's latest post
| on Claude 4 where it learns to play Pokemon. The only
| remaining issue here is maybe how to diffuse explicit
| knowledge from the stored context into the weights.
| Andrej Karpathy wrote a good piece on this recently. But
| personally I believe this might not even be necessary if
| you can manage your context well enough and see it more
| like RAM while the LLM is the CPU. For your example you
| can then always just fetch such information from a
| permanent storage like a VDB and load it into context
| once you enter an area in the real world.
| mschuster91 wrote:
| > This kind of context management is not that hard, even
| when building LLMs.
|
| It is, at least if you wish to be in the meatspace,
| that's my point. Every day has 86400 seconds during which
| a human brain _constantly_ adapts to and learns from
| external input - either directly as it 's being awake or
| indirectly during nighttime cleanup processes.
|
| On top of that, humans have built-in filters for
| training. Basically, we see some drunkard shouting about
| the Hollow Earth on the sidewalk... our brain knows that
| this is a drunkard and that Hollow Earth is absolutely
| crackpot material, so if it stores anything at all then
| the fact that there is a drunkard on that street and one
| might take another route next time, but the drunkard's
| rambling is forgotten maybe five minutes later.
|
| AI, in contrast, needs to be hand-held by humans during
| training that annotate, "grade" or weigh information
| during the compilation of the training dataset, in order
| that the AI knows what is _written_ in "Mein Kampf" so
| it can answer questions upon it, but that it also knows
| (or at least: won't openly regurgitate) that the solution
| to economic problems isn't to just deport Jews.
|
| And huge context windows aren't the answer either. My
| wife says me, she would like to have a fruit cake for her
| next birthday. I'll probably remember that piece of
| information (or at the very least I'll write it down)...
| but an AI butler? I'd be really surprised if this is
| still in its context space in a year, and even if it is,
| I would not be surprised if it weren't able to recall
| that fact.
|
| And the final thing is prompts... also not the answer.
| We've seen it just a few days ago with Grok - someone
| messed with the system prompt so it randomly interjected
| "white genocide" claims into completely unrelated
| conversation [1] despite hopefully being trained on a ...
| more civilised dataset, and to the contrary, we've also
| seen Grok reply to Twitter questions in a way that
| suggest that it is aware its training data is biased.
|
| [1] https://www.reuters.com/business/musks-xai-updates-
| grok-chat...
| sigmoid10 wrote:
| >Every day has 86400 seconds during which a human brain
| constantly adapts to and learns from external
|
| That's not even remotely true. At least not in the sense
| that it is for context in transformer models. Or can you
| tell me all the visual and auditory inputs you
| experienced yesterday at the 45232nd second? You only
| learn permanently and effectively from particular
| stimulation coupled with surprise. That has a sample rate
| which is orders of magnitude lower. And it's exactly the
| kind of sampling that can be replicated with a run-of-
| the-mill persistent memory system for an LLM. I would
| wager that you could fit most people's core experiences
| and memories that they can randomly access at any moment
| into a 1000 page book - something that fits well into
| state of the art context windows. For deeper more
| detailed things you can always fall back to another
| system.
| bluesroo wrote:
| Your definition of "learning" is incomplete because
| you're applying LLM concepts to how human brains work. An
| LLM only "learns" during training. From that point
| forward all it has is its context and vector DBs. If an
| LLM and vector DB is not actively interacted with,
| nothing happens to it. However for the brain,
| experiencing IS learning. And the brain NEVER stops
| experiencing.
|
| Just because I don't remember my experiences at second
| 45232 on May 22, doesn't mean that my brain was not
| actively adapting to my experiences at that moment. The
| brain does a lot more learning than just what is
| conscious. And then when I went to sleep the brain
| continued pruning and organizing my unconscious learning
| for the day.
|
| Seeing if someone can go from token to freeform physical
| usefulness will be interesting. I'm of the belief that
| LLMs are too verbose and energy intensive to go from
| language regurgitation machines to moving in the real
| world according to free form prompting. It may be
| accomplishable with the vast amount of hype investment,
| but I think the energy requirements and latency will make
| an LLM-based approach economically infeasible.
| ewoodrich wrote:
| > You only learn permanently and effectively from
| particular stimulation coupled with surprise.
|
| This is just, not true. A single 2min conversation with
| emotional or intellectual resonance can significantly
| alter a human's thought process for years. There are some
| topics where every time they come up directly or
| analogously I can recall something a teacher told me in
| high school that "stuck" with me for whatever reason. And
| it isn't even a "core" experience, just something that
| instantly clicked for my brain and altered my problem
| solving. At the time, there's no heuristic that could
| predict how or why that particular interaction should
| have that kind of staying power.
|
| Not to mention, experiences that subtly alter thinking or
| behavior just by virtue of providing some baseline
| familiarity instead of blank slate problem solving or
| routine. Like how you subtly adjust how you interact with
| coworkers based on the culture of your current company
| over time vs the last without any "flash" of insight
| required.
| vectorisedkzk wrote:
| Having used vectorDBs before, we're very much not there
| yet. We don't have any appreciable amounts of context for
| any reasonable real-life memory. It works if that is the
| most recent thing you did. Have you talked to an LLM for
| a day? Stuff is gone before the first hour. You have to
| use every trick currently in the book, treat context like
| it's your precious pet
| sigmoid10 wrote:
| VectorDBs are basically just one excuse of many to make
| up for a part of the system that is lacking capability
| due to technical limitations. I'm currently at 50:50 if
| the problems will be overcome directly by the models or
| by such support systems. Used to be 80:20 but models have
| grown in usefulness much faster than all the tools we
| built around them.
| mr_toad wrote:
| Big context windows are a poor substitute for updating
| the weights. Its like keeping a journal because your
| memory is failing.
| fzzzy wrote:
| It reminds me of the movie Memento.
| aatd86 wrote:
| it's more than that. Our understanding from space and
| time could be stemming from continuous training. Every
| time we look at something, there seems to be a background
| process that is categorizing items that are on the
| retinal image.
|
| This is a continuous process.
| epolanski wrote:
| > Humans can use existing knowledge and apply it to new
| scenarios, and so can most AI
|
| Doesn't the article states that this is not true? AI
| cannot apply to B what it learned about A.
| mschuster91 wrote:
| Well, ChatGPT knows about the 90s Balkan wars, a topic to
| which LWT hasn't made an episode that I'm aware of, and
| yet I can ask it to write a script for a Balkan wars
| episode that reads surprisingly like John Oliver while
| being reasonably correct.
| epolanski wrote:
| Essentially Carmack pointed in the slides that teaching
| AI to play game a, b or c didn't improve AI at all at
| learning game d from scratch.
|
| That's essentially what we're looking for when we talk
| about general intelligence, the capability to adapting
| what we know to what we know nothing about.
| losvedir wrote:
| > Continuous training is the key ingredient. Humans can
| use existing knowledge and apply it to new scenarios, and
| so can most AI. But AI cannot permanently remember the
| result of its actions in the real world, and so its body
| of knowledge cannot expand.
|
| I think it depends on how you look at it. I don't want to
| torture the analogy too much, but I see the pre-training
| (getting model weights out of an enormous corpus of text)
| as more akin to the billions of years of evolution that
| led to the modern human brain. The brain still has a lot
| to learn once you're born, but it already also has lots
| of structures (e.g. to handle visual input, language,
| etc) and built-in knowledge (instincts). And you can't
| change that over the course of your life.
|
| I wouldn't be surprised if we ended up in a "pre-train /
| RAG / context window" architecture of AI, analogously to
| "evolution / long term memory / short term memory" in
| humans.
| newsclues wrote:
| Being highly used in the past is good, it's a benchmark to
| compare against.
| gregdeon wrote:
| I watched the talk live. I felt that his main argument was
| that Atari _looks_ solved, but there's still plenty of
| value that could be gained by revisiting these "solved"
| games. For one, learning how to play games through a
| physical interface is a way to start engaging with the
| kinds of problems that make robotics hard (e.g., latency).
| They're also a good environment to study catastrophic
| forgetting: an hour of training on one game shouldn't erase
| a model's ability to play other games.
|
| I think we could eventually saturate Atari, but for now it
| looks like it's still a good source of problems that are
| just out of reach of current methods.
| koolala wrote:
| Is a highly specialized bespoke robot for a Atari
| controller really that different? If anyone cared about
| latency they could have added it to the emulated controls
| and video with random noise.
| gregdeon wrote:
| I think it is. Latency was just one of the problems he
| described. A physical controller sometimes adds "phantom
| inputs" as the joystick transitions between two inputs.
| Physical actuators also slow down with wear. A physical
| Atari-playing robot needs to learn qualitatively
| different strategies that are somewhat more robust to
| these problems. Emulators also let the bot take as much
| time as it needs between frames, which is much easier
| than playing in real time. To me, all of this makes a
| physical robot seem like a decent way to start engaging
| with problems that come up in robotics but not simulated
| games.
| Buttons840 wrote:
| My impression is that Atari was 80% solved, and then
| researchers and companies moved on.
|
| A company solves self-driving 80% of the way and makes a
| lot of VC cash along the way. Then they solve intelligent
| chatbots 80% of the way and make a lot of VC cash along the
| way. Now they're working on solving humanoid robotics 80%
| of the way... I wonder why?
|
| In the end, we have technology that can do some neat
| tricks, but can't be relied upon.
|
| There are probably still some very hard problems in certain
| Atari games. Only the brave dare tackle these problems,
| because failure comes sharp and fast. Whereas, throwing
| more compute at a bigger LLM might not really accomplish
| anything, but we can make people _think_ it accomplished
| something, and thus failure is not really possible.
| cryptoz wrote:
| DeepMind's original demos were also of Atari gameplay.
| modeless wrote:
| He says they will open source it which is cool. I agree that I
| don't understand what's novel here. Playing with a physical
| controller and camera on a laptop GPU in real time is cool, and
| maybe that hasn't specifically been done before, but it doesn't
| seem surprising that it is possible.
|
| If it is substantially more sample efficient, or generalizable,
| than prior work then that would be exciting. But I'm not sure
| if it is?
| RetroTechie wrote:
| Maybe that's exactly his goal: not to come up with something
| that beats the competition, but play with the architecture,
| get a feel for what works & what doesn't, how various aspects
| affect the output, and improve on _that_. Design more
| efficient architectures, or come up with something that has
| unique features compared to other models.
|
| If so, scaling up may be more of a distraction rather than
| helpful (besides wasting resources).
|
| I hope he succeeds in whatever he's aiming for.
| tschillaci wrote:
| You will find many agents that solved (e.g., finished, reached
| high score) atari games, but there is still so much more work
| to do in the field. I wrote my Master's thesis on how to learn
| from few interactions with the game, so that if the algorithm
| is ported to actual robots they don't need to walk and fall for
| centuries before learning behaviors. I think there is more
| research to do on higher levels of generalization: when you
| know how to play a few video games, you quickly understand how
| to play a new one intuitively, and I haven't seen thorough
| research on that.
| lo0dot0 wrote:
| I can tell you right now without any research that video game
| designers reuse interface patterns and game mechanics that
| were already known when making new games. Those patterns and
| mechanics are also often analogies for real life allowing
| humans to intuitively play the games. If people can't play
| your game intuitively, they might say it's a bad game.
| Jensson wrote:
| So why can't AI learn those and reapply the same
| understanding to new games?
| albertzeyer wrote:
| His goal was not just to solve Atari games. That was already
| done.
|
| His goal is to develop generic methods. So you could work with
| more complex games or the physical world for that, as that is
| what you want in the end. However, his insight is, you can even
| modify the Atari setting to test this, e.g. to work in
| realtime, and the added complexity by more complex games
| doesn't really give you any new additional insights at this
| point.
| mike_hearn wrote:
| But how is this different to what NVIDIA have already done?
| They have robots that can achieve arbitrary and fluid actions
| in the real world by training NNs in very accurate GPU
| simulated environments using physics engines. Moving a little
| Atari stick around seems like not much compared to sorting
| through your groceries etc.
|
| The approach NVIDIA are using (and other labs) clearly works.
| It's not going to be more than a year or two now before
| robotics is as solved as NLP and chatbots are today.
| albertzeyer wrote:
| I think he argues that they would not be able to play Atari
| games this way (I don't know; maybe I also misunderstood).
|
| But also, he argues a lot about sample efficiency. He wants
| to develop algorithms/methods/models which can learn much
| faster / with much fewer data.
| gadders wrote:
| I want smarter NPCs in games.
| moralestapia wrote:
| Here's what they built,
| https://x.com/ID_AA_Carmack/status/1925243539543265286
|
| Quite exciting. Without diminishing the amazing value of LLMs, I
| don't think that path goes all the way to AGI. No idea if Carmack
| has the answer, but some good things will come out of that small
| research group, for sure.
| petters wrote:
| Isn't that what Deepmind did 12 years ago?
| willvarfar wrote:
| Playing Atari games makes it easy to benchmark and compare
| and contrast his future research with Deepmind and more
| recent efforts.
| moralestapia wrote:
| IIRC Deepmind (and OpenAI and ...) have done this on
| software-only setups (emulators, TAS, etc); while this one
| has live input and actuators in the loop, so, kind of the
| same thing but operating in the physical realm.
|
| I do agree that it is not particularly groundbreaking, but
| it's a nice "hey, here's our first update".
| hombre_fatal wrote:
| He points that out in his notes and says DeepMind needed
| specialized training and/or 200M frames of training just to
| kinda play one game.
| tsunamifury wrote:
| What deepmind accomplished with suicidal Mario was so much
| more than you probably ever will know from outside the
| company.
| mi_lk wrote:
| Do tell if you can. Were you there?
| dusted wrote:
| anywhere we can watch the presentation ? the slides alone are
| great, but if he's saying stuff alongside, I'd be interested in
| that too :)
| vasco wrote:
| Bro went his whole career and managed to somehow create a gig for
| himself where he gets the AI money while playing Atari. It's hard
| to increase the respect for someone who you already maxed out on
| but there we go. Carmack is a cool guy.
| willvarfar wrote:
| Although Carmack is the quintessential not-a-brogrammer.
| dusted wrote:
| I've never actually seen a brogrammer though, I've seen
| people who program only because they get money for it, I
| thought for a while those where it, but I'm not sure if I
| think they qualify either.
| nindalf wrote:
| It means "programmer I don't like". Very versatile insult
| and vague enough that it's impossible to defend against.
| kid64 wrote:
| No, I'm pretty sure it's just a programmer that
| understands the world in terms of bros.
| dusted wrote:
| "in the terms of bros" what does that even mean? I think
| bro is a term that's used pretty widely, for different
| things, in different cultures and contexts, I call my
| brother bro.. I've heard people call their friends bro..
| I've heard someone tell a police officer "don't tase me,
| bro"..
| floren wrote:
| My co-worker's 7 year old daughter calls him "bruh"
| lostmsu wrote:
| I call my 4yo daughter bro
| tomaytotomato wrote:
| From watching on the wall I've seen brogrammer used in
| various contexts (this is not an exhaustive list):
|
| - Someone who is a programmer but follows a hypermasculine
| cliche and makes sure everyone knows about it.
|
| - An insult used by other developers for someone who is
| more physically fit or interested in their health than
| themselves.
|
| - An insult used by engineers or other people who are not
| happy with the over representation of men in the industry.
| So everyone is lumped in the category.
|
| - Someone who is obsessed with the technology and trying to
| grind their skills on it to an excessive level.
| mi_lk wrote:
| > Someone who is obsessed with the technology and trying
| to grind their skills on it to an excessive level
|
| sounds like a person who respects their own profession
| though
| diggan wrote:
| > someone who is more physically fit or interested in
| their health than themselves
|
| Isn't "interested in their health" a signal that they are
| interested in themselves, rather than the opposite?
| oersted wrote:
| Disambigation: I believe "themselves" refers to the one
| insulting, not the one interested in their health.
|
| It tripped me up too, to be fair.
| tomaytotomato wrote:
| Apologies, grammar is hard.
| CPLX wrote:
| A. A male programmer who uses the word "bro" unironically
| in conversation.
|
| B. A person who is physically and culturally
| indistinguishable from A
| rfrey wrote:
| What physical or cultural characteristics would make a
| person "indistinguishable from A"?
| brotein wrote:
| Brogrammer here. I recognize that I spend 12 hours a day
| writing code (and loving it) as a fun thing but also a
| danger if I do it all sitting down. I stay busy and
| incorporate workouts into my day.
|
| I don't take it as a pejorative, it's an acknowledgement
| of my efforts to be even considered in this category. For
| those wondering I have a family, and have healthy
| activities otherwise. No cool diets or bioscience, just
| code, physical activity and coffee/water.
|
| This isn't a lifestyle I'm saying everyone should do,
| only that people should do what makes them happiest and
| most fulfilled for their set of goals.
| shermantanktop wrote:
| I always think of "bro" as being a hyper version of
| "dude." It's generically applied to any random person,
| but it's also exclusively male. So using it implies "this
| ingroup is assumed to be 100% male."
|
| On the other hand, I've seen and heard "dude" and "guy"
| used by and applied to women by other women. Not common
| but it happens. But I've never heard "bro" used that way.
| vasco wrote:
| Expression of endearment in this case.
| pyb wrote:
| "... Since I am new to the research community, I made an effort"
| This means they've probably submitted a paper too.
| epolanski wrote:
| It states it's a research, not a product company.
| diggan wrote:
| To be fair, OpenAI is also a "research lab" rather than
| "product company" and they still sell products for
| $200/month, not sure the distinction matters in practice much
| today as long as the entity is incorporated somehow.
| pyb wrote:
| That's what I said
| xiphias2 wrote:
| A lot of the problems John mentioned (camera jpeg, latency, real
| time decisions) have been worked on by comma.ai for many years.
| He could have just used their stack and build on it the general
| learning parts that comma is not focusing on.
| Flemlo wrote:
| And plenty of other people.
|
| It's still a lot better to really learn and discover it
| yourself to really get it.
|
| Also it's hard to determine how much time someone spent on
| particular topic.
| prosunpraiser wrote:
| Reuse is not always necessary - sometimes things are just done
| for fun and exploration, not for appeasing thirsty VCs and
| grabbing that market share.
| blitzar wrote:
| Reinventing the exact same thing and shouting from the
| rooftops about it is exactly how you appease thirsty VCs and
| grab that market share.
| CPLX wrote:
| Might I interest you in my new startup, that is a bus, but
| with technology?
| cmpxchg8b wrote:
| Why would John Carmack who is so rich that he does things
| for shits, giggles and personal development, give a hoot
| what a VC cares about?
| Cthulhu_ wrote:
| This is Carmack, who builds new things. He built one of the
| first 3D game engines based on the hard math.
| WatchDog wrote:
| Carmack himself has done a lot of work on end to end latency,
| during his time at oculus.
| aatd86 wrote:
| My own little insights and ramblings as an uninitiated quack
| (just spent the night asking Claude to explain machine learning
| to me):
|
| seems that we are learning in layers, one of the first layers
| being 2D neural net (images) augmented by other sensory data to
| create a 3D if not 4D model (neural net). HRTFs for sound
| increases the spatial data we get from images. With depth coming
| from sound and light and learnt movements(touch) we seem to
| develop a notion of space and time. (multimodality?)
|
| Seems that we can take low dimensional inputs and correlate them
| to form higher dimensional structures.
|
| Of course, physically it comes from noticing the dampening of
| visual data (in focus for example) and memorized audio data
| (sound frequency and amplitude, early reflections, doppler effect
| etc). That should be emergent from training.
|
| Those data sources can be inperfectly correlated. That's why we
| count during a lightning storm to evaluate distance. It's low
| dimensional.
|
| In a sense, it's a measure of required effort perhaps (distance
| to somewhere).
|
| What's funny is that it seems to go the other way from
| traditional training where we move from higher dimensional tensor
| spaces to lower ones. At least in a first step.
| Flamentono2 wrote:
| Its hard to follow what you try to commounicate at least the
| last half.
|
| Nonetheless, yes we do know certain brain structures like your
| image net analogy but the way you describe it, sounds a little
| bit of.
|
| Our virtual cortex is not 'just a layer' its a component i
| would say and its optimized of detecting things.
|
| Other components act differently with different structures.
| epr wrote:
| A bit confusing for sure, but I think (not sure) I get what
| they're saying. Training a nn (for visual tasks at least)
| consists of training a model with much more dimensions
| (params) than the input space (eg: controller inputs + atari
| pixels). This contrasts with a lot of what humans do, which
| is take higher dimensional information (tons of data per
| second combining visual, audio, touch/vibration, etc) and
| synthesizing much lower dimensional models / heuristics /
| rules of thumb, like the example they give of the 5 second
| per mile rule for thunder.
| abc-1 wrote:
| I was really hoping to see something cool. Such a great team of
| smart people, but this is just John reverting to what he's
| comfortable with and going off on some nonsense tangent. Hardware
| and video games. I doubt this is going to yield anything
| interesting at all.
| johnb231 wrote:
| Games are heavily used in RL research.
| abc-1 wrote:
| I understand that. Doing games in real time is just a
| performance problem that can be solved with more compute or
| inane optimizations. It's not interesting research.
| johnb231 wrote:
| I think you are trivializing the field of RL research.
| Games are not a solved problem. Doing that efficiently in
| real-time is even more difficult and is highly relevant to
| real world applications.
| abc-1 wrote:
| Right, we have not even solved games in the non-real time
| environment, so why bother adding additional constraints
| like "on real hardware in real time". This is exactly
| like Tesla trying to switch from LIDAR to cameras before
| self driving is even solved. It's avoiding the real
| harder challenge and going off on inane tangents. John is
| essentially bike shedding.
|
| In this case, John is going off on this inane tangent
| because of his prior experience with hardware and video
| games instead of challenging himself to solve the actual
| hard and open problems.
|
| I'm going to predict how this plays out for the
| inevitable screenshot in one to two years. John picks
| some existing RL algo and optimizes it to run in real
| time on real hardware. While he's doing this the field
| moves on to better and new algorithms and architectures.
| John finally achieves his goal and posts a vid of some
| (now ancient) RL algo playing some Atari game in real
| time. Everyone says "neat" and moves on. John gets to
| feel validated yet all his work is completely useless.
| johnb231 wrote:
| False dichotomy. It's not "avoiding the real harder
| challenge". It's solving a different problem and it is
| extremely relevant to real world applications. These are
| actual hard and open problems to be solved.
| criddell wrote:
| Here's a heuristic that somebody gave me a while ago: using
| the word "just" in the way you did is a signal that you
| don't understand the topic.
|
| John's document covers why he's doing what he's doing:
|
| > Fundamentally, I believe in the importance of learning
| from a stream of interactive experience, as humans and
| animals do, which is quite different from the throw-
| everything-in-a-blender approach of pretraining an LLM. The
| blender approach can still be world-changingly valuable,
| but there are plenty of people advancing the state of the
| art there.
|
| He thinks interacting with the real world and learning as
| you go isn't getting enough attention and might take us
| farther than the LLM approach. So he's applying these ideas
| to a subject that he's an expert in. You don't seem to find
| this approach interesting but John does (and I do too, for
| the record).
|
| Everybody dismissing him might be right. Those keeping
| score know that Carmack's batting average isn't one
| thousand. But those people also know Carmack has the
| resources to work on pretty much whatever he wants to work
| on. I'm happy he's still working hard at something and
| sharing his work.
| riehwvfbk wrote:
| And this is different from the argument that's being
| pooh-poed and downvoted how? You are effectively saying
| "Carmack is smart and is working on cool stuff that'll
| most likely be useless" in different words.
| criddell wrote:
| Not sure what gave you that impression. It's definitely
| not what I was trying to say.
| riehwvfbk wrote:
| > Everybody dismissing him might be right. Those keeping
| score know that Carmack's batting average isn't one
| thousand. But those people also know Carmack has the
| resources to work on pretty much whatever he wants to
| work on.
|
| To me this reads as "this is very far-fetched but he's
| got the money so golly for him"
| dylan604 wrote:
| for systems that learn in real-time, is there a way for
| the humans to know/understand how/why the system came to
| the conclusion it has? there are examples of where humans
| ran experiments that came to a conclusion for the wrong
| reasons. if an AI system thinks it knows the answer for
| the wrong reason, wouldn't that then poison its reasoning
| later as well? can an AI system learn its reasoning is
| wrong and then update it when provided better evidence?
| that seems to be something a vast majority of humans
| cannot do.
| abc-1 wrote:
| Oh darn I used the word just now everything I said is
| invalidated! That's too bad!
|
| FWIW, I'm aware of that heuristic too which is why I
| intentionally use the word "just" as a meta heuristic to
| filter a certain type of person.
| riehwvfbk wrote:
| Agreed, HN has been taken over by very pedantic weirdo
| cultists. The people they designate as holy can never be
| criticized or questioned.
| rfrey wrote:
| From chess to Go to Atari games, AI research has always been
| about games. You are unintentionally positioning yourself as
| more sophisticated than a huge number of AI luminaries,
| including Nobel and Turing winners.
| abc-1 wrote:
| Feel free to actually respond to what I'm discussing. There
| is little to no value in doing real time video games using a
| physical input device at this stage in AI.
| shermantanktop wrote:
| Maybe, but many advances in science have come from
| individuals moving backward from the popular cutting edge
| and branching off in a new direction.
| johnb231 wrote:
| There was no substance to your argument. Here's Carmack's
| response to a serious criticism from an OpenAI researcher
| https://x.com/ID_AA_Carmack/status/1925973500327591979
| brador wrote:
| I feel top level AI creation is beyond his skill set.
|
| He's a AAA software engineer but the prerequisites to build out
| cutting edge AI require deep formal math that is beyond his
| education and years at this point.
|
| Nothing to stop him playing around with AI models though.
| novosel wrote:
| There is no deep formal math in AI. It is a game of numbers.
|
| All deep formal math is a boundary to a thing.
| johnb231 wrote:
| The formal math takes a few months to learn. He is more than
| smart enough to figure that out.
| kriro wrote:
| I think you overestimate the level of math required in AI and
| at the same time I think you underestimate the math skills of
| John. AI runs on GPUs, Quake 2 engine was one of the first to
| optimized for GPUs (OpenGL).
|
| I'm pretty excited to see him in this domain. I think he'll
| focus on some DeepSeek style improvements.
| horsellama wrote:
| this.
|
| Having JC focusing on, say, writing a performant OSS CUDA
| replacement could be bigger than any of the last 20
| announcements from openai/goggle/deepmind/etc
| lyu07282 wrote:
| This feels like an understatement. At the time, young me had
| the impression Carmack came first, then the industry created
| 3dfx/OpenGL to run his games better. I still have nothing but
| respect for his skills decades later.
| cmpxchg8b wrote:
| GLQuake was released 11 months before Quake 2.
| Cthulhu_ wrote:
| What do you mean "beyond his skill set"? He effectively
| invented 3D gaming, which led to major leaps and investments
| into graphics cards which are now used for cryptocurrency and
| AI. He also did significant contributions into VR.
|
| He's probably one of the most qualified people around.
| jimbohn wrote:
| The math around machine learning is very manageable, and a lot
| of research in that area is throwing heuristics at a wall to
| see what sticks
| threeseed wrote:
| John Carmack:
|
| So I asked Ilya, their chief scientist, for a reading list.
| This is my path, my way of doing things: give me a stack of all
| the stuff I need to know to actually be relevant in this space.
|
| And he gave me a list of like 40 research papers and said, 'If
| you really learn all of these, you'll know 90% of what matters
| today! And I did. I plowed through all those things and it all
| started sorting out in my head.
| foldr wrote:
| What this misses is that research is a competitive endeavor.
| To succeed as a researcher you don't just need to know the
| bare minimum required to do research in your field. You need
| to be able to do it better than most of the people you're
| competing against. I know that HN as a collective has near-
| unlimited faith in Carmack's abilities (and he is no doubt
| Very Smart). But he's competing with other Very Smart people
| who have decades more experience of AI research.
|
| To put it another way, the idea that John Carmack is going to
| do groundbreaking research in AI is roughly as plausible as
| the idea that Yann LeCun is going to make a successful AAA
| video game. Stranger things have happened, but I won't be
| holding my breath.
| secondcoming wrote:
| Why does it need to be competitive? Maybe the guy has
| enough money to let him do whatever he wants regardless of
| the outcome and he chose AI because it's interesting to
| him.
| foldr wrote:
| In research you have to succeed before your competitors.
| It's not research if it's already been done.
| RetroTechie wrote:
| You're forgetting that a whole string of breakthroughs are
| all fairly recent (like in the last decade). _Everyone_ ,
| including the pro's, is treading new ground.
|
| In that context anyone can make progress in the field, as
| long as they understand what they're dealing with.
|
| Better regard mr. Carmack as an X factor. Maybe the experts
| will leave him in the dust. Or maybe he'll come up with
| something that none of the experts cared to look into.
| foldr wrote:
| Lots of scientific fields have seen breakthroughs in the
| past decade. Doesn't mean that any random smart person
| can jump in and start doing groundbreaking research.
| Jensson wrote:
| But a random smart person will jump in and make
| groundbreaking research.
| sergiotapia wrote:
| The difference is Carmack is literally a T-shaped dude --
| hell, he's a T-shaped dude with lots of vertical lines :P
|
| I believe all his in-depth experience in other areas will
| heavily unlock him to bring about another breakthrough.
| He's that good.
| abraxas wrote:
| I'd love to have a copy of that list. Just to see how much
| I've yet to absorb.
| akomtu wrote:
| AI creation is more like Alchemy than science, and
| breakthroughs come not from math background, but from intuition
| and a bit of math skills. Transformers behind the chatbots
| isn't a rocket science and were discovered almost by accident.
| The next breakthrough will come a similar way. I'd frankly bet
| on someone like Carmack than on some theoretical researcher who
| is churning out papers.
| MrScruff wrote:
| It's always a treat to watch a Carmack lecture or read anything
| he writes, and his notes here are no exception. He writes as an
| engineer, for engineers and documents all his thought processes
| and misteps in the exact detailed yet concise way you'd want a
| colleague to who was handing off some work.
|
| One question I would have about the research direction is the
| emphasis on realtime. If I understand correctly he's doing online
| learning in realtime. Obviously makes for a cool demo and pulls
| on his optimisation background, and no doubt some great
| innovations will be required to make this work. But I guess the
| bitter lesson and recent history also tell us that some solutions
| may only emerge at compute levels beyond what is currently
| possible for realtime inference let alone learning. And the only
| example we have of entities solving Atari games is the human
| brain, of which we don't have a clear understanding of the
| compute capacity. In which case, why wouldn't it be better to
| focus purely on learning efficiency and relax the realtime
| requirement for now?
|
| That's a genuine question by the way, definitely not an expert
| here and I'm sure there's a bunch of value to working within
| these constraints. I mean, jumping spiders solve reasonably
| complex problems with 100k neurons, so who knows.
| suddenlybananas wrote:
| It's because humans (and other animals) have enormous innate
| capacities and knowledge which makes learning new things much
| much simpler than if you start from scratch. It's not really
| because of human's computational capacity.
| MrScruff wrote:
| By innate do you mean evolved/instinctive? Surely even
| evolved behaviour must be expressed as brain function, and
| therefore would need a brain capable of handling that level
| of processing.
|
| I don't think it's clear how much of a human brains function
| exists at birth though, I know it's theorised than even much
| of the sensory processing has to be learned.
| suddenlybananas wrote:
| I'm not arguing against computational theory of mind, I'm
| just saying that innate behaviours don't require the same
| level of scale as learnt ones.
|
| Existing at birth is not the same thing as innate. Puberty
| is innate but it is not present at birth.
| MrScruff wrote:
| That's an interesting point. I can see that, as you say
| puberty and hormones impact brain function and hence
| behaviour, and those are inate and not learned. But at
| least superfically that would appear to be primarily
| broad behavioural effects, similar to what might be
| induced by medication. Rather than something that impacts
| pure abstract problem solving, which I guess is what the
| Atari games are supposed to represent?
| rafaelmn wrote:
| This is obviously wrong from genetic defects that cause
| predictable development problems in specialized areas.
| They are innate but not present at birth.
| xnx wrote:
| > enormous innate capacities and knowledge
|
| Hundreds of millions of years of trial-and-error biological
| pre-training where survival/propagation is the reward
| function
| Nopoint2 wrote:
| There is just no reason to believe that we are born with some
| insanely big library of knowledge, and it sounds completely
| impossible. How would it be stored, and how would we even
| evolve it?
|
| It just isn't needed. Just like you can find let's say
| kangaroos in the latent space of an image generator, so we
| learn abstract concepts and principles of how things work as
| a bonus of learning to process the senses.
|
| Maybe a way to AGI could be figuring out how to combine a
| video generator with a LLM or something similar in a way that
| allows it to understand things intuitively, instead of doing
| just lots and lots of some statistical bullsit.
| Jensson wrote:
| > There is just no reason to believe that we are born with
| some insanely big library of knowledge, and it sounds
| completely impossible. How would it be stored, and how
| would we even evolve it?
|
| We do have that, ever felt fear of heights? That isn't
| learned, we are born with it. Same with fear of small
| moving objects like spiders or snakes.
|
| Such things are learned/stored very different from
| memories, but its certainly there and we can see animals
| also have those. Like cats gets very scared of objects that
| are long and appear suddenly, like a cucumber, since their
| genetic instincts thinks its a snake.
| Nopoint2 wrote:
| Of course it is learned, and fear is triggered by
| anything unfamiliar, that causes a high reconstruction
| error. Because it means you don't understand it, and it
| could be dangerous. We are just not used to encoding
| anything so deep below the eye level, and it freaks us
| out.
| Jensson wrote:
| Do you really think every single ant is learning all that
| on its own? And if ants can store that in their DNA, why
| don't you think other animals can? DNA works just fine as
| generic information storage, there are obviously a ton of
| behaviors and information encoded there from hundreds of
| millions of years of survival of the fittest.
| throwup238 wrote:
| _> Like cats gets very scared of objects that are long
| and appear suddenly, like a cucumber, since their genetic
| instincts thinks its a snake._
|
| After having raised four dozen kittens that a couple of
| feral sisters gave birth to in my garage, I'm certain
| that is nonsense. It's an internet meme that became urban
| legend.
|
| I don't think they have ever even reacted to a cucumber,
| and I have run many experiments because my childhood cat
| loved cucumbers (we'd have to guard the basket of
| cucumbers after harvest, otherwise she'd bite every
| single one of them... just once).
| johnb231 wrote:
| From the notes:
|
| "A reality check for people that think full embodied AGI is
| right around the corner is to ask your dancing humanoid robot
| to pick up a joystick and learn how to play an obscure video
| game."
| throw_nbvc1234 wrote:
| This sounds like a problem that could be solved around the
| corner with a caveat.
|
| Games generally are solvable for AI because they have
| feedback loops and a clear success or failure criteria. If
| the "picking up a Joystick" part is the limiting factor,
| sure. But why would we want robots to use an interface
| (especially a modern controller) heavily optimized for human
| hands; that seems like the definition of a horseless
| carriage.
|
| I'm sure if you compared a monkey and a dolphins performance
| using a joystick you'd get results that aren't really
| correlated with their intelligence. I would guess that if you
| gave robots an R2D2 like port to jack into and play a game,
| that problem could be solved relatively quickly.
| mellosouls wrote:
| The point isn't about learning video games its about
| learning tasks unrelated to its specific competency
| generally.
| johnb231 wrote:
| No, the joystick part is really not the limiting factor.
| They've already done this with a direct software interface.
| Physical interface is a new challenge. But overall you are
| missing the point.
| xnickb wrote:
| Just like OpenAI early on promised us an AGI and showed us
| how it "solved" Dota 2.
|
| They also claimed it "learned" to play by playing itself
| only however it was clear that most of the advanced
| techniques were borrowed from existing AI and by observing
| humans.
|
| No surprise they gave up on that project completely and I
| doubt they'll ever engage in anything like that again.
|
| Money better spent on different marketing platforms.
| jsheard wrote:
| It also wasn't even _remotely_ close to learning Dota 2
| proper. They ran a massively simplified version of the
| game where the AI and humans alternated between playing
| one of two pre-defined team compositions, meaning >90%
| of the games characters and >99.999999% of the possible
| compositions and matchups weren't even on the table, plus
| other standard mechanics were also changed or disabled
| altogether for the sake of the AI team.
|
| Saying you've solved Dota after stripping out nearly all
| of its complexity is like saying you've solved Chess, but
| on a version where the back row is all Bishops.
| xnickb wrote:
| Exactly. What I find surprising in this story though is
| not the OpenAI. It's investors not seeing through these
| blatant.. lets call them exaggerations of the reality and
| still trusting the company with their money. I know I
| wouldn't have. But then again, maybe that's why I'm poor.
| ryandrake wrote:
| In their hearts, startup investors are like Agent Mulder:
| they Want To Believe. Especially after they've already
| invested a little. They are willing to overlook obvious
| exaggerations up to and including fraud, because the
| alternative is admitting their judgment is not sound.
|
| Look at how long Theranos went on! Miraculous product.
| Attractive young founder with all the right pedigree,
| credentials, and contacts, dressed in black trurtlenecks.
| Hell, she even talked like Steve Jobs! Investors never
| had a chance.
| jdross wrote:
| They already have 400 million daily users and a billion
| people using the product, with billions of consumer
| subscription revenue, faster than any company ever. They
| are also aggregating R&D talent at a density never before
| seen in Silicon Valley
|
| That is what investors see. You seem to treat this as a
| purity contest where you define purity
| xnickb wrote:
| I'm speaking about past events. Perhaps I didn't make it
| clear enough
| zaphar wrote:
| Also apparently still not making a profit.
| scotty79 wrote:
| It was 6 years ago. I'm sure now there'd be no contest
| now if OpenAI dedicated resources to it, which it won't
| because it's busy with solving entirety of human language
| before others eat their lunch.
| xnickb wrote:
| What do you base your certainty on? Were there any
| significant enough breakthroughs in the AGI?
| scotty79 wrote:
| ARC-AGI, while imagined as super hard for AI, was beaten
| enough that they had to come up with ARC-AGI-2.
| hbsbsbsndk wrote:
| "AI tend to be brittle and optimized for specific tasks,
| so we made a new specific task and then someone optimized
| for it" isn't some kind of gotcha. Once ARC puzzles
| became a benchmark they ceased to be meaningful WRT
| "AGI".
| scotty79 wrote:
| So if DOTA became a benchmark same way Chess or Go became
| earlier it would be promptly beaten. It just didn't stick
| before people moved to more useful "games".
| spektral23 wrote:
| Funnily enough, even dota2 has grown much more complex
| than it was 6 years ago, so it's a harder problem to
| solve today than it was back then
| rowanG077 wrote:
| I agree that restricting the hero pool is a huge
| simplification. But they did play full 5v5 standard dota
| with just a restricted hero pool of 17 heroes and no
| illusions/control units according to theverge
| (https://www.theverge.com/2019/4/13/18309459/openai-five-
| dota...). It destroyed the professionals.
|
| As an ex dota player, I don't think this is that far off
| from having full on, all heroes dota. Certainly not as
| far of as you are making it sound.
|
| And dota is one of the most complex games, I expect for
| example that an AI would instantly solve CS since aim is
| such a large part of the game.
| mistercheph wrote:
| Another issue with the approach is that the model had
| direct access to game data, that is simply an unfair
| competitive advantage in dota, and it is obvious why that
| advantage would be unfair in CS.
|
| It is certainly possible, but i won't be impressed by
| anything "playing CS" that isn't running a vision model
| on a display and moving a mouse, because that _is_ the
| game. The game is not abstractly reacting to enemy
| positions and relocating the cursor, it 's looking at a
| screen, seeing where the baddy is and then using this
| interface (the mouse) to get the cursor there as quickly
| as possible.
|
| It would be like letting an AI plot its position on the
| field and what action its taking during a football match
| and then saying "Look, The AI would have scored dozens of
| times in this simulation, it is the greatest soccer
| player in the world!" No, sorry, the game actually
| requires you to locomote, abstractly describing your
| position may be fun but it's not the game
| rowanG077 wrote:
| Did you read the paper? It had access to the dota 2 bot
| API, which is some gamestate but very far from all
| gamestate. It also had artifially limited reaction to
| something like 220ms, worse then professional gamers.
|
| But then again, that is precisely the point. A chess bot
| also has access to gigabytes of perfect working memory. I
| don't see people complaining about that. It's perfectly
| valid to judge the best an AI can do vs the best a human
| can do. It's not really fair to take away exactly what a
| computer is good at from an AI and then say: "Look but
| the AI is now worse". Else you would also have to do it
| the other way around. How well could a human play dota if
| it only had access to the bot API. I don't think they
| would do well at all.
| lukeschlather wrote:
| > But then again, that is precisely the point. A chess
| bot also has access to gigabytes of perfect working
| memory. I don't see people complaining about that.
|
| There are ~86 billion neurons in the human brain. If we
| assume each neuron stores a single bit a human also has
| access to gigabytes of working memory. If we assume each
| synapse is a bit that's terabytes. Petabytes is not
| unreasonable assuming 1kb of storage per synapse. (And
| more than 1kb is also not unreasonable.)
|
| The whole point of the exercise is figuring out how much
| memory compares to a human brain.
| Jensson wrote:
| > It destroyed the professionals.
|
| Only the first time, later when it played better players
| it always lost. Players learned the faults of the AI
| after some time in game and the AI had very bad late game
| so they always won later.
| rowanG077 wrote:
| Not on the last iteration.
| fennecfoxy wrote:
| To be fair humans have had quite a few million years
| across a growing population to gather all of the
| knowledge that we have.
|
| As we're learning with LLMs, the dataset is what matters
| - and what's awesome is that you can see that in us, as
| well! I've read that our evolution is comparatively slow
| to the rate of knowledge accumulation in the information
| age - and that what this means is that you can
| essentially take a caveman, raise them in our modern
| environment and they'll be just as intelligent as the
| average human today.
|
| But the core of our intelligence is logic/problem
| solving. We just have to solve higher order problems
| today, like figuring out how to make that chart in excel
| do the thing you want, but in days past it was figuring
| out how to keep the fire lit when it's raining. When you
| look at it, we've possessed the very core of that problem
| solving ability for quite a while now. I think that is
| the key to why we are human, and our close ancestors
| monkeys are...still just monkeys.
|
| It's that problem solving ability that we need to figure
| out how to produce within ML models, then we'll be
| cooking with gas!
| jappgar wrote:
| A human would learn it faster, and could immediately teach
| other humans.
|
| AI clearly isn't at human level and it's OK to admit it.
| jandrese wrote:
| > But why would we want robots to use an interface
| (especially a modern controller) heavily optimized for
| human hands; that seems like the definition of a horseless
| carriage.
|
| Elon's response to this is that if we want these androids
| to replace human jobs then the lowest friction alternative
| is for the android to be able to do anything a human can do
| in a human amount of space. A specialized machine is faster
| and more efficient, but comes with engineering and
| integration costs that create a barrier to entry. Elon
| learned this lesson the hard way when he was building out
| the gigafactories and ended up having to hire a lot of
| people to do the work while they sorted out the issues with
| the robots. To someone like Elon a payroll is an ever
| growing parasite on a companies bottom line, far better if
| the entire thing is automated.
| ferguess_k wrote:
| We don't really need AGI. We need better specialized AIs.
| Throw in a few specialized AIs and they will leave some
| impact in the society. That might not be that far away.
| bluGill wrote:
| Specialized AIs have been making an impact on society since
| at least the 1960s. AI has long suffered from every time
| they come up with something new it gets renamed and becomes
| important (where it makes sense) without giving AI credit.
|
| From what I can tell most in AI are currently hoping LLMs
| reach that point quick just because the hype is not helping
| AI at all.
| ferguess_k wrote:
| Yeah I agree with it. There is a lot of hype, but there
| is some potentials there.
| BolexNOLA wrote:
| Yeah "AI" tools (such a loose term but largely
| applicable) have been involved in audio production for a
| very long time. They have actually made huge strides with
| noise removal/voice isolation, auto
| transcription/captioning, and "enhancement" in the last
| five years in particular.
|
| I hate Adobe, I don't like to give them credit for
| anything. But their audio enhance tool is actual sorcery.
| Every competitor isn't even close. You can take garbage
| zoom audio and make it sound like it was borderline
| recorded in a treated room/studio. I've been in
| production for almost 15 years and it would take me half
| a day or more of tweaking a voice track with multiple
| tools that cost me hundreds of dollars to get it 50% as
| good as what they accomplish in a minute with the click
| of a button.
| danielbln wrote:
| Bitter lesson applies here as well though. Generalized
| models will beat specialized models given enough time and
| compute. How much bespoke NLP is there anymore?
| Generalized foundational models will subsume all of it
| eventually.
| ses1984 wrote:
| Generalized models might be better but they are rarely
| more efficient.
| johnecheck wrote:
| You misunderstand the bitter lesson.
|
| It's not about specialized vs generalized models - it's
| about how models are trained. The chess engine that beat
| Kasparov is a specialized model (it only plays chess),
| yet it's the bitter lesson's example for the smarter way
| to do AI.
|
| Chess engines are better at chess than LLMs. It's not
| close. Perhaps eventually a superintelligence will
| surpass the engines, but that's far from assured.
|
| Specialized AI are hardly obsolete and may never be. This
| hypothetical superintelligence may even decide not to
| waste resources trying to surpass the chess AI and
| instead use it as a tool.
| CrimsonCape wrote:
| I think your point that AI would refuse to play chess is
| interesting. To humans, chess is a strategic game. To a
| mathematician, chess is an exceedingly hard game, (pretty
| sure it is EXP complete, but I'm not fully familiar with
| Np/Exp completeness). To an AI, it seems like the AI will
| side with the mathematicians. AI is like "bro you can't
| even figure out if P=NP so how am I going to, you want me
| to waste power to solve an unsolvable problem?"
|
| From Wikipedia, Garry Kasparov said it was a pleasure to
| watch AlphaZero play, especially since "its style was
| open and dynamic like his own".
|
| People can't define AI because they don't want to
| consider AI as a subset of exponentially difficult
| algorithms, but they _do_ want to consider AI as a
| generator of stylistic responses.
| Workaccount2 wrote:
| Yesterday my dad, in his late 70's, used Gemini with a
| video stream to program the thermostat. He then called me
| to tell me this, rather then call me to come stop by and
| program the thermostat.
|
| You can call this hype, maybe it is all hype until LLMs
| can work on 10M LOC codebases, but recognize that LLMs
| are a shift that is totally incomparable to any previous
| AI advancement.
| orochimaaru wrote:
| That is what open ai's non-profit economic research arm
| has claimed. LLMs will fundamentally change how we
| interact with the world like the Internet did. It will
| take time like the Internet and a couple of hype cycle
| pops but it will change the way we do things.
|
| It will help a single human do more in a white collar
| world.
|
| https://arxiv.org/abs/2303.10130
| bluGill wrote:
| There are clearly a lot of useful things about LLMs.
| However there is a lot of hype as well. It will take time
| to separate the two.
| bluefirebrand wrote:
| > He then called me to tell me this, rather then call me
| to come stop by and program the thermostat.
|
| Sounds like AI robbed you of an opportunity to spend some
| time with your Dad, to me
| jabits wrote:
| Or maybe instead of spending time with your dad on a bs
| menial task, you could spent time fishing with him...
| bluefirebrand wrote:
| It's nice to think that but life and relationships are
| also composed of the little moments, which sometimes
| happen when someone asks you over to help with a "bs
| menial task"
|
| It takes five minutes to program the thermostat, then you
| can have a beer on the patio if that's your speed and
| catch up for a bit
|
| Life is little moments, not always the big commitments
| like taking a day to go fishing
|
| That's the point of automating all of ourselves out of
| work, right? So we have more time to enjoy spending time
| with the people we love?
|
| So isn't it kind of sad if we wind up automating those
| moments out of our lives instead?
| TheGRS wrote:
| For some of us that's a plus!
| Workaccount2 wrote:
| I'm there like twice a week don't worry. He knows about
| Gemini because I was showing him it two days before hah
| ferguess_k wrote:
| Yeah. As a mediocre programmer I'm really scared about
| this. I don't think we are very far from AI replacing the
| mediocre programmers. Maybe a decade, at most.
|
| I'd definitely like to improve my skills, but to be
| realistic, most of the programmers are not top-notch.
| lexandstuff wrote:
| That is amazing. But I had a similar experience when I
| first taught my mum how to Google for computer problems.
| She called me up with delight to tell me how she fixed
| the printer problem herself, thanks to a Google search.
| In a way, LLMs are a refinement on search technology we
| already had.
| nightski wrote:
| Saying we don't "need" AGI is like saying we don't need
| electricity. Sure life existed before we had that
| capability, but it would be very transformative. Of course
| we can make specialized tools in the mean time.
| charcircuit wrote:
| Can you give an example how it would be transformative
| compared to specialized AI?
| Jensson wrote:
| AGI is transformative in that it lets us replace
| knowledge workers completely, specialized AI requires
| knowledge workers to train them for new tasks while AGI
| doesn't.
| fennecfoxy wrote:
| Because it could very well exceed our capabilities beyond
| our wildest imaginations.
|
| Because we evolved to get where we are, humans have all
| sorts of messy behaviours that aren't really compatible
| with a utopian society. Theft, violence, crime, greed -
| it's all completely unnecessary and yet most of us can't
| bring ourselves to solve these problems. And plenty are
| happy to live apathetically while billionaires become
| trillionaires...for what exactly? There's a whole
| industry of hyper-luxury goods now, because they make so
| much money even regular luxury is too cheap.
|
| If we can produce AGI that exceeds the capabilities of
| our species, then my hope is that rather than the typical
| outcome of "they kill us all", that they will simply keep
| us in line. They will babysit us. They will force us all
| to get along, to ensure that we treat each other fairly.
|
| As a parent teaches children to share by forcing them to
| break the cookie in half, perhaps AI will do the same for
| us.
| davidivadavid wrote:
| Oh great, can't wait for our AI overlords to control us
| more! That's definitely compatible with a "utopian
| society"*.
|
| Funnily enough, I still think some of the most
| interesting semi-recent writing on utopia was done ~15
| years ago by... Eliezer Yudkowsky. You might be
| interested in the article on "Amputation of Destiny."
|
| Link:
| https://www.lesswrong.com/posts/K4aGvLnHvYgX9pZHS/the-
| fun-th...
| rurp wrote:
| Who on earth has the resources to create true AGI and is
| interested in using it to create this sort of utopia for
| the masses?
|
| If AGI is created it is most likely to be guided by
| someone like Altman or Musk, people whose interests
| couldn't be farther from what you describe. They want to
| make _themselves_ gods and couldn 't care less about
| random plebs.
|
| If AGI is setting its own principles then I fail to see
| why it would care about us at all. Maybe we'll be amusing
| as pets but I expect a superhuman intelligence will treat
| us like we treat ants.
| tirant wrote:
| I still don't see an issue of billionaires becoming
| trillionaires and being able to buy hyper luxury goods.
| Good for them and good for the people selling and
| manufacturing those goods. Meanwhile poverty is in all
| time lows and there's a growing middle class at global
| level. Our middle class life conditions nowadays have a
| level of comfort that would get Kings from some centuries
| ago jealous.
| hackinthebochs wrote:
| Why on earth would you want an AI that takes away our
| autonomy? It's wild to see someone actually advocate for
| this outcome.
| johnb231 wrote:
| There are people who enjoy being dominated, kept on a
| leash like a dog. Bad idea to transfer that fetish to
| human civilization.
|
| ASI to humans would be like humans are to rats or ants.
|
| It could stomp all over us to achieve whatever goals it
| chooses to accomplish.
|
| Humans being cared for as pets would be a relatively
| benign outcome.
| brulard wrote:
| Is this meant seriously? Do we really want something more
| intelligent than us to just force on us it's rules, logic
| and ways of living (or dying), which we may be too stupid
| to understand?
| hoosieree wrote:
| The error in this argument is that _electricity is real_.
| mrandish wrote:
| Indeed, and I'd go even further. In addition to existing,
| electricity is also usefully defined - which helps
| greatly in establishing its existence. Neither unicorns
| nor AGI currently exist but at least unicorns are well
| enough defined to establish whether an equine animal is
| or isn't one.
| esafak wrote:
| Furthermore, we will be faced with it whether we want to
| or not, because others are making it happen.
| alickz wrote:
| What if AGI is just a bunch of specialized AIs put
| together?
|
| It would seem our own generalized intelligence is an
| emergent property of many, _many_ specialized processes
|
| I wonder if AI is the same
| Jensson wrote:
| > It would seem our own generalized intelligence is an
| emergent property of many, _many_ specialized processes
|
| You can say that about other animals, but about humans it
| is not so sure. No animal can be taught as general set of
| skills as a human can, they might have some better
| specialized skills but clearly there is something special
| that makes humans so much more versatile.
|
| So it seems there was this simple little thing humans got
| that makes them general, while for example our very close
| relatives the monkeys are not.
| fennecfoxy wrote:
| Humans are the ceiling at the moment yes, but that
| doesn't mean the ceiling isn't higher.
|
| Science is full of theories that are correct per our
| current knowledge and then subsequently disproven when
| research/methods/etc improves.
|
| Humans aren't special, we are made from blood & bone, not
| magic. We will eventually build AGI if we keep at it.
| However unlike VCs with no real skills except having a
| lot of money(tm), I couldn't say whether this is gonna
| happen in 2 years or 2000.
| Jensson wrote:
| Question was if cobbling together enough special
| intelligence creates general intelligence. Monkeys has a
| lot of special intelligence that our current AI models
| can't come close to, but still aren't seen as general
| intelligence like humans, so there is some little bit
| humans has that isn't just another special intelligence.
| mike_ivanov wrote:
| It may be a property of (not only of?) humans that we can
| generate specialized inner processes. The hardcoded ones
| stay, the emergent ones come and go. Intelligence itself
| might be the ability to breed new specialized mental
| processes on demand.
| babyent wrote:
| Why not just hire like 100 of the smartest people across
| domains and give them SOTA AI, to keep the AI as accurate
| as possible?
|
| Each of those 100 can hire teams or colleagues to make
| their domain better, so there's always human expertise
| keeping the model updated.
| trial3 wrote:
| "just"
| babyent wrote:
| They're spending 10s of billions. Yes, just.
|
| 200 million to have dedicated top experts on hand is
| reasonable.
| Karrot_Kream wrote:
| I think to many AI enthusiasts, we're already at the
| "specialized AIs" phase. The question is whether those will
| jump to AGI. I'm personally unconvinced but I'm not an ML
| researcher so my opinion is colored by what I use and what
| I read, not active research. I do think though that many
| specialized AIs is already enough to experience massive
| economic disruption.
| AndrewKemendo wrote:
| This debate is exhausting because there's no coherent
| definition of AGI that people agree on.
|
| I made a google form question for collecting AGI definitions
| cause I don't see anyone else doing it and I find it
| infinitely frustrating the range of definitions for this
| concept:
|
| https://docs.google.com/forms/d/e/1FAIpQLScDF5_CMSjHZDDexHkc.
| ..
|
| My concern is that people never get focused enough to care to
| define it - seems like the most likely case.
| mvkel wrote:
| It doesn't really seem like there's much utility in
| defining it. It's like defining "heaven."
|
| It's an ideal that some people believe in, and we're
| perpetually marching towards it
| theptip wrote:
| No, it's never going to be precise but it's important to
| have a good rough definition.
|
| Can we just use Morris et al and move on with our lives?
|
| Position: Levels of AGI for Operationalizing Progress on
| the Path to AGI: https://arxiv.org/html/2311.02462v4
|
| There are generational policy and societal shifts that
| need to be addressed somewhere around true Competent AGI
| (50% of knowledge work tasks automatable). Just like
| climate change, we need a shared lexicon to refer to this
| continuum. You can argue for different values of X but
| the crucial point is if X% of knowledge work is automated
| within a decade, then there are obvious risks we need to
| think about.
|
| So much of the discourse is stuck at "we will never get
| to X=99" when we could agree to disagree on that and move
| on to considering the x=25 case. Or predict our timelines
| for X and then actually be held accountable for our
| falsifiable predictions, instead of the current vide
| based discussions.
| bigyabai wrote:
| It is a marketing term. That's it. Trying to exhaustively
| define what AGI is or could be is like trying to explain
| what a Happy Meal is. At it's core, the Happy Meal was not
| invented to revolutionize food eating. It puts an
| attractive label on some mediocre food, a title that exists
| for the purpose of advertisement.
|
| There is no point collecting definitions for AGI, it was
| not conceived as a description for something novel or
| provably existent. It is "Happy Meal marketing" but aimed
| for adults.
| AndrewKemendo wrote:
| That's historically inaccurate
|
| My masters thesis advisor Ben Goertzel popularized the
| term and has been hosting the AGI conference since 2008:
|
| https://agi-conference.org/
|
| https://goertzel.org/agiri06/%5B1%5D%20Introduction_Nov15
| _PW...
|
| I had lunch with Yoshua Bengio at AGI 2014 and it was
| most of the conversation that day
| HarHarVeryFunny wrote:
| The name AGI (i.e. generalist AI) was originally intended
| to contrast with narrow AI which is only capable of one,
| or a few, specific narrow skills. A narrow AI might be
| able to play chess, or distinguish 20 breeds of dog, but
| wouldn't be able to play tic tac toe because it wasn't
| built for that. AGI would be able to learn to do
| anything, within reason.
|
| The term AGI is obviously used very loosely with little
| agreement to it's precise definition, but I think a lot
| of people take it to mean not only generality, but
| specifically human-level generality, and human-level
| ability to learn from experience and solve problems.
|
| A large part of the problem with AGI being poorly defined
| is that intelligence itself is poorly defined. Even if we
| choose to define AGI as meaning human-level intelligence,
| what does THAT mean? I think there is a simple
| reductionist definition of intelligence (as the word is
| used to refer to human/animal intelligence), but
| ultimately the meaning of words are derived from their
| usage, and the word "intelligence" is used in 100
| different ways ...
| mrandish wrote:
| > intended to contrast with narrow AI
|
| I've thought for a while that the middle letter in AGI
| ('General' vs 'Specific') would be more useful and
| helpful if it were changed to Wide vs Narrow. All AIs can
| be evaluated on a scale of narrow to wide in terms of
| their abilities and I don't think that will change
| anytime soon.
|
| Everyone understands that something is only wide or
| narrow in comparison to something else. While that's also
| true of the terms "general' and 'specific', those are
| less used that way in daily conversation these days. In
| science and tech we make distinctions about generalized
| vs specific but 'general' isn't a conversational term
| like 50 or 100 years ago. When I was a kid my
| grandparents would call the local supermarket, the
| 'general store' which I thought was an unusual usage even
| then.
| HarHarVeryFunny wrote:
| I guess "general store" made more sense back then though.
| I grew up in the UK in the 60's and food shops were
| "narrow" - fishmonger, butcher, greengrocer (fruit &
| veg), bakery, etc. From that perspective a "general
| store" would have been noteworthy!
| johnb231 wrote:
| Generalization is a formal concept in machine learning
| and is measurable.
| johnb231 wrote:
| The Wikipedia article on AGI explains it well enough.
|
| Researchers at Google have proposed a classification scheme
| with multiple levels of AGI. There are different opinions
| in the research community.
|
| https://arxiv.org/abs/2311.02462
| vonneumannstan wrote:
| Is this supposed to be a gotcha? We know these systems are
| typically trained using RL and they are exceedingly good at
| learning games...
| johnb231 wrote:
| No it is not a "gotcha" and I don't understand how you got
| that impression.
|
| Carmack believes AGI systems should be able to learn new
| tasks in realtime alongside humans in the real world.
| nlitened wrote:
| > the human brain, of which we don't have a clear understanding
| of the compute capacity
|
| Neurons have finite (very low) speed of signal transfer, so
| just by measuring cognitive reaction time we can deduce upper
| bounds on how many _consecutive_ neuron connections are
| involved in reception, cognitive processing, and resulting
| reaction via muscles, even for very complex cognitive
| processes. And the number is just around 100 consecutive
| neurons involved one after another. So "the algorithm" could
| not be _that_ complex in the end (100x matmul+tanh?)
|
| Granted, a lot of parallelism and feedback loops are involved,
| but overall it gives me (and many others) an impression that
| when the AGI algorithm is ever found, it's "mini" version
| should be able to run on modest 2025 hardware in real time.
| johnb231 wrote:
| > (100x matmul+tanh?)
|
| Biological neurons are way more complex than that. A single
| neuron has dentritic trees with subunits doing their own
| local computations. There are temporal dynamics in the firing
| sequences. There is so much more complexity in the biological
| networks. It's not comparable.
| woolion wrote:
| You could implement a Turing-machine with humans acting
| physically operating as logic gates. Then, every human is
| just a boolean function.
| Jensson wrote:
| Neurons are stateful though, it is core to their function
| and how they learn.
| neffy wrote:
| This is exactly it. Biology is making massive use of hacked
| real time local network communication in ways we haven't
| begun to explore.
| scajanus wrote:
| The granted is doing a lot of work there. In fact, if you
| imagine a computer being able to do similar tasks as human
| brain can in around 100 steps, it becomes clear that
| considering parallelism is absolutely critical.
| kilpikaarna wrote:
| I'm sure there were offline rendering and 3D graphics
| workstation people saying the same about the comparatively
| crude work he was doing in the early 90s...
|
| Obviously both Carmack and the rest of the world has changed
| since then, but it seems to me his main strength has always
| been in doing more with less (early id/Oculus, AA). When he's
| working in bigger orgs and/or with more established tech his
| output seems to suffer, at least in my view (possibly in his as
| well since he quit both Bethesda-id and Meta).
|
| I don't know Carmack and can't claim to be anywhere close to
| his level, but as someone also mainly interested in realtime
| stuff I can imagine he also feels a slight disdain for the
| throw-more-compute-at-it approach of the current AI boom. I'm
| certainly glad he's not running around asking for investor
| money to train an LLM.
|
| Best case scenario he teams up with some people who complement
| his skillset (akin to the game designers and artists at id back
| in the day) and comes up with a way to help bring some of the
| cutting edge to the masses, like with 3D graphics.
| LarsDu88 wrote:
| The thing about Carmack in the 90s... There was a lot of
| research going on around 3d graphics. Companies like SGI and
| Pixar were building specialized workstations for doing vector
| operations for 3d rendering. 3d was a thing. Game consoles
| with specialized 3d hardware would launch in 1994 with the
| Sega Saturn and the Sony Playstation (in Japan only for one
| year)
|
| What Carmack did was basically get a 3d game running on
| existing COMMODITY hardware. The 386 chip that most people
| used for their excel spreadsheets did not do floating point
| operations well, so Carmack figured out how to do everything
| using integers.
|
| May 1992 -> Wolfenstein 3d releases December 1993 -> Doom
| releases December 1994 -> Sony Playstation launches in Japan
| June 1996 -> Quake releases
|
| So Wolfenstein and Doom were actually not really 3d games,
| but rather 2.5 games (you can't have rooms below other
| rooms). The first 3d game here is actually Quake which also
| eventually also got hardware acceleration support.
|
| Carmack was the master of doing the seeminly impossible on
| super constrained hardware on virtually impossible timelines.
| If DOOM released in 1994 or 1995, would we still remember it
| in the same way?
| hx8 wrote:
| > If DOOM released in 1994 or 1995, would we still remember
| it in the same way?
|
| Maybe. One aspect of Wolfenstein and Doom's popularity is
| that it was years ahead of everyone else technically on PC
| hardware. The other aspect is that they were genre defining
| titles that set the standards for gameplay design. I think
| Doom Deathmatch would have caught on in 1995, as there
| really were very few (just Command and Conquer?) standout
| PC network multiplayer games released between 1993 and
| 1995.
| LarsDu88 wrote:
| I guess the thing about rapid change is... it's hard to
| imagine what kind of games would exist in a DOOMless
| world in an alternate 1995.
|
| The first 3d console games started to come out that year,
| like Rayman. Star Wars Dark Forces with its own custom 3d
| engine also came out. Of course Dark Forces was, however,
| an overt clone of DOOM.
|
| It's a bit ironic, but I think the gameplay innovation of
| DOOM tends to hold up more than the actual technical
| innovation. Things like BSP for level partitioning have
| slowly been phased out of game engines, we have ample
| floating point compute power and hardware acceleration
| ow, but even developers of the more recent DOOM games
| have started to realize that they should return to the
| original formula of "blast zombies in the face at high
| speed, and keep plot as window dressing"
| xh-dude wrote:
| Sort of in the middle, id games always felt tight. The
| engines were immersive not only because of graphics, but
| basic i/o was excellent.
| Narishma wrote:
| > The first 3d console games started to come out that
| year, like Rayman.
|
| Rayman was a 2D game.
| CamperBob2 wrote:
| _If DOOM released in 1994 or 1995, would we still remember
| it in the same way?_
|
| I think so, because the thing about DOOM is, it was an
| insanely good game. Yes, it pioneered fullscreen real-time
| perspective rendering on commodity hardware, instantly
| realigning the direction of much of the game industry,
| yadda yadda yadda, but at the end of the day it was a good-
| enough game for people to remember and respect even without
| considering the tech.
|
| Minecraft would be a similar example. Minecraft looked like
| total ass, and games with similar rendering technology
| could have been (and were) made years earlier, but
| Minecraft was also _good_. And that was enough.
| gjadi wrote:
| Hardware changes a lot in the time it takes to develop a
| game. When I read his plan files and interviews, I realized
| he seemed to spend a lot of time before developing the game
| thinking about what the next gen hardware was going to
| bring. Then design the best game they could think of whike
| targeting this not-yet-available hardware.
| andrepd wrote:
| > So Wolfenstein and Doom were actually not really 3d
| games, but rather 2.5 games (you can't have rooms below
| other rooms). The first 3d game here is actually Quake
|
| Ultima Underworld is a true 3D game from 1992. An
| incredibly impressive game, in more ways than one.
| leoc wrote:
| But also, he didn't do the technically hardest and most
| impressive part, Quake, on his own. IIUC he basically
| relied on Michael Abrash's help to get Quake done (in any
| reasonable amount of time).
| muziq wrote:
| The world seems to have rewritten history, and forgotten
| Ultima Underworld, which shipped prior to Doom..
| Narishma wrote:
| I think that's because it had such high system
| requirements that very few people could run it, unlike
| Wolfenstein 3D and Doom.
| zeroq wrote:
| Couple "3D" games shipped before Doom. Battlezone comes
| to mind.
|
| The difference is that id owned the natural progression
| (from Wolf3D through Doom to Quake) and laid foundation
| to what we call today a FPS genre.
| Buttons840 wrote:
| > his main strength has always been in doing more with less
|
| Carmack builds his kingdom and then runs it well.
|
| I makes me wonder how he would fare as an unknown Jr.
| developer with managers telling him "that's a neat idea, but
| for now we just need you to implement these Figma designs".
| mrandish wrote:
| A key aspect of the Carmack approach (or similar 'smart
| hacker' unconventional career approach) is avoiding that
| situation in the first place. However, this also carries
| substantial career, financial and lifestyle risks & trade-
| offs - especially if you're not both talented enough and
| lucky enough to hit a sufficiently fertile oppty in the
| right time window on the first few tries.
|
| Assuming one is willing to accept the risks and has the
| requisite high-talent plus strong work drive, the Carmack-
| like career pattern is to devote great care to evaluating
| and selecting opptys near the edges of newly emerging
| 'interesting things' which also: coincide with your
| interests/talents, are still at a point where a small team
| can plausibly generate meaningful traction, and have
| plausible potential to grow quickly and get big.
|
| Carmack was fortunate that his strong interest in graphics
| and games overlapped a time period when Moore's Law was
| enabling quite capable CPU, RAM and GFX hardware to hit
| consumer prices. But we shouldn't dismiss Carmack's success
| as "luck". That kind of luck is an ever-present
| uncontrolled variable which must be factored into your
| approach - not ignored. Since Carmack has since shown he
| can get _very interested_ in a variety of things, I assume
| he filtered his strong interests to pick the one with the
| most near-term growth potential which also matched his
| skills. I suspect the most fortunate "luck" Carmack had
| wasn't picking game graphics in the early 90s, it was that
| (for whatever reasons) he wasn't already employed in a more
| typical "well-paying job with a big, stable company, great
| benefits and career growth potential" so he was free to
| find the oppty in the first place.
|
| I had a similarly unconventional career path which,
| fortunately, turned out very well for me (although not
| quite at Carmack's scale :-)). The best luck I had actually
| looked like 'bad luck' to me and everyone else. Due to my
| inability to succeed in a traditional educational context
| (and other personal shortcomings), I didn't have a college
| degree or resume sufficient to get a "good job", so I had
| little choice but to take the high-risk road and figure out
| the unconventional approach as best I could - which
| involved teaching myself, then hiring myself (because no
| one else would) and then repeatedly failing my way through
| learning startup entrepreneurship until I got good at it. I
| think the reality is that few who succeed on the
| 'unconventional approach' consciously _chose_ that path at
| the beginning over lower risk, more comfortable
| alternatives - we simply never had those alternatives to
| 'bravely' reject in pursuit of our dreams :-).
| zeroq wrote:
| > "makes me wonder how he would fare as an unknown Jr.
| developer with managers telling him (...)"
|
| he would probably write an open letter and left Meta. /s
| saejox wrote:
| What Carmack is doing is right. More people need to get away from
| training their models just with words. AI need the physicality.
| NL807 wrote:
| >AI need the physicality.
|
| which i found interesting, because i remember Carmack saying
| simulated environments are way forward and physical
| environments are too impractical for developing AI
| SeanaldMcDnld wrote:
| Yeah in that way this demo seemed gimmicky like he
| acknowledged. He said in the past he would almost count
| people out if they weren't training RL in a virtual
| environment. I agree, still happy he's staying on the path of
| online continual learning though
| programd wrote:
| Nvidia seems to think the same thing. Here's Jim Fan talking
| about a "physical Turing test" and how embodied AI is the way
| forward.
|
| https://www.youtube.com/watch?v=_2NijXqBESI
|
| He also talks needing large amounts of compute to run the
| virtual environments where you'll be training embodied AI. Very
| much worth watching.
| johnb231 wrote:
| > More people need to get away from training their models just
| with words.
|
| They started doing that a couple of years ago. The frontier
| "language" models are natively multimodal, trained on audio,
| text, video, images. That is all in the same model, not
| separate models stitched together. The inputs are tokenized and
| mapped into a shared embedding space.
|
| Gemini, GPT-4o, Grok 3, Claude 3, Llama 4. These are all
| multimodal, not just "language models".
| timmg wrote:
| (If you know) how does that work?
|
| Are the audio/video/images tokenized the same way as text and
| then fed in as a stream? Or is the training objective
| different than "predict next token"?
|
| If the former, do you think there are limitations to "stream
| of tokens"? Or is that essentially how humans work? (Like I
| think of our input as many-dimensional. But maybe it is
| compressed to a stream of tokens in part of our perception
| layer.)
| johnb231 wrote:
| Ask Gemini to explain how it was trained
|
| https://g.co/gemini/share/f64c3358d9fa
| koolala wrote:
| I wish he did this with VR environment instead like they mention
| at the start of the slides. A VR environment with a JPEG camera
| filter, physics sim, noise, robot simulation. If anyone could
| program that well its him.
|
| Using real life robots is going to be a huge bottleneck for
| training hours no matter what they do.
| qoez wrote:
| Interesting reply from an openai insider:
| https://x.com/unixpickle/status/1925795730150527191
| andy_ppp wrote:
| My bet is on Carmack.
| speed_spread wrote:
| I suspect Carmack in the Dancehall with the BFG.
| ramesh31 wrote:
| What has he shipped in the last 20 years? Oculus is one
| thing, but that was firmly within his wheelhouse of graphics
| optimization. Abrash and co. handled the hardware side of
| things.
|
| Carmack is a genius no doubt. But genius is the result of
| intense focused practice above and beyond anyone else in a
| particular area. Trying to extend that to other domains has
| been the downfall of so many others like him.
| alexey-salmin wrote:
| Ever since Romero departed the id Software had shipped
| *checks notes* Quake II, Quake III, Doom 3 and Quake 4.
|
| Funnily enough Romero himself didn't ship much either. IMO
| it's one of the most iconic "duo breakups". The whole is
| greater than the sum of the parts.
| johnb231 wrote:
| Rage was Carmack's last big game at id Software before
| leaving.
|
| Romero is credited on 27 games since he left id Software.
|
| https://en.wikipedia.org/wiki/John_Romero#Games
| WithinReason wrote:
| "Graphics Carmack" is a genius but that doesn't mean that "AI
| Carmack" is too.
| MrLeap wrote:
| I wouldn't bet against him. "The Bitter Lesson" may imply
| an advantage to someone who historically has been at the
| tip of the spear for squeezing the most juice out of GPU
| hosted parallel computation.
|
| Graphics rendering and AI live on the same pyramid of
| technology. A pyramid with a lot of bricks with the
| initials "JC" carved into them, as it turns out.
| kadushka wrote:
| Only if computation is the bottleneck. GPT-4.5 shows it's
| not.
| mhh__ wrote:
| I would be long carmack in the sense that I think he will
| have good judgement and taste running a business but I
| really don't see anything in common between AI and
| graphics.
|
| Maybe someone better at aphorisms than me can say it
| better but I really don't see it. There are definitely
| mid-level low hanging fruits that would look like the
| kinds of things he did in graphics but the game just
| seems completely different.
| KerrAvon wrote:
| I think people would do well to read about Philo
| Farnsworth in this context.
| cheschire wrote:
| Carmack is always a genius, but like most people he
| requires luck, and like most people, the house always wins.
| Poor Armadillo Aerospace.
| dumdedum123 wrote:
| Exactly. I know him and like him. He is a genius programmer
| for sure BUT people forget that the last successful
| _product_ that he released was Doom 3 over 20 years ago.
| Armadillo was a failure and Oculus went nowhere.
|
| He's also admitted he doesn't have much of math chops,
| which you need if you want to make a dent in AI. (Although
| the same could have been said of 3D graphics when he did
| Wolfenstein and Doom, so perhaps he'll surprise us)
|
| I wish him well TBH
| mrguyorama wrote:
| What has "Graphics Carmack" actually done since about 2001?
|
| So, his initial tech was "Adaptive tile refresh" in
| Commander Keen, used to give it console style pixel-level
| scrolling. Turns out, they actually hampered themselves in
| Commander Keen 1 by not understanding the actual tech, and
| implemented "The Jolt", a feature that was not necessary.
| The actual hardware implemented scrolling the same way that
| consoles like the NES did, and did not need "the jolt", nor
| the limitations it imposed.
|
| Then, Doom and Quake was mostly him writing really good
| optimizations of existing, known and documented algorithms
| and 3D techniques, usually by recognizing what assumptions
| they could make, what portions of the algorithm didn't need
| to be recalculated when, etc. Very talented at the time,
| but in the software development industry, making a good
| implementation of existing algorithms that utilize your
| specific requirements is called _doing your job_. This is
| still the height of his relative technical output IMO.
|
| Fast Inverse Square Root was not invented by him, but was
| floating around in industry for a while. He still gets
| kudos for knowing about it and using it.
|
| "Carmack's reverse" is a technique for doing stencil
| shadows that was a minor (but extremely clever)
| modification to the "standard" documented way of doing
| shadow buffers. There is evidence of the actual technique
| from a decade before Carmack put it in Doom 3 and it was
| outright patented by two different people the year before.
| There is no evidence that Carmack "stole" or anything this
| technique, it was independent discovery, but was clearly
| also just a topic in the industry at the time.
|
| "Megatextures" from Rage didn't really go anywhere.
|
| Did Carmack actually contribute anything to VR rendering
| while at Oculus?
|
| People treat him like this programming god and I just don't
| understand. He was well read, had a good (maybe too good)
| work ethic, and was very talented at writing 386 era
| assembly code. These are all laudable, but doesn't in my
| mind imply that he's some sort of 10X programmer who could
| revolutionize random industries that he isn't familiar
| with. 3D graphics math isn't exactly difficult.
| WithinReason wrote:
| AI math isn't exactly difficult either.
| cmpxchg8b wrote:
| Appeal to authority is a logical fallacy. People often fall
| into the trap of thinking that because they are highly
| intelligent and an expert in one domain that this makes them
| an expert in one or more other domains. You see this all the
| time.
| edanm wrote:
| Expecting an expert in one thing to also be pretty good at
| other domains, especially when they're relatively related,
| isn't a fallacy.
| rurp wrote:
| Bayesian reasoning isn't a fallacy. A known expert in one
| domain is often correct about things in a related one. The
| post didn't claim that Carmack is right, just that that
| he's who they would bet on to be right, which seems
| perfectly reasonable to me.
| mrandish wrote:
| > People often fall into the trap of thinking that because
| they are highly intelligent and an expert in one domain
| that this makes them an expert in one or more other
| domains.
|
| While this is certainly true, I'm not aware of any evidence
| that Carmack thinks this way about himself. I think he's
| been successful enough that's he's personally 'post-
| economic' and is choosing to spend his time working on
| unsolved hard problems he thinks are extremely interesting
| and potentially tractable. In fact, he's actively sought
| out domain experts to work with him and accelerate his
| learning.
| roflcopter69 wrote:
| Funny, I was just commenting something similar here, see
| https://news.ycombinator.com/item?id=44071614
|
| And I say this while most certainly not being as knowledgeable
| as this openai insider. So it even I can see this, then it's
| kinda bad, isn't it?
| fmbb wrote:
| Can you explain which parts you think are bad and why?
| jjulius wrote:
| Right? "Even I can see this" isn't exactly enlightening.
| johnb231 wrote:
| https://x.com/ID_AA_Carmack/status/1925973500327591979
| zeroq wrote:
| >> "they will learn the same lesson I did"
|
| Which is what? Don't trust Altman? x)
| epr wrote:
| Actually no, it's not interesting at all. Vague dismissal of an
| outsider is a pretty standard response by insecure academic
| types. It could have been interesting and/or helpful to the
| conversation if they went into specifics or explained anything
| at all. Since none of that's provided, it's "OpenAI insider" vs
| John Carmack AND Richard Sutton. I know who I would bet on.
| kadushka wrote:
| He did go into specifics and explained his point. Or have you
| only read his first post?
| ActivePattern wrote:
| It's a OpenAI researcher that's worked on some of their most
| successful projects, and I think the criticism in his X
| thread is very clear.
|
| Systems that can learn to play Atari efficiently are
| exploiting the fact that the solutions to each game are
| simple to encode (compared to real world problems).
| Furthermore, you can nudge them towards those solutions using
| tricks that don't generalize to the real world.
| dgb23 wrote:
| That sounds like an extremely useful insight that makes
| this kind of research even more valuable.
| lairv wrote:
| Alex Nichol worked on "Gotta Learn Fast" in 2018 which
| Carmack mentions in his talk, he also worked on foundational
| deep learning methods like CLIP, DDPM, GLIDE, etc. Reducing
| him to a "seething openai insider" seems a bit unfair
| MattRix wrote:
| It's not vague, did you only see the first tweet or the
| entire thread?
| quadrature wrote:
| Do you have an X account, if you're not logged in you'll only
| see the first post in the thread.
| threatripper wrote:
| x.com/... -> xcancel.com/...
| ewoodrich wrote:
| I use a Chrome extension to auto replace the string in
| the URL, works very well.
| handsclean wrote:
| It seems that you've only read the first part of the message.
| X sometimes aggressively truncates content with no indication
| it's done so. I'm not sure this is complete, but I've
| recovered this much:
|
| > I read through these slides and felt like I was transported
| back to 2018.
|
| > Having been in this spot years ago, thinking about what
| John & team are thinking about, I can't help but feel like
| they will learn the same lesson I did the hard way.
|
| > The lesson: on a fundamental level, solutions to these
| games are low-dimensional. No matter how hard you hit them
| with from-scratch training, tiny models will work about as
| well as big ones. Why? Because there's just not that many
| bits to learn.
|
| > If there's not that many bits to learn, then researcher
| input becomes non-negligible.
|
| > "I found a trick that makes score go up!" -- yeah, you just
| hard-coded 100+ bits of information; a winning solution is
| probably only like 1000 bits. You see progress, but it's not
| the AI's.
|
| > In this simplified RL setting, you don't see anything close
| to general intelligence. The neural networks aren't even that
| important.
|
| > You won't see _real_ learning until you absorb a ton of
| bits into the model. The only way I really know to do this is
| with generative modeling.
|
| > A classic example: why is frame stacking just as good as
| RNNs? John mentioned this in his slides. Shouldn't a better,
| more general architecture work better?
|
| > YES, it should! But it doesn't, because these environments
| don't heavily encourage real intelligence.
| cmiles74 wrote:
| From a marketing perspective, this strikes me as a very
| predictable response.
| jjulius wrote:
| I appreciate how they don't tell us what lesson they learned.
| dcre wrote:
| It is a thread. You may have only seen the first tweet
| because Twitter is a user-hostile trash fire.
|
| "The lesson: on a fundamental level, solutions to these games
| are low-dimensional. No matter how hard you hit them with
| from-scratch training, tiny models will work about as well as
| big ones. Why? Because there's just not that many bits to
| learn."
|
| https://unrollnow.com/status/1925795730150527191
| jjulius wrote:
| Thank you for clarifying. I don't have a Twitter account,
| and the linked tweet genuinely looks like a standalone
| object. Mea culpa.
| dcre wrote:
| Not your fault. They are the worst.
| alexey-salmin wrote:
| Each of these games is low-dimensional and require not the
| "intelligence" but more like "reflexes", I tend to agree.
|
| However making a system that can beat an unknown game does
| require generalization. If not real a intelligence (whatever
| that means) but at the level of say "a wolf".
|
| Whether it can arise from RL alone is not certain, but it's
| there somewhere.
| lancekey wrote:
| I think some replies here are reading the full twitter thread,
| while others (not logged in?) see only the first tweet. The
| first tweet alone does come off as a dismissal with no insight.
| mannycalavera42 wrote:
| indeed, this is pure walled garden sh*t
| johnb231 wrote:
| Carmack replied to that
| https://x.com/ID_AA_Carmack/status/1925973500327591979
| threeseed wrote:
| Direct links:
|
| https://docs.google.com/presentation/d/1GmGe9ref1nxEX_ekDuJX...
|
| https://docs.google.com/document/d/1-Fqc6R6FdngRlxe9gi49PRvU...
| roflcopter69 wrote:
| Honestly, having gone through the slides, it's a bit painful to
| see Carmack "rediscover" stuff I've learned in a reinforcement
| learning lecture like ten years ago.
|
| But don't get me wrong! Since this is a long-term research
| endeavor of his, I believe really starting from the basics is
| good for him and will empower him to bring something new to the
| table eventually.
|
| I'm surprised though that he "only" came so far as of now. Maybe
| my slight idolization of Carmack made me kinda of blind to the
| fact that this kind of research is a mean beast after all and
| there is a reason that huuuuge research labs dump countless of
| man-decades into this kind of stuff with no guaranteed
| breakthroughs.
| Cipater wrote:
| https://x.com/unixpickle/status/1925795730150527191
| roflcopter69 wrote:
| I was just going to answer
| https://news.ycombinator.com/item?id=44071595 who mentioned
| exactly the same tweet.
|
| I'm nowhere as good at my craft as someone who works for
| openai, which the author of that tweet seems to be, but if
| even I can see this, then it's bad, isn't it?
| andy_ppp wrote:
| I still don't think we have a clear enough idea of what a concept
| is to be able to think about AGI. And then being able to use
| concepts from one area to translate into another area, what is
| the process by which the brain combines and abstracts ideas into
| something new?
| throw310822 wrote:
| Known entities are recurring patterns (we give names to things
| that occur more than once, in the world or in our thoughts).
| Concepts are recurring thought patterns. Abstractions,
| relations, metaphors, are all ways of finding and transferring
| patterns from one domain to another.
| andy_ppp wrote:
| Sure, I understand what the terminology means but I don't
| believe we get to AGI without some ability to translate the
| learning of say using a mouse to using a trackpad in a simple
| way. Humans make these translations all the while, you know
| how to use a new room and the items in it automatically but I
| personally see the systems we have built are currently very
| brittle when they see new environments because they can't
| simplify everything to its fundamentals and then extrapolate
| back to more complex tasks. You could train a human on using
| an Android phone and give them an iPhone and they would do
| pretty well, if you did this with modern machine learning
| systems you will get an extremely high error rate. Or say you
| train an model on how to use a sword, I'm not convinced it
| would know how to use and ax or pair of crutches as a weapon.
|
| Maybe it will turn out to simply be enough artificial neurons
| and everything works. But I don't believe that.
| steveBK123 wrote:
| Another thought experiment - if OpenAI AGI was right around the
| corner, why are they wasting time/money/energy buying a product-
| less vanity hardware startup run by Ive?
|
| Why not tackle robotics if anything. Or really just be the best
| AGI and everyone will be knocking on your door to license it in
| their hardware/software stacks, you will print infinite money.
| soared wrote:
| Or have your AGI design products
| steveBK123 wrote:
| All the more reason not to acquihire Ive for $6.5B, if true
| tiahura wrote:
| Does AGI necessarily mean super-genius? Was KITT AGI? I'm not
| sure he could design products?
| steveBK123 wrote:
| Is VC really funding a trillion dollars of GPU purchases to
| replace labor that could instead be bid out to developing
| world mechanical turks for $1/hr?
| trendoid wrote:
| No the term for that is ASI...artifical super intelligence.
| People in AI community have different timelines for that
| than AGI.
| j_timberlake wrote:
| This line of thought doesn't work, because any company
| approaching AGI might be actively trying to hide that
| information from regulators and the military. Being the 1st AGI
| company is actually pretty risky.
| steveBK123 wrote:
| VCs are far too conditioned as hype men to hide the ball like
| that.
|
| After generations of boastful over-promising, do you really
| believe THIS time they are underpromising?
| foobiekr wrote:
| However their actual actions resemble companies who know AGI
| isn't even on the horizon and moreso they are acting as
| exactly as if they believe the AI hype bubble is coming to an
| end and they need to dump the stuff into the public markets
| asap.
|
| There really isn't any other way to interpret OpenAI's
| actions for the last few months.
|
| Sure it could all be a feint to hide their amazing progress.
| Or it could be what it looks like.
|
| Given the hype cycles of the last 20 years, I'm going with
| the second.
| j_timberlake wrote:
| Name any other company acting like that besides OpenAI. Or
| any person besides Sam Altman, the guy who screwed up
| OpenAI's structure/board/funding. You just have a narrative
| you want to be true, and one company half fits into that
| narrative.
| mindwok wrote:
| AGI is not enough. Seriously, imagine if they had an AGI in
| their ChatGPT interface. It's not enough to do anything truly
| meaningful. It's like a genius in the woods somewhere. For AGI
| to have an impact it needs to be everywhere.
| steveBK123 wrote:
| Robotics to navigate the physical world seems more impactful
| than some pin/glasses product to provide a passive
| audio/visual interface to the chatbot doesn't seem so earth
| shattering either though.
|
| What would you do with a 10x or 100x smarter Siri/Alexa? I
| still don't see my life changing.
|
| Give me a robot that can legitimately do household errands
| like the dishes, laundry, etc.. now we are talking.
| Jensson wrote:
| > Seriously, imagine if they had an AGI in their ChatGPT
| interface. It's not enough to do anything truly meaningful
|
| If they had that people would make agents with it and then it
| can do tons of truly meaningful things.
|
| People try to make agents with the current one but its really
| difficult since its not AGI.
| joshstrange wrote:
| Once AGI is accomplished I can't imagine what else it would
| do but bootstrap itself up which, depending on compute, could
| scale quite far. OpenAI would only need to feed it compute
| for the most part.
|
| I don't think AGI is close, but once it happens it's hard to
| imagine it not "escaping" (whenever we want to define that
| as).
| Flamentono2 wrote:
| I find it interesting that he dismisses LLMs.
|
| I would argue that if he wants to do AGI through RL, a LLM could
| be a perfect teacher or oracle.
|
| After all i'm not walking around as a human and not having
| guidance. It should/could make RL a lot faster leveraging this.
|
| My logical part / RL part does need the 'database'/fact part and
| my facts are trying to be as logical as possible but its just
| not.
| akomtu wrote:
| IMO, he's right. LLMs can't be AI because they don't create a
| model of observations to predict things, they just imitate
| observations based on their likeness to each other. When you
| play Quake, you create a simple model of the game physics and
| use that fast model to navigate the game. Your equivalent of
| LLM has a role too: it's a fuzzy detector of things you
| encounter in the game, sounds, images and symbols, but once
| detected, those things are fed into the fast and rigid physics
| model.
| mkoubaa wrote:
| > It is worth trying out one of the many web based reaction time
| testers - you will find that you average over 160 milliseconds.
|
| TIL JC has elite reflexes
| 2OEH8eoCRo0 wrote:
| And nerves of steel
| soci wrote:
| > Fundamentally, I believe in the importance of learning from a
| stream of interactive experience, as humans and animals do, which
| is quite different from the throw-everything-in-a-blender
| approach of pretraining an LLM. The blender approach can still be
| world-changingly valuable, but there are plenty of people
| advancing the state of the art there.
|
| It's a shame that pretrained approach leads to such good enough
| result. The learning-from-experience, or what should be the
| "right" approach, will stagnate. I might be wrong, but it seems
| that aside from Carmack and a small team, "the world" is just not
| looking/investing on that side of the AI anymore.
|
| However, I find it funny that Carmack is now researching for such
| approach. At the end of the day, he was the one who invented
| Portals, an algorithm to circumvent the need to reproduce the
| whole 3D world and therefore making 3D games computationally
| possible.
|
| As a side note, I wonder what models are to come once we see the
| latest state of the art AI Video training technologies, in synch
| with the joystick movements from a real player. Maybe the results
| are so astonishing that even Carmack changes his mind on the
| subject.
|
| EDIT::grammar & typos
| anthonypasq wrote:
| i think you're overstating this. Yann LeCun (chief scientist at
| Meta) is firmly in this camp, and i think most companies trying
| to bring AI into the real world via some sort of robotics
| technology are thinking about and testing this approach.
| soci wrote:
| Thank you. You are right, most likely the ones working in the
| field haven't switched. But the truth is that big bucks are
| in pretrained technologies. As Carmack himself said, "there
| are plenty of people advancing the state of the art there".
| koolala wrote:
| Humans had 500 million * 8670 hours of Pre-Training.
|
| I don't get why Carmack would say things should be learned in
| hours or upper bounds it to human lifetime.
| flipnotyk wrote:
| I think there's a difference between "should be learned" and
| "should be able to be learned" here.
| vlovich123 wrote:
| I'm not necessarily convinced despite my human bias that it's a
| superior mechanism. Humans work the way they do and learn the
| way they do in no small part because of biological limitations
| and a physical reality. It's not clear that a virtual entity
| needs to face the same limitations, although clearly learning
| from feedback that's not available to an AI is important. It is
| true though that humans are more energy efficient learners, but
| letting the AI experiment with the real world and get feedback
| that's way may be the only missing piece rather than a problem
| with the "blender" approach.
| tshaddox wrote:
| > It's a shame that pretrained approach leads to such good
| enough result. The learning-from-experience, or what should be
| the "right" approach, will stagnate.
|
| We'll see. I'm skeptical that you'll ever get novel theories
| like special and general relativity out of LLMs. For stuff like
| that I suspect you need the interactive learning approach, and
| perhaps more importantly, the ability to reject the current
| best theories and invent a replacement.
| xnx wrote:
| I'm surprised there's as much interest in looking at the
| structure/behavior of the biological brain, and less interest in
| considering the behavior of our vision system. Our brains are not
| CPUs, and our eyes are definitely not a grid of pixels with a
| fixed framerate.
| lostmsu wrote:
| I'm with OpenAI folks on this one: Atari just won't cut it for
| AGI. My layman intuition is that RL works well when rewards give
| good signal all the time. Until it does RL is basically random
| search. That's where massive data diversity like we have in text
| comes in handy.
|
| In a game there might be a level with a door and a key, and
| because there's no reward for getting the key closer to the door,
| bridging this gap requires random search in a massive state
| space. But in the vast sea of scenarios that you can find in
| Common Crawl there's probably one, where you are 1 step from the
| key, and the key is 1 step from the door, so you get the reward
| signal from it without having to search an enormous state space.
|
| You might say "but you have to search through the giant Common
| Crawl". Well yes, but while doing so you will get reward signal
| not just for the key and door problem, but for nearly every
| problem in the world.
|
| The point is: pretraining teaches models to extract signal that
| can be used to explore solutions to hard search problems, and if
| you don't do that you are wasting your time enumerating giant
| state spaces.
| lostmsu wrote:
| You can actually easily test and overcome this by training a
| model simultaneously on a massive of text and Atari while
| carefully balancing learning rates between the two.
| ploden wrote:
| Why would AGI choose to be embodied? We talk about creating a
| superior intelligence and having it drive our cars and clean our
| homes. The scenario in Dan Simmons' Hyperion seems much more
| plausible: we invent AGI and it disappears into the cloud and
| largely ignores us.
| fusionadvocate wrote:
| It doesn't need to be permanent. If humans could escape from
| their embodiment temporarily they would certainly do so. Being
| permanently bounded to a physical interface is definitely a
| disadvantage.
| jwmcq wrote:
| Looking at other examples in sci-fi, perhaps to stop _my_ body
| from pressing its off-switch?
| ploden wrote:
| With distributed backups in place, AIs will be much less
| worried about self-preservation than we are.
___________________________________________________________________
(page generated 2025-05-23 23:00 UTC)