[HN Gopher] The Matrix: Infinite-Horizon World Generation with R...
___________________________________________________________________
The Matrix: Infinite-Horizon World Generation with Real-Time
Interaction
Author : lnyan
Score : 175 points
Date : 2024-11-21 04:26 UTC (18 hours ago)
(HTM) web link (thematrix1999.github.io)
(TXT) w3m dump (thematrix1999.github.io)
| jerpint wrote:
| I'm really excited for where this is going. From the demo videos,
| it seems to be a step up from Oasis, which itself came out only 2
| weeks ago. I expect to see a lot of innovative use cases in this
| field
| shmerl wrote:
| _> Click to play_
|
| Clicking - nothing works.
| tsaoyu wrote:
| Click to play [...the video]
| efitz wrote:
| Prediction: in 20 years, I'm going to be reading about some dude
| who wrote a program to drive the car continuously until it ran
| into some surreal edge condition, and finally hit it. There will
| be a subculture of "matrix glitchers" who spend much of their
| time doing these kinds of experiments.
| soulofmischief wrote:
| People have been doing that with Minecraft for over a decade.
| In the old days, once you got far away enough, the terrain
| generation would go haywire. Lots of videos from that time
| period of people exploring the "edge of the world".
|
| Personally, these were the kind of glitches which made games
| feel magical and "real" to me as a kid. Being able to analyze a
| system by breaking it made it seem so much more tangible, like
| an actual place I had an NTSC-sized porthole into.
| billisonline wrote:
| Cf. the "Minus World" in Super Mario Bros. for the NES.
|
| https://en.wikipedia.org/wiki/Minus_World
| soulofmischief wrote:
| Ha! I remember being either 5 or 6 when my uncle showed me
| Minus World and it blowing my mind. That might have
| actually been my first exposure to "backrooms" glitches
| like that. What an amazing glitch. It even worked on my
| combo Super Mario Bros / Duck Hunt cartridge
|
| MissingNo. is another good example. I have fond memories
| spending untold hours in my favorite game engines trying to
| break free. The Jak and Daxter series were some of my
| favorite to break, due to the uniqueness and flexibility of
| the engine and the weird ways that the chunk loading system
| could be broken.
| axus wrote:
| Ahh, "Mountain King" on the Atari 2600 was the game for
| me finding a cool bug. If you bounced just right, you'd
| soar over the mountain into the glitches far above. Games
| didn't crash, they just worked with what they had.
| soulofmischief wrote:
| I didn't have Mountain King for my 2600 so I looked it
| up. What a neat glitch. Platformer glitches are fun, I
| really enjoyed breaking the early Sonic games for things
| like the Hyper Sonic glitch, or some of the map glitches.
|
| I think this is one thing about Super Mario Bros. 3 that
| felt so magical to me. With the addition of the hidden
| whistles and intentional "glitches" like crouching for an
| extended time on a white platform, running behind map
| elements, etc. you felt like some kind of plane walker
| just bending time and space to your will. Fantastic
| implementation of a level skip mechanism for veteran
| players. It gave an already incredibly expansive game
| quite a lot of extra replay value, just like Minus World.
| sunnybeetroot wrote:
| Thank you for the reminder of MissingNo! Takes me back to
| when I was a child and received a Gameboy Color without
| any games. I spent months just watching the start up
| animation on repeat before I got Pokemon yellow.
| soulofmischief wrote:
| That is one of the saddest thing I've ever heard. Did
| your parents just not know it needed games, or was it a
| budget thing?
|
| I was extremely poor growing up but I did get lucky and
| get a Gameboy Color for Christmas with a copy of Pokemon
| Gold at age 5, right before my guardians went insane and
| forbade any non-Christian media such as "Pocket Demons"
| or any fantasy content. That game expanded my mind so
| much, introduced me to a lot of things I'd never
| encountered before. It seemed so mysterious and huge,
| especially with the entire extra Kanto campaign. Still
| one of the greatest and most complete games ever made.
| pugworthy wrote:
| Far Lands or Bust is a great YouTube channel for this. He's
| been walking in one direction in a Minecraft world since
| 2011.
| bobim wrote:
| "there's no bugs in there!" a chick with a star on the cheek
| probably.
| ilaksh wrote:
| That community already exists because the current version of
| these types of AI game engines are constantly running into a
| surreal edge condition since they don't track things
| consistently when they go off frame.
| grodes wrote:
| unreadable website
| ribcage wrote:
| Someone should ban "AI" articles on Hacker News.
| ferguu_ wrote:
| But then there would be no articles!
| Cthulhu_ wrote:
| Why? It's a mostly democratic news aggregator site without much
| editorial overview, if you don't like it, downvote / don't
| upvote it, or write a client that filters topics you don't like
| out based on some keywords. You didn't need to open this page
| and comment on it if you don't like it, nobody's making you
| read things you don't like.
| steego wrote:
| I don't know if you're actually new here, or you've been
| reading HN for years, but your account is only 11 months old.
|
| We talk about a lot of things here, but when we do talk about
| AI, we tend to prefer talk about things with code or papers.
|
| This particular project has a paper. They're expecting to
| publish their code soon.
|
| Here's the paper:
|
| https://thematrix1999.github.io/article/the_matrix.pdf
|
| You don't have to read it, but you may want to consider it if
| you want to learn something and/or contribute something
| meaningful to the conversation.
| olddustytrail wrote:
| Yes, and that someone should be you. Be the change you want to
| be.
| ilaksh wrote:
| I'm into VR and mixed reality, and I think this is headed to
| making the Holodeck real in an immersive way. That's the concept
| of the Matrix and what they are demoing, just in 2d.
|
| I am guessing the main thing holding this stuff back in terms of
| fidelity and consistency or generalization is just compute. But
| the new techniques they have here have just dramatically lowered
| the compute costs and increased the generalization.
|
| Maybe just something like the giant Cerebras SRAM chips will get
| to the next 10 X in scale that smooths this out and pushes it
| closer to Star Trek. Or maybe some new paradigm like memristors.
|
| But I'm looking forward to within just a few years being able to
| put on some fairly comfortable mixed reality glasses and just
| asking for whatever or whoever I want to appear in my home (for
| example) according to my whim.
|
| Or, train it on a lot of how-to videos such as cooking. It just
| materializes an example of someone showing you exactly what you
| need to do right in your kitchen.
|
| Here's another crazy idea: train on videos and interactions with
| productivity applications rather than games. In the future, for
| small businesses, we skip having the AI generate source code and
| just describe how the application works. The data and program
| state are just stored in a giant context window, and the
| application functionality changes the instant you make a request.
| scotty79 wrote:
| I think we need devices that can produce gaussian splat videos
| to become as common as 2d cameras.
| alkonaut wrote:
| What does "world" mean here? How does the spatiality fit into
| some latent space? Or what constitutes the "world"? If the answer
| is, there is none, the world is just frames of video and any
| consistency quickly blurs out after a few seconds. That's not a
| world generation, that's just generation of video frames
| following frames. Not that it isn't cool, but it has almost zero
| usability for generating a "world" simulation. The key to a
| realistic world is that you can reliably navigate it. Visit and
| revisit places. If you modify anything, those modifications are
| persisted. If you leave a room and re-enter it hours later, the
| base expectation is that the same objects are in that room.
|
| Wouldn't a working approach be to just create a really low
| resolution 3D world in the traditional "3D game world" sense to
| get the spatial consistency. Then this crude map with attributes
| is fed into frame generation to create the resulting world? It
| wouldn't be infinite, but on the other hand no one has a need for
| an infinite world either. A spherical world solves the border
| issue pretty handily. As I understood it, there was some element
| of that in the new FS2024 (discussed yesterday on HN).
| QuantumG wrote:
| What you really want is a story telling engine informing the
| world model and the humans need to generate power.
| scotty79 wrote:
| It's basically a dream. Maybe that's what dreams are? World
| prediction engine running with no sensor input and memory in
| read only mode?
| withinboredom wrote:
| When you go to sleep, your brain stem disconnects from your
| body and your brain enters a feedback loop. The sensors are
| very much connected, just to whatever. Hence why you can grow
| dragon wings in your dreams and feel them. Memories can be
| made as well. I used to be really into lucid dreaming and
| time compression. My longest dream was nine years compressed
| into a 12-hour sleep period.
| Helonomoto wrote:
| Where did you get the 'brain stem disconnects from your
| body'? Because thats not how it works in the brain.
|
| We have the part which controls your muscles and we have
| another part which simulates the movement. Not executing on
| it has nothing to do with the brain stem disconnecting from
| the 'body'.
|
| Its the same mechanism as you thinking about a movement but
| not doing it.
| lxgr wrote:
| I believe it's a much more fundamental difference than
| just the distinction between ideating and acting.
|
| Many people occasionally experience the transition
| between "conencted" and "disconnected" states as a sudden
| jerk or loud noise just at the moment of falling asleep.
|
| Sleep paralysis is another "failure mode" of this
| mechanism that reveals what's going on. (I'm not sure if
| there is a reverse to it, i.e. whether sleepwalking could
| be explained as a drastic fail-open of the same
| mechanism).
| jebarker wrote:
| > My longest dream was nine years compressed into a 12-hour
| sleep period.
|
| Does the brain change in response to that sleep period? Or
| is there no change because there's no new information
| input?
| withinboredom wrote:
| I dunno, I wasn't hooked up to any kind of measurement
| device to measure the changes in my brain. From a
| subjective point of view, I miss that place. I've been
| writing a book about my adventure there, off-and-on for
| years now. Maybe someday, I will finish it. If I could
| return, I'd do it in a heartbeat at any cost.
|
| I _feel_ older than I am because there are a couple extra
| decades in my brain than in real life. Most of my time
| compression experiments were only a few months or weeks.
| That one long one changed me forever, and I've never done
| it on purpose since then.
|
| I still have time compressed dreams from time to time,
| and when I wake up, two or three weeks have subjectively
| passed, but only a night has passed in the real world.
| There's a period of time, no more than 10-30 minutes,
| while the brain tries to reconcile two different and
| overlapping pasts. It can be a bit disorienting. My wife
| knows when I have these dreams because when I wake up,
| she says I look around surprised or confused to be there.
| The absolute worst is when you lay down to go to bed in
| the dream and wake up in the real world. Those will mess
| you up.
|
| So, maybe my brain did change. Who knows? Maybe someone
| should study it.
| roboboffin wrote:
| Not an expert, but I think procedurally generated terrain is
| generally fractal in nature, and is reproducible in that sense
| from a seed that is used in the generation. It is therefore
| recursive, as fractals are recursive.
|
| A traditional neural network is a universal function
| approximator, however it is not recursive in nature, unless it
| is some sort of RNN. The transformer architecture, which this
| seems fairly similar to this one, is also not recursive in
| nature; although I believe, limited recursion can come about
| through CoT.
|
| Therefore, I don't believe this could match the
| reproducibility, in an infinite sense, of a traditional
| procedural generator.
| jrvieira wrote:
| "Fractals are not self-similar"
|
| https://m.youtube.com/watch?v=gB9n2gHsHN4
|
| If you want to learn more about the fascinating world of
| fractals.
| falcor84 wrote:
| The parent comment didn't mention self-similarity; did you
| maybe intend to reply to another comment?
| jerf wrote:
| Procedurally generated terrain is whatever it wants to be. It
| isn't necessary for it to be fractal in any particular sense.
| Or even arguably all that useful, at least at the present
| time.
| astrobe_ wrote:
| Luanti [1] (Minetest) has a fractal map generator ("mapgen").
| You can test it for yourself. It's funny at first but becomes
| eventually boring.
|
| Its other mapgens massively some kind of Perlin noise in
| various ways, so that you can have "realistic" landscapes
| (e.g. the Carpethian mapgen) or landscapes with impossible
| mountains and floating rocks sometimes (e.g. the V7 mapgen)
| that are good for fantasy/sci-fi worlds.
|
| Noise is a pretty efficient way to fake complexity, and it's
| not a coincidence [2].
|
| [1] https://www.luanti.org/ [2]
| https://en.wikipedia.org/wiki/Kolmogorov_complexity
| jerf wrote:
| "Wouldn't a working approach be to just create a really low
| resolution 3D world in the traditional "3D game world" sense to
| get the spatial consistency. Then this crude map with
| attributes is fed into frame generation to create the resulting
| world?"
|
| That's Minecraft. It does work. And when you turn around, then
| turn back to where you were facing, you don't feel like you're
| in a fever dream, because the landscape hasn't completely
| shifted.
|
| The tech is neat and I'm sure worth publishing. The rhetoric
| accompanying it is terribly overblown. Of course that's as
| likely to be their University PR department as the authors.
| astrobe_ wrote:
| > Wouldn't a working approach be to just create a really low
| resolution 3D world in the traditional "3D game world" sense to
| get the spatial consistency. Then this crude map with
| attributes is fed into frame generation to create the resulting
| world?
|
| If you go "ultra-low" resolution, that's basically Minecraft or
| Luanti (formerly Minetest). There some mods for Luanti that let
| one generate a less "squary" world by adding slopes, but that's
| still heavily polygonal.
|
| > It wouldn't be infinite, but on the other hand no one has a
| need for an infinite world either.
|
| Minecraft has an infinite world (with glitches when you go very
| far from origin due to floating point errors). Luanti's world
| is finite (around 64000 m^3 because 16 bits coordinates;
| players move at 2-4m/s usually), some people are working to
| make it infinite. In my opinion, you are right that nobody
| needs an infinite world; the argument for an infinite world is
| that a finite world can be a problem on popular servers but in
| my eyes servers where this is a problem don't use enough the
| available space (in particular the huge vertical space that can
| be used to create multiple worlds) or shoot themselves in the
| foot by providing players with ways to move fast (mainly
| transports); increasing the player's speed actually shrinks the
| world.
|
| I think there is a conundrum from the players and gameplay
| perspective. On one hand, you want a lot of space, but on the
| other hand you want to be close to other players (for commerce,
| play together, etc.). It's not enough to have an infinite
| world, you also have to have the gameplay that goes with it.
|
| Another thing to consider are the interaction of mobiles (NPCs,
| enemies) with the terrain. Because nobody likes an infinitely
| empty world either. I haven't played an AAA game recently, but
| a decade ago even with precomputed terrain, some mobiles would
| look silly sometimes with steep terrains.
| abricq wrote:
| This is surely really cool. Just a bit sad that, as phrased by
| the authors, the "First Real-Time" virtual world created for the
| demo is a fat & fast SUV driving on virgin lands.
| spuz wrote:
| Why is that sad?
| pmayrgundter wrote:
| Sigh, agreed!
| akutlay wrote:
| Why would you want to generate all the pixels using this model
| instead of generating all the art, physics, and objects in the
| world using a game engine? The engine does so much of the physics
| and keeps everything stable for very cheap.
| FeepingCreature wrote:
| Generating coherent objects in an infinite world is probably
| harder.
| 01HNNWZ0MV43FF wrote:
| Because you wouldn't have to wait for a human to write the
| engine and write the game
| wavemode wrote:
| I wish researchers would spend more time on using generative
| models to create level geometry, rather than trying to generate
| video from scratch. It would be both cheaper and more effective
| for stable gameplay.
| jebarker wrote:
| There's so many researchers working in AI now that we can
| afford to explore all avenues. This probably isn't going to
| lead to a fully generative AI game engine, but I bet there'll
| be useful learnings along the way.
| pedalpete wrote:
| This is the future I was trying to pitch in 2018 when we had
| built Ayvri and had every paraglider in the world, the world's
| largest ultramarathons, drone operators, and lots of other users
| of our real-world 3D environment.
|
| Though we were using map tiles at the time, we were developing a
| model that took photos and a GPS track to add information that
| better matched environmental conditions (cloud, better lighting,
| etc).
|
| People still ask me to open-source or give them our source code,
| but the code was acquired, so that isn't possible. But I do
| regularly say that if I were to rebuild Ayvri today, I'd do it as
| an interactive video rather than loading tiles.
| darepublic wrote:
| I didn't fully grok what this was about from the website. Though
| just last night I was talking to a friend about that quote from
| the Matrix that Morpheus tells Neo, so some nice synchronicity
| there. The sense I got from this is that they are developing a
| triple AAA type virtual world that can get generated on the fly
| based on text prompts? When the authors say frame level control
| do they mean that at any point, the next frame can be
| manipulated, either to be completely new or to influence the
| current story or context that is being played out?
___________________________________________________________________
(page generated 2024-11-21 23:00 UTC)