[HN Gopher] Show HN: Stable Diffusion powered level editor for a...
___________________________________________________________________
Show HN: Stable Diffusion powered level editor for a 2D game
Hey folks, I've been working on using control-net to take in a
video game level (input as a depth image) and output a beautiful
illustration of that level. Play with it here: dimensionhopper.com
or read the blog post about what it took to get it to work. Been a
super fun project.
Author : robobenjie
Score : 245 points
Date : 2023-06-12 15:31 UTC (7 hours ago)
(HTM) web link (generalrobots.substack.com)
(TXT) w3m dump (generalrobots.substack.com)
| mock-possum wrote:
| Seems like a good source of cheap visual design!
| robobenjie wrote:
| Sure, but I'm more interested in things that were just
| impossible before. You can hire a artist to illustrate a level,
| or have AI do it cheaper, but you can't have an artist
| illustrate a level the player made while they wait. I think
| there are whole play patterns that are possible because the
| cost and especially speed of creating the art are many orders
| of magnitude different.
| fnordpiglet wrote:
| I think even more interesting is generating entire styles and
| story lines that evolve infinitely and coherently based off
| of seeds. Players could even inject a concept - "steampunk"
| or "discworld" and an LLM could construct the story, with
| characters, and visual themes.
| robobenjie wrote:
| Several of the themes (including "alien jungle", which is
| my favorite) we're created by chatGpt. I totally want to
| try evolving the game in that direction.
| zacharycohn wrote:
| I didn't think of that while reading your article. You should
| plaster that comment all over this project to help people see
| the possibilities.
| robobenjie wrote:
| Great feedback. I added it to the start of the article.
| Thanks!
| kraftman wrote:
| I think this could be great for letting the players design
| custom weapons/Armor/spells very roughly and then using AI to
| convert it to something that looks good in the game
| erezsh wrote:
| Very cool! When building it, did you encounter any unexpected
| obstacles, or found any cool tips that you'd like to share?
| robobenjie wrote:
| The main one is that making the control-net depth input look
| like something helps a ton. You can creating levels that have
| more 'structure' (large flat platforms, platforms that line up
| with others) and levels that are more random and see that the
| structure works way better. I played around a lot with turning
| the control-net on and off at the beginning and end of
| generation, which seemed to help when I was playing in the
| webui but then I didn't immediately find the API in diffusers
| and the results I was getting were great so I didn't keep
| looking.
| q_andrew wrote:
| Depth is a useful parameter for controlnet, especially when
| you want really specific forms. I've found that it can hamper
| outputs because blank sections of solid color are interpreted
| as flat walls, when really I'm trying to make those parts
| ambiguous!
| robobenjie wrote:
| I'm also curious if anyone has made a level that worked
| particularly well/poorly or has a great custom theme (that maybe
| I should add to the dropdown) :)
| nico wrote:
| This is such a cool project. Very inspiring, thank you for
| posting it
|
| Great example of using AI as a tool to make something exceptional
| robobenjie wrote:
| Thanks! It has been a ton of fun too.
| spywaregorilla wrote:
| I wish this was around when I was a kid :(
| [deleted]
| fnordpiglet wrote:
| Protip: you don't have to grow up
| gadflyinyoureye wrote:
| Why not dimension jump?
| https://www.reddit.com/r/DimensionalJumping/
| anhner wrote:
| what the fuck did i just read
| gadflyinyoureye wrote:
| The internet is a weird and wonderful place.
| spywaregorilla wrote:
| It appears to be a group of people who are more willing
| to believe in transdimensional travel being responsible
| for changes in their life than things happening as a
| result of their outlook, luck, or their own agency.
| robobenjie wrote:
| While I have folks attention, I want to try training a model to
| generate monster/creature walk animations. Anyone know of a
| dataset of walk cycle sprite sheets that I could massage and
| label to see if I can make that work?
| psychphysic wrote:
| Does https://opengameart.org/ work?
| __loam wrote:
| Might be nice to actually ask for permission here before
| scraping everything and feeding it into a training set.
| doctorpangloss wrote:
| There are a lot of sprites to work with. As I'm sure you're
| aware, there are artists known for making animations, like
| Pedro Medeiros; spriters-resource.com has material from
| thousands of games; you can buy the Unity Asset Store, itch.io
| and stock art pixel art assets; and you can use DevX Tools Pro
| to extract assets from hundreds of 2D pixel art Unity games.
| All told, there are maybe 100,000-1m examples of high quality
| pixel art you can scrape. It is additionally possible that it
| already exists in the major crawls and needs to be labeled
| better.
|
| A few people have tried training on sprite sheets and emitting
| them directly, and it did not work.
|
| A few people have been working specifically on walking cycles,
| and it has a lot of limitations.
|
| In my specific experience with other bespoke pixel art models,
| if you ask for a "knight," you're going to get a lot of the
| same looking knight. Fine-tuning will unlearn other concepts
| that are not represented in your dataset. LORAs have not been
| observed to work well for pixel art. You can try the Astropixel
| model, the highest quality in my opinion, for prototyping.
|
| Part of this is you're really observing how powerful
| ControlNet, T2I-Adapters and LORAs are and you may have the
| expectation that something else you, a layperson, can do will
| be similarly powerful. Your thing is really cool. But is there
| some easy trick without doing all this science, for animation?
| No. Those are really big scientific breakthroughs, and with all
| the attention on video - maybe 100-1,000 academic and industry
| teams working on it - there still hasn't been something super
| robust for animation that uses LDMs. The most coherent video is
| happening in with NeRF, and a layperson isn't going to make
| that coherent with pixel art. Your best bet is to wait. That
| said, I'm sure people are going to link here to some great
| hand-processed LDM videos, and maybe there's a pipeline with
| hand artwork a layperson can do today that would work well.
| nightski wrote:
| It's not a full dataset per say, but you might be interested in
| Sebastian Starke's DeepPhase work [1].
|
| [1] https://youtu.be/wAbLsRymXe4
|
| [2] https://github.com/sebastianstarke/AI4Animation
| Ireallyapart wrote:
| This is good for procedural generated 2D worlds. Think Hollow
| Knight, but expansive across infinite environments. Just randomly
| generate the control image and have the LLM generate the theme.
| Combine that with LLM generated lore and the possibilities are
| unlimited.
|
| We have the technology to do this right now.
| 6510 wrote:
| think random events that change the entire world
| robobenjie wrote:
| Yeah, in the map editor there is, in fact a random button that
| generates. I havn't gotten around to making sure that the
| random level is playable (and about 1 in 4 have unreachable
| areas) but that wouldn't be that hard to add. (I've been
| focused on the creative aspect of creating your own levels
| because right now that part is more fun).
| zokier wrote:
| I have far more simpler (I imagine?) case already in mind, from
| the recent Cities discussion thread:
|
| >> I would've expected at least a not grid-based zoning so that
| buildings on curves look more natural. All these empty pieces
| of land in between buildings look really bad and kind of force
| us to make grid cities. And that is not even an innovation, it
| was already present in the SimCity series. But some
| procedurally generated buildings for smooth corners and
| connecting buildings would be nice.
|
| > Its hard to make assets that would work with every curve.
| When you see screenshots of nice cities like this, people are
| using mods to hand place assets with them clipping into each
| other to make a unified wall of buildings along the curve or
| corner.
|
| https://news.ycombinator.com/item?id=36294742
|
| ML generated building configurations for city builder games.
| Readily adaptable to any shape, and as a bonus can break up
| excessive repetition a bit. If you want to be ambitious, train
| a model on real-world aerial photos.
| mywittyname wrote:
| Would this require ML? I would think this could be
| accomplished with some rudimentary procedural generation.
| Divide the large space up into building sized plots, then
| generate a building to fit the space.
| arijo wrote:
| Any books, articles you could suggest?
| Ireallyapart wrote:
| There's no books man. This stuff is too new. But all the
| components are in place and exist.
|
| This guy just demonstrated what's required to generate a
| theme, and it's not far off from extrapolating further from
| that in using a LLM to generate Lore and just some random
| maze generator to create the base control image.
| baq wrote:
| Yeah. Show this to the Dwarf Fortress team. I hope they're
| already working on it anyway.
| Ireallyapart wrote:
| Doubtful. The developer is really old school.
| Additionally he's already spent years on integrating his
| huge lore generator into the fabric of the game. An LLM
| will throw a wrench into the whole process.
| tweetle_beetle wrote:
| I feel like Jump 'n Bump probably has a special place in the
| hearts of people who had access to the internet at a particular
| time. The internet was available, but multiplayer online gaming
| was still out of reach for many with it - there was an amazing
| niche of fairly polished indie local multiplayer games. Imagine
| being told while playing it back then what would be possible a
| few decades later.
|
| (see also https://github.com/midzer/jumpnbump)
|
| Now need someone to do other games of that time, place and genre:
| Tremor 3, C-Dogs, etc.
| ionwake wrote:
| amazing advances this year. Remember the guy who created the 2d
| platformer thats based on time, what was it called again? He
| spent around $100k+ just for the art, which I am pretty sure was
| a huge expenditure for him, with this software he could have done
| it virtually for free without much artistic talent at all.
| brucethemoose2 wrote:
| Would that even be legal for a commercial game?
|
| Personal use, a mod, or a free experiment is one thing, but a
| shipping game is a different can of worms.
| drorco wrote:
| At the quality of the current output, I think players still
| easily differentiate between AI generated art and hand-created
| art. Maybe in future versions this will be less noticeable.
|
| As a game dev, I think at this stage AI can be a helpful
| utility, but it does not replace a designer's touch for
| professionally looking games.
| AlecSchueler wrote:
| The AI stuff is a style though. I'm seeing it happening now
| in the art world, where the quirks of the model become part
| of the appeal of the work. Won't be long until a game with
| good enough mechanics comes along and blows up I think.
| __loam wrote:
| For me the model "quirks", eg. all the mistakes they make,
| are a huge turnoff.
| fnordpiglet wrote:
| The sad thing though is that $100k on art wasn't wasted. It
| allowed an artist to make art and a living. I'm down the with
| tech and I've even written a typing game that generates
| Minecraft stories with a LLM and imagines them with stable
| diffusion for my daughter to learn typing with. But - my mom is
| an artist and she spent her career starving between sales and
| commissions. The fact the models have ingested their work and
| careers and can now replace them is sad.
|
| On the other hand, I've never been an artist myself. So I've
| never been able to make my game ideas come true until now. The
| world is much more open to me in a creative side that my
| mechanical skills prevented.
|
| Artists will continue to make art because it's a compulsion.
| But I wish we had a world that was less oriented towards
| rewarding meaningless toil and would at least allow our born
| artists, writers, and creators the chance to do their
| obsessions to our benefit. Especially as we move post scarcity,
| I hope we can build a WPA like entity - perhaps, in a crazy
| twist, funded by AI?
| adventured wrote:
| We're never moving post scarcity. There is no scenario where
| that outcome is going to occur. Logistics alone guarantee
| that won't happen. Existence by its very nature is bound up
| in scarcity. There will always be various critical elements
| that provide a scarcity restriction on humanity and the rest
| (the less scarce) will always collide against that scarcity
| bottleneck.
|
| The artists can now create on the tech side too, courtesy of
| inbound GPT-like LLMs. This isn't a one way street. The
| techies can craft art, the artists can craft tech.
|
| It's opening up enormous pathways whether you're a programmer
| or artist. The artist has to be willing to expand and take on
| more responsibilities, just as the techie does if they want
| to craft quality AI art for a game.
|
| With all the various game engines available now, we're not
| far away from being able to relatively easily have an LLM
| build nearly all the software side for you via prompting.
| From there you can bring whatever your strength is to
| customizing, implementing the game. Maybe you're good at
| ensuring high quality gameplay, maybe you're an artist that
| has an elite eye for how things should look, maybe you're a
| programmer and your game will be better optimized (and so
| on).
| fragmede wrote:
| > We're never moving post scarcity.
|
| We already _are_ with digital goods, society just hasn 't
| caught up yet. I can make essentially infinite many copies
| of, say, Braid, and give every person in the world with an
| Internet connection a copy of it for a couple thousand
| dollars by using Cloudflare unlimited bandwidth R2 and
| bittorrent. A couple thousand dollars is basically a
| rounding error in the scheme of things. As I am not
| Jonathan Blow, distributing Braid would be a violation of
| copyright law, but copyright law is just a social contract
| that we entered into to incentivize the creation of work.
| If Jonathan Blow were compensated for _every_ copy of Braid
| out there, I 'm sure he would be quite happy to be (even
| more) rich.
|
| So even in a post-digital-scarcity world, artists and
| programmers need to get paid, and so we have various DRM
| schemes, the first of which is the copyright system in the
| first place, but that works about as well as trying to make
| water not wet. Movies are leaked onto torrent sites like
| Rarbg (RIP) and people make copies all day long. libgen
| mirrors are still around despite the best efforts of the
| copyright regime. But let's be honest with ourselves,
| digital goods themselves are already post scarcity, we just
| haven't figured out how to incentivize the creation of
| works in our half-post-scarcity world and have no idea on
| how to move forwards.
|
| Alternate solutions are out there, but we have no
| experience as a society in upending large social contracts
| (like copyright). You can easily imagine a system where
| what's popular gets tracked, and money flow to the creators
| of the media that people are actually watching and
| consuming. It would be a more draconian system than the DRM
| we have right now, but on the other hand, if it promotes
| the arts, then maybe it's worth it.
| __loam wrote:
| It feels like a lot of people on the pro-AI side of this
| debate think making art is some kind of magical activity that
| is totally inaccessible to them. The reality is the barrier
| to entry is a pen and paper and 10 minutes a day. There are a
| ton of resources online to learn about how to produce art
| without surrendering all your creativity to the AI slot
| machine, that don't involve stealing 400 million images.
| flatline wrote:
| I could glibly say the same about programming but it's just
| not true. Anyone can learn to write a script to do some
| simple task, but they are not a programmer. The ability to
| go from a vision to a high quality output that accurately
| represents your vision requires years of work for most
| people, in any field.
| __loam wrote:
| And yet, so ready are we to discard those millennia of
| earned experience in favor of statistically generated
| sludge. Instead of glibly saying the same about
| programming, we glibly dismiss the role of the artist in
| making art. In another comment on this thread, someone
| talks about how Jon Blow could have saved $100,000 on art
| for Braid if he had used an ai system like this, perhaps
| not understanding that the art of Braid is a critical
| component of what makes it Braid.
|
| If Karma is real, games made with uninteresting AI art
| will fail as we generate a sea of low-effort sludge.
| GaggiX wrote:
| History has shown that with every new tool people are
| going to make some crap, but also some wonderful things.
| tltimeline2 wrote:
| Yeah- imagine paying people for their work.
| DeRock wrote:
| The game was called "Braid"
| bsenftner wrote:
| Love to see a write up on your Hugging Face Diffusers experience,
| setting that up, what your dev cycle & stack look like, if you're
| hosting that server on a GPU cloud instance or what. Those kind
| of details are very interesting.
| thih9 wrote:
| Does the 2d data like platforms and hit boxes still match the
| input entered by the human? If yes, I wouldn't say this is using
| AI for level editing, this seems using AI for level artwork
| generation. Impressive nonetheless, just different.
|
| HN's submission title ("Show HN: Stable Diffusion powered level
| editor for a 2D game") made me think of the former. Article title
| ("2D Platformer using Stable Diffusion for live level art
| creation") was more accurate to me.
| AlecSchueler wrote:
| Your quote says represent the levels not generate the levels
| right?
| porcc wrote:
| To get even less "structured" backgrounds you should try
| replacing the support backgrounds and the far backgrounds with a
| light bit of depth noise.
| tltimeline2 wrote:
| [flagged]
___________________________________________________________________
(page generated 2023-06-12 23:00 UTC)