[HN Gopher] Small neural network enables realistic rendering of ...
___________________________________________________________________
Small neural network enables realistic rendering of woven fabrics
in real-time
Author : wglb
Score : 151 points
Date : 2024-07-25 23:23 UTC (2 days ago)
(HTM) web link (techxplore.com)
(TXT) w3m dump (techxplore.com)
| dartos wrote:
| This is what I was expecting from AI in games
| ffsm8 wrote:
| Well, the holy grail in games for AI will always be ... AI.
|
| Like, actually really pseudo intelligent companions you can
| converse with. Not gonna happen in the foreseeable future, and
| the job markets are gonna be... Well... essentially gone once
| that exist, so it's probably for the best that this tech isn't
| _actually_ anywhere close to AGI
| fhd2 wrote:
| Didn't downvote you (and not sure why anyone would), but I
| still disagree: Aren't games ultimately all about interacting
| with other people?
|
| It's already possible to play with actual other people, and
| quite popular too. Sure, those guys might not be very in-
| character or prioritise you having fun over them, but there's
| genuine intelligent interaction.
|
| The other way in which games make us interact with other
| humans is by experiencing things designed by them. Story,
| dialogues, game design, atmosphere etc. I think we might see
| more ad-hoc generated world/story, but I think many people
| would want to experience the _same_ game, in some way, being
| social animals.
|
| I'm not sure if and in what way generative AI or even AGI is
| going to disrupt the game industry. I can certainly picture
| the jobs of many people changing, from artists to voice
| actors.
|
| But I'm not sure I see a holy gray here. Lots of interesting
| novelties that I'm sure will be tried, but not really a
| fundamental change to why we play games and consequently,
| what games are like.
| throwthrowuknow wrote:
| There are plenty of games where the main feature is
| emergent gameplay resulting from a world simulator and the
| interplay of basic or complex rules. Dwarf Fortress is the
| OG here but also Sim City and Elite. Single player
| experiences are a huge segment and these will get better
| with LLM integration for world building and NPC agents.
| Fine tuned or custom models that operate on some basic
| state and rules could also be explored. You could train
| BERT family models to choose interesting state changes or
| actions based on the current game state for simple things
| like a dog NPC that would seem more intelligent by choosing
| the next action it should take e.g. deciding to chase a
| squirrel or growl at a new NPC you encountered based on its
| awareness of the context rather than following some rule
| system plus RNG. You could get rid of a lot of the stupid
| seeming game AI behaviour.
| fhd2 wrote:
| True, though in my mind I put these things into my "lots
| of interesting novelties" category. Everything in games
| (including the physics engine!) is typically fine-tuned
| to provide fun, more than realism. The same is currently
| true for AI (especially enemy AI), and I believe it'll
| continue to be that way.
|
| Good point on the procedural aspect, Minecraft and
| similar games came to mind when I wrote my first post.
| I'd argue they are still fundamentally a designed,
| shareable experience, and that's my main point really. In
| Minecraft, you can't talk about the world much with
| someone else since it's generated, but the mechanics,
| materials, recipes etc. are the same, and provide
| exchange between humans playing the same game.
|
| I believe we'll even see some successful AI heavy games
| in the near future, but I also believe we'll cycle back
| to games being, first and foremost, designed upfront. I
| might be wrong of course and we might see some kind of
| holodeck 100% custom game experience become dominant.
| It's just my hunch that we won't, even if it's feasible.
| Thinking as far back as I can, people always seem to have
| wanted two things from games: Fun, and connection to
| other people.
| MonortYp wrote:
| I'd love to see a real improvement in games where the NPCs
| think further than a few if statements. If AI could track
| context of conversations and include player actions such as
| quests, relationships, deaths etc to influence decisions or
| outcomes it would add lots of enjoyment for me.
|
| For me it's either Stardew Valley or Paradox type games for
| my single player itch or in the past I've ventured off to
| MMORPGs like Eve or Wurm and play in "single player" mode.
|
| MMORPGs have a greater sense of world change as players can
| carve mountains and build estates while slaying champions and
| such in Wurm online and you can stick to your plot of land
| and have nothing to do with it while enjoying the benefits.
| This doesn't exist in single player games but it could and
| should.
|
| Although I don't agree AI will be taking jobs but it will
| just be another tool used in the industry. We need to embrace
| change and use these new tools to make better content and not
| just assume jobs will be lost.
| dartos wrote:
| It depends on the game.
|
| That'd be great for an RPG game, but if I'm playing a action
| adventure game or a style fighter, I generally care about how
| cool it looks and how smooth it plays than I do about the
| intelligence of the side characters.
| Modified3019 wrote:
| We'll have made it as a species when the harvester in C&C no
| longer gets stuck or wanders through the enemy base.
|
| In all seriousness though, I think RTS games (I only do single
| player) would be more fun for me personally if I could leave
| the economy/build order stuff to a reasonable AI and focus
| instead on military unit control. I'm just not someone that can
| handle all of that at once.
|
| I think the potential for an enemy tailored to challenging and
| occasionally surprising me without simply overwhelming me would
| also be great.
| Epa095 wrote:
| What about RTS games specialising on the military part? I
| haven't played them myself, but I think the Company of Heroes
| series is like this, light on base building and focusing more
| on the military aspect.
| dartos wrote:
| Totally off topic, but RTS is a weird split genre.
|
| I really like the economy / base building part, but care less
| for unit control. I'm more of a macro player than micro.
|
| I think a lot of players fall on one side or the other.
| thebruce87m wrote:
| Did you ever play settlers? I loved building the towns.
| dartos wrote:
| Settlers of catan?
|
| Weirdly enough, I've never played it
| MaKey wrote:
| Probably this one or one of its successors:
| https://en.wikipedia.org/wiki/The_Settlers_II
| thebruce87m wrote:
| That's the one
| fragmede wrote:
| thankfully, the genre has been dissected so there are games
| like revolution that focus on the warring armies aspect,
| and games like sim city that focus on the building aspect.
| and then there are factory games, which is a whole genre
| unto itself.
| seedboot wrote:
| The factory must grow.
| jorvi wrote:
| With more and more fixes for "hallucinations", and with smaller
| models getting more capable (or bigger models getting more
| efficient, depending on how you look at it), I have a feeling
| that eventually it'll be cheaper to render things neurally
| rather than rasterized.
|
| Imagine if you could describe what is happening on the screen,
| including the state of the world and everything, with a really
| long prompt. You dynamically alter that prompt continuously
| based on inputs received.
|
| There will be a point in the future where the prompt will
| render with 99% consistency. You'd probably still need some
| sort of barebones state server for things like hitboxes and to
| prevent cumulative divergence.
|
| Still a ways away though :)
| johanvts wrote:
| > works by encoding the patterns and parameters of fabrics as a
| small latent vector, which can later be interpreted by a decoder
| to produce realistic representations of various fabrics.
|
| > Woven fabric patterns [...] tend to be regular and repetitive.
| Chen and his colleagues set out to leverage their repetitive
| nature
|
| Is this "just" using the NN to approximately compress and
| decompress things? That doesn't seem to explain the performance
| gains, unless some work can be done in the reduced space?
| Anotheroneagain wrote:
| I believe that results like these show that our algorithms are
| woefully inadequate, and something much faster has to exist, if
| a NN can do so much better with all the overhead.
| Arwill wrote:
| Light/material calculation in shaders requires
| exponentiation, which is a costly operation. If a clever
| lookup table can get around that, it sure can be faster.
| magicalhippo wrote:
| Using a lookup table[1] or other approximations like
| wavelets[2] for complex BRDFs is a well-known strategy.
|
| [1]: https://github.com/HectorMF/BRDFGenerator
|
| [2]: https://hal.science/hal-01518547/document
| esperent wrote:
| This isn't efficient at all. It requires a 3090 to get 60fps
| - and presumably it requires all the GPU's power to do that
| for the simple scene in the article: a couple of sheets and
| cushions on a sofa.
|
| Possibly it's RAM rather than compute limited but still. You
| couldn't use that in a game. Existing techniques like PBD or
| verlet integration would be far more efficient.
|
| Maybe on future devices with separate neural processor units
| this could be useful. But even then I bet those same chips
| could run standard real-time cloth algorithms really really
| fast too.
|
| Edit: I downloaded their video and there's no movement at
| all, only still images. Maybe I misunderstood the purpose of
| the technique. But I re-read the article and it certainly
| _seems_ like they 're claiming to do full simulation, not
| just static scenes.
|
| That scene with the sofa isn't even getting 60fps! It kinda
| feels like a disingenuous article after watching the video.
| zokier wrote:
| I don't see how something like PBD helps at all with
| rendering.. are you thinking that this work relates somehow
| to physics simulation? Considering that the result is
| pretty much just a shader, I imagine it works just as well
| on animated scenes as static ones.
| brookst wrote:
| Yeah it's kind of like a FPGA implementation: it proves the
| concept but is by definition not optimized for efficiency.
| There must be a way to go from the weights to an algorithm.
| jacoblambda wrote:
| > As for NeuMIP, it can produce results with a higher quality
| than ours in terms of MSE, at the cost of a higher storage and
| per-material training.
|
| > Furthermore, our network can be applied to three typical
| types of woven fabrics once trained, but NeuMIP have to be
| trained per material which takes a long time
|
| > Specifically, we introduce a simple encoder-decoder
| structure, where the encoder compresses the fabric pattern and
| other parameters into a material latent vector, and the decoder
| interprets the material latent vector with a spatial fusion
| component and a small angular decoder. Thanks to the
| lightweight encoder and decoder, our network is able to achieve
| real-time rendering and editing. Meanwhile, our network only
| occupies a small storage of 5 MB, even for scenes with multiple
| fabric materials.
|
| Yeah. This is really just a simple solution that can be quickly
| trained and that compresses the problem space better at the
| cost of some fidelity.
| the8472 wrote:
| NNs/learning/function approximation _are_ compression.
| johanvts wrote:
| I think more often it's about translation. From image to
| classification ex. with no hope of mapping back.
| nightski wrote:
| Yes, but not solely. They are also generalization.
| plorg wrote:
| If you let your eyes roll to the back of your head and let
| the spice take over generalization and compression are two
| sides of the same coin.
| RandomThoughts3 wrote:
| Everything NN-related is compress and decompress if you squint
| hard enough.
|
| You are trying to map inputs to outputs in an efficient way
| space-wise or time-wise. The NN is doing something which allows
| it to skip complex calculations at the expense of fidelity in a
| way which is not far from a more advanced look up table.
|
| The fact that what it does is so opaque remains the main flaw
| of it.
| loufe wrote:
| I'm curious if the neural net ran wide and used the majority of
| the VRAM to achieve 60fps, or if it ran narrow and had more of a
| long-thread-length dependency.
| sgt101 wrote:
| Resnet + 16 layers isn't GPT4 but it's not small.
|
| I think that the 60fps on a consumer grade card is the win, but
| that says a lot about the power of consumer grade cards in 2024.
| ahoka wrote:
| Only 60 fps on a 3090 is not exactly real time in practice.
| johann8384 wrote:
| Why? 60fps is twice the framerate of common video streams. It
| seems like it's pretty realtime to me.
| jsheard wrote:
| They're spending about 14ms on inferencing and 3ms on
| everything else, which technically is about 60fps, but in
| any real world application you're probably not going to be
| able to blow almost your entire GPU compute budget on
| shading a few pieces of cloth.
| DistractionRect wrote:
| To their point, 30 fps is commonly acceptable, in which
| case you're spending 7-14ms on this, and 26-19ms on
| everything else. You can also use frame gen cheat 30 fps
| back up to 60 if one is so inclined.
| refulgentis wrote:
| > 30 fps is commonly acceptable
|
| Interesting to hear that, consumer market has been well
| past 30 fps for years, to my knowledge. For example:
|
| - Phones are widely at 120 fps as of a couple years ago.
|
| - Gamers have been targeting 240 fps for years and years,
| since 2018?
|
| - On the low end, ex. Steam Deck requires you manually
| opt-in if you're happy with less than 60.
|
| I'm curious for some groups you're aware of that commonly
| accept 30. (note: not a trick question, ex. we know
| movies are 24 fps, and professional 3D renderers are
| happy with not-real-time, but we're thinking about 3D
| rendering for games)
| kennyadam wrote:
| It may be the case that high end PCs with the latest
| expensive graphics cards can run modern games like
| cyberpunk at 240fps using things like DLSS, but console
| games regularly run at 30fps in "quality" modes and 60fps
| in "performance" modes. 30fps is absolutely an acceptable
| frame rate for the most people. If we didn't have Digital
| Foundry and the like nitpicking dropped frames and
| counting pixels for dynamic resolution, I think even less
| people would care about high framerates, as long as the
| game is fun.
| Vecr wrote:
| The end-to-end latency is horrible with 30fps. Move your
| mouse and you can feel the lag. In theory you can over-
| render the sides of the screen and move the rendered
| image around with the mouse but almost no one does that.
| refulgentis wrote:
| This isn't true, though.
|
| PS5 came out 4 years ago (November 2020) and has 4 games
| at 30 fps. Of 3140.[1]
|
| PS5 and Series X both have an infamous problem with ports
| from older generations coming over at 30fps, because they
| were render-locked to that in the previous console
| generation, and no upgrade work was done during the port.
|
| Beyond that, 30 fps comes about via opting into it via
| choosing raytracing at 4K.
|
| It's a massive reach to describe 30 fps as commonly
| accepted. And that's without considering the context:
| it's being used to argue for the usefulness of a cloth
| simulation neural network that takes 18 ms to render on a
| 3090, not 2020 consoles.
|
| Moreover, those screenshots aren't showing a game with
| some clothing simulation. They're showing a nearly-empty
| render except with some cloth simulation.
|
| I think steelmanning is good and important for
| conversation, but its hard to swallow here. Blithe while
| taking too many liberties that are likely to shade rather
| than shed light.
|
| [1] https://www.reddit.com/r/PS5/comments/1biskb4/there_a
| re_3140...
| DistractionRect wrote:
| All fair points. Phones have 120fps displays, but
| obviously can't run much at that rate. Some web pages,
| basic note taking, etc. But pretty much anything 3d is
| 30-60fps.
|
| Consoles, hand helds usually aim for 30+
|
| Real time rendering, video streaming etc, are 24-30.
|
| If the nit is my phrasing, let me rephrase, the point is
| that it's usable in an interactive/real-time application.
| It's perhaps at the lower bar, but 30 fps is far from a
| rarity. Though not always ideal, it's certainly
| acceptable.
| astrange wrote:
| > - Phones are widely at 120 fps as of a couple years
| ago.
|
| They're only briefly at 120fps during important things
| like full screen scrolling. Phones are battery powered
| and passively cooled, so rendering at 120fps would be
| very wasteful even if you can do it.
|
| The important thing about 120fps is that a lot of things
| are factors of it; you can't play a 24fps movie on a
| 60fps screen.
| LoganDark wrote:
| 30 fps should not be acceptable given a 4090. Remember,
| most people have cards that are barely 15% the
| performance of a 4090, or less. The fact that almost all
| modern titles target the 4090 is a huge problem; only a
| very small number like Roblox actually understand how to
| optimize.
| zokier wrote:
| There was also another neural cloth rendering paper released
| recently, focusing on more micro-scale details (and not being
| real-time) https://doi.org/10.1111/cgf.15156
| brcmthrowaway wrote:
| How would you a neural network ever be faster than a closed form
| solution for fabric rendering?
|
| Neural network is the big hammer.
| speps wrote:
| Paper link: https://arxiv.org/abs/2406.17782
___________________________________________________________________
(page generated 2024-07-28 23:06 UTC)