[HN Gopher] Trellis - 3D mesh generative model
___________________________________________________________________
Trellis - 3D mesh generative model
Author : tarr11
Score : 425 points
Date : 2024-12-09 19:19 UTC (1 days ago)
(HTM) web link (trellis3d.github.io)
(TXT) w3m dump (trellis3d.github.io)
| tarr11 wrote:
| Saw this submitted a few days ago [0], but it's a very impressive
| demo and would like to see it get discussed here.
|
| [0] https://news.ycombinator.com/item?id=42342557
| vessenes wrote:
| Um, wowowow. This is a huge leap forward in 3D Asset generation.
| I'm downloading it right now. This feels like a combination of
| ergonomics - pulling together multiple somewhat janky workflows -
| better training data - and solid architecture innovation. Thanks
| Microsoft, seriously. I'm looking forward to playing with this.
| tmilard wrote:
| It's a game changer
| airstrike wrote:
| Wow, these look amazing. I'm a layman, but I think this is what
| everyone's been thinking about ever since the first NeRF demos?
|
| EDIT: I went looking for those threads and found my own comment
| wishing for this 5 years ago
| https://news.ycombinator.com/item?id=22642628
|
| The next step is to automatically add "nodes" to the 3D images
| where the model can pivot, rotate and whatnot and then boom, you
| have on-demand animated, interactive content.
|
| Feed it some childhood photos and recreate your memories. Add an
| audio sample from a loved one and have them speak to you. Drop
| into VR with noise-cancelling headphones for extra immersion.
| Coming soon! Click here to join the "Surrender Reality" waitlist
| movedx wrote:
| I'm interested to see how this affects 3D artists in game
| development studios. Will those studios use these tools and
| keep their artists, allowing them to push out more and more
| content, faster, and easier, or just keep a bunch of artists
| around, drop the other 80% of them, and use the tools to
| _replace_ those artists?
| pfisch wrote:
| Last time I looked at these the lighting was in the textures,
| also the meshes were asymmetrical and insane. Not usable by a
| game dev studio.
| bilbo0s wrote:
| These models will get better. And in answer to the previous
| question, of course they will get rid of artists. They will
| keep just enough to do the work necessary with the help of
| generative models, and let go of the rest.
|
| The rest of the artists are not dumb, and they have a lot
| of talent. I'm sure many of them will use the same models
| to come up with their own games.
|
| How this whole thing will play out is anyone's guess. Long
| term, I'm hoping for new _types_ of jobs to come into
| being. But who can say?
| 8n4vidtmkvmk wrote:
| I hope they use it to create a bigger variety of assets. In
| lots of large games you start to notice where they've reused
| assets that could have used some more variation.
| DecoySalamander wrote:
| Studios don't have that many artists already - most of the
| "heavy lifting" is outsourced to asset production companies
| in China. I can see a future where these are replaced by AI
| and the main job of the in-house artists is to fix problems
| with the generated output.
| TacticalCoder wrote:
| > ... and then boom, you have on-demand animated, interactive
| content.
|
| And in addition to that it's also useful for rending still
| pictures. 2D generated images by AI so far have incorrect
| lighting and many errors. Once it's a 3D scene rendered by
| something like Blender (which is free), it's going to look
| flawless: lighting is going to be correct (and configurable)
| and all the little details that are wrong are going to be
| easily fixed.
|
| We already have insanely powerful tools and apparently from
| here it's only going to get way more powerful real quick.
| sangnoir wrote:
| > The next step is to automatically add "nodes" to the 3D
| images where the model can pivot, rotate and whatnot and then
| boom, you have on-demand animated, interactive content
|
| My gut says a 3D engine + this would be a superior solution to
| the current approach of rendering rasterized video directly
| from the latents (coincidentally, Sora got released today).
|
| It may not be tractable to train a network to rig and animate
| meshes, as well as setting up an entire scene to be a "digital
| twin" of random videos, bit I imagine such a set up would have
| finer-grained control over the created video while keeping
| everything else in it the unchanged
| Kaijo wrote:
| >The next step is to automatically add "nodes" to the 3D images
| where the model can pivot, rotate and whatnot and then boom,
| you have on-demand animated, interactive content.
|
| The next step is to generate models with higher quality mesh
| topology that allows animation and editing without breaking the
| mesh. I've done a lot of retopologizing and if I (or AI) were
| to rig these models as-is there would be all kinds of shading
| and deformation issues. Even without animating they are
| glaringly triangulated up close. But I suspect really high
| quality 3D asset generation is just around the corner because
| all you'd have to do is join up the approach seen here with AI
| quad re-meshing based on estimated direction fields and feature
| detection, which is also getting scarily good.
| taikon wrote:
| Anywhere you'd recommend a hobbyist can learn more about
| remeshing or feature detection?
| xi_design wrote:
| https://blog.yarsalabs.com/blender-topology-guide/
|
| https://cgcookie.com/posts/guide-to-clean-topology
| Kaijo wrote:
| This is a technical paper, but it has a quite
| conversational abstract and introduction that is easy
| enough to follow if you have some experience with mesh
| modelling: https://www.graphics.rwth-
| aachen.de/media/papers/337/learnin...
| woctordho wrote:
| At this point maybe meshes are not the best representation
| for animation and editing. We can just use latents of neural
| networks
| vunderba wrote:
| > _" The next step is to automatically add "nodes" to the 3D
| images where the model can pivot, rotate and whatnot and then
| boom, you have on-demand animated, interactive content."_
|
| Well not really sure what you're talking about here wrt nodes
| (adding in arbitrary rotation/zoom sounds great in theory if
| all you're looking for is a lazy susan or spinning exorcist
| heads), but the next steps will likely be more around ensuring
| _sane symmetrical_ topologies, better UV maps, and
| automatically building rigging (FK /IK) to allow for easy
| animation.
| airstrike wrote:
| I meant rigging, but I'm a layman so I don't know the
| terminology. But yes, symmetrical models with simpler meshes
| and better UV maps would definitely be needed to make it work
| as I'm imagining it
| horsawlarway wrote:
| Mixamo is pretty close to this (Auto-rigging tool from
| Adobe). https://www.mixamo.com/#/
|
| It's limited to mostly human shapes, but I've personally
| used it in combination with the 3d-pack in comfyUI to
| generate 3d models and rig them, starting from a text
| prompt.
|
| Trellis looks like a more capable model generation tool
| than TripoSR and marching cubes, which is what I was doing
| in Comfy: https://github.com/flowtyone/ComfyUI-Flowty-
| TripoSR - It worked but models ended up having a slightly
| "melted wax" appearance.
|
| This type of flow is definitely already here for low
| quality assets (think mobile games). I'm excited to go play
| around with Trellis, looks like a significant bump up in
| quality.
| airstrike wrote:
| Super cool stuff! Thanks for sharing. It's exciting to
| see how fast this space is developing.
|
| Hopefully one day we get some open source alternative to
| Mixamo that plays nicely with the rest of the open
| ecosystem.
| baq wrote:
| As a newly minted 3d printer owner next step is accurate
| dimensions, material and nozzle diameter awareness ;) then some
| CAD-like support where you can specify constraints on...
| things?
| spaceships wrote:
| This isn't anything parametric but I 3d printed a model from
| Trellis https://news.ycombinator.com/item?id=42375951
| kfarr wrote:
| Hey this is actually really good, one of the best image to 3D
| models I've seen so far. From an image of a bollard I was able to
| generate a pretty good mesh. The GLB as generated was 1.2MB,
| after compression I got it down to 35kb (!) and the fidelity is
| good enough for most projects.
| summarity wrote:
| Pretty amazing. Scale consistency is an issue looking at the
| scene examples, but for one-off or background assets, pretty
| neat!
| robinduckett wrote:
| I can see the potential, but the images I give it must be very
| far outside of its training because all it generates are weird
| flat planes
| Etherlord87 wrote:
| Yea another miracle tool... Until you test it.
| andybak wrote:
| I've been testing it and it's the best one I've tried so far.
|
| It does have failure cases but the success rate is fairly
| high and when it works, the resulting meshes are reasonably
| usable (maybe not to game dev production standards - but that
| still leaves plenty of other use cases)
| d0100 wrote:
| I just asked for low poly plant on Adobe Firefly, then
| uploaded it to Trellis
|
| The result was pretty good for the mesh, at least 100x faster
| than having to do it from scratch
| tosmatos wrote:
| I managed to get it to work with images that were looking down
| on the character / thing, like in an isometric game. Using any
| image that was facing the front was giving flat results
| gsuuon wrote:
| The ability to edit parts of the model after the fact using
| prompts is pretty incredible
| wkat4242 wrote:
| This is great! Love it, 3D visualising can really benefit from
| the use of Generative AI.
| LarsDu88 wrote:
| Very cool. I wonder if the fact that it featurizes 3d objects
| using voxels can be memory intensive considering only the surface
| voxels are actually relevant for 3d structure makin'
| DaiPlusPlus wrote:
| Octrees are memory-efficient for this kind of thing.
| LarsDu88 wrote:
| Yeah I guess they probably did the standard thing... whatever
| minecraft uses.
| ThrowawayTestr wrote:
| Think the server is overloaded, I keep getting an error
| gunalx wrote:
| Worked pretty terrible at trying to model the nix snowflake.
|
| Guess its more trained on natural and biologic structures and
| textrues, rather than more structural or symetric data.
| gmueckl wrote:
| Looks like the generated wireframes are still terrible and would
| need to be redone by hand. So no need for artists to worry about
| their job security just yet.
| AnotherGoodName wrote:
| Question: How much do crappy wireframes actually matter?
|
| From a performance pov I've imported ridiculous meshes not
| intended for games into unreal. It worked.
|
| I can see this being ok for anything other than triple A games
| right?
| wg0 wrote:
| They matter as much as good quality code matters. Even more
| so actually.
| vunderba wrote:
| Three things:
|
| 1. Unless we're strictly talking about static meshes (such as
| for scenery), building out the rigging/animation for garbage
| topologies is going to be absolute hell.
|
| 2. Your performance POV was for _ONE_ mesh, but if your quad
| /triangle count is 10x higher than it needs to be, how well
| is it going to scale particularly on low-end gaming
| platforms?
|
| 3. Lighting/shadows/reflections can produce some pretty UGLY
| effects when your surface polygons are a mess.
| hi_hi wrote:
| I tried an image of a F-117 stealth bomber from wikipedia. The
| output was a complete fail, to the point where I have no idea how
| they managed to generate the examples on their project page. The
| basic silhouette was completely inaccurate.
|
| I was hoping you could upload several images from different
| angles to help it, but that doesn't appear to be a feature.
| regularfry wrote:
| The F117 is _weird_. Unless you already know what it looks
| like, any single view from any particular angle is quite hard
| even for a human to extrapolate from. If it wasn 't in its
| dataset then I can forgive it that, particularly because the
| angular nature of it means that it could easily be tripped up
| into thinking it's not looking at an aircraft at all.
|
| I'm not saying anything about the quality of the model here -
| just that the F117 is almost certainly going to be an unfair
| test.
| LarsDu88 wrote:
| It's interesting how Alphafold... which is a 3d generative model
| from 1d protein sequences is all fancy and complicated with its
| internal data representation in comparison to this paper which
| basically just voxelizes the input data and takes a bunch of
| pictures from various angles to build its training set.
| 9cb14c1ec0 wrote:
| It's not perfect, but significantly better than most that I've
| tried. Every time I've tried a 3d model generator up to this
| point, the result was unbelievably bad. This time it was medium
| good. All, give me a file format I can drop right into Orca
| Slicer.
| vunderba wrote:
| Tried it with a few simple images - not trying to deliberately
| confuse it. Seems to handle angular objects better (buildings,
| ships, etc.) as opposed to something more biological (lot of
| animal failures).
|
| I haven't cracked any of these open in Blender to look at the
| topology so they could be (and probably are) an absolute mess, so
| it's likely that attempting to make changes to the model may
| prove to be more trouble than it's worth, but still neat to play
| around with.
|
| Images / Resulting models:
|
| https://imgur.com/a/eSWJhMj
| tosmatos wrote:
| I've just put one in blender, indeed the topology is all over
| the place, but only triangles at least. Seems really hard to
| edit it in a 3D program after the fact because the vertices are
| placed seemingly at random
| yodon wrote:
| Nice work from Microsoft Research, including it being open
| source, and with open sourcing of the training dataset to come
| (which I suspect will actually be the most valuable contribution
| long term)
| jjcm wrote:
| I'm impressed. I used layer diffusion to make this low poly
| airship:
| https://image.non.io/b3f843be-b1b4-468a-a0ec-9d58b191beee.we...
|
| Which resulted in this: https://video.non.io/video-2732101706.mp4
|
| Honestly not bad at all. Getting to the point of being able to
| use this as game assets.
| dusted wrote:
| Wow, that's amazing.. I think this is the first time I've felt
| sick to my stomach watching AI generated content.. A sadness rush
| over me. Because these look so good, like every soul-less super-
| high quality shovel-ware asset ever made.
|
| I'm not putting it down, it truly is an amazing achievement, and
| it feels like it marks the end of hand-made assets. I don't even
| feel bad for the artists, I just feel bad for myself, because I
| want things made by people, for the inherent property that they
| were made by people. This is the same reason I don't care for
| procedurally generated games, I want to walk worlds that arose in
| the minds of others, not just worlds, if I wanted a procedually
| generated world that just exist for no reason in particular, I'd
| walk outside..
|
| I don't want content, I don't just want stuff to sift through, I
| want pieces of art, made by my peers with their own visions,
| ideas, values, insights and personalities. They don't need to
| look that good, they just need to have been made with purpose.
| qwertox wrote:
| > if I wanted a procedually generated world that just exist for
| no reason in particular, I'd walk outside..
|
| I don't know why, but this comment reminded me of an experience
| I had a few years ago, when I started exercising outdoors. I
| rarely went outside prior to that and stayed in a relatively
| dark room.
|
| One day I looked at the sky and thought: Wow, these clouds do
| look like the ones in video games, thinking of Horizon and
| Assassins Creed. This just pertaining the comment about the
| "procedurally generated" outside world.
|
| While looking at the assets I also felt a bit of sadness. I was
| looking at the "Two-story brick house with red roof and fence."
| and was then thinking about how it reminded me of the three.js
| animation/keyframes example [0].
|
| I asked myself if we will lose something very valuable. The
| three.js example was hand-crafted by persons, a real intention
| behind every choice made, while with Trellis it's just "poof,
| there it is", an amalgamation of all work found in the internet
| and possibly in games.
|
| Some value will be lost through AI, but this makes handcrafted
| content even more valuable. The question is, if we will really
| value this enough for it to be sustainable for the artists.
|
| [0] https://threejs.org/examples/#webgl_animation_keyframes
| kurokikaze wrote:
| It's a really good prototyping tool for those who cannot do 3D
| assets. Like visual scripting opened up the game
| development/modding for those not really familiar with the
| prigramming (Unreal Blueprints for example). So yeah, I'm okay
| with models I can throw into my prototypes without learning
| Blender/Maya/whatever. Sure, it may look uneven and strange but
| at least it's content.
| Kiro wrote:
| > I want to walk worlds that arose in the minds of others, not
| just worlds
|
| A significant portion of game developers hate level design and
| the only reason they don't do procedural is because it's hard,
| so they are forced to build hand-crafted worlds. I'm one of
| those and I would find it pretty hilarious if anyone played my
| game thinking the levels "arose" in my mind, like I'm some kind
| of profound artist. I take great pride in other aspects of game
| development, but my level design is not one of those.
| dusted wrote:
| > A significant portion of game developers hate level design
|
| And that shows, that really, really shows :)
|
| Now they get to make even more of the soul-less trash in
| shorter time..
|
| I'm not putting this tool down, it's an amazing technical
| achievement, and the results are absolutely mind-blowing,
| but, to me, it is what it is, and it's just like, my opinion,
| dude, not some statement of absolute truth.
|
| You hating level design and wishing you didn't have to do it
| at all has absolutely no bearing on my wanting games where
| the assets are made by hand.
|
| Conversely,me not wanting products made by people who don't
| like making them, should have absolutely no influence on you.
| I don't care if your passion is some other field of game
| creation, go do that, and have someone who enjoys level
| design do the levels, if you can't, well, then I guess you'll
| have to just accept that I might not want your game, and
| that's okay too, for both of us, you don't have to make
| something _I_ in particular like, and I don't have to accept
| your criteria for what I like.
|
| I want, as an inherent property of the stuff I consume, a few
| things whose merits can be argued endlessly about, but I'm
| not arguing about their merit, my opinion, my criteria for
| selection is inherent property itself.
|
| I'm not arguing whether there are any difference, I'm not
| arguing one is better than the other, I'm not arguing why one
| should be chosen over the other, I'm simply stating that
| among my selection criteria is that particular property of
| origin. It in itself, alone, nothing about it, just it.
|
| I want movies recorded on actual film, not movies that look
| like it, inherent property not its merit.
|
| I want books written by human minds, not books that "you
| can't prove was not".
|
| I want paintings painted by pencils held in human hands,
| guided by human hearts and minds, regardless of whether I am
| looking at a photograph of that painting, the property of
| it's origin is important to me, not its merits of lack
| thereof.
|
| So yeah, you can attack the merits of doing things one way or
| another all day long, but you don't get to say what I can an
| can not chose as my selection criteria.
| Kiro wrote:
| Relax, no one's policing your opinions. You're not under
| attack.
| pythonaut_16 wrote:
| Do you find it soulless when painters don't mix their own
| paints or weave and stretch their own canvases?
|
| Is it soulless when a sculptor doesn't source their own
| clay and marble directly from the earth?
|
| Or when a musician uses an instrument made by someone else,
| or a composer uses digital sounds recorded by someone else?
|
| There are many different forms of artistic expression and
| many rely on relatively mechanical and "soulless" work
| being done for us by someone or something else.
|
| Tools like this can open up new and creative world building
| options to people who previously didn't have access to 3D
| models. It increases the opportunities for creative
| expression rather than diminishing them.
| InDubioProRubio wrote:
| I poured ten years of my life into a open source game near
| nobody played. Screw all that handcrafted lovechild of
| artist-brain shit. My 2nd game was using store-assets and i
| was free to get things done.
| themaninthedark wrote:
| Do you mind sharing a link to the game?
| elif wrote:
| What if 2,000 people from your community collaborated on an art
| piece that spoke to their own personal experience?
|
| The artistic message would be disjointed, muddy, but
| indisputably an unmitigated human expression.
|
| So you put an artistic director in charge of curating and
| unifying the collective work.
|
| Still a human expression.
|
| This is what AI represents, and what the prompt writer
| represents.
|
| The data in LLM is undeniably human. Everything it "knows" is
| an extension of and exclusively composed of real human data.
|
| The prompt writer has a choice how much of his own human input
| to prioritize, and how much raw humanity to allow
| spontaneously.
| BenoitP wrote:
| > if I wanted a procedually generated world that just exist for
| no reason in particular, I'd walk outside..
|
| Oops, old.reddit.com/r/outside/ is leaking again
| danielvaughn wrote:
| If it makes you feel any better, the arena of human competition
| isn't going to fundamentally change just because of this
| technology, IMO. Yes, we'll see a flood of slop as it becomes
| more widely available. But the real artists, the ones who want
| to make things with purpose, will learn how to use this
| technology as a stepping stool towards something even greater.
|
| Look at people like Martin Nebelong - they're learning how to
| leverage AI without losing the human in the loop.
|
| https://x.com/martinnebelong?s=21&t=cTpE-rRbCiocUlN0VaSheQ
| jeffhuys wrote:
| Think of it this way: AAA games will now have to do something
| MORE than just "amazing graphics" in order to set themselves
| apart. Because if I'm honest, almost all of the newest games
| coming out is just the same gameplay + updated graphics.
|
| Well, guess what, very soon even I could do that. So what do
| these studios have in store to make us come back to them?
| nashashmi wrote:
| Very soon, the AI generator in your Xbox 2030 could upscale
| every object to incredible resolution.
|
| This shifts the attention to story development and away from
| graphic designers. It does not mean cheaper games. It means
| more successful indie games with fewer team members. It also
| means fewer games because as I understand it right now, the
| only reason new games are pumped out is to keep the larger
| industry perpetually employed and other more time consuming
| projects funded enough to be developed.
| plasticeagle wrote:
| Do not worry.
|
| Art is much more than pictures on your monitor. If you want
| pieces of art, made by your peers, visit your local galleries
| and buy it. I don't know who you are or where you live, but I'm
| willing to bet that where-ever it is those local galleries
| exist - and the artists that exhibit there would love to sell
| some of their work.
|
| And you can be sure that human-made art will remain, and be
| valued, because art is what humans love to make most of all.
| TinkersW wrote:
| It looks very nice in the examples, but I tested it by dropping a
| Chrono Trigger character art in, and it generated some 2D planes
| instead of a 3D mesh..so ugh.. what?
| tobyhinloopen wrote:
| This is incredible. I uploaded a picture of a cable with plug and
| it generated a mesh with individual wires and a plug with proper
| holes
| singularity2001 wrote:
| Now we just need a physical VR environment where we can upload
| these and play with, similar to Genie 2 but mesh based(?).
| abbe98 wrote:
| Is there a demo of its "Text to 3D Asset" capabilities somewhere?
| Lucasoato wrote:
| Yes, check it out here:
| https://huggingface.co/spaces/JeffreyXiang/TRELLIS
| rodnim wrote:
| Where exactly? All I see is "Image to 3D Asset".
| tarr11 wrote:
| You'd have to combine it with one of the text to image
| models.
| mock-possum wrote:
| Wow those mech examples are pretty boss lookin
| spaceships wrote:
| I was able to use this to go from an AI generated image to 3d
| print. Steps documented here:
| https://x.com/ryanlanciaux/status/1866163343788007619
| fourside wrote:
| What a future. We can generate images with words, turn those
| into physical objects at home, but I can't read the plain text
| and images describing it because of a broken site.
| nickpinkston wrote:
| Anyone see anything like this for GenAI for BREP CAD?
|
| This is using meshes which seem a lot easier, and I'm unsure
| who's working on BREP or at least models that have some feel for
| dimensions and physics.
|
| ie It could still be meshes, but the designs would be physically
| reasonable like how a skilled tradesperson may make a device with
| their intuition without running FEA, etc.
| unholiness wrote:
| As excited as I am about this jump from the fuzzy
| NeRFs/gaussian splatting to real meshes, I'm not holding my
| breath for BREP generation. Mesh to BREP has always been
| fraught because for anything beyond "find the cylinders", it
| becomes really subjective what a good representation is, and
| your average mesh likely doesn't have any simple representation
| that captures the full organic shape with analytic definitions.
|
| With mesh faces now supported in BREP, I'm more optimistic
| about a mixed modeling approach, where you can do the braindead
| find-the-cylinders conversion but keep the rest mesh, not
| needing to force it into some eldritch contortion of BSurfs.
| RobotToaster wrote:
| If this can be improved to accept multiple photos from different
| angles, it could make a great method for "pseudo-photogrammetry".
| spyder wrote:
| It really depends on the image, but WOW I was really surprised
| that it reproduced animal fur with proper combination of polygon
| mesh and transparent texture, and this kind of capability isn't
| even demonstrated in the examples on the page.
|
| https://imgur.com/a/qJp4HNX
___________________________________________________________________
(page generated 2024-12-10 23:01 UTC)