[HN Gopher] Artistic Radiance Fields
___________________________________________________________________
Artistic Radiance Fields
Author : Mizza
Score : 117 points
Date : 2022-06-21 15:38 UTC (7 hours ago)
(HTM) web link (www.cs.cornell.edu)
(TXT) w3m dump (www.cs.cornell.edu)
| [deleted]
| amelius wrote:
| It looks cool, but like PhotoShop filters, the coolness factor
| will wear off quickly.
| raptortech wrote:
| This is excellent! I can't wait for Pixar to pick this up and
| release a gorgeous hand-painted film.
| ipsin wrote:
| @ak92501 posted this one last week. This isn't to say "it's old
| news" (it isn't!) but that if you're on Twitter and like
| summaries of ML papers, they're a great follow.
|
| https://twitter.com/ak92501/status/1536518567688122368
| hansword wrote:
| Just yesterday someone asked me if "AI style transfer"* would
| work for 3d models or video. I said "if the current speed of
| progress holds, i think we will see this within the year".
|
| And here it is.
|
| "What a time to be alive!"
|
| (*) this isn't style transfer on video, as far as i understand,
| this is styling the neural radiance field before rendering it to
| images.
| dimatura wrote:
| Adaptations of image style transfer specifically for video came
| out pretty quick after the image ones, and there's been a few
| iterations. For the most part they use the same ideas, but add
| some extra steps to "stabilize" the result of the stylization
| across time so that it doesn't jump all over the place from
| frame to frame.
|
| For 3D, it's more complicated as there isn't an obvious
| translation of image-based stylization to the 3D realm - if we
| are talking 3D mesh models, I've seen similar ideas applied to
| generating textures as well as geometry.
|
| In this case, the novelty is combining the stylization with the
| NERF, which does result in a special type of 3D model (not a
| mesh, but a volumetric representation that lends itself well to
| rendering). There isn't really any video aspect to it, other
| than that videos of the result from different viewpoints are a
| good way of visualizing the result.
| trhway wrote:
| how Van Gough would paint an excavator? Now we know :)
| pavlov wrote:
| Note that the input here is not any arbitrary video, but a
| radiance field, a type of data structure that's halfway between
| photos and a 3D model:
|
| "The radiance field representation can be seen as a 5D function
| that maps any 3D location x and viewing direction d to volume
| density s and RGB color c."
|
| The radiance field is typically reconstructed from multiple
| photos of a scene, which implies that either the subject must be
| static or you need a fancy camera rig to capture the views
| simultaneously.
|
| We don't have any content pipelines for creating and editing
| radiance fields, so movies produced with this approach are a ways
| off. But one could imagine AR glasses in a couple of years that
| capture the different views as you move around, then the radiance
| field is built in the cloud and you can create artistic re-
| renders of what you just saw.
| robocat wrote:
| > We don't have any content pipelines for creating and editing
| radiance fields
|
| I presume radiance fields can be generated from computer
| graphic scenes? It would be awesome to choose your own style
| while playing a video game! Real-time generation is unlikely,
| but game replay videos in a surreal style would be tres cool.
| Imnimo wrote:
| Radiance fields really feel like they're taking off. Reminds me
| of GANs a few year ago, where the number of papers seemed to
| start growing exponentially one day.
| fxtentacle wrote:
| In my opinion, this looks like a 2D style transfer, not a true 3D
| style transfer. In the horse statue example, a human artist would
| add a separator line between the white box and the grey
| background, to illustrate the depth disparity. But this approach
| seems to somewhat ignore depth cues when styling.
| rektide wrote:
| I always pictured eventually Augmented Reality would have
| artistic overlays, reinterpretations of the world, and this sure
| does seem like a feasible implementation of that.
| Wistar wrote:
| Yes! Perhaps AAR: Artistically Augmented Reality.
| timmg wrote:
| Would love to have this in VR. Dumb question: is there a VR
| "player" for NERFs?
| andybak wrote:
| I don't think consumer GPUs can render this in real-time yet.
| I've seen around 5 to 10fps framerates on a 3080 just rendering
| monoscopic output (from memory)
| kleer001 wrote:
| cool stuff, but isn't style-transfer pretty old hat? even on
| moving images?
| bckr wrote:
| it seems like this is more stable. the style applies over a 3D
| world without noticeable artifacts or distortions
| wwalexander wrote:
| This was my impression as well. It seems to have some sort of
| object permanence when the camera is moved rather than just
| convoluting a single frame; seems significantly more
| difficult to me as a layman.
| corysama wrote:
| Yep. There has been some work to figure out frame-to-frame
| coherence on a sequence of images in 2D. But, I think this
| skips over that problem by working in 3D.
| killerstorm wrote:
| It's not a "moving image", it's a neural radiance field (NeRF),
| a way to represent a 3D scene using a neural network.
|
| Style transfer worked on pixels, this approach works on the
| scene itself, before it's rasterized.
___________________________________________________________________
(page generated 2022-06-21 23:01 UTC)