[HN Gopher] Neural Geometric Level of Detail: Real-Time Renderin...
___________________________________________________________________
Neural Geometric Level of Detail: Real-Time Rendering with Implicit
3D Surfaces
Author : ArtWomb
Score : 144 points
Date : 2021-01-28 13:57 UTC (9 hours ago)
(HTM) web link (nv-tlabs.github.io)
(TXT) w3m dump (nv-tlabs.github.io)
| AbrahamParangi wrote:
| I bet you could take advantage of the fact that the function
| generated with feedforward + relu is piecewise linear to mesh the
| resulting SDF really efficiently.
| zqfm wrote:
| As an aside, this page is great, I was able to watch the videos
| and see all the images without needing to enable javascript for
| half a dozen 3rd party URLs!
| ilaksh wrote:
| They site the Occupancy Networks paper as [33]. But I thought
| that one also used octrees? Which they are saying is supposed to
| be novel.
| sxp wrote:
| If you don't have an understanding of signed distance fields,
| https://www.youtube.com/watch?v=8--5LwHRhjk is an amazing demo of
| their power. There are more videos about the fundamentals of SDFs
| at https://www.iquilezles.org/live/index.htm
|
| The interesting thing about this paper is that they replace the
| mathematical SDF with a neural net that computes SDF_NN(x,y,z) =>
| -1/1 and use this as a compression system for mesh based inputs.
| This feels like a graphical version of the
| https://en.wikipedia.org/wiki/Hutter_Prize for NN-based text
| compression.
| folli wrote:
| Inigo truly is a modern-day magician.
| mortenjorck wrote:
| Seriously. I've never seen anyone who is simultaneously this
| fluent in applied mathematics while having this level of
| visual sensibility as an illustrator. Truly a polymath.
| tovacinni wrote:
| Here's a Twitter thread that explains what this paper is about:
| https://twitter.com/yongyuanxi/status/1354478763065528320?
| baxuz wrote:
| I'm trying to understand this but I'm not sure I'm following.
|
| As far as I understand as a complete beginner in 3D rendering:
|
| Regular 3d models/meshes are just a collection of
| points/triangles in a 3d plane, along with some additional data
| like normal orientations. For rendering, the mesh is
| constructed and then rendered with some
| perspective/distance/occlusion settings etc. for each pixel
| painted on screen.
|
| This format isn't a discrete collection of points but some data
| set where every point queried during 2d painting returns a
| distance (if it exists) based on some function or ML algorithm,
| basically returning a point cloud?
| GreenHeuristics wrote:
| Think of it like so:
|
| float distance_to_surface_of_model(x, y, z);
|
| For every point in 3d space you can get the distance to the
| surface. So, if you have a camera you can shoot a ray into
| the scene and pick some points on it to ask if it is inside
| the model. if so you paint that pixel with the surface
| material of the model. Then it's just a matter of picking a
| good way of sampling points along the camera ray. usually
| this distance function is some basic math (distance to sphere
| or so). But here they take a 3d mesh and create a neural
| network that can answer the question of distance to it (since
| meshes are very slow at that).
| xpe wrote:
| After skimming here is my (imperfect) summary. Comments
| encouraged!
|
| Technique: "Neural Geometric LOD (level of detail)"
|
| Summary: This technique provides a machine-learning approach to
| give more efficient representation for 3D surfaces at interactive
| rates.
|
| Related Work: "Our work is most related to prior research on mesh
| simplification for level of detail (LOD), 3D neural shape
| representations, and implicit neural rendering."
|
| Method: "Our goal is to design a representation which
| reconstructs detailed geometry and enables continuous level of
| detail, all whilst being able to render at interactive rates."
|
| Experiments: "Across all [tested] datasets [(ShapeNet, Thingi10K,
| TurboSquid, Shadertoy)] and metrics, we achieve state-of-the-art
| results" relative to DeepSDF, Fourier Feature Networks, SIREN,
| and Neural Implicits (NI).
|
| Generalization: "surface extraction mechanism can generalize to
| multiple shapes, even from being trained on a single shape."
| gdubs wrote:
| If I understand correctly, this is essentially using neural nets
| to compress the information representing a 3D mesh [1], similar
| to how neural nets can do image upscaling on 2D images?
|
| Either way, the technology of Star Trek The Next Generation went
| through a period in the late 90s where it all seemed ridiculous,
| and now we've come full circle and it all seems plausible again.
|
| More specifically: I'd love to see a generative adversarial
| network that can materialize 3D worlds in real time, like the
| holodeck. More and more the holodeck seems plausible as a system
| of nets upon nets upon nets. Nets for generating models, nets for
| generating stories...
|
| 1: In this case the meshes are not polygons, but rather signed
| distance fields. Metaballs are commonly made with this technique.
|
| Edit: thanks for catching the spelling mistake. Metaballs
| misspelled as Meatballs is a plague that continues for over two
| decades. Usually I try to avoid the trap altogether and say
| Implicit Surfaces :)
| ben_w wrote:
| For TNG-style holodecks, I think the future is smart-dust
| drones and beamed power.
| ryandamm wrote:
| Re: GANNs generating 3D worlds, there are a couple startups
| that appear to be... well, sniffing around the edges, anyway:
|
| https://www.wunderparc.com/ https://www.prometheanai.com/
|
| I haven't tried either one out; someone in my industry pointed
| them out to me recently.
| harperlee wrote:
| Metaballs also, not only meatballs ;)
| whelming_wave wrote:
| Not at all an expert in this, but I'm curious how this compares
| to non-NN solutions like
| https://research.nvidia.com/publication/efficient-sparse-vox...
|
| Presumably it takes less memory, letting a more complex scene be
| transferred to the GPU more quickly, at the tradeoff of time
| spent training the model?
| skye-adaire wrote:
| Memory savings are the main potential gain, since you won't
| need as deep a tree. This is not the main issue with this tech
| atm.
|
| Also, their distance output will be an estimate, it won't be
| exact, which will cause render time to underperform vs exact
| representations. Worse, the NN could return a value greater
| than actual distance, which will cause artifacts.
___________________________________________________________________
(page generated 2021-01-28 23:01 UTC)