https://research.nvidia.com/labs/dir/neuralangelo/

Neuralangelo: High-Fidelity Neural Surface Reconstruction

Zhaoshuo Li^1,2
Thomas M&uumlller^1
Alex Evans^1
Russell H. Taylor^2
Mathias Unberath^2
Ming-Yu Liu^1
Chen-Hsuan Lin^1
^1NVIDIA Research      ^2Johns Hopkins University
IEEE Conference on Computer Vision and Pattern Recognition (CVPR),
2023
Paper (arXiv) Paper Supplementary Code

Neuralangelo is a framework for high-fidelity 3D surface
reconstruction from RGB video captures. Using ubiquitous mobile
devices, we enable users to create digital twins of both
object-centric and large-scale real-world scenes with highly detailed
3D geometry.

For business inquiries, please visit our website and submit the form:
NVIDIA Research Licensing.

(best viewed with Google Chrome on a desktop/laptop)

Abstract

Neural surface reconstruction has shown to be powerful for recovering
dense 3D surfaces via image-based neural rendering. However, current
methods struggle to recover detailed structures of real-world scenes.
To address the issue, we present Neuralangelo, which combines the
representation power of multi-resolution 3D hash grids with neural
surface rendering. Our approach is enabled by two key ingredients:
(1) numerical gradients for computing higher-order derivatives as a
smoothing operation and (2) coarseto-fine optimization on the hash
grids controlling different levels of details. Even without auxiliary
depth, Neuralangelo can effectively recover dense 3D surface
structures from multi-view images with a fidelity that significantly
surpasses previous methods, enabling detailed large-scale scene
reconstruction from RGB video captures.

---------------------------------------------------------------------

Video

X
---------------------------------------------------------------------

Large-scale Reconstruction

  * NVIDIA HQ Park
  * Johns Hopkins University

  * Barn
  * Caterpillar
  * Courthouse
  * Ignatius
  * Meetingroom
  * Truck

RGB view synthesis 3D surface reconstruction Surface normal
---------------------------------------------------------------------

Object-centric Reconstruction

  * Scan24
  * Scan37
  * Scan40
  * Scan63

NeuS Neuralangelo (Ours) NeuralWarp
---------------------------------------------------------------------

Approach

Neuralangelo builds on top of multi-resolution hash encoding and
SDF-based volume rendering.


1.  Numerical gradients to compute higher-order derivatives

Using numerical gradients with step size matching the spatial
resolutions of hash grid optimizes beyond the local cells. The
numerical gradients act as a smoothing operation on the SDF in
comparison to the analytical gradients.


2.  Progressive level of details

By progressively decreasing the step size for numerical gradient and
enabling higher resolution hash grids, the optimization landscape is
better shaped to recover both large smooth surfaces and fine
geometric details. Such a learning curriculum enables progressive
level of details.


3.  Optimization

Neuralangelo uses three optimization objectives: $$\mathcal{L} = \
mathcal{L}_{rgb} + w_\text{eik} \mathcal{L}_{eik} + w_\text{curv} \
mathcal{L}_{curv}.$$

  * RGB synthesis loss \( \mathcal{L}_{rgb} \) :  RGB reconstruction
    loss between the input image and synthesized images.
  * Eikonal loss \( \mathcal{L}_{eik} \) :  regularize underlying SDF
    such that the surface normals are unit-norm.
  * Curvature loss \( \mathcal{L}_{curv} \) :  regularize underlying
    SDF such that the mean-curvature is not arbitrarily large.

---------------------------------------------------------------------

Presentation

---------------------------------------------------------------------

Poster

(Click image to enlarge)
[poster]
---------------------------------------------------------------------

Citation

@inproceedings{li2023neuralangelo,
  title={Neuralangelo: High-Fidelity Neural Surface Reconstruction},
  author={Li, Zhaoshuo and M\"uller, Thomas and Evans, Alex and
Taylor, Russell H and Unberath, Mathias and Liu, Ming-Yu and Lin,
Chen-Hsuan},
  booktitle={IEEE Conference on Computer Vision and Pattern
Recognition ({CVPR})},
  year={2023}
}
Web Analytics