[HN Gopher] Show HN: Real-Time Gaussian Splatting
___________________________________________________________________
Show HN: Real-Time Gaussian Splatting
LiveSplat is a system for turning RGBD camera streams into Gaussian
splat scenes in real-time. The system works by passing all the RGBD
frames into a feed forward neural net that outputs the current
scene as Gaussian splats. These splats are then rendered in real-
time. I've put together a demo video at the link above.
Author : markisus
Score : 117 points
Date : 2025-05-15 13:26 UTC (9 hours ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| sreekotay wrote:
| This is realtime capture/display? Presumable (at this stage) for
| local viewing? Is that right?
| markisus wrote:
| Yes realtime capture and display. Locality is not required. You
| can send the source RGBD video streams over IP and in fact I
| have that component working in the larger codebase that this
| was split off from. For that use case, you need to do some sort
| of compression. The RGB stream compression is a pretty solved
| problem, but the depth channel needs special consideration
| since "perceptual loss" in the depth space is not a well
| researched area.
| patrick4urcloud wrote:
| nice
| echelon wrote:
| OP, this is incredible. I worry that people might see a "glitchy
| 3D video" and might not understand the significance of this.
|
| This is getting unreal. They're becoming fast and high fidelity.
| Once we get better editing capabilities and can shape the
| Gaussian fields, this will become the prevailing means of
| creating and distributing media.
|
| Turning any source into something 4D volumetric that you can
| easily mold as clay, relight, reshape. A fully interactable and
| playable 4D canvas.
|
| Imagine if the work being done with diffusion models could read
| and write from Gaussian fields instead of just pixels. It could
| look like anything: real life, Ghibli, Pixar, whatever.
|
| I can't imagine where this tech will be in five years.
| markisus wrote:
| Thanks so much! Even when I was putting together the demo video
| I was getting a little self-critical about the visual glitches.
| But I agree the tech will get better over time. I imagine we
| will be able to have virtual front row seats at any live event,
| and many other applications we haven't thought of yet.
| echelon wrote:
| > I imagine we will be able to have virtual front row seats
| at any live event, and many other applications we haven't
| thought of yet.
|
| 100%. And style-transfer it into steam punk or H.R. Giger or
| cartoons or anime. Or dream up new fantasy worlds
| instantaneously. Explore them, play them, shape them like
| Minecraft-becomes-holodeck. With physics and tactile
| responses.
|
| I'm so excited for everything happening in graphics right
| now.
|
| Keep it up! You're at the forefront!
| _verandaguy wrote:
| I know enough about 3D rendering to know that Gaussian
| splatting's one of the Big New Things in high-performance
| rendering, so I understand that this is a big deal -- but I
| can't quantify why, or how big a deal it is.
|
| Could you or someone else wise in the ways of graphics give me
| a layperson's rundown of how this works, why it's considered so
| important, and what the technical challenges are given that an
| RGB+D(epth?) stream is the input?
| markisus wrote:
| Gaussian Splatting allows you to create a photorealistic
| representation of an environment from just a collection of
| images. Philosophically, this is a form of geometric scene
| understanding from raw pixels, which has been a holy grail of
| computer vision since the beginning.
|
| Usually creating a Gaussian splat representation takes a long
| time and uses an iterative gradient-based optimization
| procedure. Using RGBD helps me sidestep this optimization, as
| much of the geometry is already present in the depth channel
| and so it enables the real-time aspect of my technique.
|
| When you say "big deal", I imagine you are also asking about
| business or societal implications. I can't really speak on
| those, but I'm open to licensing this IP to any companies
| which know about big business applications :)
| corysama wrote:
| So, is there some amount of gradient-based optimization
| going on here? I see RGBD input, transmission, RGBD output.
| But, other than multi-camera registration, it's difficult
| to determine what processing took place between input and
| transmission. What makes this different from RGBD camera
| visualizations from 10 years ago?
| markisus wrote:
| There is no gradient-based optimization. It's (RGBD
| input, Current Camera Pose) -> Neural Net -> Gaussian
| Splat output.
|
| I'm not aware of other live RGBD visualizations except
| for direct pointcloud rendering. Compared to pointclouds,
| splats are better able to render textures, view-dependent
| effects, and occlusions.
| rsp1984 wrote:
| Except that no view-dependent effects that would benefit
| multi-view consistency are present in your splats.
|
| So yes, it's very much like the RGB-D visualizations from
| 10 years ago, just with splats instead of points.
| _verandaguy wrote:
| Thanks! That makes a lot of sense, I might dig into this
| after work some more.
|
| By "big deal," I meant more for people specializing around
| computer graphics, computer vision, or even narrower
| subfields of either of those two -- a big deal from an
| academic interest perspective.
|
| Sure, this might also have implications in society and
| business, but I'm a nerd, and I appreciate a good nerding
| out over something cool, niche, and technically impressive.
| sendfoods wrote:
| Please excuse my naive question - isn't Gaussian Splatting
| usually used to create 3D imagery from 2D? How does providing 3D
| input data make sense in this context?
| ttoinou wrote:
| Well if you have the D channel you might as well benefit from
| it and have better output
| markisus wrote:
| Yes, the normal case uses 2D input, but it can take hours to
| create the scene. Using the depth channel allows me to create
| the scene in 33 milliseconds, from scratch, every frame. You
| could conceptualize this as a compromise between raw pointcloud
| rendering and fully precomputed Gaussian splat rendering. With
| pointclouds, you have a lot visual artifacts due to sparsity
| (low texture information, seeing "through" objects"). With
| Gaussian splatting, you can transfer a lot more of the 2D
| texture information into 3D space and render occlusion and
| view-dependent effects better.
| Retr0id wrote:
| How do the view-dependent effects get "discovered" from only
| a single source camera angle?
| markisus wrote:
| Actually there are multiple source cameras. The neural net
| learns to interpolate the source camera colors based on
| where the virtual camera is. Under the hood it's hard to
| say exactly what's going on in the mind of the neural net,
| but I think it's something like "If I'm closer to camera A,
| take most of the color from camera A."
| ttoinou wrote:
| So we're not sure how it works exactly ?
| markisus wrote:
| Yup, this is the case for all neural nets.
| jayd16 wrote:
| Splatting is about building a scene that supports synthetic
| view angles.
|
| The depth is helpful to properly handle the parallaxing of the
| scene as the view angle changes. The system should then ideally
| "in-paint" the areas that are occluded from the input.
|
| You can either guess the input depth from matching multiple RGB
| inputs or just use depth inputs along with RGB inputs if you
| have them. It's not fundamental to the process of building the
| splats either way.
| yuchi wrote:
| The output looks terribly similar to what sci-fi movies
| envisioned as 3D reconstruction of scenes. It is absolutely
| awesome. Now, if we could _project_ them in 3D... :)
| tough wrote:
| Apple Vision maybe?
| mandeepj wrote:
| Another implementation of splat
| https://github.com/NVlabs/InstantSplat
| jasonjmcghee wrote:
| The quality is better, no doubt, but this method (from the
| paper) takes on the order of 10-45s depending on input from
| their table. Which is much better than 10 minutes etc.
|
| That being said, afaict OP's method is 1000x faster, at 33ms.
| markisus wrote:
| Note that the method you linked is "Splatting in Seconds" where
| as real-time requires splatting in tens of milliseconds.
|
| I'm also following this work https://guanjunwu.github.io/4dgs/
| which produces temporal Gaussian splats but takes at least half
| an hour to learn the scene.
| metalrain wrote:
| How did you train this? I'm thinking there isn't reference output
| for live video frame to splats so supervised learning doesn't
| work.
|
| Is there some temporal accumulation?
| markisus wrote:
| There is no temporal accumulation, but I think that's the next
| logical step.
|
| Supervised learning actually does work. Suppose you have four
| cameras. You input the three of them into the net and use the
| fourth as the ground truth. The live video aspect just emerges
| from re-running the neural net every frame.
| corysama wrote:
| So, I see livesplat_realsense.py imports livesplat. Where's
| livesplat?
| IshKebab wrote:
| The README says it's closed source.
| markisus wrote:
| I've tried to make it clear in the link that the actual
| application is closed source. I'm distributing it as a .whl
| full of binaries (see the installation instructions).
|
| I've considered publishing the source but the source code is is
| dependent on some proprietary utility libraries from my bigger
| project and it's hard to fully disentangle it and I'm not sure
| if this project has some business applications but I'd like to
| keep that door open at this time.
| armchairhacker wrote:
| Gaussian Splatting looks pretty and realistic in a way unlike any
| other 3D render, except UE5 and some hyper-realistic not-realtime
| renders.
|
| I wonder if one can go the opposite route and use gaussian
| splatting or (more likely) some other method to generate 3D/4D
| scenes from cartoons. Cartoons are famously hard to emulate in 3D
| even entirely manually; like with traditional realistic renders
| (polygons, shaders, lighting, post-processing) vs gaussian
| splats, maybe we need a fundamentally different approach.
| spyder wrote:
| Correct me if I'm wrong but looking at the video this just looks
| like a 3D point cloud using equal-sized "gaussians" (soft
| spheres) for each pixel, that's why it looks still pixelated
| especially at the edges. Even when it's low resolution the real
| gaussian splatting artifacts look different with spikes an soft
| blobs at the lower resolution parts. So this is not really doing
| the same as a real gaussian splatting of combining different
| sized view-dependent elliptic gaussians splats to reconstruct the
| scene and also this doesn't seem to reproduce the radiance field
| as the real gaussian splatting does.
| markisus wrote:
| I had to make a lot of concessions to make this work in real-
| time. There is no way that I know to replicate the fidelity of
| "actual" Gaussian splatting training process within the 33ms
| frame budget.
|
| However, I have not baked in the size or orientation into the
| system. Those are "chosen" by the neural net based on the input
| RGBD frames. The view dependent effects are also "chosen" by
| the neural net, but not through an explicit radiance field. If
| you run the application and zoom in, you will be able to see
| the splats of different sizes pointing in different directions.
| The system as limited ability to re-adjust the positions and
| sizes due to the compute budget leading to the pixelated
| effect.
| kookamamie wrote:
| [flagged]
| dang wrote:
| Whoa--please don't be a jerk on HN and especially not when
| discussing other people's work.
|
| You broke the site guidelines badly here
| (https://news.ycombinator.com/newsguidelines.html), and the
| Show HN guidelines even more so
| (https://news.ycombinator.com/showhn.html).
|
| If you wouldn't mind reviewing those links and sticking to the
| rules when posting to HN, we'd appreciate it.
| whywhywhywhy wrote:
| Would be good to see how it's different from just the depth
| channel applied to the Z of the RGB pixels. Because it looks very
| similar to that.
| markisus wrote:
| The application has this feature and lets you switch back and
| forth. What you are talking about is the standard pointcloud
| rendering algorithm. I have an older video where I display the
| corresponding pointcloud [1] in a small picture in picture
| frame so you can compare.
|
| I actually started with pointclouds for my VR teleoperation
| system but I hated how ugly it looked. You end up seeing
| through objects and objects becoming unparseable if you get too
| close. Textures present in the RGB frame also become very hard
| to make out because everything becomes "pointilized". In the
| linked video you can make out the wood grain direction in the
| splat rendering, but not in the pointcloud rendering.
|
| [1] https://youtu.be/-u-e8YTt8R8?si=qBjYlvdOsUwAl5_r&t=14
| badmonster wrote:
| What is the expected frame rate and latency when running on a
| typical setup with one Realsense camera and an RTX 3060?
| markisus wrote:
| I don't have a 3060 at hand so I'm not sure. Ideally someone
| with that setup will try it out and report back. There is no
| noticeable latency when comparing visually with standard
| pointcloud rendering.
|
| With framerate, there are two different frame rates that are
| important. One is the splat construction framerate, which the
| speed that an entirely new set of Gaussian's can be
| constructed. LiveSplat can usually maintain 30fps in this case.
|
| The second important splat rendering framerate. In VR this is
| important to prevent motion sickness. Even if you have a static
| set of splats, you need the rendering to react to the user's
| minor head movements at around 90fps for the best in-headset
| experience.
|
| All these figures are on my setup with a 4090 but I have gotten
| close results with a 3080 (maybe 70fps splat rendering instead
| of 90fps).
| smusamashah wrote:
| The demo video does not show constructing 3d from input. Is it
| possible to do something like that with this? Take a continus
| feed of a static scene and keep improving the 3D view?
|
| This is what I thought from the title, but the demo video is just
| a conitnuously changing stream of points/splats with the video.
| markisus wrote:
| If the scene is static, the normal Gaussian splatting pipeline
| will give much better results. You take a bunch of photos and
| then let the optimizer run for a while to create the scene.
| drewbeck wrote:
| imo this is a key component of a successful VR future for live
| events. Many cameras at a venue, viewers strap on a headset at
| home and get to sit/stand anywhere in the room and see the show.
|
| Also I love the example video. Folks could make some killer music
| videos with this tech.
| asadm wrote:
| This is amazing! Video calls of the future (this + vision pro)
| would be lovely.
___________________________________________________________________
(page generated 2025-05-15 23:00 UTC)