[HN Gopher] Deblur-GS: 3D Gaussian splatting from camera motion ...
       ___________________________________________________________________
        
       Deblur-GS: 3D Gaussian splatting from camera motion blurred images
        
       Author : smusamashah
       Score  : 91 points
       Date   : 2024-05-13 17:16 UTC (5 hours ago)
        
 (HTM) web link (chaphlagical.icu)
 (TXT) w3m dump (chaphlagical.icu)
        
       | nathancahill wrote:
       | I know it's a meme at this point but this is real life "Enhance
       | please". Incredibly impressive what we're able to do to
       | reconstruct missing data.
        
       | zevv wrote:
       | My friend doesn't quite grasp this yet, can someone explain? Is
       | the reconstructed detail all "real" and extracted from the
       | blurred input, or is there some model at work here, filling in
       | the image with plausible details, but basically making up stuff
       | that was not really there to start with?
        
         | karmakaze wrote:
         | That's accurate. What's worth nothing though is that everything
         | we 'see' with our own eyes is constructed from sampling our
         | environment. The image we construct is what we expected to see
         | given the sample data. This is one reason why eyewitness
         | testimony can be vivid and false without any foul play.
        
         | barrysteve wrote:
         | Both. The paper mentions using a deblurrer and novel view
         | synthesis model(ExBluRF).
        
         | tomp wrote:
         | I skimmed the Overview and am not an expert.
         | 
         | It seems to me they don't use any ML at all. They use
         | backpropagation to jointly optimise the entire physics/motion
         | model, which models camera motion and the generated blurry
         | images (they generate multiple images for each camera frame
         | along the path of motion of the camera, and then merge them,
         | simulating motion blur)
        
           | dheera wrote:
           | It is ML in the sense of optimizing a nonconvex loss function
           | over a dataset. It is not a fancy diffusion model or even a
           | generative model, but it is no less a machine learning
           | problem.
        
             | tomp wrote:
             | "Not ML" as in "not learning from data to apply in new
             | situations" but rather they do "mathematical optimisation".
             | 
             | The data they optimise over is just the images of the
             | current camera trajectory (as far as I understand)
        
         | chpatrick wrote:
         | Gaussian blur is a reversible operation, but in practice it's
         | not possible on still images. With multiple pictures you might
         | have enough information.
        
         | peppertree wrote:
         | No it does not "make up things" using generative AI. Current GS
         | implementations assume camera poses are static. This paper
         | assigns a linear motion trajectory to camera during training.
        
           | creativeSlumber wrote:
           | So can it handle when both camera and multiple objects in
           | scene are moving in different trajectories?
        
             | dheera wrote:
             | Not with traditional 3D Gaussian splatting, but it is
             | potentially possible to separate the time axis and do a 4D
             | Gaussian splatting with some regularization to accommodate
             | dynamic scenes.
             | 
             | Here's some early work in this area which seems promising:
             | https://guanjunwu.github.io/4dgs/
        
       | emilk wrote:
       | Very cool! A next step could be to model a rolling-shutter
        
       | tomaskafka wrote:
       | Absolutely impressive - seems on par with what's happening in our
       | eyes and brain. If this becomes realtime, we could turn the noisy
       | low fps image from cameras on AR headsets in dark environments
       | into smooth bright image.
        
       | borgchick wrote:
       | finally, all the UFO videos can be clear!
        
         | bee_rider wrote:
         | The aliens are actually pan-dimensional light beings. That is
         | why they are afraid of high quality cameras, if they get caught
         | in a photo they are stuck here forever. Running this algorithm
         | on pictures of UFOs is actually an intergalactic warcrime.
        
       | germinator wrote:
       | I really want to be impressed, but I've been reading papers about
       | breakthroughs in deblurring and upscaling for two decades now,
       | and the state of the art in commercial and open-source tools is
       | still pretty underwhelming. Chances are, if you have a low-res
       | keepsake photo, or take a blurry nature shot, you're gonna be
       | stuck with that.
       | 
       | Video, where the result needs to be temporally coherent and make
       | sense in 3D, can't be the easier one.
        
         | tomp wrote:
         | this work won't solve that. it requires a video (sequence of
         | images)
        
         | IshKebab wrote:
         | > Video, where the result needs to be temporally coherent and
         | make sense in 3D, can't be the easier one.
         | 
         | Why not? Video is a much more tractable problem because you
         | have much more information to go on.
        
       | adkaplan wrote:
       | https://web.archive.org/web/20240511220923/https://chaphlagi...
       | 
       | Down for me, archive above.
        
       | smusamashah wrote:
       | The reconstruction looks even better than ground truth images in
       | their examples.
        
       ___________________________________________________________________
       (page generated 2024-05-13 23:00 UTC)