https://sifeiliu.net/CosAE-page/

 

CosAE: Learnable Fourier Series for Image Restoration

Sifei Liu, Shalini De Mello, Jan Kautz
NVIDIA
Paper arXiv Video Code

Abstract

In this paper, we introduce CosAE (Cosine Autoencoder), a novel,
generic Autoencoder that seamlessly leverages the classic Fourier
series with a feed-forward neural network. CosAE represents an input
image as a series of 2D Cosine time series, each defined by a tuple
of learnable frequency and Fourier coefficients. This method stands
in contrast to a conventional Autoencoder that often sacrifices
detail in their reduced-resolution bottleneck latent spaces. CosAE,
however, encodes frequency coefficients, i.e., the amplitudes and
phases, in its bottleneck. This encoding enables extreme spatial
compression, e.g., 64x downsampled feature maps in the bottleneck,
without losing detail upon decoding.

We showcase the advantage of CosAE via extensive experiments on
flexible-resolution super-resolution and blind image restoration, two
highly challenging tasks that demand the restoration network to
effectively generalize to complex and even unknown image
degradations. Our method surpasses state-of-the-art approaches,
highlighting its capability to learn a generalizable representation
for image restoration.

Video

Network Architecture

Network Architecture

CosAE produce an extremely compressed representation of the input
image in the bottleneck, while still being able to be expended via
HCM, and decoded to a high-quality image. The HCM is a group of
learnable Fourier basis functions, used to expand the compressed
representation to the original image space.

Flexible-Resolution Super-Resolution

Network Architecture

CosAE LR

8x upsampled image. Drag to compare results

CosAE LR

8x upsampled image. Drag to compare results

CosAE LR

8x upsampled image. Drag to compare results

CosAE LR

8x upsampled image. Drag to compare results

CosAE LR

8x upsampled image. Drag to compare results

ImageNet 4x Super-Resolution

LR CosAE
LR CosAE
LR CosAE
CosAE LR

4x upsampled ImageNet results.

Blind Image Restoration

CosAE LR
CosAE LR

Blind image restoration with unknown degradations. Drag to compare.

BibTeX

@article{liu2024cosae,
  author    = {Sifei Liu, Shalini De Mello, Jan Kautz},
  title     = {CosAE: Learnable Fourier Series for Image Restoration},
  journal   = {NeurIPS},
  year      = {2024},
}

  

This website is licensed under a Creative Commons
Attribution-ShareAlike 4.0 International License.

This means you are free to borrow the source code of this website, we
just ask that you link back to this page in the footer. Please
remember to remove the analytics code included in the header of the
website which you do not want on your website.