[HN Gopher] What is residual vector quantization?
___________________________________________________________________
What is residual vector quantization?
Author : gregdx
Score : 35 points
Date : 2023-09-04 16:11 UTC (6 hours ago)
(HTM) web link (www.assemblyai.com)
(TXT) w3m dump (www.assemblyai.com)
| waffletower wrote:
| These are interesting developments. We are seeing much interest
| in codec representations utilized in Generative AI applications.
| My specific exposure to EnCodec comes from MusicGen, and my
| initial response is that some combination of the perceptual
| tuning of the codec itself, or the chosen quantization seriously
| limits the perceptual audio quality of MusicGen outputs. I have
| yet to use EnCodec in isolation, but its utilization in MusicGen
| does not showcase a perceptually lossless codec.
| zbobet2012 wrote:
| This is interesting, but it's key to remember that audio and
| video codecs have three primary constraints in there design:
| compression efficiency and encode, and decode complexity
|
| Simply addressing compression efficiency without considering the
| other constraints makes for an impractical codec.
|
| If your encode complexity is too high, then you can't handle
| real-time (live) use cases, and your compute requirements may
| make practical use for large libraries like social media
| impractical.
|
| If your decode complexity is too high mobile devices will suffer
| from severe battery drain or simply won't be able to decode, even
| in hardware. Dedicated hardware may also have to much complexity
| to implement in a cost effective way.
| 26fingies wrote:
| It can be worth deprioritizing encode complexity since the
| majority of audio and video streaming is non-live content.
| Youtube, Netflix, TikTok, Instagram, and so on can all benefit
| even if encode is slower than realtime. Obviously there are
| cost-benefit considerations here but they are doing AV1 so they
| are willing to accept some hit on compute costs as a trade for
| bandwidth costs.
| sobriquet9 wrote:
| A lot of that footage is originated on smartphones, GoPro,
| drone cameras, etc. where hardware and power limitations do
| not allow one to run expensive encoding algorithms.
| stefanpie wrote:
| Another approach to neural-based data representation / encoding /
| decoding is Implicit Neural Representations (INRs). For INRs, you
| overfit a neural network, typically a Multilayer Perceptron
| (MLP)[1], to a single data point. For instance, in the case of a
| single image (the data point), the inputs would be the xy pixel
| coordinates--usually scaled from [0, n_pixels]^2 to either [0,
| 1]^2 or [-1, 1]^2--and the outputs would be the RGB values of
| each pixel. Once trained on pixels from this singular image, the
| INR model essentially becomes a representation of that data
| point. The image can then be re-rendered or reconstructed by
| inputting all the pixel coordinates and receiving the
| corresponding RGB values. This approach even allows for sampling
| at intermediate coordinates or a subset of coordinates for
| partial, progressive, or super-resolution decoding.
|
| This technique is generally effective as long as you can specify
| a reasonable coordinate system for your data, whether it's audio
| (n_samples, n_channels), 3D models (x, y, z)(x, y, z, th, ph)
| [2], geospatial data (lat, lon, altitude), and so on.
| Interestingly, it can also be applied even if there isn't a well-
| defined or non-Euclidean coordinate system for your data [3].
|
| Additionally, you can meta-learn an initial set of weights
| through meta-learning techniques tailored for a particular
| distribution of data points [4]. This enables you to quickly fit
| an INR model to a new data point that comes from the same
| distribution, such as microscope images of cells, MRI scans, or
| scenes for self-driving cars.
|
| Most pertinent to the post, INRs can be compressed and quantized
| to achieve comparable or even superior performance to traditional
| data compression techniques, at various quality levels for
| different modalities [5]. By no means is it competitive for long
| standing approaches like for image compression but it works well
| nonetheless. This is an area that we have explored extensively in
| my lab, enabling end-to-end encoding, processing, and decoding of
| INRs in hardware for a variety of applications.
|
| The most interesting topic to me is the fact that you can edit
| the weights of the INR to edit the underlying data [6][7], or use
| the weights of the INR to compute tasks on the data without ever
| having to decode it. For example, I can perform image
| classification on the weights of the INR model rather than on the
| actual images. This classification model is a hypernetwork that
| takes the INR parameters as inputs instead of the RGB image.
| These hypernetworks can also be designed in a way that preserves
| the invariance and equivariance properties of MLP structures,
| which is a very cool idea to me [8].
|
| [1] https://arxiv.org/abs/2006.09661 [2]
| https://www.matthewtancik.com/nerf [3]
| https://arxiv.org/abs/2205.15674 [4]
| https://www.matthewtancik.com/learnit [5]
| https://arxiv.org/abs/2201.12904 [6]
| https://arxiv.org/abs/2201.12204 [7]
| https://arxiv.org/abs/2210.08772 [8]
| https://developer.nvidia.com/blog/designing-deep-networks-to...
___________________________________________________________________
(page generated 2023-09-04 23:01 UTC)