http://richg42.blogspot.com/2023/04/faster-lz-is-not-answer-to-150-250-gb.html

Richard Geldreich's Blog

Co-owner of Binomial LLC, working on GPU texture interchange. Open
source developer, graphics programmer, former video game developer.
Worked previously at SpaceX (Starlink), Valve, Ensemble Studios
(Microsoft), DICE Canada.

Friday, April 21, 2023

Faster LZ is not the answer to 150-250+ GB video game downloads

When the JPEG folks were working on image compression, they didn't
create a better or faster LZ. Instead they developed new approaches.
I see games growing >150GB and then graphs like this, and it's
obvious the game devs are going in the wrong direction:

https://aras-p.info/blog/2023/01/31/Float-Compression-2-Oodleflate/

(Note these benchmarks are great and extremely useful.)

Separating out the texture encoding stage from the lossless stage is
a compromise. I first did this in my "crunch" library around 15 years
ago. It was called "RDO mode". You can swizzle the ASTC/BC1-7 bits
before LZ, and precondition them, and that'll help, but the two steps
are still disconnected. Instead combine the texture and compression
steps (like crunch's .CRN mode - shipped by Unity for BC1-5.) 

Alternatively: defer computing the GPU texture data until right
before it's actually needed and cache it. Ship the texture signal
data using existing image compression technology, which at this point
is quite advanced. For normal maps, customize or tune existing tech
to handle them without introducing excessive angular distortion. I
think both ideas are workable.

Also, these LZ codecs are too fast. They are designed for fast
loading and streaming off SSD's. Who cares about saving off a few
hundred ms (or a second) when it takes hours or days to download the
product onto the SSD?

Somebody could develop a 10x faster Oodle (or an Oodle that
compresses 1-2% better) and we're still going to wait many hours or
days to actually use the product. And then there's the constant
updates. This approach doesn't scale.

It's fun and sexy to work on faster LZ but the real problem (and
value add) doesn't call for better or more lossless tech. This is a
distraction. If trends continue the downloads and updates will be
measured in terms of fractional or 1+ week(s).

Posted by Rich Geldreich at 8:56 AM #
Email ThisBlogThis!Share to TwitterShare to FacebookShare to
Pinterest

No comments:

Post a Comment

 

Older Post Home
Subscribe to: Post Comments (Atom)

Blog Archive

  * V  2023 (3)
      + V  April (2)
          o Faster LZ is not the answer to 150-250+ GB video g...
          o The Dark Horse of the Image Codec World: Near-Loss...
      + >  March (1)

  * >  2022 (7)
      + >  January (7)

  * >  2021 (25)
      + >  November (1)
      + >  February (23)
      + >  January (1)

  * >  2020 (12)
      + >  September (1)
      + >  August (1)
      + >  April (5)
      + >  February (3)
      + >  January (2)

  * >  2019 (12)
      + >  October (2)
      + >  September (1)
      + >  April (9)

  * >  2018 (43)
      + >  October (2)
      + >  July (3)
      + >  June (6)
      + >  May (9)
      + >  April (17)
      + >  March (2)
      + >  February (4)

  * >  2017 (24)
      + >  November (5)
      + >  September (1)
      + >  August (1)
      + >  June (2)
      + >  April (2)
      + >  March (4)
      + >  February (2)
      + >  January (7)

  * >  2016 (77)
      + >  December (3)
      + >  November (2)
      + >  October (19)
      + >  September (33)
      + >  August (5)
      + >  July (3)
      + >  June (1)
      + >  May (4)
      + >  April (3)
      + >  March (1)
      + >  January (3)

  * >  2015 (62)
      + >  December (12)
      + >  November (9)
      + >  October (6)
      + >  June (2)
      + >  May (6)
      + >  April (1)
      + >  March (1)
      + >  February (5)
      + >  January (20)

  * >  2014 (34)
      + >  November (1)
      + >  October (2)
      + >  June (6)
      + >  May (3)
      + >  April (2)
      + >  March (11)
      + >  February (5)
      + >  January (4)

  * >  2013 (6)
      + >  November (1)
      + >  October (4)
      + >  August (1)

  * >  2012 (4)
      + >  November (1)
      + >  August (1)
      + >  July (2)

About Me

Rich Geldreich
    Seattle, WA, United States
    Back in the day I worked for several years at Digital Illusions
    on things like the first shipping deferred shaded game ("Shrek" -
    2001), software renderers, and game AI. Then, after working for
    Microsoft at Ensemble Studios for 5 years as engine lead on Halo
    Wars, I took a year off to create "crunch", an advanced DXTc
    texture compression library. I then worked 5 years at Valve,
    where I contributed to Portal 2, Dota 2, CS:GO, and the Linux
    versions of Valve's Source1 games. I was one of the original
    developers on the Steam Linux team, where I worked with a
    (somewhat enigmatic) multi-billionare on proving that OpenGL
    could still hold its own vs. Direct3D. I also started the vogl
    (Valve's OpenGL debugger) project from scratch, which I worked on
    for over a year. In my spare time I work on various open source
    lossless and texture compression projects: crunch, LZHAM, miniz,
    jpeg-compressor, and picojpeg.

View my complete profile
 
Simple theme. Powered by Blogger.