[HN Gopher] Faster LZ is not the answer to 150-250 GB video game...
___________________________________________________________________
Faster LZ is not the answer to 150-250 GB video game downloads
Author : ingve
Score : 88 points
Date : 2023-04-21 19:52 UTC (3 hours ago)
(HTM) web link (richg42.blogspot.com)
(TXT) w3m dump (richg42.blogspot.com)
| terafo wrote:
| He completely misses the point. Games don't take hundreds of gigs
| because they have sub-optimal compression. They take hundreds of
| gigs because they don't use ANY compression. Jedi Survivor takes
| 139 gigs on Series X and mere 44 gigs on Series S. It didn't come
| out yet, but I'm 99.99% certain that this difference is just from
| inclusion of 8k textures into Series X version. That kind of
| scaling means that they don't use any compression beyond
| something like BC7. And some games are not only without any
| compression, they ACTIVELY duplicate their assets on disk so they
| load a bit faster on HDD. And these builds end up on modern
| consoles that have ludicrously fast SSDs.
| jjoonathan wrote:
| In 2003, I found out that America's Army (the US Army
| videogame) had compressed and decompressed all their textures
| at some point, so they were paying the quality hit for
| compression _and_ the space hit for RGBA. This was a time when
| the game still took a day to download, even on a half decent
| connection, so >200% footprint inflation kinda mattered. The
| more things change, the more they stay the same!
| taeric wrote:
| This feels wrong. In that, I'd be surprised if they are not
| using compression. I think folks are just drastically
| underestimating how many resources are under discussion. And
| how big they are.
|
| Heck, just the geometry is probably far larger than folks
| anticipate.
| sudosysgen wrote:
| It's true. I've seen tons of games use basically uncompressed
| audio, badly compressed textures, etc...
|
| People often repack video games, making programs to compress
| and decompress them so they are easier to pirate. It's not
| uncommon to see a 2-3x reduction in size, for bit-identical
| results.
| taeric wrote:
| That is a little surprising to me. Especially if we are
| talking about games that are delivered online. Thinking on
| it, I'm actually only surprised for online delivered games.
| I don't see any incentive to reduce size on delivered
| disks.
|
| I'd expect that they use a compression that is good for
| transport, at transport time. I'm not too shocked that they
| have the assets in a way that is easy to use at runtime.
| coffeebeqn wrote:
| Games are duct taped together at last minute on
| impossible deadlines pretty much every time.
| sroussey wrote:
| Textures are not compressed for size on disk, they are
| compressed to the format that can be directly loaded into
| the GPU.
| terafo wrote:
| You can add lossless compression, like Kraken, on top of
| that. PS5 has hardware accelerated Kraken decompression.
| And you can encode textures for GPU in a way that is
| better compressable with lossless algorithms.
| sudosysgen wrote:
| Yes, that's the problem. You can compress them for size
| at download time, and recompress them to a GPU-optimal
| format at the time of installation. You may even be able
| to do it at runtime with modern hardware, shortening
| loadtimes and sometimes even improving performance
| indirectly.
| coffeebeqn wrote:
| Then you have to have the game reprocess all the assets
| on the users machine which could easily take an hour
| sudosysgen wrote:
| Essentially any modern image compression format is going
| to be limited, or close to, by storage speed (for
| decompression). So unless the user has insanely fast
| internet (and the servers are also able to saturate their
| connection), it will be faster in the end.
| lpghatguy wrote:
| You can combine the approaches! KTX2, for example,
| supports "supercompressing" GPU block compressed formats
| with LZ4 or Zstandard. The project I have been working on
| ships BCn textures compressed with zstd and it has been
| pretty dang good.
| marcellus23 wrote:
| Maybe not for that particular game, but a lot of games have
| been shown to use uncompressed assets. It's not just
| speculation.
| forrestthewoods wrote:
| > It's not just speculation.
|
| So it's exactly speculation. A lot of games have done a lot
| of things for a lot of reasons over the decades.
|
| Why are new games so large? Good question. This entire
| thread is speculative.
| marcellus23 wrote:
| Um, no. It's not speculation. Many games have been shown
| to use uncompressed assets, that's an objective fact.
|
| The thread might be speculating about _why_ they do that,
| or whether this particular game does that, but that's a
| non sequitur, that isn't what I was talking about.
| terafo wrote:
| Jedi Survivor having uncompressed textures is a
| speculation, but I'm extremely confident that it is this
| way. The fact that games nowadays ship without LOD
| systems, without texture compression, without adequate
| streaming is not just speculation. It is a fact.
| kevingadd wrote:
| You're saying Jedi Survivor doesn't use block texture
| compression? On which target? All targets?
|
| Can you capture a trace in RenderDoc to prove this? Or
| show raw bmp/uncompressed dds files on disk?
| terafo wrote:
| By texture compression I meant something like LZ4 or
| Kraken on top of block texture compression, sorry for
| ambiguity. Of course they use block level compression.
| But it seems that it's the only thing they use, as I said
| in my top level comment.
| taeric wrote:
| Even for transferring the game to local?
|
| I can see some benefits to having uncompressed assets in
| that it would help with asset loading. I'd still be
| surprised to not have any form of compression at all
| happening. I'd also be curious to know just how much the
| assets could be compressed.
| uoaei wrote:
| The geometry would be hard to compress, but I suspect there's
| some clever tricks with some kind of fractal compression[1]
| that could drastically reduce texture sizes for camo,
| foliage, etc. without any effect on aesthetics.
|
| Encoding is the hard part, but only need be done once when
| building the final release.
|
| [1] https://en.wikipedia.org/wiki/Fractal_compression
| gatane wrote:
| Wait, are there more papers on this? Seems interesting.
| xaduha wrote:
| > This feels wrong. In that, I'd be surprised if they are not
| using compression.
|
| Check this out, you can sort by the number of submissions to
| filter out indie games https://docs.google.com/spreadsheets/d
| /14CVXd6PTIYE9XlNpRsxJ...
|
| from https://github.com/IridiumIO/CompactGUI
| taeric wrote:
| Thinking more on it, I am not too shocked that they would
| use uncompressed assets "at rest" on the machine. This is
| especially true for games mailed out on disks or
| cartridges. I'm still a little surprised, so the links are
| neat to read on.
|
| I am kind of curious on why they wouldn't use compression
| in transport, as it were. I'm still mostly assuming that
| they do.
| kevingadd wrote:
| This list appears to be largely indie games, many with
| sizes under a gigabyte. I'm not sure why at-rest
| compression would be a high priority for a small team when
| their game fits on a $5 thumbdrive and will be compressed
| during the install process by Steam or Itch.
| xaduha wrote:
| You can sort by the number of submissions to see more
| popular games.
| Waterluvian wrote:
| In terms of attracting engineering talent, how does the video
| games industry fare?
|
| In a crunch situation, I can think of how easy it is for some
| to be able to circumnavigate various pitfalls while others fall
| into them and have no time to address it.
|
| Very much a skill/experience thing. There's countless examples
| of games that run horribly given their level of presentation,
| or have no business being so impressively performant on the
| hardware they're running.
|
| I can guess what execs will pick when faced with "delay the
| game or ship a massive bundle."
| PeterisP wrote:
| They can get great people but then choose to spend their
| effort solely on features and ignoring things like size and
| speed - for example, the GTA V loading time issue
| (https://nee.lv/2021/02/28/How-I-cut-GTA-Online-loading-
| times...) is a good example, where GTA V did many things
| really well technically but then completely ignored a huge
| loading performance issue for many years; things like that
| aren't "very much a skill/experience thing", the difference
| is whether the company cares.
| syntheweave wrote:
| Game programmers can do some amazing things, but they
| generally aren't given the schedules to do them "properly".
|
| If most commercial software is a feature factory, games are
| "megafactories" because they are not really competing on the
| robustness of the implementation, just on delivering a
| certain kind of experience that delivers some interactivity
| and shows off some assets, and that tends to lead towards
| approaches that hack one feature on top of another and then
| lean on manual testing to catch "enough" errors. Many games
| also take characteristic shortcuts that eliminate some
| options for game design in order to simplify the tech.
|
| The larger companies will often keep their tech teams
| separated from their game teams and treat the tech coders
| more generously. This comes with all the pluses and minuses
| of siloing the work - sometimes they solve the wrong problem,
| but do it well.
| klodolph wrote:
| > In terms of attracting engineering talent, how does the
| video games industry fare?
|
| When I talk to people in the game industry, it sounds a lot
| like my experiences outside it. Plenty of programming skill
| to go around, but the skill is often allocated inefficiently,
| and everyone's under pressure to just ship stuff and move on
| to the next deliverable.
| jjoonathan wrote:
| It attracts truly extraordinary talent. One of many examples:
| https://www.youtube.com/watch?v=eviSykqSUUw . There is more
| technical ambition in a single slide of that project than an
| entire department at places I've worked.
|
| However, it also tends to demand more and pay less, so there
| is churn and burnout and when it comes time to triage, the
| triage can cut very deep.
|
| If I were to dive into the JIRA for any one of these "missing
| compression" incidents, I wouldn't expect to find an easy
| win, I would expect to find "missing compression" beneath a
| pile of much worse bugs like "level 2 crashes on AMD cards."
| So it goes.
| Sohcahtoa82 wrote:
| > However, it also tends to demand more and pay less, so
| there is churn and burnout and when it comes time to
| triage, the triage can cut very deep.
|
| Yeah, I've often heard that the advice for people who
| really want to make games is to get some friends together
| and try to put out something together as a side gig while
| you work somewhere boring to pay the bills.
|
| SWE at a game studio, especially any AAA game studio, is an
| awful experience. The expectations are high and the
| salaries are low because there's a seemingly unlimited
| bunch of young adults that want to make games.
|
| And yes, it takes extraordinary talent. Your code needs to
| be able to generate a frame in under 16 milliseconds in
| order to maintain 60 fps. If you want to appease the
| hardcore players with these crazy 360 hz monitors, you need
| to get your game loop down to a mere 3 ms.
|
| The amount of shortcuts and optimizations needed to pull
| that off is insane. At that level, you're trying to figure
| out how to minimize cache misses and branch mispredictions.
| How many Node devs are thinking about those?
| gabereiser wrote:
| This is mostly true. The packages of assets for games are
| compressed but the individual assets are usually not. It's
| mostly fancy zip of files with maybe a different header.
| kevingadd wrote:
| So they're compressed? What's wrong with compressing
| packages? That usually produces a better compression ratio by
| exploiting redundancy cross-file.
| bentcorner wrote:
| Agree - OP even links to this[1] PC gamer article which calls
| out Hitman 3 as having a much smaller size compared to Hitman 2
| (and Hitman 3 _includes all of Hitman 2_ ). How did they do it?
| Compression[2].
|
| [1] https://www.pcgamer.com/biggest-game-install-sizes/
|
| [2] https://www.pcgamer.com/how-hitman-3s-devs-shrank-the-
| entire...
| jsheard wrote:
| There was more to it than just compression. Hitman 2 allowed
| the player to install/uninstall each level individually, and
| the way that was implemented meant that each level had its
| own copy of every asset it contained. Hitman 3 removed that
| option and simply shipped every level in one unified package,
| with assets used in multiple levels only stored once.
| yjftsjthsd-h wrote:
| > they ACTIVELY duplicate their assets on disk so they load a
| bit faster on HDD
|
| How does that work?
| sudosysgen wrote:
| Shorter seek times because the bits are likely to be on the
| same or nearby tracks.
| nijave wrote:
| Yup, basically converting random IO to sequential IO
| babypuncher wrote:
| There's no excuse for Jedi Survivor not to compress it's
| assets. Respawn is skipping last-gen consoles with this game.
| The modern consoles have significantly more CPU power at their
| disposal, and dedicated hardware for decompressing texture and
| sound data.
|
| There is no excuse for them to be shipping a 150GB game on the
| PS5 when so many cross-gen games are 40-60% smaller on PS5 than
| PS4
| ehaliewicz2 wrote:
| It's easy to say something like that, but as mentioned above,
| games are generally duct taped together at the literal last
| minute and it's easy to miss things that, to outsiders, seem
| obvious. The difference is that we aren't looking at
| literally every piece of the game and rushing to fix as many
| bugs as possible, like the devs, we're just looking at the
| file size.
| terafo wrote:
| And, besides, his solutions aren't really good. Radical
| solution would be shipping image generating model with the game
| and generating textures on the fly. That's how all texture-
| related data goes from being measured in hundreds of gigs to
| being measured in hundreds of megs. So basically becomes non-
| issue.
| circuit10 wrote:
| That would be far, far slower than just downloading the image
| terafo wrote:
| 8k texture with mipmaps takes around 100 megs uncompressed.
| Downloading it would be 8-10 seconds on 100 mbit
| connection. It is definitely possible to generate that
| image in 8 to 10 seconds on consumer GPU if you are using
| some kind of consistency models that are trained to
| generate specific game style and inferences in 8 bit.
| circuit10 wrote:
| I doubt you can do it in 8K at that speed and also why
| not just compress it? Any quality degradation from
| compression will be less than the artifacts produced by
| an AI image generator
| terafo wrote:
| _> I doubt you can do it in 8K at that speed_
|
| All inference of Stable Diffusion is done in FP16 at
| best, with multi-step models, and models themselves are
| highly inefficient. I would be surprised if in May 2024
| 4090 isn't capable of generating 8k(8192x8192) Stable
| Diffusion quality image in less than 1 second. And also
| you wouldn't realistically need to
|
| _> also why not just compress it? Any quality
| degradation from compression will be less than the
| artifacts produced by an AI image generator_
|
| Because it allows for much more variance in content. I'm
| a bit sick of seeing tiled textures everywhere TBH. Also
| virtually all textures would be AI generated in less than
| 5 years.
| dalmo3 wrote:
| > on consumer GPU
|
| You make it sound like my electricity bill would be lower
| by just downloading the bigger files...
| kevingadd wrote:
| Saying "They take hundreds of gigs because they don't use ANY
| compression." is absurd when you immediately have to caveat
| that codecs like BC7 are used for texture data, which makes up
| a large portion of the shipped data.
|
| Sony included _dedicated compression support_ in the PS5,
| licensed from a leading middleware company, specifically to
| make it easier to use compression in games. Games are using it.
|
| I don't know why you're trotting out this misinformed BS.
|
| You can also simply watch the statistics when Steam is
| installing a game - the amount written to disk is larger than
| the amount downloaded over the network, because they also
| compress assets in flight to reduce bandwidth usage (the most
| important thing, since raw assets often load faster from an SSD
| as long as you have available storage.)
|
| You can argue every single file should be compressed at rest on
| persistent storage after install, but that's a losing argument.
| People _really_ hate longer load times, so compression at rest
| is only appropriate if it improves them.
| charcircuit wrote:
| At least for steam all chunks of a game download are compressed
| and encrypted.
| ricardobayes wrote:
| I remember Fallout New Vegas was almost 50 GB because it was a
| parallel install of all language/localization files. Once you
| manually deleted the languages you didn't want it went back to
| like 10 GB.
| terafo wrote:
| And there was a hilarious situation with audio in Titanfall,
| which included 35 gigs of uncompressed audio(whole game took
| 48 gigs).
| klodolph wrote:
| As I recall, this was a purposeful choice. Titanfall came
| out in 2014. The minimum requirements were a Core 2 Duo at
| 2.4 GHz or Athlon X2 at 2.8 GHz. I think that puts it on
| processors dating back to 2006 or so.
|
| Games sometimes have a lot of sound effects. It's easier to
| stream 35 GB of data from disk than it is to decompress,
| like, 10 streams of audio at the same time with whatever
| CPU you can dedicate to audio.
|
| I do think they could have figured something out, like
| compressing the right subset of audio assets, but I just
| want to give the context for why they made that choice.
| recursive wrote:
| That's like 50 hours of uncompressed stereo audio at CD
| quality. (2x16bit@44.1kHz) Do games really have that much
| stuff in them these days? I mean, I wouldn't doubt it,
| but it's surprising.
| kevingadd wrote:
| The uncompressed audio was on purpose to manage CPU usage,
| you can find quotes explaining this in interviews.
|
| https://www.escapistmagazine.com/titanfall-dev-explains-
| the-...
|
| I can attest that MP3 and OGG decompression (the two audio
| codecs that would have likely been used at the time) are
| both pretty slow. It adds up.
| Sohcahtoa82 wrote:
| What a bunch of absolute horse shit.
|
| MP3 isn't slow. I was playing MP3s on a 100 Mhz 486 DX4
| in 1996. Certainly a system in 2014 can handle
| decompressing MP3s on the fly.
|
| Even if you didn't want to do it on the fly during game
| play, they easily could have been decompressed at load
| time and it would have taken only a couple seconds. It's
| not like ENTIRE 35 GB of audio needed to be loaded all at
| once, nobody had that RAM back then, and very few people
| (especially gamers) have that much now.
| overthrow wrote:
| If you compress data, it will be smaller and require less i/o
| to load, so it will load faster. That's ESPECIALLY true on
| HDDs, but even on SSDs there are decompression algos that run
| faster than SSD bandwidth. The uncompressed asset cargo cult is
| hurting everyone who plays their games, no matter what kind of
| disk they use.
| babypuncher wrote:
| Jedi Survivor isn't shipping on any platforms constrained by
| an HDD, so it would be odd if they were making any
| optimizations for spinning rust, especially when they come at
| the cost of precious SSD space.
| falcolas wrote:
| It's releasing on PC, which could be constrained. It wont
| be for a majority of players, but HDD setups still exist.
| kevingadd wrote:
| Compression does not guarantee faster loads anymore.
| Uncompressed assets can be DMA'd directly to their
| destination. This is part of why GPU decompression (gdeflate
| like in directstorage, or the special decompression tech in
| the PS5) is on the rise because it allows you to DMA
| resources to their destination on the GPU and then decompress
| them there to avoid the loading time hit involved in
| decompressing them on-CPU after copying them disk->page
| cache->userspace app memory.
|
| For HDDs, yes, compression is generally a win. HDDs are dead
| as far as game development is concerned, all modern targets
| have flash storage.
|
| Some compression codecs that produce smaller files are also
| MUCH slower to decompress, which swamps any benefits from the
| reduced i/o. This is part of why sometimes people pick less
| efficient formats for use in games.
| blibble wrote:
| if it's on a modern SSD you can DMA it directly into the GPU
|
| no CPU required
| ByThyGrace wrote:
| Interesting. How modern are we talking here? Is this a
| "devil's in the details" kind-of feature?
| [deleted]
| mxmlnkn wrote:
| Slow downloads wouldn't be a problem if shipping actually
| finished games on Blu-rays were still in vogue. And the disc
| limitation would make some publishers think twice about shipping
| 250 GB on 3 or more discs.
| [deleted]
| rektide wrote:
| Richg42 is great & his pieces are fun to read.
|
| I do think in this case he misses a bit of the point, goes too
| far:
|
| > _They are designed for fast loading and streaming off SSD 's.
| Who cares about saving off a few hundred ms (or a second) when it
| takes hours or days to download the product onto the SSD?_
|
| Most games will have only one two or three downloads over the
| user's lifetime, and it'll never be instant, will always require
| some wait.
|
| But we load textures many times if we keep playing a game.
|
| Rich's overall point remains, but I think some balance of
| optimizing load speed is also very appreciated & very necessary.
| overthrow wrote:
| That can be solved on first load with a texture cache. Shader
| caches are pretty universal at this point. Texture caches
| aren't (afaik), but I think they should be.
| rektide wrote:
| So now we would have faster download & faster load, but we'd
| have significantly bugger hard drive space usage than we
| have? That also seems like a concern.
| hellotomyrars wrote:
| Game size is a perennial talking point and some games earn it and
| some games don't.
|
| There was a phase of games shipping with uncompressed audio in
| every language they support for no good reason that ballooned
| file size (Titanfall)
|
| Some games have gotten around this problem by having higher
| resolution textures or additional languages being optional
| installs within whatever ecosystems DLC implementation is. I
| think this is the smart way to do it.
|
| Examples of games shrinking I are hard to come by but special
| credit to IO interactive and Hitman 3.
|
| The install size with all the content of the previous games
| included (If you own them) is less than the install size of the
| first game. That is impressive and commendable.
|
| Games are weird and some people will look at the file size and
| think it means the game is better, just the way some people think
| games with more hours are better. The psychology of people who
| play video games is weird.
|
| Duplication of assets on disk was a very weird byproduct of HDD
| file access conventions, but we shouldn't need that anymore.
| There is still a shrinking, but valid crossover window between
| devs being able to assume all storage is solid state. I think
| things will get better but the debate of file size vs
| average/expects storage capacity vs Network speed will never end
| but there is certainly work to be done.
| RajT88 wrote:
| The .kkreiger devs really showed how far you can go to reduce
| download size:
|
| https://en.wikipedia.org/wiki/.kkrieger
|
| https://www.youtube.com/watch?v=ya_MUKc343U
|
| It is dated, but looks pretty good by 2004 standards.
| thequux wrote:
| While I'm all for reducing download sizes, perhaps lekktor[1]
| is a bridge too far.
|
| [1]:
| https://fgiesen.wordpress.com/2012/04/08/metaprogramming-
| for...
| kevingadd wrote:
| Titanfall shipped uncompressed audio on purpose to cut out
| audio decompression CPU usage. You can disagree with the
| decision, but it wasn't an accident or negligence.
| haswell wrote:
| I realize they're trying to optimize for a wide base, but I
| have plenty of spare cores to decompress audio, but SSD space
| is limited.
|
| For me, the net result is that Titanfall was one of the first
| games to go when I needed space. Low hanging fruit.
|
| Even if it was deliberate, I have to question the logic and
| wonder how much usage they enabled vs. uninstalls from file
| size over time.
| KMnO4 wrote:
| Games want to be more photorealistic and the hardware to fake it
| is getting faster and cheaper.
|
| I remember in 2008 when Intel demoed ray tracing[0] for Quake
| Wars. It ran on a quad-socket (4 CPUs x 2.7GHz) at around 25 FPS,
| and even Intel admitted that ray tracing is just "brute forcing"
| the complex optical physics of a scene.
|
| Considering between then and now, we have much prettier looking
| games that do NOT require ray tracing for their visual effects
| (and can run on much fewer CPU cycles), thanks to much more
| "clever" techniques and algorithms (DoF, bloom, subsurface
| scattering, PBR, etc), and if it weren't for the RTX graphics
| cards setting the precedent, there would probably be more
| research into making things prettier without having to "brute
| force" things.
|
| Nowadays, you can drop a 100gb mesh into UE5 Nanite[1] and skip a
| lot of the work involved in making things pretty. You don't need
| to generate or even understand displacement maps if you can just
| add more polygons to the model. You don't need to billboard the
| mountain way in the distance if Unreal does it on the fly. Who
| cares if that means including a 250mb mesh in your assets, versus
| a 5mb HDRI.
|
| So yes, producers have either become lazy or are simply unable to
| meet the demands of the hardware by coming up with better
| algorithms. That's unlikely to change until either hardware stops
| advancing and either A) can't run the latest games or B) games
| become too large that they're impossible to download.
|
| [0]: https://www.gamedeveloper.com/programming/sponsored-
| feature-...
|
| [1]: https://docs.unrealengine.com/5.0/en-US/nanite-
| virtualized-g...
|
| [2]:
| https://twitter.com/mariobrothblog/status/163604089376436224...
| Sohcahtoa82 wrote:
| > clever" techniques and algorithms
|
| Lately I've become hyper-aware of games doing screen-space
| reflections and suddenly finding it distracting when I point my
| view down and the reflection of the scene at the horizon
| vanishes off the water/floor.
|
| I upgraded from a 1070 to a 3080 right when the 3080 came out,
| and I actually have yet to play an RTX game.
|
| > bloom
|
| I hate that this ever became a thing.
|
| Bloom just makes the scene blurry and bright parts end up blown
| out.
| terafo wrote:
| > _and if it weren 't for the RTX graphics cards setting the
| precedent, there would probably be more research into making
| things prettier without having to "brute force" things._
|
| Not really. A lot of this research included making rudimentary
| versions of RTGI that could run on hardware of that age(see
| SVOGI). Of course you can bake everything and make amazing
| looking levels, that are completely static. But that doesn't
| work with open-world games, especially if they have different
| times of day. And it doesn't work in highly dynamic scenes. We
| basically hit the end of the road for scaling graphics without
| RT. And now we have hardware that can do realtime path-traced
| rendering of major AAA titles, and it looks miles better than
| anything smoke and mirrors could've ever delivered.
| W4RH4WK55 wrote:
| We recently had blog post closely related to the topic:
| https://ph3at.github.io/posts/Asset-Compression/
| kevingadd wrote:
| Great post that helps people understand why smaller isn't
| always better.
| arthur2e5 wrote:
| Thing should get its own submission.
| pornel wrote:
| Compression optimizes for wall clock time which is (roughly
| speaking) sum of time it takes to get the compressed data and
| time it takes to decompress it. Downloads over the network are
| almost always bottlenecked on the network speed, so the heavier
| compression you have the better. However, when loading data from
| the local disk, the I/O can be so fast that the decompression
| time becomes the bottleneck and slows down loading.
|
| Because of that optimization for download time and startup time
| end up optimizing for different things.
|
| With loading speeds so different in these two cases, I wonder if
| the best solution would be to _recompress_ the data with a
| different compression level during download and installation. For
| example, download textures compressed with a good but slow and
| non-random-access image compression format, and then convert them
| to a lighter GPU-friendly compression format locally.
| kevingadd wrote:
| Heavy compression for the first install is a common tactic.
| Steam and itch do it, and brotli transfer compression is widely
| used on the web.
| LinuxBender wrote:
| I recall a lot of hype more than a decade ago about all the
| textures going away and being replaced with math formulas to
| generate everything on the fly. Games were supposed to get
| dramatically smaller. What ever happened to that?
| GuB-42 wrote:
| And there was also the opposite, John Carmack in particular
| didn't have a problem with using huge amount of data, "Rage" is
| an application of this idea. The game uses a single, huge, non-
| repeating "megatexture", with elaborate caching strategies to
| keep it playable even on slow media (HDD/optical). The idea was
| to give complete freedom to artists and not require them to be
| mathematicians. Further down the line, he even considered using
| sparse voxel octrees, i.e. even more huge 3D textures.
|
| On the opposite end of the spectrum, you have Inigo Quilez (
| https://iquilezles.org/ ). He literally draws with maths, and
| he is famous for his 4k demoscene productions. The problem is
| that about no one else is capable of doing what he does.
| gatane wrote:
| I see that Inigo Quilez work is more in the "render a scene
| not necesarily in real time" realm. Like, you could make a
| raytracing program, make it render off camera a shiny scene,
| and only after rendering it shows you the image.
|
| Because of this, its very suited to math and procedural
| generation. You just change some parameters, render and show.
| Games instead have to do this in real time, with less latency
| and CPU/GPU usage. And procedural/math generated content
| tends to be CPU heavy (more usage+latency), if not also on
| the GPU (cant be cached, etc).
|
| Shaders are more on the middle of this, because you want to
| apply certains transformations to and from rendering data.
|
| Just my armchair observation.
| babypuncher wrote:
| It is an order of magnitude more expensive to produce art
| assets that way than to just hire people familiar with
| photoshop and maya. Those procedurally generated assets in tech
| demos like .kkrieger were just that: tech demos.
|
| Asset production is already the bulk of the cost of AAA video
| game production.
| db48x wrote:
| Economics. Supply and demand. To create good procedural art you
| have to hire software engineers with artistic talent, which are
| more expensive than the texture and model artists you can find
| by the hundreds coming out of art schools.
| mabbo wrote:
| Games will be as large as they can be until it starts causing
| people not to buy the game. Before that point, no one will care.
|
| And as someone who just got 1.5Gbps internet installed, my
| tolerance for file size is continually increasing.
| MrFoof wrote:
| The Moore's Law is Dead podcast recently did a podcast with an
| Infinity Ward award-winning effects artist:
| https://www.youtube.com/watch?v=tmfHxJT1I3I.
|
| The crazy thing he mentioned is that if there was zero
| compression, the latest COD games genuinely had over 1 terabyte
| of uncompressed assets.
| teaearlgraycold wrote:
| But why?
| MrFoof wrote:
| Listen to the answer to the question, _" How long until our
| game installs take up 500 gigabytes of space"_:
| https://youtu.be/tmfHxJT1I3I?t=7152s
|
| The reality is a game has increasingly more content nowadays.
| Remember, there's a lot of "games as a service". So you have
| many "seasons" of additional content. You may not play all
| the content, but _someone_ is. And when people want 4K and 8K
| textures, different variations so effects don 't look the
| same twice, on top of all the other sizes, plus all the other
| effects ... it adds up. Never mind huge open worlds, where
| the asset footprint for just the map might be 100GB.
|
| Having been in a few AAA studios myself _(doing database
| stuff)_ and sitting next to artists and even art directors,
| the sheer amount of effort, headcount, etc. put into the art
| pipeline is genuine madness. In a large studio, art is over
| HALF the production cost, sometimes more towards 60-65%.
| Heck, one of the things sometimes brought up about Red Dead
| Redemption II is that there were three full-time employees in
| which all they were responsible for was horses. 6000 man-
| hours a year, on just horses.
|
| To sum up what Chris said towards the end, _" The only thing
| keeping these numbers down is the compression algorithms. If
| we didn't have these newer compression algorithms, we'd
| probably legitimately be at a terabyte. I can actually tell
| you for a fact, that there are over a terabyte of assets in
| the COD engine."_
| ok_dad wrote:
| Have you played a modern game? they have amazing levels of
| detail in the tiniest places. One game I play, Gran Turismo
| 7, claims that a single car model start to finish takes
| months to model for a single modeler. Some of the details in
| those cars that you can see in VR you can absolutely not see
| in-game normally, such as the door lock controls or the key-
| hole for the driver's door being exactly like on the real
| car, even for something so rare as a McLaren F1 longtail.
| There is so much waste of time and space dedicated to this
| stuff, but I love it personally!
| taeric wrote:
| Gran Turismo in VR is an amazing experience. I'm not even a
| racing car fan, but holy crap does that game really capture
| an experience. I'm somewhat afraid to find out just how
| much data goes into a single model.
| crummy wrote:
| https://acronyms.thefreedictionary.com/LZ
|
| I think the right definition is this one:
|
| Lempel-Ziv (dictionary-based lossless data compression)
| thfuran wrote:
| It is
| charlie0 wrote:
| If users hate large file sizes, they should just stop buying
| them. Unfortunately, hitting the bottom line is really the only
| way to get this resolved.
| tmilard wrote:
| Chocked at FAT size video games file ? Me too. Giga bytes Damned
| !
|
| To reduce the size, I decided : 1) To use the standard PNG
| compression. 2) To limit mesh size at less than 300 points for a
| scene.
|
| - Result : a) File size < 1 mega Byte ( really). b) It downloads
| in 4 seconds.
|
| Check it here : https://free-visit.net/fr/demo01 (pleaseload url
| twice : I still have a bug)
|
| Question : - why video games STUDIOS do not use this strategy ?
| kevingadd wrote:
| PNG decompression is way too slow. Some games use it but it's
| on the way out, either being replaced by codecs like Basis or
| QOI in most cases
| tmilard wrote:
| Really ? It does not feel this way on my scene that has more
| than 90 .png files used for texture.
|
| The time to : download the globel zip file + open and reading
| the points and all textures => 5 to 7 seconds.
| egypturnash wrote:
| _To limit mesh size at less than 300 points for a scene._
|
| I am pretty sure most modern video games use more mesh points
| than that for, like, a hat your character can wear.
| tmilard wrote:
| I it is true : I was very agressive on this one. But still,
| thousands and tousands of triangle feels like so much
| overweight for me. Specialy for indoor scenes. Quality for
| the eye is a lot to do with texture, NOT the number of
| triangle.
| tmilard wrote:
| And I do not even talk about the GPU work to handle those
| thousands of triangles. With 300 points, my scene runs
| quick&fast even on budget Smartphones ! This is an
| important issues I can tell you. A video game that runs on
| all devices ...
| Sohcahtoa82 wrote:
| At 300 points, a computer from 1996 could run your game.
| terafo wrote:
| Because PNG isn't a good format for GPU textures. Oodle
| texture+Kraken+something like BC7 is an optimal solution and is
| used in many games.
| tmilard wrote:
| I think in today's machine (be it PC, Mac or even
| Smartphones) the reading-and-transform of a png asset for the
| GPU is hard coded in the processor : so VERY fast. It works
| well.
| mrkramer wrote:
| Cloud gaming is.
| adgjlsfhk1 wrote:
| I really want to see someone make a renderer that can work on
| compressed textures. No need to decompress if you can just read
| the data on the fly.
| aappleby wrote:
| Renderers have been using compressed textures for literal
| decades. Not JPG, alas, because the compression formats have to
| support random access to individual texels.
| corysama wrote:
| https://www.reedbeta.com/blog/understanding-bcn-texture-comp...
|
| BC1-5 have been standard practice since before the original
| XBox. I think when people here are talking about "uncompressed
| assets" they mean "not applying variable-rate zip style
| compression on top of BCn compression."
| codedokode wrote:
| Why not simply make several versions? Like 1 Gb, 10 Gb and 100 Gb
| versions?
| jsheard wrote:
| Some games do this by splitting off the highest resolution
| textures into a free "DLC" that the user can opt not to install
| if they want to save space. It's sometimes done for audio as
| well, e.g. optional Japanese voices for the English release of
| a JRPG.
|
| Remedy's Quantum Break had an unusual spin on letting users
| choose the install size, it made heavy use of live action FMVs
| which could either be downloaded ahead of time or streamed on
| the fly as they played. Downloading the FMVs more than doubled
| the games footprint.
| tmilard wrote:
| good idea : one why chitty etxture quality, one with average,
| one with quality texture. They have done this on web-videos.
| Why not on video games files ?
| kevingadd wrote:
| They do, the Xbox calls it smart delivery.
| bick_nyers wrote:
| Faster internet and cheaper storage is the answer :)
|
| Or at the very least just offer two different downloads of the
| game, one with lossless textures and one with lossy textures...
| topspin wrote:
| Guild Wars 2 and others solve this with progressive downloads.
| You may start playing after a fraction of the content is
| downloaded. There is effectively no downside: new players can't
| access most of the content anyhow. It also applies to updates:
| you can start playing after a big update with a partial
| download. The background process can be safely interrupted and
| resumed without glitches.
| arthur2e5 wrote:
| War Thunder kind of has that, except that the "progressive"
| bit means you need to download the full thing to progress in
| the game beyond a certain point.
|
| Enlisted, which uses the same engine, is fine though.
| topspin wrote:
| The design of ArenaNet's GW/GW2 has always impressed me.
| 99.4% of all of the game is contained in one file: GW2.DAT,
| which is (currently) 66.7 GB. I/O for updates is extremely
| efficient.
|
| I've seen many games create a vast file system mess, and
| they're forever plagued by "corruption" or something,
| complete with "fix" tools, "cache" clearing push-ups and
| players reinstalling after big updates that don't install
| correctly for whatever reason. Somehow ArenaNet surrounded
| this whole problem and solved it optimally.
| kevingadd wrote:
| A massive amount of engineering went into making it work
| and some of the stuff we did was downright evil :) in
| gw1, a lot of data was compiled into the exe as c
| constants so that we wouldn't have to do a bunch of file
| management and I/o to locate and load it. We ended up
| hitting limitations in the C compiler.
| kevingadd wrote:
| Rich is a compression expert who wrote a good post, and there are
| some good comments in this thread already, but a few key summary
| points to help people understand the tradeoffs here:
|
| * Steam, Itch, etc all compress game files before sending them to
| you over the network for an install. The amount of stuff you
| download is usually much less than what gets stored on disk.
| There are still games with unusually massive updates though,
| especially ones where the files change enough that delta
| compression fails. This is difficult to avoid when using some
| middleware due to the design of its file formats. Compressed
| archives actually _make this worse_ because you can 't delta
| compress big archives as well as individual files.
|
| * Smaller on disk doesn't mean faster to load anymore. In the era
| of spinning rust HDDs, smaller usually loaded faster, but there
| were extremes - some compression formats that produced massive
| size reductions were also 10x slower to decompress or worse,
| which could produce slower overall game load times. Now that most
| targets have SSDs, this means that less compression often
| produces faster loads. See https://ph3at.github.io/posts/Asset-
| Compression/ (from elsewhere in this thread) for one example.
|
| * For image and geometry data that is headed for the GPU, it's
| best to pick compression formats that can either be natively
| consumed by the hardware (block texture formats like BC7, etc) or
| that can be efficiently decompressed on the GPU once you copy it
| there, like gdeflate. This means the file on disk will probably
| be bigger, unless you supercompress it using something like
| Basis. Basis adds measurable CPU time to loading, though, and as
| a result I'm considering doing an uncompressed texture cache in
| my game to improve load times even though I plan to continue
| shipping textures as basis KTX2.
|
| * For streaming asset loads, the CPU usage to decompress the
| assets during gameplay could lead to frame drops, inconsistent
| frame pacing, and pop-in. My understanding is that Titanfall
| opted for uncompressed audio in order to keep CPU usage low when
| streaming in all the sound effects and voices that could play
| during a hectic match - at that point in time CPU usage for audio
| mixing was a serious concern; modern CPUs more headroom so it's
| less of a problem now. It made sense for them to choose this as a
| trade-off because high-framerate twitch online shooters
| prioritize low consistent latency.
|
| * In some cases the install you get depends on the hardware you
| have. It's not uncommon for older platforms to see 'high res
| texture packs' that could be installed optionally. The Xbox
| platform has something called "smart delivery" where it will
| deliver a smaller game bundle depending on whether your
| particular console needs high-def textures. This takes real
| engineering work to do, so many games opt not to support it
| because that time is better spent elsewhere.
|
| * Ultimately, unless your game is too big to fit on people's hard
| drives at all, there is usually something more important for you
| to work on vs implementing some complex compression algorithm for
| every one of your files at rest. You are probably also not as
| smart as the compression experts who wrote the compression codecs
| that Steam/Itch/Xbox/Playstation offer, either for install-time
| compression (to reduce network bandwidth) or texture compression,
| etc. So in practice you probably just pick an off-the-shelf
| compression solution and apply it where it improves load times
| and you leave everything else alone. It doesn't make sense to
| spend weeks shrinking your on-disk size by 5% when you could
| spend that time fixing crash bugs, improving accessibility with
| better control customization, or improving the game's framerate
| instead.
|
| * Lots of people decide a game "isn't compressed" because you can
| shove it into an lzma archive and make it smaller. This
| misunderstands the issue. Many compressed or optimized-for-size
| formats can still be made smaller by using a really heavyweight
| compression codec like lzma or brotli, but critically, there are
| tradeoffs involved - slower compression means slower build times
| means longer development times, and slower decompression means
| worse load times. The best compression typically also requires
| you to bundle all your files together, which loses the ability to
| load only the data you want when you want it.
|
| An anecdote: Back when I was working on the original WebAssembly
| spec, I spent multiple weeks doing lots of research into custom
| compression and how we could design the format to make it
| compress more efficiently. I even had some chats with the Brotli
| team. In the end, we barely did anything, because even months of
| hard work barely produced better results than just Brotli-
| compressing our files at maximum settings. We were able to
| improve gzipped sizes tremendously by filtering and rearranging
| the data, but there's no point doing it if you can just use the
| browser's built in brotli decoder instead.
___________________________________________________________________
(page generated 2023-04-21 23:01 UTC)