[HN Gopher] Faster LZ is not the answer to 150-250 GB video game...
___________________________________________________________________
Faster LZ is not the answer to 150-250 GB video game downloads
Author : ingve
Score : 178 points
Date : 2023-04-21 19:52 UTC (1 days ago)
(HTM) web link (richg42.blogspot.com)
(TXT) w3m dump (richg42.blogspot.com)
| terafo wrote:
| He completely misses the point. Games don't take hundreds of gigs
| because they have sub-optimal compression. They take hundreds of
| gigs because they don't use ANY compression. Jedi Survivor takes
| 139 gigs on Series X and mere 44 gigs on Series S. It didn't come
| out yet, but I'm 99.99% certain that this difference is just from
| inclusion of 8k textures into Series X version. That kind of
| scaling means that they don't use any compression beyond
| something like BC7. And some games are not only without any
| compression, they ACTIVELY duplicate their assets on disk so they
| load a bit faster on HDD. And these builds end up on modern
| consoles that have ludicrously fast SSDs.
| jjoonathan wrote:
| In 2003, I found out that America's Army (the US Army
| videogame) had compressed and decompressed all their textures
| at some point, so they were paying the quality hit for
| compression _and_ the space hit for RGBA. This was a time when
| the game still took a day to download, even on a half decent
| connection, so >200% footprint inflation kinda mattered. The
| more things change, the more they stay the same!
| sgtnoodle wrote:
| I have fond memories of that game. 80% of the time I would
| play very patiently and carefully. Occasionally, though, I
| would go Rambo and take out the whole enemy team in 30
| seconds.
| blarg1 wrote:
| I use to pretend to go afk, and while the enemy was doing a
| little dance in front of me I would shoot them :)
| taeric wrote:
| This feels wrong. In that, I'd be surprised if they are not
| using compression. I think folks are just drastically
| underestimating how many resources are under discussion. And
| how big they are.
|
| Heck, just the geometry is probably far larger than folks
| anticipate.
| sudosysgen wrote:
| It's true. I've seen tons of games use basically uncompressed
| audio, badly compressed textures, etc...
|
| People often repack video games, making programs to compress
| and decompress them so they are easier to pirate. It's not
| uncommon to see a 2-3x reduction in size, for bit-identical
| results.
| taeric wrote:
| That is a little surprising to me. Especially if we are
| talking about games that are delivered online. Thinking on
| it, I'm actually only surprised for online delivered games.
| I don't see any incentive to reduce size on delivered
| disks.
|
| I'd expect that they use a compression that is good for
| transport, at transport time. I'm not too shocked that they
| have the assets in a way that is easy to use at runtime.
| coffeebeqn wrote:
| Games are duct taped together at last minute on
| impossible deadlines pretty much every time.
| bombcar wrote:
| This is exactly it - and once released there is rarely
| the motivation to go back and optimize side things. The
| only game I know that has done anything like that is
| Factorio, or the open source reimplementations of various
| engines.
|
| The crackers who remove copy protection often repackage
| (or someone else does) to greatly reduce size; since
| they're in there poking around the code they can see what
| isn't used.
|
| This isn't new, either, ancient console games have unused
| assets and even entire levels in the code but disabled.
| sroussey wrote:
| Textures are not compressed for size on disk, they are
| compressed to the format that can be directly loaded into
| the GPU.
| terafo wrote:
| You can add lossless compression, like Kraken, on top of
| that. PS5 has hardware accelerated Kraken decompression.
| And you can encode textures for GPU in a way that is
| better compressable with lossless algorithms.
| midnightclubbed wrote:
| Hardware accelerated is still not zero cost. Plus if you
| are publishing for the other console and/or PC there is a
| question as to whether you want to do the extra work for
| one platform.
| maccard wrote:
| This is where using something like unreal shines. You get
| the correct default for all the platforms.
| sudosysgen wrote:
| Yes, that's the problem. You can compress them for size
| at download time, and recompress them to a GPU-optimal
| format at the time of installation. You may even be able
| to do it at runtime with modern hardware, shortening
| loadtimes and sometimes even improving performance
| indirectly.
| coffeebeqn wrote:
| Then you have to have the game reprocess all the assets
| on the users machine which could easily take an hour
| xboxnolifes wrote:
| With the size of games nowadays, that's likely a net time
| save for many people from the discounted download time.
| jon-wood wrote:
| Gamers being gamers though they're not going to say
| "well, we on reflection I've not had to wait so long",
| they're going to slam you with one star reviews for the
| fact there's an hour long loading screen on first start.
| xboxnolifes wrote:
| The recompression/decompression logic could (and I would
| argue, should) be packaged separately for distribution
| platform instead of being embedded as part of the game's
| logic. In the case of using Steam, it would be no
| different than how it shows separation of time spent
| downloading vs time spent installing.
|
| This allows it to be ran automatically after download
| without having the user need to launch the game.
| sudosysgen wrote:
| Essentially any modern image compression format is going
| to be limited, or close to, by storage speed (for
| decompression). So unless the user has insanely fast
| internet (and the servers are also able to saturate their
| connection), it will be faster in the end.
| lpghatguy wrote:
| You can combine the approaches! KTX2, for example,
| supports "supercompressing" GPU block compressed formats
| with LZ4 or Zstandard. The project I have been working on
| ships BCn textures compressed with zstd and it has been
| pretty dang good.
| marcellus23 wrote:
| Maybe not for that particular game, but a lot of games have
| been shown to use uncompressed assets. It's not just
| speculation.
| forrestthewoods wrote:
| > It's not just speculation.
|
| So it's exactly speculation. A lot of games have done a lot
| of things for a lot of reasons over the decades.
|
| Why are new games so large? Good question. This entire
| thread is speculative.
| marcellus23 wrote:
| Um, no. It's not speculation. Many games have been shown
| to use uncompressed assets, that's an objective fact.
|
| The thread might be speculating about _why_ they do that,
| or whether this particular game does that, but that's a
| non sequitur, that isn't what I was talking about.
| forrestthewoods wrote:
| Your statement that some games in the past have used
| uncompressed assets is a non-sequitur.
|
| Has any game in the past 5 years that has an install size
| >100gb used uncompressed assets? I would be very very
| surprised! The likelihood of this being the case on PS5
| or XSX hardware is approximately zero.
|
| Yes I'm familiar with the infamous uncompressed audio in
| Titanfall.
| terafo wrote:
| Hitman would be very notable example. Fortnite almost got
| to 100 gigs when they enabled some kind of asset
| compression.
| forrestthewoods wrote:
| Hitman was a special case of bundling older games
| designed designed for spinning disks.
|
| Fortnite didn't publish what they did. But I promise it's
| not "bEnableCompression=true". Compression is used up and
| down the stack with the utmost of care. Even the older
| games that didn't use compression did so only after great
| consideration of the trade offs.
|
| "Games as a Service" games like Fortnite, Destiny, and
| Call of Duty have often bloated and then been optimized.
| The fix is never bEnableCompression. It's more about how
| the content is sliced and packaged. How to efficiently
| serve new content is a really tricky problem. And a
| somewhat new problem that didn't exist 10 years ago. At
| least for consoles.
| terafo wrote:
| Jedi Survivor having uncompressed textures is a
| speculation, but I'm extremely confident that it is this
| way. The fact that games nowadays ship without LOD
| systems, without texture compression, without adequate
| streaming is not just speculation. It is a fact.
| kevingadd wrote:
| You're saying Jedi Survivor doesn't use block texture
| compression? On which target? All targets?
|
| Can you capture a trace in RenderDoc to prove this? Or
| show raw bmp/uncompressed dds files on disk?
| terafo wrote:
| By texture compression I meant something like LZ4 or
| Kraken on top of block texture compression, sorry for
| ambiguity. Of course they use block level compression.
| But it seems that it's the only thing they use, as I said
| in my top level comment.
| kevingadd wrote:
| You'll find that's pretty common since block compression
| already reduces sizes pretty effectively. At install time
| the files will be compressed further over the wire before
| being decompressed, so I imagine the payoff is not that
| big in many cases. I do wonder how much of it is that a
| Kraken license costs money and it takes time to
| integrate, though...
| terafo wrote:
| _> already reduces sizes pretty effectively_
|
| 8k texture with mipmaps takes about 100mb with BC7. Which
| is huge. Should be around 30-40mb with kraken and oodle
| texture.
|
| _> I do wonder how much of it is that a Kraken license
| costs money and it takes time to integrate, though..._
|
| Kraken license doesn't cost anything for Unreal
| developers. And Sony licensed it for all PS5 games.
| taeric wrote:
| Even for transferring the game to local?
|
| I can see some benefits to having uncompressed assets in
| that it would help with asset loading. I'd still be
| surprised to not have any form of compression at all
| happening. I'd also be curious to know just how much the
| assets could be compressed.
| uoaei wrote:
| The geometry would be hard to compress, but I suspect there's
| some clever tricks with some kind of fractal compression[1]
| that could drastically reduce texture sizes for camo,
| foliage, etc. without any effect on aesthetics.
|
| Encoding is the hard part, but only need be done once when
| building the final release.
|
| [1] https://en.wikipedia.org/wiki/Fractal_compression
| gatane wrote:
| Wait, are there more papers on this? Seems interesting.
| zimpenfish wrote:
| When I was messing around with this (~1995?), there was a
| Yuval Fisher paper floating around which did a good job
| of explaining it. Although it looks like it now costs
| ~$40 to get that paper or ~$100 for his book. Helpful!
| zimpenfish wrote:
| > there's some clever tricks with some kind of fractal
| compression
|
| It's a shame that IFS hamstrung the adoption of fractal
| compression with their expensive license fees - wouldn't
| necessarily have been The Future Of Images but at least
| we'd have had the option.
| xaduha wrote:
| > This feels wrong. In that, I'd be surprised if they are not
| using compression.
|
| Check this out, you can sort by the number of submissions to
| filter out indie games https://docs.google.com/spreadsheets/d
| /14CVXd6PTIYE9XlNpRsxJ...
|
| from https://github.com/IridiumIO/CompactGUI
| taeric wrote:
| Thinking more on it, I am not too shocked that they would
| use uncompressed assets "at rest" on the machine. This is
| especially true for games mailed out on disks or
| cartridges. I'm still a little surprised, so the links are
| neat to read on.
|
| I am kind of curious on why they wouldn't use compression
| in transport, as it were. I'm still mostly assuming that
| they do.
| MetaWhirledPeas wrote:
| Yeah I don't understand why they don't just decompress
| the audio into RAM as the map loads. Or even just to some
| swap space.
| kevingadd wrote:
| This list appears to be largely indie games, many with
| sizes under a gigabyte. I'm not sure why at-rest
| compression would be a high priority for a small team when
| their game fits on a $5 thumbdrive and will be compressed
| during the install process by Steam or Itch.
| xaduha wrote:
| You can sort by the number of submissions to see more
| popular games.
| Waterluvian wrote:
| In terms of attracting engineering talent, how does the video
| games industry fare?
|
| In a crunch situation, I can think of how easy it is for some
| to be able to circumnavigate various pitfalls while others fall
| into them and have no time to address it.
|
| Very much a skill/experience thing. There's countless examples
| of games that run horribly given their level of presentation,
| or have no business being so impressively performant on the
| hardware they're running.
|
| I can guess what execs will pick when faced with "delay the
| game or ship a massive bundle."
| naikrovek wrote:
| game developers work so much harder than enterprise
| developers, and enterprise developers look down on game
| developers as is they are sub-human or something. it's
| completely inverted to reality.
|
| I'm not saying that you are doing this, I am saying that this
| is done.
|
| watch some GDC developer videos on YouTube. these people work
| HARD for performance.
| tester756 wrote:
| Are you sure those people were talking about "game
| developers", not "game industry"?
| PeterisP wrote:
| They can get great people but then choose to spend their
| effort solely on features and ignoring things like size and
| speed - for example, the GTA V loading time issue
| (https://nee.lv/2021/02/28/How-I-cut-GTA-Online-loading-
| times...) is a good example, where GTA V did many things
| really well technically but then completely ignored a huge
| loading performance issue for many years; things like that
| aren't "very much a skill/experience thing", the difference
| is whether the company cares.
| syntheweave wrote:
| Game programmers can do some amazing things, but they
| generally aren't given the schedules to do them "properly".
|
| If most commercial software is a feature factory, games are
| "megafactories" because they are not really competing on the
| robustness of the implementation, just on delivering a
| certain kind of experience that delivers some interactivity
| and shows off some assets, and that tends to lead towards
| approaches that hack one feature on top of another and then
| lean on manual testing to catch "enough" errors. Many games
| also take characteristic shortcuts that eliminate some
| options for game design in order to simplify the tech.
|
| The larger companies will often keep their tech teams
| separated from their game teams and treat the tech coders
| more generously. This comes with all the pluses and minuses
| of siloing the work - sometimes they solve the wrong problem,
| but do it well.
| klodolph wrote:
| > In terms of attracting engineering talent, how does the
| video games industry fare?
|
| When I talk to people in the game industry, it sounds a lot
| like my experiences outside it. Plenty of programming skill
| to go around, but the skill is often allocated inefficiently,
| and everyone's under pressure to just ship stuff and move on
| to the next deliverable.
| jjoonathan wrote:
| It attracts truly extraordinary talent. One of many examples:
| https://www.youtube.com/watch?v=eviSykqSUUw . There is more
| technical ambition in a single slide of that project than an
| entire department at places I've worked.
|
| However, it also tends to demand more and pay less, so there
| is churn and burnout and when it comes time to triage, the
| triage can cut very deep.
|
| If I were to dive into the JIRA for any one of these "missing
| compression" incidents, I wouldn't expect to find an easy
| win, I would expect to find "missing compression" beneath a
| pile of much worse bugs like "level 2 crashes on AMD cards."
| So it goes.
| Sohcahtoa82 wrote:
| > However, it also tends to demand more and pay less, so
| there is churn and burnout and when it comes time to
| triage, the triage can cut very deep.
|
| Yeah, I've often heard that the advice for people who
| really want to make games is to get some friends together
| and try to put out something together as a side gig while
| you work somewhere boring to pay the bills.
|
| SWE at a game studio, especially any AAA game studio, is an
| awful experience. The expectations are high and the
| salaries are low because there's a seemingly unlimited
| bunch of young adults that want to make games.
|
| And yes, it takes extraordinary talent. Your code needs to
| be able to generate a frame in under 16 milliseconds in
| order to maintain 60 fps. If you want to appease the
| hardcore players with these crazy 360 hz monitors, you need
| to get your game loop down to a mere 3 ms.
|
| The amount of shortcuts and optimizations needed to pull
| that off is insane. At that level, you're trying to figure
| out how to minimize cache misses and branch mispredictions.
| How many Node devs are thinking about those?
| Waterluvian wrote:
| Just trying to maintain 60fps with a mapping Library like
| OpenLayers is a humbling experience.
| gambiting wrote:
| >>SWE at a game studio, especially any AAA game studio,
| is an awful experience.
|
| Just as a counter point - I work at one of the largest
| AAA game companies and we have great work life
| balance(been here 9 years). Don't remember when was the
| last time I worked more than the required 7.5h/day, never
| worked on the weekend, the people I work with are all
| passionate about their work and want to deliver something
| great. The pay is "ok" - it's not exceptional but it
| allows for good quality of life. In comparison all my
| friends who went to work in traditional software
| development are paid somewhat more than I am but they all
| 100% hate their jobs, so......I don't think it's a bad
| trade off.
|
| As for the infinite supply of people wanting to do this -
| it's only really true at junior level, as soon as you
| talk anyone above junior it's extremely hard to recruit
| and you might be getting a handful of candidates a year.
| We've been trying to hire a senior rendering programmer
| for over a year now and we've had like 5 candidates
| total.
| gabereiser wrote:
| This is mostly true. The packages of assets for games are
| compressed but the individual assets are usually not. It's
| mostly fancy zip of files with maybe a different header.
| kevingadd wrote:
| So they're compressed? What's wrong with compressing
| packages? That usually produces a better compression ratio by
| exploiting redundancy cross-file.
| gabereiser wrote:
| When optimizing load times, it's usually console that's the
| benchmark. PC is assumed to have faster or equal hardware
| so they optimize for the lowest denominator. Consoles
| aren't PC's but they are very close. They lack some
| instructions while gaining others. I can't speak on current
| gen consoles but older ones like the Xbox 360, only could
| handle zip format and at a slow pace. You could show a
| loading screen while you unpack the data in memory or you
| can unpack on disk on installation saving some loading
| time, or it's all uncompressed on consoles leaving the
| console to load the data as fast as console do.
|
| The issue at hand around install size comes down to the
| engines, their ability to decompress asset bundles, the
| fidelity of the assets (8k vs 8k,4k,2k,1024) if no runtime
| optimization is done. WAV sounds vs OGG vs MP3 vs flac,
| there's a lot of shit in those asset packs. You could
| compress it all like they do, into a folder structure you
| can index and load from, or you can write that game code to
| make boss fights cause camera shake, the decisions you make
| making a game are really all about making the game and
| little to do with how best to deliver your content.
|
| Some texture formats like DXT are already compressed. Raw
| PNG's could have compression. All these things are possible
| if the engine supports it.
| dontlaugh wrote:
| It does vary a bit. One exception is that current
| consoles guarantee fast SSDs and DMA to GPU, which can't
| be said of PCs.
| bentcorner wrote:
| Agree - OP even links to this[1] PC gamer article which calls
| out Hitman 3 as having a much smaller size compared to Hitman 2
| (and Hitman 3 _includes all of Hitman 2_ ). How did they do it?
| Compression[2].
|
| [1] https://www.pcgamer.com/biggest-game-install-sizes/
|
| [2] https://www.pcgamer.com/how-hitman-3s-devs-shrank-the-
| entire...
| jsheard wrote:
| There was more to it than just compression. Hitman 2 allowed
| the player to install/uninstall each level individually, and
| the way that was implemented meant that each level had its
| own copy of every asset it contained. Hitman 3 removed that
| option and simply shipped every level in one unified package,
| with assets used in multiple levels only stored once.
| Dylan16807 wrote:
| Hitman 2 had 6-8? locations. They could have pretty easily
| packed the assets into a hundred files based on use and
| made each level install a different subset.
| yjftsjthsd-h wrote:
| > they ACTIVELY duplicate their assets on disk so they load a
| bit faster on HDD
|
| How does that work?
| sudosysgen wrote:
| Shorter seek times because the bits are likely to be on the
| same or nearby tracks.
| nijave wrote:
| Yup, basically converting random IO to sequential IO
| babypuncher wrote:
| There's no excuse for Jedi Survivor not to compress it's
| assets. Respawn is skipping last-gen consoles with this game.
| The modern consoles have significantly more CPU power at their
| disposal, and dedicated hardware for decompressing texture and
| sound data.
|
| There is no excuse for them to be shipping a 150GB game on the
| PS5 when so many cross-gen games are 40-60% smaller on PS5 than
| PS4
| ehaliewicz2 wrote:
| It's easy to say something like that, but as mentioned above,
| games are generally duct taped together at the literal last
| minute and it's easy to miss things that, to outsiders, seem
| obvious. The difference is that we aren't looking at
| literally every piece of the game and rushing to fix as many
| bugs as possible, like the devs, we're just looking at the
| file size.
| ehaliewicz2 wrote:
| Of course, the other possibility is that they were aware of
| it, but it was somewhere in a long list of things that they
| would have liked to fix, but did not have time for.
| naikrovek wrote:
| that IS his point. the problem is not the compression, the
| problem is the size of the assets. compression can not solve
| this. alleviate it a bit, yes, but not solve.
|
| he's saying that game asset paradigms are headed in the wrong
| direction rapidly.
| terafo wrote:
| The problem is that it wouldn't help. We have good enough
| solutions right now, but not many are using them. Better
| compression will be nice, but it will yield less results than
| adoption of something that is already here.
| terafo wrote:
| And, besides, his solutions aren't really good. Radical
| solution would be shipping image generating model with the game
| and generating textures on the fly. That's how all texture-
| related data goes from being measured in hundreds of gigs to
| being measured in hundreds of megs. So basically becomes non-
| issue.
| circuit10 wrote:
| That would be far, far slower than just downloading the image
| terafo wrote:
| 8k texture with mipmaps takes around 100 megs uncompressed.
| Downloading it would be 8-10 seconds on 100 mbit
| connection. It is definitely possible to generate that
| image in 8 to 10 seconds on consumer GPU if you are using
| some kind of consistency models that are trained to
| generate specific game style and inferences in 8 bit.
| circuit10 wrote:
| I doubt you can do it in 8K at that speed and also why
| not just compress it? Any quality degradation from
| compression will be less than the artifacts produced by
| an AI image generator
| terafo wrote:
| _> I doubt you can do it in 8K at that speed_
|
| All inference of Stable Diffusion is done in FP16 at
| best, with multi-step models, and models themselves are
| highly inefficient. I would be surprised if in May 2024
| 4090 isn't capable of generating 8k(8192x8192) Stable
| Diffusion quality image in less than 1 second. And also
| you wouldn't realistically need to
|
| _> also why not just compress it? Any quality
| degradation from compression will be less than the
| artifacts produced by an AI image generator_
|
| Because it allows for much more variance in content. I'm
| a bit sick of seeing tiled textures everywhere TBH. Also
| virtually all textures would be AI generated in less than
| 5 years.
| WithinReason wrote:
| I think this is exactly right. The next step is generated
| meshes/textures and generating entire worlds. You can do
| generation in a smart way so that performance is not an
| issue. Predict what you will need before you need it.
| Generally you would almost never need to generate an 8K
| texture from scratch, you would generate it from a 4K
| texture you already generated earlier, which would be
| much faster. You could have specialised asset generation
| HW on a GPU.
| dalmo3 wrote:
| > on consumer GPU
|
| You make it sound like my electricity bill would be lower
| by just downloading the bigger files...
| kevingadd wrote:
| Saying "They take hundreds of gigs because they don't use ANY
| compression." is absurd when you immediately have to caveat
| that codecs like BC7 are used for texture data, which makes up
| a large portion of the shipped data.
|
| Sony included _dedicated compression support_ in the PS5,
| licensed from a leading middleware company, specifically to
| make it easier to use compression in games. Games are using it.
|
| I don't know why you're trotting out this misinformed BS.
|
| You can also simply watch the statistics when Steam is
| installing a game - the amount written to disk is larger than
| the amount downloaded over the network, because they also
| compress assets in flight to reduce bandwidth usage (the most
| important thing, since raw assets often load faster from an SSD
| as long as you have available storage.)
|
| You can argue every single file should be compressed at rest on
| persistent storage after install, but that's a losing argument.
| People _really_ hate longer load times, so compression at rest
| is only appropriate if it improves them.
| ehaliewicz2 wrote:
| >(the most important thing, since raw assets often load
| faster from an SSD as long as you have available storage.)
|
| >You can argue every single file should be compressed at rest
| on persistent storage after install, but that's a losing
| argument. People really hate longer load times, so
| compression at rest is only appropriate if it improves them.
|
| Is this really true? I feel like you'd need an absolutely
| blisteringly fast SSD to match something like an modern,
| optimized lz decompressor's speed.
| terafo wrote:
| _> immediately have to caveat_
|
| Yeah, I could've indicated better that I'm talking about on-
| disk compression of files.
|
| _> Sony included dedicated compression support in the PS5,
| licensed from a leading middleware company, specifically to
| make it easier to use compression in games. Games are using
| it._
|
| Not really if we look at 3rd party games. Jedi Survivor is
| likely not using that since it takes 147 gigs on PS5 and only
| 139 on Series X. Warzone doesn't use it, Fortnite doesn't use
| it(despite Epic's acquisition of RAD), Hogwarts Legacy
| doesn't use it, Elden Ring doesn't use it, FIFA doesn't use
| it. These are some of the most popular games on the platform.
| My point was that he is talking about something beyond LZ
| compression when most games aren't using tools that are
| available off-the-shelf.
|
| _> You can argue every single file should be compressed at
| rest on persistent storage after install, but that's a losing
| argument. People really hate longer load times, so
| compression at rest is only appropriate if it improves them._
|
| We have hardware-accelerated decompression in all current gen
| consoles and GPU decompression on PC. We would be in much
| better world if everyone used that.
| bananaboy wrote:
| > Not really if we look at 3rd party games.
|
| I find this extremely unlikely. The packaging tools by
| _default_ will compress files when building a package.
| Decompression is transparent and done by the hardware; the
| game doesn 't have to implement anything or link any
| libraries. You would have to explicitly select "no
| compression" during packaging. Warzone definitely uses it -
| I worked on Cold War, Vanguard, and MW2 specifically on
| streaming and decompression across our supported consoles.
| terafo wrote:
| Sorry, the fact that Xbox version is smaller confused me.
| What might be the reason for Xbox install being 20 gigs
| smaller than PS5 one given that PS5 compression is
| generally better?
| bananaboy wrote:
| Great question, I'm not sure off the top of my head
| though. Maybe this is really the annoying copying that
| ps4/5 do where you basically need twice the disk space
| when installing https://www.pushsquare.com/guides/why-
| does-ps5-ps4-copying-u...
| maccard wrote:
| Imagine you have a 1GB game, with a 500MB patch. To be
| able to play the game while downloading, you need the
| full patch space available. To apply the patch to the
| game, you need to grab the old 500MB, stuff it somewhere,
| and replace it with the new 500MB (in that order,
| otherwise if the update is interrupted, the game is
| screwed and you need to redownload the whole thing).
| maccard wrote:
| > Fortnite doesn't use it(despite Epic's acquisition of
| RAD),
|
| You're talking nonsense. I worked on Fortnite and have
| direct experience with it and the HW decompression on PS5.
| I can even point you to the patch [0] where Fortnite
| enabled oodle compression and shrunk the game by 60GB.
|
| [0] https://twitter.com/FortniteStatus/status/1318677841253
| 45382...
| terafo wrote:
| I apologize, it appears that my way of estimating whether
| game uses HW compression is garbage.
| maccard wrote:
| Hey no worries, it can be hard to parse intent from text.
| My response was a bit harsh too!
| charcircuit wrote:
| At least for steam all chunks of a game download are compressed
| and encrypted.
| ricardobayes wrote:
| I remember Fallout New Vegas was almost 50 GB because it was a
| parallel install of all language/localization files. Once you
| manually deleted the languages you didn't want it went back to
| like 10 GB.
| terafo wrote:
| And there was a hilarious situation with audio in Titanfall,
| which included 35 gigs of uncompressed audio(whole game took
| 48 gigs).
| klodolph wrote:
| As I recall, this was a purposeful choice. Titanfall came
| out in 2014. The minimum requirements were a Core 2 Duo at
| 2.4 GHz or Athlon X2 at 2.8 GHz. I think that puts it on
| processors dating back to 2006 or so.
|
| Games sometimes have a lot of sound effects. It's easier to
| stream 35 GB of data from disk than it is to decompress,
| like, 10 streams of audio at the same time with whatever
| CPU you can dedicate to audio.
|
| I do think they could have figured something out, like
| compressing the right subset of audio assets, but I just
| want to give the context for why they made that choice.
| recursive wrote:
| That's like 50 hours of uncompressed stereo audio at CD
| quality. (2x16bit@44.1kHz) Do games really have that much
| stuff in them these days? I mean, I wouldn't doubt it,
| but it's surprising.
| midnightclubbed wrote:
| For story based games (with voice acting) absolutely yes.
| Dozens of story characters, hundreds of supporting npc
| and all of them talking as either part of your missions
| or just to fill in the world.
|
| Maybe 50 hours is a lot for a non story based game but
| even then once you factor in all the sounds and all their
| variations to reduce repetition you might be up there.
| ant6n wrote:
| Do they all talk at the same time?
| klodolph wrote:
| Most of it is probably mono, so double the estimate.
| kevingadd wrote:
| The uncompressed audio was on purpose to manage CPU usage,
| you can find quotes explaining this in interviews.
|
| https://www.escapistmagazine.com/titanfall-dev-explains-
| the-...
|
| I can attest that MP3 and OGG decompression (the two audio
| codecs that would have likely been used at the time) are
| both pretty slow. It adds up.
| Sohcahtoa82 wrote:
| What a bunch of absolute horse shit.
|
| MP3 isn't slow. I was playing MP3s on a 100 Mhz 486 DX4
| in 1996. Certainly a system in 2014 can handle
| decompressing MP3s on the fly.
|
| Even if you didn't want to do it on the fly during game
| play, they easily could have been decompressed at load
| time and it would have taken only a couple seconds. It's
| not like ENTIRE 35 GB of audio needed to be loaded all at
| once, nobody had that RAM back then, and very few people
| (especially gamers) have that much now.
| jayd16 wrote:
| Ok, how much uncompressed audio does need to be resident
| in RAM, do you think? Last gen hardware had 8GB total. I
| think the XBONE could only use 5GB for games. That was
| shared for CPU and GPU. Is it worth kicking out a lot of
| texture data so you don't have to stream audio?
| kevingadd wrote:
| I've benchmarked it myself in my game. It adds a
| measurable amount of time to loads if you're
| decompressing all the audio up front to reduce latency,
| and during streaming it increases CPU usage. Obviously a
| modern machine can handle a few ogg/mp3 streams in the
| background during gameplay, but some games are playing
| dozens or hundreds of sounds at once.
|
| "It's not like ENTIRE 35 GB of audio needed to be loaded
| all at once" misunderstands the nature of games
| fundamentally. Some games do potentially need the ability
| to access any of thousands of sound effects at a moment's
| notice with the lowest possible latency.
| mirsadm wrote:
| Plus last gen consoles have extremely slow CPUs. They
| were already slow when they were released.
| bombcar wrote:
| This is likely where some of this comes from - trying to
| keep the PC and console versions in sync and having to do
| tricks like swapping space for speed.
| izacus wrote:
| Why are you assuming MP3/OGG formats are the ones to use
| for game audio compression ?
|
| It's like trying to use PNG for texture compression
| instead of better suited formats.
| midnightclubbed wrote:
| What formats are better suited? To my knowledge most
| games use mp3-like compression for all their audio.
| Console hardware often has decompression support, it's
| reasonably compact, quality/bitrate can be tuned by the
| sound engineers and supports a wide range of content.
| jpetso wrote:
| It seems like on PC one would want to use Opus instead of
| the older MP3 or Ogg Vorbis. It has a number of
| performance-related improvements in addition to better
| audio per bitrate, although the higher-quality codec out
| of its two internal modes is still conceptually similar
| to MP3 & Co. so I figure it wouldn't necessarily solve
| all performance issues either.
| ad133 wrote:
| Let me tell you a story about playing The Settlers IV on
| a 233Mhz 64MB RAM Voodo 3 2000. This game was cool in
| that it had two audio features: a soundtrack on the CD
| itself, played using the CD drive's DAC [^1], and you
| could also drop MP3 files into a folder where the game
| would play those instead. It was common practice to use a
| NoCD crack when playing online, because the CD check took
| long enough you could time out of the lobby, and if you
| forgot the CD you got booted. That meant most online
| gamers had MP3 files, and no CD.
|
| The minimum requirements to play this game were a 200MHz
| CPU w/ MMX and 64MB RAM - I was pretty close to this
| baseline. So anyway, I discovered that the game played at
| much better FPS (like 30 instead of 5) if you turned the
| music off - but only when playing MP3 - CD Audio had no
| hit. Now perhaps the game used a sub-par audio codec, but
| that single MP3 decode stream was enough to make the game
| unplayable.
|
| Anyway that's not to say that I would expect MP3 decoding
| to be a problem in 2014, in fact you can likely play
| audio with no noticable increase on CPU usage, but when
| you have multi-stream audio (think voices, background
| music, sound effects from various channels - guns,
| explosions, etc.) I can see it starting to add up -
| especially when the CPU is already constrained for the
| graphics, game logic and perhaps of course everyone's
| favourite anti-piracy/anti-cheat logic.
|
| [^1] For younger readers, yes, CD Drives used to come
| with built-in DACs and a special cable you could hook
| directly into the audio card, allowing you to listen to
| CD Audio on PC for "basically" free in terms of CPU
| cycles.
| bombcar wrote:
| Yeah CD drives were also CD players - I had rigged my
| computer with a separate tiny power supply for the drive
| so I could have it on without the computer being on - put
| a disk in and it would start playing audio through the
| headphone port on the front and out the special wires on
| the back, and if you had the right audio card that would
| even play out the speakers with the computer off.
| overthrow wrote:
| If you compress data, it will be smaller and require less i/o
| to load, so it will load faster. That's ESPECIALLY true on
| HDDs, but even on SSDs there are decompression algos that run
| faster than SSD bandwidth. The uncompressed asset cargo cult is
| hurting everyone who plays their games, no matter what kind of
| disk they use.
| babypuncher wrote:
| Jedi Survivor isn't shipping on any platforms constrained by
| an HDD, so it would be odd if they were making any
| optimizations for spinning rust, especially when they come at
| the cost of precious SSD space.
| falcolas wrote:
| It's releasing on PC, which could be constrained. It wont
| be for a majority of players, but HDD setups still exist.
| kevingadd wrote:
| Compression does not guarantee faster loads anymore.
| Uncompressed assets can be DMA'd directly to their
| destination. This is part of why GPU decompression (gdeflate
| like in directstorage, or the special decompression tech in
| the PS5) is on the rise because it allows you to DMA
| resources to their destination on the GPU and then decompress
| them there to avoid the loading time hit involved in
| decompressing them on-CPU after copying them disk->page
| cache->userspace app memory.
|
| For HDDs, yes, compression is generally a win. HDDs are dead
| as far as game development is concerned, all modern targets
| have flash storage.
|
| Some compression codecs that produce smaller files are also
| MUCH slower to decompress, which swamps any benefits from the
| reduced i/o. This is part of why sometimes people pick less
| efficient formats for use in games.
| blibble wrote:
| if it's on a modern SSD you can DMA it directly into the GPU
|
| no CPU required
| ByThyGrace wrote:
| Interesting. How modern are we talking here? Is this a
| "devil's in the details" kind-of feature?
| blibble wrote:
| drives since 2018 or so
|
| it's marketed by Microsoft as "direct storage"
| [deleted]
| mxmlnkn wrote:
| Slow downloads wouldn't be a problem if shipping actually
| finished games on Blu-rays were still in vogue. And the disc
| limitation would make some publishers think twice about shipping
| 250 GB on 3 or more discs.
| bhouston wrote:
| 250gB? No problem with 1gb/S to 3gb/s fiber to the home. At least
| here in Ottawa, Canada Fiber to the home is prevalent and
| affordable.
|
| But I understand where fiber isn't available these are large
| downloads.
| dendriti wrote:
| Well shit. Does your "Canada Fiber to the home" come with
| unlimited fast storage for free?
| nottorp wrote:
| At 1 Gbps unmetered deleting and reinstalling games later is
| not much of a problem.
| bhouston wrote:
| The article mentioned primarily download sizes and times.
| Honestly having +1Gb/S download speeds is quite nice. I also
| run a local 10Gb/S local network and again it makes large
| files seem small.
| qtzaz wrote:
| That's what I was thinking. Game size is not the problem, the
| problem is that you have a shit internet connection and/or a
| small hard disk.
| cultureswitch wrote:
| Maybe I misunderstand something but to me it seems the premise is
| wrong.
|
| What's most important about game textures is that they can load
| fast at a quality good enough not to disturb gameplay. Absolutely
| everything else is a secondary concern.
| [deleted]
| justeleblanc wrote:
| This whole thing is an environmental disaster.
| langsoul-com wrote:
| I think games should have advanced textures optional by default.
| 4k textures are unnecessary if it results in a 50gig+ download.
| kevingadd wrote:
| It used to be more common to have optional texture packs like
| this. It seems to have fallen out of favor, I can offer some
| informed speculation as to the reasons:
|
| * Every additional configuration i.e. 'texture pack vs no
| texture pack' is more QA workload to verify that everything
| works as expected. So adding this option is more work for your
| already-busy testers.
|
| * Adding DLCs to deployment systems like Steam or PSN is
| additional work and then when patching the game you would also
| need to patch the DLC. I can imagine the coordination overhead
| here being a pain.
|
| * Some users will fail to install the DLC and then complain
| that the game is ugly or that the screenshots were misleading.
| This is ultimately a user education problem but it's also the
| case that the flow for installing DLC on most platforms is
| awkward.
|
| * The DLC install process adds additional points of failure, so
| you need to handle all of that now too.
|
| * Users installing a new graphics card will probably be annoyed
| to have to suddenly wait for a HD texture pack to download
| before they can test it out.
|
| All pretty minor individually, but it probably adds up to
| explain why most studios don't do this.
|
| I wonder how widespread developer support for Xbox Smart
| Delivery (which is basically what you describe) actually is at
| this point?
| bno1 wrote:
| Saving a few ms during decompression actually matters when the
| game streams data from the disk in real time. People also hate
| loading screens and objects popping-in because the streaming
| pipeline is slow.
| faebi wrote:
| Don't want to blame too much, but it seems like your internet got
| stuck in the past. I have 10gbits. I used to have 1gbits for a
| long time before. 100mbits is not that much these days.
| Switzerland is talking about a government mandated minimum speed
| of 80mbits in 2024. That's a 7 hour download on minimum speed for
| an unusually large game. Switzerland is not alone in this regard.
| That could be a view of privilege, but average internet speeds
| are still progressing forward fast.
| SergeAx wrote:
| I beleive it is your internet is a bit in a future. Good for
| you, but not a reason to blame others. See
| https://www.fastmetrics.com/internet-connection-speed-map-us...
| , for example. 30-50 Mbps was median in 2020, even with COVID-
| induced improvements I doubt things became significantly better
| today.
| MMAesawy wrote:
| It is an extremely privileged view. In my home country of
| Egypt, a population of 100 million lives on an average home
| internet speed of 30mbps with about 300gb monthly data cap, in
| 2023. I'm sure most of the world is similar.
| ProtoAES256 wrote:
| You're correct about the internet stucked the past, but for a
| lot of places in the world even with a lot of money you won't
| get fast speeds! My hometown is only getting 100/25Mbits max
| after the previous max 30/5Mbits 2 years ago!
|
| It's not like we don't have the money, I bought a 10900k and
| 2080 super for my brother back then, it's just that the speed
| is unavailable. A "git gud" simply don't do that justice.
| timwaagh wrote:
| I think honestly the problem is small disks. Games are achieving
| almost photorealism on 4k with maybe 150gb. That's quite honestly
| not bad. You can't store the universe on a floppy disk.
| bick_nyers wrote:
| Blu-ray movies are like 50GB after all, and are significantly
| less hours of entertainment.
| cabirum wrote:
| I suspect it's not the devs should be blamed, but marketing and
| management. "A newer and a better game must also take more space,
| right?"
| rektide wrote:
| Richg42 is great & his pieces are fun to read.
|
| I do think in this case he misses a bit of the point, goes too
| far:
|
| > _They are designed for fast loading and streaming off SSD 's.
| Who cares about saving off a few hundred ms (or a second) when it
| takes hours or days to download the product onto the SSD?_
|
| Most games will have only one two or three downloads over the
| user's lifetime, and it'll never be instant, will always require
| some wait.
|
| But we load textures many times if we keep playing a game.
|
| Rich's overall point remains, but I think some balance of
| optimizing load speed is also very appreciated & very necessary.
| overthrow wrote:
| That can be solved on first load with a texture cache. Shader
| caches are pretty universal at this point. Texture caches
| aren't (afaik), but I think they should be.
| rektide wrote:
| So now we would have faster download & faster load, but we'd
| have significantly bugger hard drive space usage than we
| have? That also seems like a concern.
| jeroenhd wrote:
| > Most games will have only one two or three downloads over the
| user's lifetime, and it'll never be instant, will always
| require some wait.
|
| Are there any statistics on that? Because I have deleted and
| redownloaded games a whole bunch of times because you can't fit
| more than two or three games on a single SSD anymore.
|
| I've played large games off of hard drives because I simply
| didn't want to delete 240GB of other files off of my main
| drive. The game experience was clearly much worse than the
| developers intended, but their space demands are unreasonable.
| Especially considering I'm only playing at 1080p!
|
| Give me a 1080p installer that shaves off 75% of the texture
| size and you'll win more than a few milliseconds because the
| stupid thing can actually fit on my drive and doesn't need to
| come off the internet again.
| hellotomyrars wrote:
| Game size is a perennial talking point and some games earn it and
| some games don't.
|
| There was a phase of games shipping with uncompressed audio in
| every language they support for no good reason that ballooned
| file size (Titanfall)
|
| Some games have gotten around this problem by having higher
| resolution textures or additional languages being optional
| installs within whatever ecosystems DLC implementation is. I
| think this is the smart way to do it.
|
| Examples of games shrinking I are hard to come by but special
| credit to IO interactive and Hitman 3.
|
| The install size with all the content of the previous games
| included (If you own them) is less than the install size of the
| first game. That is impressive and commendable.
|
| Games are weird and some people will look at the file size and
| think it means the game is better, just the way some people think
| games with more hours are better. The psychology of people who
| play video games is weird.
|
| Duplication of assets on disk was a very weird byproduct of HDD
| file access conventions, but we shouldn't need that anymore.
| There is still a shrinking, but valid crossover window between
| devs being able to assume all storage is solid state. I think
| things will get better but the debate of file size vs
| average/expects storage capacity vs Network speed will never end
| but there is certainly work to be done.
| RajT88 wrote:
| The .kkreiger devs really showed how far you can go to reduce
| download size:
|
| https://en.wikipedia.org/wiki/.kkrieger
|
| https://www.youtube.com/watch?v=ya_MUKc343U
|
| It is dated, but looks pretty good by 2004 standards.
| thequux wrote:
| While I'm all for reducing download sizes, perhaps lekktor[1]
| is a bridge too far.
|
| [1]:
| https://fgiesen.wordpress.com/2012/04/08/metaprogramming-
| for...
| Dylan16807 wrote:
| Sure. Focus on the procedural textures and see where that
| gets you.
| spatulon wrote:
| Note that one of the .kkreiger devs (ryg aka Fabian Giesen)
| works at Epic Game Tools on the Oodle compressors mentioned
| in OP's post.
| hinkley wrote:
| Blizzard sorted the high res textures and some of the raids
| last so that you could start playing the game halfway through
| the download.
|
| Another problem is chunking the data properly so that
| incremental updates are not hugs.
| rvba wrote:
| Blizzard us a bad example. Their games take incredible
| amounts of time to start. I think they dont play their own
| games and nobody gives a shit.
|
| The launcher is bugged and downloads either empty or 137mb
| patches (maybe bad CRC at some server? Different CRCs at
| some?). Every time.
|
| Starcraft remastered was bugged at some point and couldnt be
| played for a week.
|
| If you statt Heroes of the storm, it takes few minutes to
| download some 600 byte ghost patch, then does some
| (antihack?) processing... then you can start the queue after
| like 10 minutes.
|
| It's probably like the GTA5 that also took ages to start due
| to poorly done code and devs not caring at all.
| saberience wrote:
| Your experience is not my experience at all. Blizzard is
| the gold standard for me in starting games faster than any
| other launcher. In fact they are the only one I've seen
| that lets you play the game part way into the download. No
| other dev seems to have this technology.
|
| I waited hours to play Witcher 3, Cuberpunk, Assassins
| Creed Odyssey. With WoW I was playing within minutes.
| lakomen wrote:
| You waited hours... to play those games? From double
| clicking the icon to launch? I find that extremely hard
| to believe.
|
| If you include download time, even that doesn't hold up
| to your claim.
| lathiat wrote:
| Assassins creed odyssey was apparently 44-50GB at launch.
| That's 3 hours @ 45 megabits/second.
|
| This is not an uncommon internet download speed even
| today (let alone 5 years ago when it launched) according
| to the steam download stats (it varies by country, check
| australia for example, US is a little over 100 average):
| https://store.steampowered.com/stats/content/
|
| Even at 100Mb, 45GB is 1 hour. Witcher 3 looks to be
| about 30GB but Cyberpunk is apparently 70-100GB.
|
| Do the math next time :)
| dunefox wrote:
| I regularly needed multiple hours (sometimes up to 12) to
| download large games until 2-3 years ago. So he's
| definitely correct.
| BlueTemplar wrote:
| In the Beyond All Reason (still "alpha") RTS, the game
| itself is only a couple Go, but maps are downloaded on
| the fly, because downloading all of the featured ones
| would take a dozen Go.
| wakaru44 wrote:
| what is 'Go' as a unit? size? time?
| mastensg wrote:
| gigaoctets
| ahartmetz wrote:
| Probably Giga-octets. Either French or very formal.
| cliff wrote:
| Other games have 'start with partial download'
| technology. In fact, the core tech of the team that
| eventually created Valve's Steam was downloading assets
| on-the-fly so that you could start playing a game before
| everything was downloaded.
|
| I worked on Guild Wars 2, which has this feature. I made
| a first prototype of it that streamed all content on-the-
| fly. It's pretty easy to implement - you have an
| abstraction that asynchronously loads a file off of the
| disk, and you can just make that download from the
| network instead.
|
| The tricky part is when you want to ensure all the assets
| are there for a specific area before you load in, or
| simply knowing what order to download things in. For
| example, there was a starter area of Guild Wars 2 that
| spawned monsters from many other areas, this meant that
| the manifest of what was needed was enormous for that
| area.
|
| So the 'playability' threshold becomes a trade-off
| between game experience (assets popping in as you play)
| and quick entry.
| SketchySeaBeast wrote:
| Guild Wars 2 needs more respect from the MMO community.
| The ability to do testing on the live servers, or enable
| patches with a client reboot and no server downtime, is
| great.
| nazgulsenpai wrote:
| Guild Wars 2, EverQuest 2 also allow launching after a
| relatively small (maybe 10%) base amount of data has been
| downloaded. Its never ideal IMO as it creates longer
| loading times to areas not already downloaded, but I
| appreciate that its an option.
| BlueTemplar wrote:
| It's so different from game to game, some games are still only
| a couple Go despite having "4k" graphics...
| kevingadd wrote:
| Titanfall shipped uncompressed audio on purpose to cut out
| audio decompression CPU usage. You can disagree with the
| decision, but it wasn't an accident or negligence.
| terafo wrote:
| It could've been handled as DLC that downloads with a game by
| default and can be uninstalled/disabled, if we are talking
| low-effort solutions.
| haswell wrote:
| I realize they're trying to optimize for a wide base, but I
| have plenty of spare cores to decompress audio, and SSD space
| is limited.
|
| For me, the net result is that Titanfall was one of the first
| games to go when I needed space. Low hanging fruit.
|
| Even if it was deliberate, I have to question the logic and
| wonder how much usage they enabled vs. uninstalls from file
| size over time.
| kristianp wrote:
| You've already bought it, so the game studio got their
| money, despite having a "bloated" install. So the
| incentives aren't quite there to reduce the size.
|
| I was thinking about buying Mass Effect yesterday, the
| recent update, but it requires 150GB and I don't have a
| Windows partition that big! (I'm Dual booting Linux) So I
| think money is being lost in people who aren't often gamers
| and haven't bought big disks for gaming.
| haswell wrote:
| > _You 've already bought it, so the game studio got
| their money, despite having a "bloated" install. So the
| incentives aren't quite there to reduce the size._
|
| They may have _some_ of my money, but not much
| (discount!), and they're not getting money for DLCs,
| arguably the reason the base game was so cheap, and I'm
| not in a rush to buy more stuff that I can't fit on my
| PC.
|
| Prior to the era of the initial game purchase being just
| the entry point I'd have agreed with you, but many modern
| games have monetization strategies that make
| uninstallation a problem.
|
| And if they have a strong enough incentive to care about
| CPU usage, they're clearly putting effort into optimizing
| for _some_ audience.
| yellowapple wrote:
| Do you have a big enough Linux partition? Mass Effect
| Legendary Edition runs pretty great via Proton.
| tzs wrote:
| Why not ship the audio compressed and decompress it once when
| the game gets installed?
|
| That wouldn't help people whose issue with large game size is
| the amount of space used on their disk, but it would help
| people whose issue is the long download times.
| kevingadd wrote:
| For all we know it actually worked that way. Steam
| definitely decompresses game assets at install time. The
| people complaining about game sizes aren't looking at the
| depot files you download when installing the game, they're
| looking at their HDD (which is fair!)
|
| My guess is Valve doesn't offer a 'decompress your ogg
| files to wav at install time' feature though.
| ntauthority wrote:
| Titanfall _did_ work this way on the EA launcher. There
| was a tool that it 'd run on first launch to decompress
| the compressed audio files.
| Dylan16807 wrote:
| But why.
|
| Even when you do want lossless assets, as long as you
| compress in reasonable chunks there is no downside.
| usefulcat wrote:
| Are spare cores really such a scarce resource? I thought most
| games didn't come close to using all available cores. And
| audio decompression is not known for being that cpu
| intensive.
| kevingadd wrote:
| It was a long time ago, the CPU usage was an actual
| problem.
| bullen wrote:
| I use zip (in flight) and 2 different lz (on disk) for my 3D
| action voxel MMO.
|
| But the characters are flat shaded so those textures are tiny.
|
| You need to look at the data, animation weights compresses 95%
| with a simple lz.
|
| One big dissappointment was half-float support having low
| performance on the some drivers, that would have saved 50% off
| the models.
|
| One advantage of not compressing is that your disk shows the
| (V)RAM usage you'll have.
|
| I'm aiming for 1GB total (to support 2GB Raspberry 4) and my
| biggest concern is uncompressed 120Hz animations! (to save CPU)
|
| But so many of those can be hot-loaded off disk.
| amelius wrote:
| In the future assets will be created using a prompt the size of a
| tweet. It would be silly to send the pixels instead of the
| prompt.
| sdfjkl wrote:
| Well, whatever the repackers (e.g. Fitgirl) use seems to work a
| lot better than what the game publishers use (often no
| compression at all).
|
| Original Size: 93.1 GB -> Repack Size: from 35.5 GB [Selective
| Download]
|
| Original Size: 3.1 GB -> Repack Size: 844 MB
|
| And these are lossless, bit for bit reproductions. In many cases
| assets (audio, video, images/textures) could be using lossy
| compression for even greater reduction in size with no
| significant loss in perceived quality (but maybe some loss of
| advertising/bragging rights).
|
| The downside is, some effort is required packing it:
| https://fitgirl-repacks.site/new-compression-algorithm/
| SSLy wrote:
| Another downside is obscene time it takes to unpack FG
| releases.
| sdfjkl wrote:
| That depends. If you're on a slow connection, the 40 minutes
| to unpack something are nothing in comparison to the week it
| took you to download it. If used by the game publisher, the
| decompression could happen during download (I believe this is
| how steam does it).
|
| If you're on a metered connection (common in many of the more
| forgotten parts of the world), the bloated download costs you
| real money or is simply not possible (or you have to spread
| it over several periods of data allowance).
| KMnO4 wrote:
| Games want to be more photorealistic and the hardware to fake it
| is getting faster and cheaper.
|
| I remember in 2008 when Intel demoed ray tracing[0] for Quake
| Wars. It ran on a quad-socket (4 CPUs x 2.7GHz) at around 25 FPS,
| and even Intel admitted that ray tracing is just "brute forcing"
| the complex optical physics of a scene.
|
| Considering between then and now, we have much prettier looking
| games that do NOT require ray tracing for their visual effects
| (and can run on much fewer CPU cycles), thanks to much more
| "clever" techniques and algorithms (DoF, bloom, subsurface
| scattering, PBR, etc), and if it weren't for the RTX graphics
| cards setting the precedent, there would probably be more
| research into making things prettier without having to "brute
| force" things.
|
| Nowadays, you can drop a 100gb mesh into UE5 Nanite[1] and skip a
| lot of the work involved in making things pretty. You don't need
| to generate or even understand displacement maps if you can just
| add more polygons to the model. You don't need to billboard the
| mountain way in the distance if Unreal does it on the fly. Who
| cares if that means including a 250mb mesh in your assets, versus
| a 5mb HDRI.
|
| So yes, producers have either become lazy or are simply unable to
| meet the demands of the hardware by coming up with better
| algorithms. That's unlikely to change until either hardware stops
| advancing and either A) can't run the latest games or B) games
| become too large that they're impossible to download.
|
| [0]: https://www.gamedeveloper.com/programming/sponsored-
| feature-...
|
| [1]: https://docs.unrealengine.com/5.0/en-US/nanite-
| virtualized-g...
|
| [2]:
| https://twitter.com/mariobrothblog/status/163604089376436224...
| phire wrote:
| _> and even Intel admitted that ray tracing is just "brute
| forcing" the complex optical physics of a scene_
|
| Modern ray traced games are much less "brute force" than what
| they were doing back in 2008. The use clever techniques and
| algorithms like temporal denoising and radiance caching to
| achieve high quality ray tracing effects with a tiny number of
| secondary rays per pixel (The budget is often under one ray per
| pixel).
|
| It's not really fair to call it "brute force" at all, just a
| different set of clever algorithms.
| Sohcahtoa82 wrote:
| > clever" techniques and algorithms
|
| Lately I've become hyper-aware of games doing screen-space
| reflections and suddenly finding it distracting when I point my
| view down and the reflection of the scene at the horizon
| vanishes off the water/floor.
|
| I upgraded from a 1070 to a 3080 right when the 3080 came out,
| and I actually have yet to play an RTX game.
|
| > bloom
|
| I hate that this ever became a thing.
|
| Bloom just makes the scene blurry and bright parts end up blown
| out.
| terafo wrote:
| > _and if it weren 't for the RTX graphics cards setting the
| precedent, there would probably be more research into making
| things prettier without having to "brute force" things._
|
| Not really. A lot of this research included making rudimentary
| versions of RTGI that could run on hardware of that age(see
| SVOGI). Of course you can bake everything and make amazing
| looking levels, that are completely static. But that doesn't
| work with open-world games, especially if they have different
| times of day. And it doesn't work in highly dynamic scenes. We
| basically hit the end of the road for scaling graphics without
| RT. And now we have hardware that can do realtime path-traced
| rendering of major AAA titles, and it looks miles better than
| anything smoke and mirrors could've ever delivered.
| Animats wrote:
| I'd rather have more RAM in the GPU and give up the ray
| tracing hardware.
|
| The real problem in the industry is that GPUs are not getting
| cheaper. It cost about $250 for an 6GB NVidia 1060 in 2016,
| and it costs about $250 for a basic NVidia card today that
| can do slightly better. Meanwhile, people are buying $100
| WalMart laptops and trying to play games on them.
| terafo wrote:
| You can buy RX 6600, which is roughly 2x faster than 1060
| and has 8 gigs of VRAM for 200 dollars[1]. If we take
| inflation into account, 1060 did cost around 300$ in
| today's money. So basically you are getting 30% more ram
| and 2x performance for 30% less money. In other words you
| are getting performance 20% better than GTX 1080 for 4x
| less money(launch price of 1080 was $600). And it uses less
| energy too, so TCO difference is even higher. And I'm not
| even talking about expanded feature set.
|
| [1] https://www.newegg.com/gigabyte-radeon-
| rx-6600-gv-r66eagle-8...
| Retric wrote:
| Blame CPU manufacturers. There isn't a market for 50$
| graphics cards when Intel and AMD have integrated graphics
| that can handle low end 3D games.
|
| There's still a gap between integrated and 250$ cards, but
| that's mostly filled by used last generation cards.
| aseipp wrote:
| I mean, you could have a bunch of other things in the GPU
| instead of those cores, but the reality is that ray tracing
| serves multiple target markets for GPUs very well, and
| actually meets a demand people have: better graphical
| quality. In contrast, while I'm jealous of my friend with a
| 16GB 4080, versus my paltry 10GB 3080, it does not
| realistically impact anything I do outside of occasional ML
| retraining tasks. Neither did my old 8GB card. Gaming
| workloads just generally don't need it, because people
| there are mostly bound by compute, not texture or RAM
| limits -- and solutions like super-resolution/upscaling
| instead improve performance _and_ memory usage all at once.
|
| > and it costs about $250 for a basic NVidia card today
| that can do slightly better
|
| Eh, the 3050 is like 30% more efficient in every single
| benchmark versus a 1060, with 30% more VRAM, and costs like
| $280. I don't think 30% is "slightly better". The Arc A750
| also is around the same price point and is nearly 80%
| better, but I admit Nvidia has the best software stack on
| the market at the moment so the 3050 is a more fair
| comparison.
| cubefox wrote:
| > Considering between then and now, we have much prettier
| looking games that do NOT require ray tracing for their visual
| effects (and can run on much fewer CPU cycles), thanks to much
| more "clever" techniques and algorithms (DoF, bloom, subsurface
| scattering, PBR, etc), and if it weren't for the RTX graphics
| cards setting the precedent, there would probably be more
| research into making things prettier without having to "brute
| force" things.
|
| Unreal Engine 5 has a new lighting system called "Lumen". It
| seems to be visually almost on the level of normal brute force
| ray tracing, while not requiring any special ray traycing
| hardware and still running with decent performance on mid range
| hardware. It's a smarter approach which tries to accumulate
| lighting information on textures over time instead of computing
| everything again for each frame.
| starkparker wrote:
| There's a good breakdown from one of the Lumen devs for more
| detail on how it came about:
| https://knarkowicz.wordpress.com/2022/08/18/journey-to-
| lumen...
| dahart wrote:
| > Considering between then and now, we have much prettier
| looking games that do NOT require ray tracing for their visual
| effects (and can run on much fewer CPU cycles), thanks to much
| more "clever" techniques and algorithms (DoF, bloom, subsurface
| scattering, PBR, etc), and if it weren't for the RTX graphics
| cards setting the precedent, there would probably be more
| research into making things prettier without having to "brute
| force" things.
|
| Despite being a graphics researcher, I've been a little
| skeptical in the past about ray tracing in games. Recently I've
| started hearing for the first time from some hard-core gamers
| that turning on ray tracing makes a "massive" difference in the
| quality of the experience, and that they'll use it as long as
| the frame rate can stay above 30 fps most of the time.
|
| That said, with all due respect, your speculation about clever
| non-ray tracing techniques seems a little off-target from my
| perspective. There has been, and still is, a _huge_ amount of
| research into these, and the entire field of techniques is
| hitting limits and failing to progress as fast as ray tracing.
| There are multiple reasons for this. One problem is that many
| of these one-off raster effects require their own memory &
| compute. Another is that they each require their own
| implementation, each require their own tuning, they impose
| constraints on the art and the artists, and they only work in
| narrow ranges of curated input. Shadow maps are a good example
| - needs a buffer of it's own, and get too close or too far away
| from the shadows, or if the scene is too big, and they look
| terrible. You have to spend precious art+dev time just to make
| them work, you have to spend precious memory for this one
| effect, it eats into your rendering budget, it only takes care
| of the shadows, it's quality has a hard ceiling limit, and you
| still have 13 other effects to work on that have the same
| story. With ray tracing, once you setup the memory for it, it
| can render all the effects at the same time, it's the unified
| theory of rendering, and doesn't require per-effect memory or
| much per-effect tuning, not like the grab bag of raster and
| screen-space effects does. Ray tracing has it's downsides, and
| it's certainly cycle-hungry, but there is a good reason that
| game devs are gravitating toward using ray tracing, it's
| because ray tracing is _fundamentally_ easier and
| _fundamentally_ looks better than the one-off raster effects.
|
| Regarding ray tracing being 'brute force', keep in mind that if
| you're rendering many instances, or have lots more geometry
| than pixels visible to the camera, ray tracing can be more
| efficient than raster. Is usually isn't in today's games,
| because games have hard budgets that limit the amount of
| geometry - mostly due to raster algorithms! - but the trend of
| increasing geometric complexity is clear, and that ray tracing
| is the likely path forward. Nanite is very interesting and has
| made waves - it's being adopted and used for the same reason
| that use of ray tracing is growing, primarily because it makes
| game development easier. Maybe we'll see these two things merge
| in the future, they are not mutually exclusive. Nanite is
| carefully designed to make sure you're always rendering
| polygons that are about 1 pixel in size, plus or minus, and
| Nanite has pushed scene complexity, but there are certainly
| open questions about whether this approach will continue to
| work as complexity grows.
| W4RH4WK55 wrote:
| We recently had blog post closely related to the topic:
| https://ph3at.github.io/posts/Asset-Compression/
| kevingadd wrote:
| Great post that helps people understand why smaller isn't
| always better.
| arthur2e5 wrote:
| Thing should get its own submission.
| kristianp wrote:
| https://news.ycombinator.com/item?id=35662548
| pornel wrote:
| Compression optimizes for wall clock time which is (roughly
| speaking) sum of time it takes to get the compressed data and
| time it takes to decompress it. Downloads over the network are
| almost always bottlenecked on the network speed, so the heavier
| compression you have the better. However, when loading data from
| the local disk, the I/O can be so fast that the decompression
| time becomes the bottleneck and slows down loading.
|
| Because of that optimization for download time and startup time
| end up optimizing for different things.
|
| With loading speeds so different in these two cases, I wonder if
| the best solution would be to _recompress_ the data with a
| different compression level during download and installation. For
| example, download textures compressed with a good but slow and
| non-random-access image compression format, and then convert them
| to a lighter GPU-friendly compression format locally.
| bick_nyers wrote:
| Recompression is a fantastic idea.
| langsoul-com wrote:
| There's a limit to compression. At some point, too much
| compression takes too long to unpack. The normal person
| wouldn't really understand that.
| kevingadd wrote:
| Heavy compression for the first install is a common tactic.
| Steam and itch do it, and brotli transfer compression is widely
| used on the web.
| pornel wrote:
| Lossless transfer compression is good to have, but it's not
| enough. Lossy compression can give you over 10 times smaller
| files, but it's format-specific and needs to be carefully
| applied by the developer.
| AniseAbyss wrote:
| [dead]
| LinuxBender wrote:
| I recall a lot of hype more than a decade ago about all the
| textures going away and being replaced with math formulas to
| generate everything on the fly. Games were supposed to get
| dramatically smaller. What ever happened to that?
| GuB-42 wrote:
| And there was also the opposite, John Carmack in particular
| didn't have a problem with using huge amount of data, "Rage" is
| an application of this idea. The game uses a single, huge, non-
| repeating "megatexture", with elaborate caching strategies to
| keep it playable even on slow media (HDD/optical). The idea was
| to give complete freedom to artists and not require them to be
| mathematicians. Further down the line, he even considered using
| sparse voxel octrees, i.e. even more huge 3D textures.
|
| On the opposite end of the spectrum, you have Inigo Quilez (
| https://iquilezles.org/ ). He literally draws with maths, and
| he is famous for his 4k demoscene productions. The problem is
| that about no one else is capable of doing what he does.
| rootw0rm wrote:
| I love Inigo's work! I've been learning about shaders, and
| being able to edit and reload his code on shadertoy has been
| really cool
| gatane wrote:
| I see that Inigo Quilez work is more in the "render a scene
| not necesarily in real time" realm. Like, you could make a
| raytracing program, make it render off camera a shiny scene,
| and only after rendering it shows you the image.
|
| Because of this, its very suited to math and procedural
| generation. You just change some parameters, render and show.
| Games instead have to do this in real time, with less latency
| and CPU/GPU usage. And procedural/math generated content
| tends to be CPU heavy (more usage+latency), if not also on
| the GPU (cant be cached, etc).
|
| Shaders are more on the middle of this, because you want to
| apply certains transformations to and from rendering data.
|
| Just my armchair observation.
| Sohcahtoa82 wrote:
| > all the textures going away and being replaced with math
| formulas to generate everything on the fly.
|
| Anyone who thought this was incredibly naive.
|
| I look around my room and try to think about what textures
| could be generated procedurally. My walls, carpet, and ceiling?
| Yeah, sure. The stickers on my computer? No way.
| LinuxBender wrote:
| Does it have to be one or the other and not both? I think
| your room could have textures for small items like stickers
| but the house itself could be a semi-static procedure? Maybe
| the tricky part would be group parties that have to share the
| procedure, maybe sharing a set of procedural seeds?
| babypuncher wrote:
| It is an order of magnitude more expensive to produce art
| assets that way than to just hire people familiar with
| photoshop and maya. Those procedurally generated assets in tech
| demos like .kkrieger were just that: tech demos.
|
| Asset production is already the bulk of the cost of AAA video
| game production.
| db48x wrote:
| Economics. Supply and demand. To create good procedural art you
| have to hire software engineers with artistic talent, which are
| more expensive than the texture and model artists you can find
| by the hundreds coming out of art schools.
| nottorp wrote:
| Sheesh. Please no. We have enough asset reuse as it is.
| mabbo wrote:
| Games will be as large as they can be until it starts causing
| people not to buy the game. Before that point, no one will care.
|
| And as someone who just got 1.5Gbps internet installed, my
| tolerance for file size is continually increasing.
| shmde wrote:
| Your tolerance for file size is increasing but what about your
| SSD/HDD tolerance for file size ?
| maccard wrote:
| A 2TB SSD is available on Amazon for roughly the cost of 2
| AAA games.
| MrFoof wrote:
| The Moore's Law is Dead podcast recently did a podcast with an
| Infinity Ward award-winning effects artist:
| https://www.youtube.com/watch?v=tmfHxJT1I3I.
|
| The crazy thing he mentioned is that if there was zero
| compression, the latest COD games genuinely had over 1 terabyte
| of uncompressed assets.
| landr0id wrote:
| >The crazy thing he mentioned is that if there was zero
| compression, the latest COD games genuinely had over 1 terabyte
| of uncompressed assets.
|
| I think what this shows is there should be smart delivery of
| assets. Before installing perhaps you should be able to choose
| your graphical fidelity and only download assets related to
| that fidelity level and below.
|
| This makes manifest building and delivery more complicated, and
| probably UX too, but I'm sure it's a problem that the
| infra/marketplace provider (Valve, Epic, Microsoft, Sony, etc.)
| could solve and make easy for devs.
| comex wrote:
| CoD itself can download some high-resolution textures _during
| gameplay_ :
|
| https://www.charlieintel.com/warzone-on-demand-texture-
| strea...
|
| I've never tested how much bandwidth it uses. It seems a
| little crazy but also neat.
| chrisan wrote:
| > I think what this shows is there should be smart delivery
| of assets. Before installing perhaps you should be able to
| choose your graphical fidelity and only download assets
| related to that fidelity level and below.
|
| The Diablo 4 beta had this option, drastically reducing the
| download size if you didn't need 4K
| KronisLV wrote:
| > I think what this shows is there should be smart delivery
| of assets. Before installing perhaps you should be able to
| choose your graphical fidelity and only download assets
| related to that fidelity level and below.
|
| War Thunder does this - in the game launcher you can choose
| whether you want the highest resolution textures and most
| detailed models or not. You can even get a minimal client
| without vehicle interiors as well (only 3rd person camera
| available).
|
| It makes a lot of sense to me, to the point where it's
| surprising that all games aren't embracing this approach. I
| mean, I don't have a lot of storage, nor good hardware - why
| should I be forced to download high detail assets that I'll
| never even see because instead all of the models will be at
| lower LODs and all the textures will be of a lower
| resolution.
|
| It feels like the game engine developers don't really care
| about bundling assets by fidelity and each game acts like it
| just owns your entire disk.
|
| That's precisely why I don't really play AAA titles anymore
| and mostly go for pixel graphics and indie titles. There are
| actually surprisingly few well optimized games out there.
| genocidicbunny wrote:
| Microsoft already has something like this for Xbox -- you can
| mark certain assets as not required or only needed for
| higher-end xboxes or languages.
| teaearlgraycold wrote:
| But why?
| MrFoof wrote:
| Listen to the answer to the question, _" How long until our
| game installs take up 500 gigabytes of space"_:
| https://youtu.be/tmfHxJT1I3I?t=7152s
|
| The reality is a game has increasingly more content nowadays.
| Remember, there's a lot of "games as a service". So you have
| many "seasons" of additional content. You may not play all
| the content, but _someone_ is. And when people want 4K and 8K
| textures, different variations so effects don 't look the
| same twice, on top of all the other sizes, plus all the other
| effects ... it adds up. Never mind huge open worlds, where
| the asset footprint for just the map might be 100GB.
|
| Having been in a few AAA studios myself _(doing database
| stuff)_ and sitting next to artists and even art directors,
| the sheer amount of effort, headcount, etc. put into the art
| pipeline is genuine madness. In a large studio, art is over
| HALF the production cost, sometimes more towards 60-65%.
| Heck, one of the things sometimes brought up about Red Dead
| Redemption II is that there were three full-time employees in
| which all they were responsible for was horses. 6000 man-
| hours a year, on just horses.
|
| To sum up what Chris said towards the end, _" The only thing
| keeping these numbers down is the compression algorithms. If
| we didn't have these newer compression algorithms, we'd
| probably legitimately be at a terabyte. I can actually tell
| you for a fact, that there are over a terabyte of assets in
| the COD engine."_
| ok_dad wrote:
| Have you played a modern game? they have amazing levels of
| detail in the tiniest places. One game I play, Gran Turismo
| 7, claims that a single car model start to finish takes
| months to model for a single modeler. Some of the details in
| those cars that you can see in VR you can absolutely not see
| in-game normally, such as the door lock controls or the key-
| hole for the driver's door being exactly like on the real
| car, even for something so rare as a McLaren F1 longtail.
| There is so much waste of time and space dedicated to this
| stuff, but I love it personally!
| taeric wrote:
| Gran Turismo in VR is an amazing experience. I'm not even a
| racing car fan, but holy crap does that game really capture
| an experience. I'm somewhat afraid to find out just how
| much data goes into a single model.
| bick_nyers wrote:
| It's the same deal with Netflix etc. shows before they are re-
| encoded to 20Mbits (or whatever bitrate).
| crummy wrote:
| https://acronyms.thefreedictionary.com/LZ
|
| I think the right definition is this one:
|
| Lempel-Ziv (dictionary-based lossless data compression)
| thfuran wrote:
| It is
| charlie0 wrote:
| If users hate large file sizes, they should just stop buying
| them. Unfortunately, hitting the bottom line is really the only
| way to get this resolved.
| fbdab103 wrote:
| That seems impossibly black or white. I am not sure there is
| any product which is perfect in all dimensions. At some point
| you have to compromise with the state of the world and accept
| some bad with the good.
| Animats wrote:
| Meanwhile, over in metaverse land, there are petabytes of content
| in the big worlds.
|
| I've been working on a faster Second Life / Open Simulator
| client. Those have a reputation for being sluggish. That's
| rapidly changing.
|
| All content is loaded on demand. Textures are stored in JPEG 2000
| format. JPEG 2000 decoders come in several flavors - slow and
| flaky, fast but need the GPU, and midrange but expensive. If you
| want to rewrite OpenJPEG in Rust, go for it. You can download
| part of a JPEG 2000 file and get a lower resolution texture, and
| that's what's done for distant objects. The general idea is to
| download textures at a size that gets one texel per screen pixel.
| (Yes, subpixel textures would be better.) Textures are locally
| cached to SSD. Background threads are frantically loading and
| unloading textures in the GPU at different resolutions as the
| viewpoint moves. The highest resolution is compressed losslessly;
| the others are lossy.
|
| In metaverse systems with user generated content, there's not
| much instancing. So the texture load is much higher. Wander
| around Cyberpunk 2077 and you'll see the same railing show up in
| completely different contexts. This doesn't happen in worlds with
| user-created content. There's no producer or art director to
| insist on reusing content.
|
| If your game downloads content on the fly, you need substantial
| content delivery capacity. Second Life front-ends Amazon AWS with
| Akamai. Open Simulator, on the other hand, often just hosts the
| content on a single game server, along with the the simulator
| itself. My own viewer pulls about 200Mb/s from the content
| servers for about 15 seconds when you first connect. Then it's
| done until the viewpoint moves.
| bick_nyers wrote:
| Check out HTJ2K if you haven't already, it's J2K with SIMD
| instructions, it will have solid support because the medical
| industry is (slowly) moving towards adopting it. Sounds like
| you already saw NVIDIA's J2K decoder. There are FPGA solutions
| as well that are insanely fast (CAST and some others).
|
| HTJ2K has support in OpenJPEG (master branch at least), might
| need to compile with a specific flag or something I can't
| remember.
|
| I think JPEG-XL is ideal for these types of applications in my
| opinion, even if you ship the decoder with your client. Their
| progressive encoding/decoding is much better than J2K in my
| opinion.
| Animats wrote:
| The content already exists. I'm just decoding it.
| Gelenoto wrote:
| So you compare a highly detailed state of the art game with
| gigabytes if proper assets to the 'art' dessert of vr?
| bick_nyers wrote:
| Second Life has been around since before Oculus was born, it
| is an OG MMO
| yboris wrote:
| What about using JPEG XL - .jxl format instead?
| https://jpegxl.info/
| ghilston wrote:
| I used to love Second Life back in the day. Can you tell us
| more about your "game" , and less about the technical details?
| I know this is HN, but what would you tell a prospective player
| about your game?
| Animats wrote:
| I'm interested in metaverses that work. The NFT industry has
| crashed and burned [1], and Facebook/Meta has bombed in that
| space [2]. With the clown car out of the way, it's now
| possible to make progress. The lesson of these failures is
| that there's not much of a role for ads or brands in the
| metaverse. They're distracting and don't fit in. The
| successes, from Roblox to Fortnite to Second Life, charge
| users a modest fee each month.
|
| A metaverse is not a game. Games can be built within it by
| users. This has been done in Second Life, but the games are
| 1) sluggish, and 2) space-constrained, because land is
| expensive. Those are scaling problems which can be solved.
|
| This requires solving the scaling problems that led to the
| original article here. You can't just download everything in
| advance. There's too much stuff in the larger games.
|
| Open Simulator is an open source re-implementation of Second
| Life servers, written in C#. It's been around for a while,
| and now it's getting a bit more developer attention. There
| are multiple federated grids of Open Simulator servers, and
| content stores where you can buy items, all under different
| management. Land is much cheaper than in Second Life, but
| servers tend to be under-resourced and slow. There are some
| people working quietly on trying to improve the Open
| Simulator technology to work better.
|
| Other attempts to solve this problem include Improbable's
| system. Improbable managed to blow through $400 million on
| the scaling problem, producing a system that's too expensive
| to run.[3] (They're funded by SoftBank.) Some good indy games
| tried to use their system, but the server bill was too high.
| Their approach is a general-purpose distributed object
| manager, which seems to be the wrong tool for the job.
| Otherside uses Improbable, and they only turn on their world
| maybe twice a year for a few hours for special events.
|
| [1] https://web3isgoinggreat.com/
|
| [2] https://www.wsj.com/articles/meta-metaverse-horizon-
| worlds-z...
|
| [3] https://www.improbable.io/
|
| [4] https://www.ft.com/content/3508bec7-a2f8-414e-8059-7b96b2
| 700...
| tmilard wrote:
| Chocked at FAT size video games file ? Me too. Giga bytes Damned
| !
|
| To reduce the size, I decided : 1) To use the standard PNG
| compression. 2) To limit mesh size at less than 300 points for a
| scene.
|
| - Result : a) File size < 1 mega Byte ( really). b) It downloads
| in 4 seconds.
|
| Check it here : https://free-visit.net/fr/demo01 (pleaseload url
| twice : I still have a bug)
|
| Question : - why video games STUDIOS do not use this strategy ?
| kevingadd wrote:
| PNG decompression is way too slow. Some games use it but it's
| on the way out, either being replaced by codecs like Basis or
| QOI in most cases
| tmilard wrote:
| Really ? It does not feel this way on my scene that has more
| than 90 .png files used for texture.
|
| The time to : download the globel zip file + open and reading
| the points and all textures => 5 to 7 seconds. This is super
| quick !
| kevingadd wrote:
| Switching from PNG to basis for my game (currently with a
| pretty minimal set of textures) was like a 50% reduction in
| load times with no other changes.
|
| Note that I'm not storing assets at rest in a ZIP file
| because there's no reason to. If I was doing that it might
| close the gap some.
| tmilard wrote:
| Yes I agree but the PNG compresion size is so great ! The
| Textures in total reduce enormously my game file size. By
| 15 times : so good. And because it is small the web
| download is VERY quick. Very.... So in a way I loose a
| bit on decompresing textures time, BUT a gain a lot od
| seconds in dowloading reduced textures files.
| ehaliewicz2 wrote:
| How many times do you decompress the textures, and how
| many times do you download them?
| tmilard wrote:
| Only once :
|
| - Step 1: download zip file containing my XML point file
| and my 90 PNG texture files.
|
| - Step 2: open zip file and dezip in 91 filed
|
| - Step 3: run 3D games with all the 91 Assets. The
| (small) game engine reads each 90 PNG files and - of
| xccourse- transform each one in a GPU compatible texture.
|
| All zip takes about 6 seconds. Not so bad
| egypturnash wrote:
| _To limit mesh size at less than 300 points for a scene._
|
| I am pretty sure most modern video games use more mesh points
| than that for, like, a hat your character can wear.
| tmilard wrote:
| I it is true : I was very agressive on this one. But still,
| thousands and tousands of triangle feels like so much
| overweight for me. Specialy for indoor scenes. Quality for
| the eye is a lot to do with texture, NOT the number of
| triangle.
| tmilard wrote:
| And I do not even talk about the GPU work to handle those
| thousands of triangles. With 300 points, my scene runs
| quick&fast even on budget Smartphones ! This is an
| important issues I can tell you. A video game that runs on
| all devices ...
| Sohcahtoa82 wrote:
| At 300 points, a computer from 1996 could run your game.
| tmilard wrote:
| But not the high texture
| terafo wrote:
| Because PNG isn't a good format for GPU textures. Oodle
| texture+Kraken+something like BC7 is an optimal solution and is
| used in many games.
| tmilard wrote:
| I think in today's machine (be it PC, Mac or even
| Smartphones) the reading-and-transform of a png asset for the
| GPU is hard coded in the processor : so VERY fast. It works
| well.
| terafo wrote:
| There is no hardware acceleration for PNG in any consumer
| device that I know of. And there is such thing as VRAM
| usage, if you decompress PNG to bitmap it would use 3x-4x
| more VRAM than BC7. If you decompress it from PNG and then
| compress to BC7 it would increase loading times and
| basically destroys all point of using PNG instead of lossy
| compression.
| mrkramer wrote:
| Cloud gaming is.
| adgjlsfhk1 wrote:
| I really want to see someone make a renderer that can work on
| compressed textures. No need to decompress if you can just read
| the data on the fly.
| aappleby wrote:
| Renderers have been using compressed textures for literal
| decades. Not JPG, alas, because the compression formats have to
| support random access to individual texels.
| tmilard wrote:
| I use this one. Web compliant. https://www.babylonjs.com/
| corysama wrote:
| https://www.reedbeta.com/blog/understanding-bcn-texture-comp...
|
| BC1-5 have been standard practice since before the original
| XBox. I think when people here are talking about "uncompressed
| assets" they mean "not applying variable-rate zip style
| compression on top of BCn compression."
| codedokode wrote:
| Why not simply make several versions? Like 1 Gb, 10 Gb and 100 Gb
| versions?
| jsheard wrote:
| Some games do this by splitting off the highest resolution
| textures into a free "DLC" that the user can opt not to install
| if they want to save space. It's sometimes done for audio as
| well, e.g. optional Japanese voices for the English release of
| a JRPG.
|
| Remedy's Quantum Break had an unusual spin on letting users
| choose the install size, it made heavy use of live action FMVs
| which could either be downloaded ahead of time or streamed on
| the fly as they played. Downloading the FMVs more than doubled
| the games footprint.
| tmilard wrote:
| good idea : one why chitty etxture quality, one with average,
| one with quality texture. They have done this on web-videos.
| Why not on video games files ?
| kevingadd wrote:
| They do, the Xbox calls it smart delivery.
| tmilard wrote:
| Yeah you are right
| bick_nyers wrote:
| Faster internet and cheaper storage is the answer :)
|
| Or at the very least just offer two different downloads of the
| game, one with lossless textures and one with lossy textures...
| topspin wrote:
| Guild Wars 2 and others solve this with progressive downloads.
| You may start playing after a fraction of the content is
| downloaded. There is effectively no downside: new players can't
| access most of the content anyhow. It also applies to updates:
| you can start playing after a big update with a partial
| download. The background process can be safely interrupted and
| resumed without glitches.
| arthur2e5 wrote:
| War Thunder kind of has that, except that the "progressive"
| bit means you need to download the full thing to progress in
| the game beyond a certain point.
|
| Enlisted, which uses the same engine, is fine though.
| topspin wrote:
| The design of ArenaNet's GW/GW2 has always impressed me.
| 99.4% of all of the game is contained in one file: GW2.DAT,
| which is (currently) 66.7 GB. I/O for updates is extremely
| efficient.
|
| I've seen many games create a vast file system mess, and
| they're forever plagued by "corruption" or something,
| complete with "fix" tools, "cache" clearing push-ups and
| players reinstalling after big updates that don't install
| correctly for whatever reason. Somehow ArenaNet surrounded
| this whole problem and solved it optimally.
| kevingadd wrote:
| A massive amount of engineering went into making it work
| and some of the stuff we did was downright evil :) in
| gw1, a lot of data was compiled into the exe as c
| constants so that we wouldn't have to do a bunch of file
| management and I/o to locate and load it. We ended up
| hitting limitations in the C compiler.
| arthur2e5 wrote:
| > I/O for updates is extremely efficient.
|
| This happens to touch upon a pain I keep getting with
| Steam. Games like squad keep getting tiny patches
| (compared to the whole game size), which is great, until
| you see that steam needs to read and write the whole file
| again to finish the patch. It's not as simple as their
| "dumb allocation wearing out SSD" problem either --
| there's no API to do this anywhere unlike the common
| posix_fallocate(). Linux FALLOC_FL_INSERT_RANGE comes
| close, but no studio will tend their files so they make
| exact filesystem block-sized units.
| kevingadd wrote:
| Rich is a compression expert who wrote a good post, and there are
| some good comments in this thread already, but a few key summary
| points to help people understand the tradeoffs here:
|
| * Steam, Itch, etc all compress game files before sending them to
| you over the network for an install. The amount of stuff you
| download is usually much less than what gets stored on disk.
| There are still games with unusually massive updates though,
| especially ones where the files change enough that delta
| compression fails. This is difficult to avoid when using some
| middleware due to the design of its file formats. Compressed
| archives actually _make this worse_ because you can 't delta
| compress big archives as well as individual files.
|
| * Smaller on disk doesn't mean faster to load anymore. In the era
| of spinning rust HDDs, smaller usually loaded faster, but there
| were extremes - some compression formats that produced massive
| size reductions were also 10x slower to decompress or worse,
| which could produce slower overall game load times. Now that most
| targets have SSDs, this means that less compression often
| produces faster loads. See https://ph3at.github.io/posts/Asset-
| Compression/ (from elsewhere in this thread) for one example.
|
| * For image and geometry data that is headed for the GPU, it's
| best to pick compression formats that can either be natively
| consumed by the hardware (block texture formats like BC7, etc) or
| that can be efficiently decompressed on the GPU once you copy it
| there, like gdeflate. This means the file on disk will probably
| be bigger, unless you supercompress it using something like
| Basis. Basis adds measurable CPU time to loading, though, and as
| a result I'm considering doing an uncompressed texture cache in
| my game to improve load times even though I plan to continue
| shipping textures as basis KTX2.
|
| * For streaming asset loads, the CPU usage to decompress the
| assets during gameplay could lead to frame drops, inconsistent
| frame pacing, and pop-in. My understanding is that Titanfall
| opted for uncompressed audio in order to keep CPU usage low when
| streaming in all the sound effects and voices that could play
| during a hectic match - at that point in time CPU usage for audio
| mixing was a serious concern; modern CPUs more headroom so it's
| less of a problem now. It made sense for them to choose this as a
| trade-off because high-framerate twitch online shooters
| prioritize low consistent latency.
|
| * In some cases the install you get depends on the hardware you
| have. It's not uncommon for older platforms to see 'high res
| texture packs' that could be installed optionally. The Xbox
| platform has something called "smart delivery" where it will
| deliver a smaller game bundle depending on whether your
| particular console needs high-def textures. This takes real
| engineering work to do, so many games opt not to support it
| because that time is better spent elsewhere.
|
| * Ultimately, unless your game is too big to fit on people's hard
| drives at all, there is usually something more important for you
| to work on vs implementing some complex compression algorithm for
| every one of your files at rest. You are probably also not as
| smart as the compression experts who wrote the compression codecs
| that Steam/Itch/Xbox/Playstation offer, either for install-time
| compression (to reduce network bandwidth) or texture compression,
| etc. So in practice you probably just pick an off-the-shelf
| compression solution and apply it where it improves load times
| and you leave everything else alone. It doesn't make sense to
| spend weeks shrinking your on-disk size by 5% when you could
| spend that time fixing crash bugs, improving accessibility with
| better control customization, or improving the game's framerate
| instead.
|
| * Lots of people decide a game "isn't compressed" because you can
| shove it into an lzma archive and make it smaller. This
| misunderstands the issue. Many compressed or optimized-for-size
| formats can still be made smaller by using a really heavyweight
| compression codec like lzma or brotli, but critically, there are
| tradeoffs involved - slower compression means slower build times
| means longer development times, and slower decompression means
| worse load times. The best compression typically also requires
| you to bundle all your files together, which loses the ability to
| load only the data you want when you want it.
|
| An anecdote: Back when I was working on the original WebAssembly
| spec, I spent multiple weeks doing lots of research into custom
| compression and how we could design the format to make it
| compress more efficiently. I even had some chats with the Brotli
| team. In the end, we barely did anything, because even months of
| hard work barely produced better results than just Brotli-
| compressing our files at maximum settings. We were able to
| improve gzipped sizes tremendously by filtering and rearranging
| the data, but there's no point doing it if you can just use the
| browser's built in brotli decoder instead.
| ksec wrote:
| Not sure why this isn't top post. Also not sure why the author
| of the post wasn't clear in a lot of these trade offs.
|
| It is either the author is missing the point, or am I missing
| something?
| kevingadd wrote:
| I think the post just assumes context that most people don't
| have.
___________________________________________________________________
(page generated 2023-04-22 23:02 UTC)