[HN Gopher] How I cut GTA Online loading times by 70%
___________________________________________________________________
How I cut GTA Online loading times by 70%
Author : kuroguro
Score : 3593 points
Date : 2021-02-28 19:38 UTC (1 days ago)
(HTM) web link (nee.lv)
(TXT) w3m dump (nee.lv)
| Parsnip1 wrote:
| There's a decent chance that this is also just an issue within a
| JSON parsing library that someone at R* decided to throw in and
| use. The parsing library was probably never intended to handle
| such large blobs, and R* didn't want to mess with the the OSS
| code.
|
| I wonder if it might even be an incompatible OSS license?
| lumberingjack wrote:
| Red Dead redemption 2 online also does this exact same thing the
| first time it did it I shut the game off and then charged back my
| credit card I dealt with it for GTA and I'm just not going to
| deal with it anymore.
| VectorLock wrote:
| Its a bit disheartening to think about the person-centuries of
| time wasted on what wouldn't be unfair to characterize as sloppy
| code.
| lidotajs wrote:
| He updated his article and added an option to donate him. Check
| the bottom of his website! ;)
| johnnyAghands wrote:
| Face palm. Thanks for doing this, maybe Rockstar will finally fix
| this bullshit.
| ufo wrote:
| The part that puzzles me the most was this comment about sscanf:
|
| > To be fair I had no idea most sscanf implementations called
| strlen so I can't blame the developer who wrote this.
|
| Is this true? Is sscanf really O(N) on the size of the string?
| Why does it need to call strlen in the first place?
| JdeBP wrote:
| I think that the author hasn't checked them all. Even this
| isn't checking them all.
|
| The MUSL C library' sscanf() does not do this, but does call
| memchr() on limited substrings of the input string as it
| refills its input buffer, so it's not _entirely_ free of this
| behaviour.
|
| * https://git.musl-libc.org/cgit/musl/tree/src/stdio/vsscanf.c
|
| The sscanf() in Microsoft's C library does this because it all
| passes through a __stdio_common_vsscanf() function which uses
| length-counted rather than NUL-terminated strings internally.
|
| *
| https://github.com/tpn/winsdk-10/blob/master/Include/10.0.16...
|
| *
| https://github.com/huangqinjin/ucrt/blob/master/inc/corecrt_...
|
| The GNU C library does something similar, using a FILE
| structure alongside a special "operations" table, with a
| _rawmemchr() in the initialization.
|
| *
| https://github.com/bminor/glibc/blob/master/libio/strops.c#L...
|
| *
| https://github.com/bminor/glibc/blob/master/libio/strfile.h#...
|
| The FreeBSD C library does not use a separate "operations"
| table.
|
| * https://github.com/freebsd/freebsd-
| src/blob/main/lib/libc/st...
|
| A glib summary is that sscanf() in these implementations has to
| set up state on every call that fscanf() has the luxury of
| keeping around over multiple calls in the FILE structure.
| They're setting up special nonce FILE objects for each sscanf()
| call, and that involves finding out how long the input string
| is every time.
|
| It is food for thought. How much could life be improved if
| these implementations exported the way to set up these nonce
| FILE structures from a string, and callers used fscanf()
| instead of sscanf()? How many applications are scanning long
| strings with lots of calls to sscanf()?
| pja wrote:
| OpenBSD is also doing the same thing. It seems almost
| universal, unless the libc author has specifically gone out
| of their way to do something different!
| pdw wrote:
| > How much could life be improved if these implementations
| exported the way to set up these nonce FILE structures from a
| string
|
| That's fmemopen. Not widespread, but at least part of POSIX
| these days.
| JdeBP wrote:
| Addendum: There are C library implementations that definitely
| do not work this way. It is possible to implement a C library
| sscanf() that doesn't call strlen() first thing every time or
| memchr() over and over on the same block of memory.
|
| Neither P.J. Plauger's nor my Standard C library (which I
| wrote in the 1990s and used for my 32-bit OS/2 programs) work
| this way. We both use simple callback functions that use
| "void*"s that are opaque to the common internals of *scanf()
| but that are cast to "FILE*" or "const char*" in the various
| callback functions.
|
| OpenWatcom's C library does the same. Things don't get
| marshalled into nonce FILE objects on every call. Rather, the
| callback functions simply look at the next character to see
| whether it is NUL. They aren't even using memchr() calls to
| find a NUL in the first position of a string. (-:
|
| * http://perforce.openwatcom.org:4000/@md=d&cd=//depot/V2/src
| /...
| JdeBP wrote:
| Addendum: The C library on Tru64 Unix didn't work that way
| either, reportedly.
|
| * https://groups.google.com/g/comp.lang.c/c/SPOnRZ3nEHk/m/d
| AoB...
| wnoise wrote:
| Wow. Thanks for looking.
|
| > limited substrings of the input string as it refills its
| input buffer,
|
| As far as I can tell, that copying helper function set to the
| read member of the FILE* never actually gets called in this
| path. I see no references to f->read() or anything that would
| call it. All of the access goes through shgetc and shunget,
| shlim, and shcnt, which directly reference the buf, with no
| copying. The called functions __intscan() and __floatscan()
| do the same. __toread() is called but just ensures it is
| readable, and possibly resets some pointers.
|
| Even if it did, that pretty much does make it entirely free
| of this behavior, though not of added overhead. That
| operations structure stuffed into the file buffer doesn't
| scan the entire string, only copying an at most fixed amount
| more than asked for (stopping if the string terminates
| earlier than that). That leaves it linear, just with some
| unfortunate overhead.
|
| I do find the exceedingly common choice of funneling all the
| scanf variants through fscanf to be weird. But I guess if
| they already have one structure for indirecting input, it's
| easy to overload that. (And somehow _not_ have a general
| "string as a FILE" facility, and building on top of that.
| (Posix 2008 does have fmemopen(), but it's unsuitable, as it
| is buffer with specified size (which would need to be
| calculated, as in the MS case), rather than not worried about
| until a NUL byte is reached.))
| JdeBP wrote:
| You've missed what happens in __uflow() when __toread()
| does not return EOF. (And yes, that does mean occasional
| memchr() of single characters and repeated memchr()s of the
| same memory block.)
| wnoise wrote:
| Ah, I did indeed. Wacky.
| ufo wrote:
| Oh dear... that's one of the biggest footguns I've ever seen
| in all my years of working with C.
| pja wrote:
| It is! Not mentioned anywhere in the manpages either &
| there's no a priori reason for sscanf() to need to call
| strlen() on the input string, so most programmers would
| never expect it to.
|
| Pretty sure I would have made this error in the same
| situation, no question.
| daodedickinson wrote:
| I remember when someone explained here eloquently and inticately
| explained why Twitter constantly breaks in various ways, such at
| saying I'm not allowed to see a tweet, but then I merely hit
| reload and it works... easy fix posted around a year ago at least
| but still not implemented. Don't hold your breath.
| mstade wrote:
| This is some first rate debugging, and great writing to boot. I
| hope Rockstar sees this, fixes the bug and then pays this fella
| _something_. Great post, thanks for sharing!
| petros786 wrote:
| Brilliant...
| JoshMcguigan wrote:
| > Normally Luke would group the same functions together but since
| I don't have debugging symbols I had to eyeball nearby addresses
| to guess if it's the same place.
|
| I really enjoyed this article, but I do have a question about
| this part. Why would a single function be listed at mutliple
| addresses?
| kuroguro wrote:
| Well it's not actually the function's start address, it's the
| instruction pointer's address for the top function (so it moves
| around while executing).
|
| And going down the call tree, it's also not the start address,
| but the return address - so the place where in the previous
| function called this one.
|
| Without debug symbols there's no way to tell if we're inside
| the same function block or not - it's all just addresses
| somewhere in the machine code.
| tiddles wrote:
| I'd guess it's just using the value of the instruction pointer
| at each point it samples, and the way to resolve the function
| from that is to look backwards to find the symbol of the
| function it's in. As he has no symbols Luke has no (easy) way
| of knowing where functions start so it can't do this lookup.
| tomgs wrote:
| I'm going to make the valid assumption that someone, at some
| point, was assigned to fix this. The issue here, IMHO, is why
| they _didn 't_. It comes down to how complicated it was to fix
| this from the organizational perspective, and not from the
| technical perspective. I'll explain.
|
| First, a disclaimer: I have no idea how much Rockstar employees
| are paid, nor how their workdays look like. I don't know what
| their team sizes are, where they worked before, or who manages
| who and in what fashion. I actually don't know anything about the
| Rockstar engineering organization at all.
|
| I am also not a GTA player (or, more accurately, haven't been
| since San Andreas came out many moons ago). This is my
| perspective as someone who has worked for various organizations
| in his life ( SWE-centered ones but also other, more
| "traditional" ones too).
|
| We're all familiar with technical debt - it's a well established
| concept by now. Reducing it is part of the normal workload for
| well-functioning organizations, and (good) tech leads think about
| it often.
|
| What isn't talked about as often is the "organizational" debt.
| Some things are "verboten" - you don't touch them, you don't talk
| about them, and you run like hell if you're ever assigned to deal
| with them.
|
| Every large enough company has (at least) one of those. It might
| be a critical service written in a language nobody in the team
| knows anymore. Maybe it's a central piece of infra that somebody
| wrote and then left, and it's seen patch after patch ever since.
| These are things that end your career at the company if the BLAME
| points to you after you attempted to fix them.
|
| I have a gut feeling - not based on anything concrete, as I
| mentioned - that the loading part for GTA Online might be one of
| those things. If someone breaks the process that loads the game -
| that's a big deal. No one would be able to play. Money would be
| lost, and business-folk will come knocking.
|
| So sure, there might be some mitigations in place - if some part
| fails to load they allow the game to load anyways, and then
| attempt to fix it mid-fly. It's not black and white. But it feels
| like one of those things, and so people might have just been
| running like hell from it for years and years. Teams change.
| Projects change hands. People leave, more people join. It's life
| in the industry.
|
| I would be REALLY interested in learning how software orgs deal
| with these types of behemoths in their projects. I have yet to
| find someone who knows how to - methodically and repetitively -
| break these apart when they appear.
| op00to wrote:
| > I would be REALLY interested in learning how software orgs
| deal with these types of behemoths in their projects. I have
| yet to find someone who knows how to - methodically and
| repetitively - break these apart when they appear.
|
| I ignore them until a big, big customer complains and threatens
| revenue, then I scream and scream until it gets fixed.
| will4274 wrote:
| Doesn't surprise me at all. It's an O(n^2) algorithm (strlen
| called in a loop) in a part of the code where N is likely much
| smaller in the test environment (in-app purchases).
|
| Overwatch is another an incredibly popular game with obvious bugs
| (the matchmaking time) front and center. And gamers are quick to
| excuse it as some sort of incredibly sophisticated matchmaking -
| just like the gamers mentioned in OP.
|
| It's easy to to say it's something about gamers / gaming / fandom
| - but I have a locked down laptop issued by my bigcorp which is
| unbelievably slow. I'd bet a dollar there's a bug in the
| enterprise management software that spins the CPU. A lot of
| software just sucks and people use it anyway.
| jrockway wrote:
| I am not sure Overwatch's matchmaking time is a bug per se. The
| time estimates are bad for sure. But the matchmaker can really
| only be sure of one state -- if you queue for a match, a match
| will be made. The rest is predicting who will show up, along
| with some time-based scale for making a suboptimal match in the
| interest of time. Players absolutely hate these suboptimal
| matches, so the time threshold ends up being pretty high. The
| rest seems to just be luck; will the right combination of 11
| other people be in the right place at the right time?
|
| I think it could be improved, but it doesn't strike me as being
| buggy.
|
| (Overwatch itself, lots of bugs. Tons of bugs. If they have any
| automated tests for game mechanics I would be pretty
| surprised.)
| will4274 wrote:
| No, it doesn't add up. I can see dozens of groups filling up
| and queueing at my groups level as I wait for matches. Worse,
| many of my matches just aren't that evenly balanced. Even if
| you believe the game is dead now, things were just as bad
| (10+ minute queues for full 6 stacks) at the peak. They don't
| do tight geographic bindings - I live on the US west coast
| and regularly get Brits and Aussies in my games.
|
| I guess what they are probably doing is batching groups of
| games and then matching for the entire batch, to ensure
| nobody gets a "bad" match. What they've missed is that well -
| 5% of matches are bad baseline because somebody is being a
| jerk or quits or has an internet disconnect or smurfs or
| whatever other reasons. They could have picked an algorithm
| that gave fast matches 99% of the time at the cost of having
| bad matches 1% of the time and nobody would have noticed,
| because their baseline bad match rate is so high.
| Optimization from too narrow a perspective.
|
| Honestly, the OW matches I get aren't any more balanced that
| the COD matches I used to get, and I got those in a minute,
| not 15.
| jrockway wrote:
| I think that player variance is the hardest thing for the
| matchmaker to account for, and a factor that makes a big
| difference in the quality of the match. Bronze duo'd with
| Top 500 is always a shit show, even if the other team has
| the same duo. It don't think it can be made to work.
|
| (My experience shows this is the case; quickplay matches,
| with no grouping restrictions, are always more of a
| shitshow than ranked, which has some grouping
| restrictions.)
|
| Similarly, an individual's performance variance makes a big
| difference that a mere arithmetic average can't account
| for. A 3900 player probably plays like a 4100 player when
| fresh and warmed up, but like a 3500 player when drunk,
| sleepy, and mad. The actual SR/MMR will converge to a time
| weighted average of those two states, but if you're in a
| 4100 game with that player when they're in their 3500
| state, it's probably unwinnable. Maybe the matchmaker could
| attempt to predict this, but it would probably make people
| mad. Personally, I think that's the nature of a 12 person
| game composed of 12 randos. There is very little an
| algorithm can do to tune that (short of changing
| abilities/hitboxes/cooldowns of each player, which would
| make people mad).
|
| Then there are people that abuse the matchmaker by
| exploiting uncertainty in their favor (creating a new
| account). Every time I have played in a 6 stack in ranked,
| we've always played against 6 brand new accounts that are
| clearly better than the matchmaker thinks they are, which
| sucks. (In quickplay, I generally have very good games in a
| 6 stack though.)
|
| Overall, I hate to say this, but I think they're doing the
| best they can. I don't think there's enough data to make a
| good match 100% of the time, which is why people find 11
| players and play "scrims" instead of ranked/quickplay. I
| don't know where you're able to see other groups and decide
| that they're a suitable match -- 90% of players have
| private profiles, and those that don't only play like 10
| ranked games a season, so you can't really gauge how good
| or bad they are from public data.
|
| (As for brits and aussies joining your west coast games...
| people VPN in to do that on purpose. Legend has it that
| west coast gamers are better than east coast gamers, so
| people all over the world VPN to the west coast and make it
| a self-fulfilling prophecy.)
| will4274 wrote:
| I think you might be missing my point. Maybe an analogy -
| the classic problem in distributed database is CAP -
| consistency, availability, partition tolerance - pick
| two. Historically, a lot of people picked AP to build
| their highly available databases. In the context of
| Spanner, Google basically said this was the wrong
| tradeoff - availability is more highly impacted by
| external events like client to server networking
| incidents than by the design architecture - so you should
| pick CP instead.
|
| I'm making the same point with regard to matchmaking.
| Overwatch has tried to optimize for good matches,
| ignoring all the issues you rightly describe above, and
| thinking about the tradeoff between time and good matches
| without regard to external events that impact matches
| like smurfs, uneven play, DCs, etc. They'd have been
| better off optimizing for fast matchmaking. It's bad
| engineering in plain sight, and gamers go out of their
| way to justify it.
| jrockway wrote:
| That makes sense. The question is would more players quit
| the game because of long queue times, or because they got
| stomped when they were new. (That would be the result of
| just picking the first 12 people in the queue and
| throwing them in a match.)
|
| I think there is some value in caring about that case. I
| started playing Overwatch with no FPS background (at 31!)
| and I never felt like I was in unwinnable games. All the
| players were as bad as me. (I still remember my first
| games when a D.va bomb would reliably get 6 kills.)
| qyi wrote:
| It's certainly not unique to game consumers. People in general
| just blame every fault on "it's physically impossible to
| solve". One big reason why corporations get away with creating
| non stop worse and worse products.
| ufo wrote:
| The surprising part is that sscanf calls strlen behind the
| scenes.
| saagarjha wrote:
| Reading things like these is bittersweet. One one hand, I am glad
| to see that the art of figuring out "why is this thing slow" is
| still alive and well, even in the face of pushback from multiple
| fronts. On the other hand, it's clear that the bar is continually
| rising for people who are looking to do this as a result of that
| pushback. Software has always had a bottleneck of the portion of
| the code written by the person with the least knowledge or worst
| priorities, but the ability to actually work around this as an
| end user has gotten harder and harder.
|
| The first hurdle is the technical skill required: there has
| always been closed source software, but these days the software
| is so much more complex, often even obfuscated, that the level of
| knowledge necessary to diagnose an issue and fix it has gone up
| significantly. It used to be that you could hold and entire
| program in your head and fix it by patching a couple bytes, but
| these days things have many, many libraries and you may have to
| do patching at much stranger boundaries ("function level but when
| the arguments are these values and the call stack is this"). And
| that's not to say anything of increasing codesigning/DRM schemes
| that raise the bar for this kind of thing anyways.
|
| The other thing I've been seeing is that the separation between
| the perceived skills of software authors and software users has
| increased, which I think has discouraged people from trying to
| make sense of the systems they use. Software authors are
| generally large, well funded teams, and together they can make
| "formidable" programs, which leads many to forget that code is
| written by individuals who write bugs like individuals. So even
| when you put in the work to find the bug there will be people who
| refuse to believe you know what you are doing on account of "how
| could you possibly know better than $GIANT_CORPORATION".
|
| If you're looking for ways to improve this, as a user you should
| strive to understand how the things you use work and how you
| might respond to it not working-in my experience this is a
| perpetually undervalued skill. As a software author you should
| look to make your software introspectable, be that providing
| debug symbols or instructions on how users can diagnose issues.
| And from both sides, more debugging tools and stories like this
| one :)
| EdSchouten wrote:
| Both of these issues are good candidates for
| https://accidentallyquadratic.tumblr.com
| JyB wrote:
| I am absolutely shocked about this finding. The amount of money
| on microtransactions Rockstar lost because of this single issue
| must be gigantic. The amount of people that got turned off by the
| loading times over the years is massive. It's mind boggling.
| linuxhansl wrote:
| I always cringe when I come across any kind of loop with
| unnecessary polynomial or even exponential complexity, even when
| it is found in a non-critical part of the code.
|
| It offends my taste as a software engineer and I always fix
| these.
| Geee wrote:
| I think the developers knew it was slow, but they assumed that it
| was because 'parsing JSON is slow and we can't do anything about
| it'. Sometimes you're just too sure what's wrong.
| w_for_wumbo wrote:
| How many bets they'll leave it like this on the current version
| and fix it for the PS5 version, to show how dramatic of a change
| it is between consoles?
| ksec wrote:
| Even after cutting loading by 70% it still take a minute? I
| haven't played any AAA titles for a long time. But even 30s is
| way too long. Especially I used to play with HDD. Considering
| modern SSD can be 30x Faster in Seq Read and Random Read up to
| 200x.
|
| Is 1 min loading time even normal? Why did it take so long? I
| never played GTA Online so could someone explain?
| Red_Leaves_Flyy wrote:
| Many AAA games take minutes to load.
|
| https://hardcoregamer.com/2020/11/06/ps5-ssd-vs-ps4-hdd-load...
| iknowstuff wrote:
| Could be due to decompression of lots of huge assets.
| fl0tingh0st wrote:
| This is awesome! Great write up. Just loved it. Thank you :)
| awinter-py wrote:
| woo open source
| wun0ne wrote:
| Incredible work. Hopefully R* acknowledge this and compensate you
| in some way. I won't be holding my breath.
|
| Maybe set up a donation page? I'd be more than happy to send some
| beer money your way for your time spent on this!
| emptyparadise wrote:
| The author's reward will probably be a DMCA takedown.
| sundvor wrote:
| Agree, this is up there in the top tier of amazing stories I've
| read here on HN. I admire T0ST's technical and writing skills;
| first rate combination. Massive kudos, would like to shout a
| cup of coffee.
|
| (I also really like the design and presentation of the article;
| I'm running out of superlatives here.)
| kuroguro wrote:
| Thanks for the suggestion, probably missed most of the traffic
| but just added it :)
|
| https://buymeacoffee.com/t0st
| berkayozturk wrote:
| Hi,
|
| I loved the article. Are you planning to write any
| guides/tutorials about reverse engineering games? Seems like
| you have a lot of practical experience. I (and probably many
| other people) would be really excited if you started writing
| about how you do all these in detail with practical examples.
| I would even be glad to pay for such content.
| noobermin wrote:
| I hate to be the guy but I have to. I don't know how to
| accomplish this but if the part in question was open source
| someone would have noticed.
| raggi wrote:
| Spicy hot take: the root cause here is the awful c++ library
| ecosystem.
| steerablesafe wrote:
| Yeah no. While the C++ library ecosystem is painful to use, it
| still doesn't justify hand-rolling a JSON parser and there are
| certainly high quality hash-based container implementations
| available too, but even the standard one should beat the one
| used here.
| waprin wrote:
| But there is no "standard one"...reaffirming the point that
| you disagreed with. C++ standard library is blatantly missing
| key pieces given how long the language been around.
| Blumkratz wrote:
| Red Dead Redemption 2 introduced a bug where the exe would reduce
| the volumne on start to 19%.
|
| Its now at least 4 month old.
|
| So whats the problem? When you tab+alt out of fullscreen, to
| change your soundvolumen everytime you start the game, you have
| to rechange the graphics configuration as well.
|
| I solved it by .net code i found on github which would run fur 5
| minutes and would put the volumne up as soon as it ifinds the
| rdd2 process...
| Aissen wrote:
| It would make a great addition to the Accidentally Quadratic
| blog: https://accidentallyquadratic.tumblr.com/ (haven't been
| updated in ~2 years, but maybe the author is around).
| Dragonai wrote:
| There's some great information here. Thanks so much for
| sharing!
| cwojno wrote:
| Fantastic research and write-up
| capableweb wrote:
| > the problems shouldn't take more than a day for a single dev to
| solve
|
| It bothers me that so many of us developers are annoyed by
| managers saying stupid stuff like "shouldn't be too hard" about
| things they don't understand, but then other developers walk into
| the same trap.
|
| Yes, it looks simple at the surface. It most likely isn't. The
| problem is not that it looks simple, the problem is that you
| assume it is simple, probably because you don't have the full
| context.
| asadlionpk wrote:
| The person posted a PoC that works. Surely they have context
| now?
| capableweb wrote:
| That's very true, and the PoC works for that person. It's not
| that easy in real-life development though, you can't just
| switch out the JSON parser and call it a day. Lots of testing
| has to be done and you have to go through all the previous
| changes to make sure you're not missing what the previous
| maintainers did to fix some one-in-a-million issues.
|
| I'm not saying it's impossible for this to be as easy as the
| author claims it to be. I'm just saying that it might not
| actually be that easy in reality, if you're on the inside.
| segmondy wrote:
| Let's say that JSON parser is used in so many other places
| and we are afraid of regression. Guess what? We can make a
| new one. Let's call it newImprovedJsonParser(). Use that
| only at this spot, and you have fixed the issue without
| breaking other parts. You can then replace the other's once
| you vet that they are compatible.
| jhncls wrote:
| You seem to be hinting to Chesterton's fence [0].
|
| [0]: https://fs.blog/2020/03/chestertons-fence/
| resonantjacket5 wrote:
| Accept in this case I don't really see why you'd want a
| naive o(n^2) parser when it isn't needed. Well besides
| for slow loading times.
| asadlionpk wrote:
| Thanks for sharing this, didn't have a name for this
| scenario.
| asadlionpk wrote:
| But you do understand the ROI of this relatively simple
| fix?
| rvnx wrote:
| At such big amount of scale of revenues, I'm sure that
| GTA Online has feature flags that can act as canary
| deploys or A/B tests groups.
|
| Deploy one hacked version (with the dirty fix / hack for
| this case) to one group, deploy the standard version to
| another group, measure the ROI and the crash-rate.
|
| If the results are similar, then you validate permanently
| the fix and live happily after.
|
| 6 minutes to 1 minutes 10 is massive, it will even bring
| new players to join the game (I don't like to play GTA
| Online, though it's a great game, just because of loading
| times).
| toast0 wrote:
| OK, don't switch out the JSON parser. Just patch in the
| caching strlen before parsing the json, and set it back to
| normal strlen after; and fiddle with the hashmap stuff.
| fctorial wrote:
| This bug might be one of a two bug pair that cancel each
| other. Not very uncommon.
| stevemk14ebr wrote:
| it _is_ simple, as evident by the article if you read it in
| full
| capableweb wrote:
| I _did_ read it. Did I miss the part where Rockstar confessed
| it was a really easy issue to fix and they apologised or
| what's your point?
| stevemk14ebr wrote:
| 'Yes, it looks simple at the surface. It most likely isn't'
| my point is the issue is simple, as the article proves.
| blt wrote:
| I agree "shouldn't be too hard" should be avoided, but in this
| case the developers should fix it even if it _is_ hard.
| cma wrote:
| Back of the envelope (and if the same problem applies to
| consoles in addition to PC), the fix would save 30+ human
| waking lifetimes per year.
| bspammer wrote:
| I hope you at least agree this isn't a bug that should go
| unfixed for _7 years_.
| bedbuggy221 wrote:
| I'm a complete noob at all this, si there an easy way to try this
| out at all? This is an awesome find!
| endoelenar58 wrote:
| Developers probably knew about this, but executives didn't care
| when getting int monetary calculations.
| bedbuggy221 wrote:
| I'm a zero to no knowledge on this in terms of implementation, is
| there like an easy execute at all?
| trollied wrote:
| Wow. I always assumed that profiling would be part of the pre-
| release test processes for AAA games...
| gbl08ma wrote:
| I had heard about this giant JSON from friends in the GTA V
| modding community. OP's idea of what it is used for is right.
| My guess is that this JSON was quite smaller when the game
| released and has been increasing in size as they add more and
| more items to sell in-game. Additionally, I speculate that most
| of the people with the knowledge to do this sort of profiling
| moved on to work on other Rockstar titles, and the "secondary
| team(s)" maintaining GTA Online throughout most of its lifespan
| either didn't notice the problem, since it's something that has
| become worse slowly over the years, or don't have enough
| bandwidth to focus on it and fix it.
|
| It's also possible they are very aware of it and are saving up
| this improvement for the next iteration of GTA Online, running
| on a newer version of their game engine :)
| wruza wrote:
| _are aware of it and are saving up_
|
| Still much better than e.g. Ubisoft which repainted safety
| borders from red to blue and removed shadows in TM2 Canyon
| few years after release, also breaking a lot of user tracks.
| (If you're not sure why, it was before a new iteration of the
| game)
| Negitivefrags wrote:
| When it was released the game didn't have all the
| microtransactions so it probably took no time at all to process
| the JSON even with this issue.
|
| Then over time they slowly add data to the JSON and then this
| O(n^2) stuff starts to creep up and up, but the farther away
| from release you are, the less likely that the kind of
| engineers to who do optimisation and paying any attention.
|
| They are all off working on the next game.
| paulryanrogers wrote:
| Could it be that MMO players are just more accustomed to long
| load times? (Lack of loading patience is one of the reasons I
| don't play SWOTOR.)
| josephg wrote:
| When I used to play world of Warcraft, it never took more
| than 30 seconds or so to load. It got much faster over the
| years - when I was playing a few years ago it was more like 5
| seconds from character selection to being in the world.
|
| Nothing like the 6 minutes people are talking about for GTA.
| That's ridiculous.
| jimbob45 wrote:
| Same. I wonder if the dev didn't bother to fix it because they
| assumed profiling would identify it as a non-issue.
| bombcar wrote:
| I wonder if the list was much shorter at release and wasn't
| super slow on Developement systems.
| yread wrote:
| Could be that at release the JSON was just 200kb with 1000
| entries or something and this quadratic "algorithm" wasn't the
| slowest part of the process
| IgorPartola wrote:
| More importantly how do you release a game that takes 6 minutes
| to load? This is why mobile gaming has the advantage. In those
| 6 minutes I could have played a round of a game that's quite
| satisfying and put it down already. This just seems sloppy.
| Aerroon wrote:
| I've actually opened a game with long loading times and alt
| tabbed out, because I knew it would take a while. I booted up
| another game to play a little bit until the first game
| finishes loading. 3 hours later I was done playing and
| realized that I was supposed to play game #1.
| bluedino wrote:
| It was only one minute at first. As the game had content
| added, it got longer and longer and nobody cared.
| superjan wrote:
| It's quite possible that there were way less items in the
| store on launch, or during testing. Then it could easily be
| overlooked. Of course no excuse to not fix it later.
| IgorPartola wrote:
| OK but how come even without the store it took more than
| 20-30 seconds? This was a beefy computer 7 years ago.
| ankurdhama wrote:
| Let me guess, when they started this JSON was small and this
| parsing code worked fine for that small input but over time the
| JSON got bigger and bigger and here we are.
| faebi wrote:
| Wow, many people argue how optimized GTA was and then this. I
| wonder how much money they lost because of this. I often stopped
| playing because it just took too long to load.
| formerly_proven wrote:
| GTA V (the single player game) is quite well optimized and
| needs a frame rate limiter on most newer systems because it
| will run at over ~180 fps, at which point the engine starts to
| barf all over itself.
|
| GTA Online is a huge, enormously buggy and slow mess that will
| generally struggle to run at 80 fps on a top-of-the-line 2020
| system (think 10900K at over 5 GHz with a 3090) and will almost
| never cross the 120 fps threshold no matter how fast your
| system is and how low the settings are.
| ecf wrote:
| > at which point the engine starts to barf all over itself.
|
| I'm really confused as to why games are determining anything
| whatsoever based on the refresh rate of the screen.
|
| Skyrim has this same problem and not being able to play over
| 60fps is the reason I haven't touched the game in years.
| AndriyKunitsyn wrote:
| Because unfortunately, in order to draw something on the
| screen, you need to determine what you draw first. So,
| before each screen refresh, a thing called "game simulation
| tick" happens.
| gridspy wrote:
| Because if you run the simulation at a different frame-rate
| from the rendering it is a huge amount more work. Suddenly
| you have to start marshaling data through intermediate data
| structures and apply interpolation to everything.
|
| If you then try to run the simulation in parallel with the
| rendering (rather than between some frames) it is even more
| work, since inter-thread communication is hard.
|
| This stuff might seem easy for very good programmers,
| however on a game you want to hire a wide range of
| programmer skill and "game-play programmers" tend to be
| weaker on the pure programming front (their skills lie
| elsewhere)
| megameter wrote:
| It's a coding strategy intended to optimize around console
| refresh targets first and then "do whatever" for PC build.
|
| Generally, a AAA console game will target 30hz or 60hz.
| Therefore the timing loop is built to serve updates at a
| steady pace of 30 or 60, with limiting if it goes faster.
| Many game engines also interpolate animations separately
| from the rest of the gameplay, allowing them to float at
| different refresh rates. Many game engines further will
| also decouple AI routine tick rates from physics to spread
| out the load. Many game engines now also interleave updates
| and kick off the next frame before the first is complete,
| using clever buffering and multithreaded job code. All
| told, timing in games is one of those hazard zones where
| the dependencies are both numerous and invisible.
|
| When you bring this piece of intricate engineering over to
| PC and crank up the numbers, you hit edge cases. Things
| break. It's usually possible to rework the design to get
| better framerate independence, but doing do would be
| invasive - you'd be changing assumptions that are now baked
| into the content. It isn't just fixing one routine.
| SSLy wrote:
| >Skyrim has this same problem and not being able to play
| over 60fps is the reason I haven't touched the game in
| years.
|
| https://www.nexusmods.com/skyrimspecialedition/mods/34705
| >Refresh rate uncap for exclusive fullscreen mode.
| eznzt wrote:
| The 3D engine is highly optimised.
| SteveNuts wrote:
| They must have hired a Solarwinds intern to write the loading
| screen.
| alborzb wrote:
| It always fascinates me how sometimes people defend things just
| because they're a fan, even if the particular aspect they're
| defending doesn't make sense!
|
| I've seen this happen with some other games which are not the
| best optimised for PCs, but the fans will still defend the
| developers, just because they like the brand
| comboy wrote:
| They did some good stuff:
| http://www.adriancourreges.com/blog/2015/11/02/gta-v-
| graphic...
|
| And just thinking about the size of the game and popularity,
| to make it work at all, requires some skill. Which makes the
| OP even more unbelievable.
| ip26 wrote:
| It's part of social validation. You inherently want other
| people to like the things you like, because it validates your
| choices. This in turn means you'll defend things you like.
|
| Even "rebels", who supposedly relish having fringe tastes,
| want other rebels to approve of their tastes.
|
| The more strongly you stake your identity as "fan of X", the
| more social disapproval of X hurts.
| bredren wrote:
| This is common with disgraced entertainers.
| _the_inflator wrote:
| I disagree. There is no contradiction. JSON can be a different
| beast for C, C++, Java, backend coders. You can implement
| complex 3D graphics while struggling with JSON.
|
| For example, my backend Java guys struggled heavily with JSON
| mappers. It took them forever to apply changes safely. My
| department consumes a lot of data from backend systems and we
| have to aggregate and transform them. Unfortunately the
| consumed structure changes often.
|
| While a JSON mapper in our case in JAVA was sort of
| exceptionally hard to handle, a simple NodeJS layer in
| JavaScript did the job exceptionally easy and fast. So we used
| a small NodeJS layer to handle the mapping instead of doing
| this in Java.
|
| Moral of the story: Sometimes there are better tools outside
| your view. And this seems to be many times the case for JSON.
| JSON means JavaScript Object Notation. It is still tough for OO
| languages to handle.
|
| This is my observation.
| Koiwai wrote:
| What are you talking about? the performance improvement is
| showcased in real world and you're denying it?
| c7DJTLrn wrote:
| Yep, sometimes I feel like having a drive around, and then I
| remember how long it takes to load and play something else
| instead. If you end up in a lobby with undesirables and are
| forced to switch, you've got another long wait.
| connicpu wrote:
| I'm willing to bet the developers who wrote the in game store
| are not the same developers who optimized the rendering
| pipeline
| blibble wrote:
| I tried GTA online once and indeed got fed up of the load times
| jeroenhd wrote:
| GTA, at least the core gameplay and the single player mode, is
| quite well optimised. The game ran well even on the cheaper
| side of gaming PC hardware.
|
| This... this is GTA online. It's a cash cow designed to suck
| cash out of your pocket. Ads for things you can spend your
| money on are shown while "connecting", so if this delay wasn't
| introduced intentionally, it sure isn't a high priority fix.
| The code isn't part of the optimised, streamlined, interactive
| part of the game, it's part of the menu and loader system.
|
| Most of these online games/services have so-called "whales"
| that contribute most if not all of the income the platform
| makes. If these whales are willing to spend the wads of cash
| they throw at the platform, they won't even care for another
| five minutes of ads. The amounts of cash some of these people
| spend is obscene; the millions Take Two profit from GTA every
| year are generally generated by only a tiny (usually a single
| number percentage) of the total player base.
|
| In the end, I doubt they've lost much money on this. They
| might've even made some from the extra ads.
| jsheard wrote:
| > GTA, at least the core gameplay and the single player mode,
| is quite well optimised. The game ran well even on the
| cheaper side of gaming PC hardware.
|
| It's easy to forget that GTA5/GTA:O was originally a _360
| /PS3_ game, getting a game of that scope running at all on a
| system with just 256MB of RAM and VRAM was an incredible
| achievement.
|
| The A-Team developers who made that happen were probably
| moved over to Red Dead Redemption 2 though, with GTA5s long
| tail being handled by the B-Team.
| psyklic wrote:
| I wouldn't be surprised if the long wait increases profits --
| as you wait, Rockstar shows you ads for on-sale items.
| fastball wrote:
| Why in the world would you roll your own JSON parser?
|
| For reference, I just ran a 10MB file through the JSON parser I
| use _in Python_ and it finished in 0.047s (vs 190s)
| lordnacho wrote:
| I wonder if there's some corporate policy against using
| external libs. You'd think most of them would have solved this.
| rapsey wrote:
| Python has a package manager and a rich standard library.
| ed25519FUUU wrote:
| I love reading about this type of hacking/debugging, where they
| disassemble the binary and patch it.
|
| Can anyone recommend a youtube video where I can _watch_ (not
| necessarily learn) people doing this sort of work? I 'm in the
| mood for some vicarious hacking :-)
| notriv wrote:
| I highly recommend this forum :)
|
| https://guidedhacking.com/
| voltagex_ wrote:
| liveoverflow and stacksmashing channels on YouTube.
| dekerta wrote:
| This article is great. As a developer, I've always been curious
| about what the heck GTAO was doing that could take so long to
| load
| rewsiffer wrote:
| Maybe someone already asked(but the comments are huge). Is there
| anyway to also trim down the JSON file? Either on disk, or when
| it is fetched from remote? If you don't care about transactions
| then you could remove it completely.
| sim_card_map wrote:
| Unbelievable.
|
| Such silly mistakes causing such a huge delay for millions of
| gamers over 6 years.
| deevolution wrote:
| Wow this is absolutely fantastic investigation! I learned alot!
| Thank you.
| kevinventullo wrote:
| I gave up on playing GTA:O because everything took so long to
| load, having never spent a dime. I have to imagine there is so
| much lost revenue because of this bug; I hate to be harsh but it
| is truly an embarrassment that someone external to R* debugged
| and fixed this before they did (given they had 6 years!).
| ThomW wrote:
| Load times is absolutely the primary reason I quit playing.
| bouke wrote:
| Same here, being slow as molasses really spoiled the game for
| me. I wonder if the same software quality caused other
| unnecessary in-game loading times. I felt that time spent
| playing GTA:O was about 60% waiting in lobby/playing, and the
| remaining 40% actual play time. Same with RDR2 by the way.
| gridspy wrote:
| These problems come down to which programmers were allowed
| to remain on the team and how much they were allowed to
| change. So all similar performance problems will remain.
| jjulius wrote:
| I can stomach a long load time (but that's not to excuse how
| godawful this load time is), just as long as it's worth it. I
| need some kind of resolution/ultimate goal, but in GTA:O
| there really isn't one; it's just grind-grind-grind so that
| you can get yet another car, or another house, or another
| thing that you already have plenty of. Lather, rinse, repeat
| in perpetuity.
|
| I just felt like every time I logged in I went, "So what's
| the point here?".
| josalhor wrote:
| I have a friend who works at Rockstar. I have forwarded the blog
| post to him.
| computronus wrote:
| I don't play GTA Online, but I'll just say thanks for
| forwarding, since it's a concrete step towards possibly fixing
| the bugs.
| marshmallow_12 wrote:
| ask him from me what on earth is wrong with R*
| xdrosenheim wrote:
| Short answer: it doesn't make enough money, it doesn't
| matter.
| GhostVII wrote:
| GTA V has made 6 billion in revenue, it's probably the most
| profitable video game in history by a large margin.
| jojobas wrote:
| GTA V is profitable, fixing bugs is not.
| josalhor wrote:
| Fixing bugs that reduce engagement is profitable. I am
| pretty confident this kind of bug has easily costed
| Rockstar in the order of 10-100 million dollars _per
| year_.
| lhl wrote:
| You could easily model how increased retention/engagement
| rates would translate to increased microtransaction
| revenue from drastically decreased load times. You could
| even bucket test it if you wanted to directly quantify
| the results, but I'm sure that the cost payback would be
| a matter of days.
| marshmallow_12 wrote:
| but why bother when you got so much money anyway? It's
| shockingly lazy, but are you as productive as you can be?
| They're no different.
| account42 wrote:
| I once came across a game that rendered progress bar frames at
| fixed points during the loading process. With VSync enabled this
| would significantly slow things down compared to what a fast
| computer could manage otherwise.
| dstola wrote:
| Hopefully they completely ignored this and other issues in GTA5
| online because they have been working on GTA6 all these years
| /end sarcasm
|
| But honestly, on a large enough team with a large enough backlog
| much worse things slip through the cracks. Doesn't excuse this at
| all though.
|
| Also kudos on the write up! Enjoyed reading it
| xeyownt wrote:
| One of the most entertaining post I read lately. Congratulations
| for the job, and the clarity of the explanations! Reading the
| story, everything looks simple where it sure is not. Lot to learn
| inside, thanks to the author for that.
| MarekKnapek wrote:
| Please don't compute speed-up like that.
|
| Old time: 6 minutes (360 seconds). New time: 1 minute and 50
| seconds (110 seconds). Speed-up calculated by article author:
| 1-110/360=0.694 (69.4% saved). Speed-up calculated by me:
| 360/110=3.27 (3 times faster).
|
| Please calculate it the other way around. It makes great
| difference when you say you made something 10x faster than when
| you say you saved 90% of work even if both mean exactly the same
| thing.
|
| Bruce Dawson has great article about this:
| https://randomascii.wordpress.com/2018/02/04/what-we-talk-ab...
| woko wrote:
| I don't think this remark is warranted here. Your sentence
| ("3.27 times faster") is clear, but conveys the same meaning.
| His sentence ("loading times are cut down by 70%") does not
| refer to "speed" or "fast" (Ctrl+F the blog post to check for
| yourself) and is technically correct and clear. So the
| criticism raised in the WordPress article (about the usage of
| the words "speed" and "fast") does not apply. The mistake would
| have been to talk about "70% faster" or a "70% speed-up",
| because then there is ambiguity for the reader.
| tgtweak wrote:
| That's how you know he's a pure engineer - he used math not
| marketing
|
| Agreed however that this is 3x faster.
| ohwaitnvm wrote:
| I haven't played GTAO in a couple years, but I vaguely remember
| using the suspend process option of windows task manager to jump
| into online much faster.
|
| Googling gave me this result, which sounds about right for what I
| remember. https://www.gfinityesports.com/grand-theft-auto/gta-
| online-h...
|
| I'm not certain it's still a valid workaround, and it's not
| nearly as sophisticated as the OP method, but at least everyone
| can do it :)
| dEnigma wrote:
| I had to use this trick to kick GTA Online back into action
| when it got stuck forever on entering buildings (mid-2020).
| Didn't know it could be used to cut the load times in general.
| pityJuke wrote:
| I remember doing this to create a private online session (i.e.
| no other users).
| powerfulclick wrote:
| This is really cool - how did you develop the background
| knowledge to solve this? I'm trying to learn more about low-level
| stuff and I would have no idea how to approach solving a problem
| like this
| spuz wrote:
| I'd recommend searching HN for threads about learning reverse
| engineering. Here are a few that I've found:
|
| Reverse Engineering Course:
| https://news.ycombinator.com/item?id=22061842
|
| Reverse Engineering For Beginners:
| https://news.ycombinator.com/item?id=21640669
|
| Introduction to reverse engineering for beginners:
| https://news.ycombinator.com/item?id=16104958
| the_only_law wrote:
| I love it but I don't have the focus or patience to ever do
| anything more than basic analysis.
|
| A lot of the reverse engineers I know seemingly have deep
| platform knowledge and can do things like cite Win32 docs
| from memory.
| mmastrac wrote:
| I was a reverse engineer for years and never was able to do
| anything like quoting docs. I'd be constantly googling or
| using reference material. The only real attribute I'd
| suggest is tenacity.
| Graziano_M wrote:
| Tenacity is everything.
|
| Both when looking at a particular problem, but also in
| sticking to RE in general for long enough to pick up the
| skills and tricks that make you quick. There are
| countless tricks you pick up that cleave off huge amounts
| of time that would otherwise be wasted.
| saagarjha wrote:
| The number one way to get the patience to do this is to
| have impatience for software that is slow.
| meibo wrote:
| And with most RE software(especially the mentioned
| inaccessible interactive disassembling tool) you'll get
| that by default, waiting for a 70mb binary to analyze
| isn't fun :)
| saagarjha wrote:
| Fortunately, software engineers can be bad at making time
| value judgments ;)
| punnerud wrote:
| tl;dr
|
| * There's a single thread CPU bottleneck while starting up GTA
| Online
|
| * It turns out GTA struggles to parse a 10MB JSON file
|
| * The JSON parser itself is poorly built / naive and
|
| * After parsing there's a slow item de-duplication routine
|
| (Copied from the END of the article)
| ozzmotik wrote:
| I find it interesting that, with ~63k entries, they didn't just
| use a bloom filter to do a lookup on if the entity they're
| working with has already been seen. granted, they stoll need to
| store the data, but I think a bloom filter would be a more
| effective way to test if the item exists already
| tyingq wrote:
| _" They're parsing JSON. A whopping 10 megabytes worth of JSON
| with some 63k item entries."_
|
| Ahh. Modern software rocks.
| LukvonStrom wrote:
| why not embed node.js to do this efficiently :D
| ed25519FUUU wrote:
| Parsing 63k items in a 10 MB json string is pretty much a
| breeze on any modern system, including raspberry pi. I wouldn't
| even consider json as an anti-pattern with storing that much
| data if it's going over the wire (compressed with gzip).
|
| Down a little in the article and you'll see one of the real
| issues:
|
| > _But before it's stored? It checks the entire array, one by
| one, comparing the hash of the item to see if it's in the list
| or not. With ~63k entries that's (n^2+n) /2 = (63000^2+63000)/2
| = 1984531500 checks if my math is right. Most of them useless._
| Slikey wrote:
| Check out https://github.com/simdjson/simdjson
|
| More than 3 GB/s are possible. Like you said 10 MB of JSON is
| a breeze.
| tyingq wrote:
| The JSON patch took out more of the elapsed time. Granted, it
| was a terrible parser. But I still think JSON is a poor
| choice here. 63k x X checks for colons, balanced
| quotes/braces and so on just isn't needed.
| Time with only duplication check patch: 4m 30s Time
| with only JSON parser patch: 2m 50s
| masklinn wrote:
| > But I still think JSON is a poor choice here.
|
| It's an irrelevant one. The json parser from the python
| stdlib parses a 10Mb json patterned after the sample in a
| few dozen ms. And it's hardly a fast parser.
| bombcar wrote:
| At least parse it into SQLite. Once.
| tyingq wrote:
| I think just using a length encoded serialization format
| would have made this work reasonably fast.
| hobofan wrote:
| Or just any properly implemented JSON parser. That's a
| laughable small amount of JSON, which should easily be
| parsed in milliseconds.
| brianberns wrote:
| They probably add more entries over time (and maybe
| update/delete old ones), so you'd have to be careful about
| keeping the local DB in sync.
| bombcar wrote:
| So just have the client download the entire DB each time.
| Can't be that many megabytes.
| Twirrim wrote:
| I did a very very ugly quick hack in python. Took the
| example JSON, made the one list entry a string (lazy
| hack), repeated it 56,000 times. That resulted in a JSON
| doc that weighed in at 10M. My initial guess at 60,000
| times was a pure fluke!
|
| Dumped it in to a very simple sqlite db:
| $ du -hs gta.db 5.2M gta.db
|
| Even 10MB is peanuts for most of their target audience.
| Stick it in an sqlite db punted across and they'd cut out
| all of the parsing time too.
| high_byte wrote:
| this is amazing. good work! unbelievable
| schemescape wrote:
| The author's frustration with GTA Online reminds me of my own
| experience with Borderlands 2 taking over 1 minute for its
| "searching for downloadable content" loading phase. I hate DLC,
| so having to wait for it was especially aggravating.
|
| I did not ever consider inspecting obfuscated assembly and
| hooking functions to fix the problem myself. Very impressive
| work!
| animal531 wrote:
| I played the game on PS3 and PC. The loading time at launch for
| PS3 (and later for PC, albeit on SSD) wasn't great, but it also
| wasn't nearly this terrible.
|
| From a game programming perspective, I'm sure at launch there
| were very few extras to obtain, so this method ran fast and
| didn't raise any red flags.
|
| But as time has worn on they've added a ton of extra items and
| it's become a real problem. What it does show is that probably
| most of their team are working on creating new things they can
| sell, vs testing and maintaining the codebase for the last N
| years.
| dijit wrote:
| It's never been "good", I played since 2013, Xbox 360 and later
| repurchased the game for PS4, the online load times were not
| just annoying they outright broke the game for me. To be having
| fun and then have a many minute delay while being pinned to the
| screen (because after loading from a mission you're never in a
| safe place).
|
| Looking down on the ground through the clouds at the san
| Andreas's streets with that wispy air sound while waiting for
| those large thud noises which could come randomly will forever
| be etched into my memory as something which completely broke
| the fun I was having, especially when playing with friends and
| trying to do Heists later in the products life.
|
| And because of this: getting people to play was really
| difficult, the combination of huge updates which took hours to
| download (PS4 has slow access to it's drive even if you upgrade
| to SSD) and the insanely long loading times once you have the
| patch culminated in many hours of lost gameplay.
|
| I remember a quote from Steve Jobs which fits here: "Let's say
| you can shave 10 seconds off of the boot time. Multiply that by
| five million users and thats 50 million seconds, every single
| day. Over a year, that's probably dozens of lifetimes. So if
| you make it boot ten seconds faster, you've saved a dozen
| lives. That's really worth it, don't you think?"[0]
|
| [0]:
| https://www.folklore.org/StoryView.py?story=Saving_Lives.txt
| forrestthewoods wrote:
| Super impressive. I am generally not impressed with community
| hacks because, as every comment thread discusses, they are
| generally hacks that do not support all players and thus can't be
| shipped.
|
| But this seems pretty damn clean. And it's egregious that no one
| at R* took a week to investigate and fix "correctly".
| sailfast wrote:
| This is such an awesome article, and I love it. Thanks to the
| author for the great digging and explanation.
| say_it_as_it_is wrote:
| Micro transactions were probably added in late stages of
| development by someone pressured to get the job done quickly and
| the developers were savvy enough to believe that the added
| overhead to slow initial loading was standard fare that users
| already accepted. Not enough time to optimize, forced to ship,
| and good enough.
| dvdbloc wrote:
| I bet if the devs did more leetcode this would have never
| happened /s
| z92 wrote:
| Also note that the way he fixed it, strlen only caches the last
| call and returns quickly on an immediate second call with the
| same string.
|
| Another reason why C styled null terminated strings suck. Use a
| class or structure and store both the string pointer and its
| length.
|
| I have seen other programs where strlen was gobbling up 95% of
| execution time.
| ip26 wrote:
| Could this be worked into a compiler/stdlib from the back-end?
| Could a compiler/stdlib quietly treat all strings as a struct
| of {length,string} and redefine strlen to just fetch the length
| field? Perhaps setting a hook to transparently update "length"
| when "string" is updated is not trivial.
|
| Edit: hah, I'm decades late to the party, here we go:
|
| _Most modern libraries replace C strings with a structure
| containing a 32-bit or larger length value (far more than were
| ever considered for length-prefixed strings), and often add
| another pointer, a reference count, and even a NUL to speed up
| conversion back to a C string. Memory is far larger now, such
| that if the addition of 3 (or 16, or more) bytes to each string
| is a real problem the software will have to be dealing with so
| many small strings that some other storage method will save
| even more memory (for instance there may be so many duplicates
| that a hash table will use less memory). Examples include the
| C++ Standard Template Library std::string..._
|
| https://en.wikipedia.org/wiki/Null-terminated_string
| toast0 wrote:
| I don't think you could do it transparently, because it's
| expected to pass the tail of a character array by doing
| &s[100] or s + 100, etc. I don't think that would be easy to
| catch all of those and turn them into a string fragment
| reference.
|
| From c++ class, std::string was easy enough to use
| everywhere, and just foo.c_str() when you needed to send it
| to a C library. But that may drags in a lot of assumptions
| about memory allocation and what not. Clearly, we don't want
| to allocate when taking 6 minutes to parse 10 megs of JSON!
| :)
| [deleted]
| magicalhippo wrote:
| If only C had introduced an actual string type...
| ziml77 wrote:
| I'm with you. I hate null terminated strings. In my first
| experiences with C, I specifically hated them because of strlen
| needing to scan them fully. When C++ introduced string_view, my
| hate grew when I realized that I could have zero-copy slices
| all the way up until I needed to interface with a C API. At
| that point you're forced to copy, even if your string_view came
| from something that was null terminated!
| wruza wrote:
| Not that C strings do not suck, but with pascal strings we
| could discuss in this thread how implicitly copying a slowly
| decreasing part of a 10mb string at every tokenizer iteration
| could miss a developer's attention. It's completely unrelated
| to C strings and is completely related to bytefucking by hand.
| Once you throw a task like "write a low-level edge-case riddled
| primitive value shuffling machine" into an usual middle level
| corporative production pipeline, you're screwed by default.
| ww520 wrote:
| Yep, O(n^2) has the problem that no matter how fast you upgrade
| your hardware it would still lag.
|
| Another pet peeve of mine is Civ 6's loading time for a saved
| game is atrocious. I'm sure there's a O(n^2) loop in there
| somewhere.
| wruza wrote:
| My personal pet peeve is Windows Update (and their products
| installation routine in general). I bet that it's n^3 somewhere
| deep and they carefully curb than n for decades.
| bscphil wrote:
| Good call. I'd love to read a post-mortem on why it was even
| _possible_ for Windows XP 's update check to be as slow as it
| was. I've definitely waited around 2 hours one time just for
| the _check_ to complete after finishing an installation.
| GuB-42 wrote:
| Maybe even more surprising to me is that sscanf() relies on
| strlen().
|
| I would have expected libc to take that use case in consideration
| and use a different algorithm when the string exceeds a certain
| size. Even if the GTA parser is not optimal, I would blame libc
| here. The worst part is that some machines may have an optimized
| libc and others don't, making the problem apparent only in some
| configuration.
|
| I believe standard libraries should always have a reasonable
| worst case by default. It doesn't have to be perfectly optimized,
| but I think it is important to have the best reasonable
| complexity class, to avoid these kinds of problems. The best
| implementations usually have several algorithms for different
| cases. For example, a sort function may do insertion (n^2, good
| for small n) -> quicksort (avg. nlog(n), worst n^2, good overall)
| -> heapsort (guaranteed nlog(n), slower than quicksort except in
| edge cases). This way, you will never hit n^2 but not at the cost
| of slow algorithm for the most common cases.
|
| The pseudo hash table is all GTA devs fault though.
| ziml77 wrote:
| I don't understand why it would need to use strlen anyway. Why
| wouldn't it treat the string like a stream when scanf is
| already coded to operate on an actual stream?
| JdeBP wrote:
| It's an implementation strategy, one of at least two,
| explained at https://news.ycombinator.com/item?id=26298300 on
| this very page.
| masklinn wrote:
| > For example, a sort function may do insertion [...]
|
| That's generally called "adaptive". A famous example of that is
| timsort.
|
| Your version has issues though: insertion sort is stable, which
| can dangerously mislead users.
| thitcanh wrote:
| This is why people should use commonly-available packages instead
| of rolling their own version of whatever dumb algorithm they
| think they can write. This happens all the time. Bugs have been
| fixed by others, but everyone is too smart to use someone else's
| code.
| xyst wrote:
| seems more like a bad copy and paste situation to me. it's sort
| of what you get when you pay for the lowest bid for
| contractors.
| mixologic wrote:
| Sometimes those commonly used packages end up being whatever
| dumb algorithm the author came up with, and nobody spends the
| time to verify if the package is worth it's popularity.
| thitcanh wrote:
| Doubt that. Popularity comes with heaps of PRs, some useful
| some less.
|
| If anyone used a JSON parser that took 4 minutes to parse a
| file you bet the author would know by the time the 100th user
| comes around.
|
| I had a tiny barely-used detection library that didn't detect
| it correctly in a new Edge browser update. Someone complained
| about it within the first month. The library has 20 stars and
| Edge has 500 users.
|
| Edit: Correction, it was 9 days after the browser release.
| breakingcups wrote:
| It is absolutely unbelievable (and unforgivable) that a cash cow
| such as GTA V has a problem like this present for over 6 years
| and it turns out to be something so absolutely simple.
|
| I do not agree with the sibling comment saying that this problem
| only looks simple and that we are missing context.
|
| This online gamemode alone made $1 billion in 2017 alone.
|
| Tweaking two functions to go from a load time of 6 minutes to
| less than two minutes is something any developer worth their salt
| should be able to do in a codebase like this equipped with a good
| profiler.
|
| Instead, someone with no source code managed to do this to an
| obfuscated executable loaded with anti-cheat measures.
|
| The fact that this problem is _caused_ by Rockstar 's excessive
| microtransaction policy (the 10MB of JSON causing this bottleneck
| are all available microtransaction items) is the cherry on top.
|
| (And yes, I _might_ also still be salty because their parent
| company unjustly DMCA 'd re3 (https://github.com/GTAmodding/re3),
| the reverse engineered version of GTA III and Vice City. A
| twenty-year-old game. Which wasn't even playable without
| purchasing the original game.)
| crazygringo wrote:
| I imagine the conversation between the programmer(s) and
| management went exactly like this:
|
| Management: So, what can we do about the loading times?
|
| Programmer(s): That's just how long it takes to load JSON.
| After all, the algorithm/function couldn't be more
| straightforward. Most of the complaints are probably coming
| from older hardware. And with new PC's and next-gen consoles it
| probably won't be noticeable at all.
|
| Management: OK, guess that's that then. Sucks but nothing we
| can do.
|
| Management had no idea of knowing whether this is true or not
| -- they have to trust what their devs tell them. And every time
| over the years someone asked "hey why is loading so slow?" they
| get told "yeah they looked into it when it was built, turns out
| there was no way to speed it up, so not worth looking into
| again."
|
| And I'm guessing that while Rockstar's best devs are put on the
| really complex in-game performance stuff... their least
| experienced ones are put on stuff like... loading a game's JSON
| config from servers.
|
| I've seen it personally in the past where the supposedly "easy"
| dev tasks are given to a separate team entirely, accountable to
| management directly, instead of accountable to the highly
| capable tech lead in charge of all the rest. I've got to assume
| that was basically the root cause here.
|
| But I agree, this is incredibly embarrassing and unforgiveable.
| Whatever chain of accountability allowed this to happen...
| goddamn there's got to be one hell of an internal postmortem on
| this one.
| o_m wrote:
| Alternatively (from my experience):
|
| Programmer(s): Can we set aside some time to fix the long
| loading times?
|
| Management: No, that won't earn us any money, focus on adding
| features
| dijit wrote:
| That's not how it's been in any of my past gamedev jobs.
|
| I work LiveOps and usually long loading times are something
| that we would take seriously, as it negatively impacts the
| reputation of the game.
| CountHackulus wrote:
| I can pretty much guarantee that there was no discussion with
| management like that. From experience, live ops games are
| essentially a perpetually broken code base that was rushed
| into production, then a breakneck release schedule for new
| features and monetization. I've personally had this
| conversation a few times:
|
| Programmer: Loading times are really slow, I want to look
| into it next sprint.
|
| Management: Feature X is higher priority, put it in the
| backlog and we'll get to it.
| Shish2k wrote:
| At my last job I had that conversation several times :( Our
| website would regularly take 30s+ for a page to load, and
| we had an hour of scheduled downtime each week, because
| that's how long it took the webapp to restart each time
| code was pushed. "Scheduled downtime doesn't count as
| downtime, we still have the three 9's that meets our SLA,
| and there's nothing in the SLA about page load times. Now
| get back to building that feature which Sales promised a
| client was already finished"...
|
| Aside from being generally shameful, the real kicker was
| that this was a "website performance & reliability" company
| x__x
| StillBored wrote:
| Reminds me of working for a company in the 1990's that
| ran constant television ads convincing everyone their
| experts could fix everyone's networking problems. OTOH,
| as an engineer working at the company the file share used
| for editing code/building/etc, died for an hour or two
| seemingly every day when the network took its daily
| vacation.
|
| Many of us, after having run out of other crap to do,
| would sit around and wonder if the "B" grade network
| engineers were assigned to run the company LAN, or the
| ones we sent onsite were as incompetent.
| dragonwriter wrote:
| > Many of us, after having run out of other crap to do,
| would sit around and wonder if the "B" grade network
| engineers were assigned to run the company LAN
|
| Internal IT is almost invariably a cost center, the
| technicians providing the service you are selling to
| customers are working in a profit center. So, yeah,
| probably that plus be managed in a way which focussed on
| minimizing cost not maximizing internal customer
| satisfaction or other quality metrics.
| nomel wrote:
| I experienced this to incredible extremes. It was taking
| our clients _two hours_ to load their data into an external
| tool that they use, daily, to view that data. This was
| caused by our use of a data store that was only half
| supported by that tool. I showed that if we took two weeks
| to switch to the other data store, their load times would
| be _30 seconds_. It took two years, and a new manager, to
| get that implemented. Unfortunately, most of the clients
| are still using the old data store. They haven 't had time
| to switch to the new version...
| sillysaurusx wrote:
| Former gamedev here. I can vouch for this conversation. The
| idea that management would ask about slow loading times
| doesn't seem to be too realistic in my experience.
| toomanybeersies wrote:
| I had that exact conversation at my old job. They only
| started listening to me when some requests started timing
| out because of Heroku's 30 second request duration limit.
|
| Same thing happened with memory consumption. The problem
| was ignored until we regularly had background jobs using >
| 4 GB of memory, causing cascading failures of our EC2
| instances and a whole bunch of strange behaviour as the OOM
| killer sniped random processes.
| crazygringo wrote:
| Normally I'd agree with you but this particularly problem
| is SO visible, and experienced by everyone, that I have to
| think management must have looked into it. I mean, it's the
| loading screen. They couldn't not encounter it themselves,
| every time.
|
| But I could be wrong. I've only worked with sites and apps
| not gaming.
| danbolt wrote:
| I think someone in QA likely might have the scenario you
| describe, but I think management might only be playing
| the game 2-3 times a day at most. They're likely trying
| to keep the studio organized enough to have the game ship
| decently.
| sagarm wrote:
| Do you really think management at Rockstar is using their
| product?
| sakarisson wrote:
| Their children are, at least.
| trissylegs wrote:
| I once did get to look an optimsing a bad load time... only
| because at that point it was crashing due to running out of
| memory. (32-bit process)
|
| In this case it was a JSON, but using .NET so Newtonsoft is
| at lease efficient. The issues were many cases of: *
| Converting string to lowercase to compare them as the keys
| were case insensitive. (I replaced with a
| StringComparison.OrderinalCaseInsensitve) * Reduntant
| Dictionary's * Using Dictionary<Key, Key> as a Set.
| replaced with HashSet.
|
| The data wasn't meant to be _that_ big, but when a big
| client was migrated it ended up with 200MB of JSON. (If
| there data was organised differently it would 've been
| split accross many json blobs instead)
|
| It would also be nice to handle it alls as UTF8 like
| System.Text.Json does. That would half all the strings
| saving a fair bit. (I mean the JSON blob it starts with is
| converted to UTF-16 because .NET)
| dec0dedab0de wrote:
| Ughh Every time I want to fix anything I have to sneak it
| in with something else I'm working on, or wait so long to
| get approval that I forgot all the details.
| nomel wrote:
| I stopped doing this after realizing that all the extra
| work was just taking time away from my family. The last
| time I did anything out of work hours was when I told my
| manager that I was doing some optimization in my free
| time and he responded "I don't want you working on that
| in your free time, you could be working on this
| instead!". I don't plan on doing anything out of work
| hours again. He left, and now we're hiring more people,
| instead, since I can't get everything done. Win for
| everyone.
| greggman3 wrote:
| Tell that to Valve. The Source engine and all its games (Half
| Life 1, 2, Portal, Alyx) have horrible load times. They might
| not be as bad as the GTA example but they're long and extremely
| frustrating.
|
| And yet, no one cares, Those games (and GTA5) all sold millions
| of copies.
|
| The only way this stuff gets fixed is if (a) some programmer
| takes pride in load times or (b) customers stop buying games
| with slow load times.
|
| (b) never happens. If the game itself is good then people put
| up with the load times. If the game is bad and it has bad load
| times they'll point to the load times as a reason it's bad but
| the truth is it's the game itself that's bad because plenty of
| popular games have bad load times
|
| Also, any game programmer loading and parsing text at runtime
| by definition, doesn't care about load times. If you want fast
| load times you setup your data so you can load it directly into
| memory, fix a few pointers and then use it where it is. If you
| have to parse text or even parse binary and move things around
| then you've already failed.
| danlugo92 wrote:
| Do you have a link to how one would archiece this data
| pointer magic? I wouldnt know what to search for.
| wmil wrote:
| In the simplest case, in C you can read file data into
| memory, cast it as a struct, then just use that struct
| without ever doing any parsing.
|
| As things get more complex you're probably going to need to
| manually set some pointers after loading blobs of data and
| casting them.
|
| It's just the standard way of dealing with binary files in
| C. I'm not sure what you'd need for search terms.
| gwd wrote:
| I think there may sort of be another thing going on:
| Basically, that the length of load time is an indicator,
| "This is a _really_ serious program. " I've sort of noticed
| the same thing with test machines: the more expensive the
| machine, the longer it takes to actually get to the grub
| prompt.
|
| Six minutes is probably excessive, but having GTA take 1-2
| minutes to load almost certainly makes people feel better
| about the money they spent on the game than if it loaded up
| in 5 seconds like some low-production 2D adventure game.
| [deleted]
| kuroguro wrote:
| Small nitpick: I believe these are items/prices for the in-game
| currency, not micro-transactions.
|
| You can buy in-game currency for real world money tho:
| https://gta.fandom.com/wiki/Cash_Cards
|
| Not 100% sure, never bought anything.
| imglorp wrote:
| > This online gamemode alone made $1 billion in 2017 alone.
|
| There's the answer right there. They figure it's making $1B/yr,
| leave it alone. Maintenance? That cuts into the billion.
| Everyone moved onto the next project.
| tyingq wrote:
| Or they fix it, see that their "in game time" average drops,
| and then back it out...
| the_gipsy wrote:
| You might be onto something here...
| themeiguoren wrote:
| I would not at all be surprised if the long load time made
| for a sunk cost that kept people playing for longer
| sessions rather than picking it up for less than half an
| hour at a time.
| iknowstuff wrote:
| I don't know. In GTA Online you encounter loading every
| 15min.
| antihero wrote:
| This is actually very possible - if they allowed people
| to dip in and dip out, they will. I remember when CP2077
| came out on GeForce NOW and the wait times were 30+mins.
| I'd play a 6 hour session until I was booted off simply
| to make it worth the wait.
| simias wrote:
| I enjoyed GTA online but haven't touched it in well over
| a year, and the insane loading times are definitely a big
| reason why. For those who haven't played the game it's
| important to emphasize that it's usually in the 5minute
| range, and even then you'll regularly end up with
| connectivity issues or other problems that will kick you
| out of the lobby for yet an other ~5 minute load.
|
| When I played it wasn't uncommon to spend 30 minutes
| mostly looking at the loading screen while you were
| trying to set up a play session with a couple of friends.
|
| If you're an adult with limited playtime it's just a
| complete dealbreaker. You can't just decide to have a
| quick 20minute play session if you know that you'll have
| to spend at least half of it looking at loading screens.
| rustybolt wrote:
| Not very surprising. Twitter doesn't work properly on my
| desktop, google freezes when showing the cookiewall, github
| freezes my phone. These are all important projects of billion
| dollar companies.
| Zak wrote:
| I played through GTA V, enjoyed it, and tried out the online
| mode afterward.
|
| I've logged in exactly twice. Load times like that may be worth
| it to a hardcore gamer, but I have no patience for it. There's
| no shortage of entertainment available to someone with a PC, a
| reasonable internet connection, and a modicum of disposable
| income. Waste my time and I'll go elsewhere for my
| entertainment.
| Thaxll wrote:
| It was probably fast 10years ago when the store had couple of
| items, the dev back then never thought that it would grow to
| 60k items. Classic programming right there.
|
| As for profiling, Windows Performance Toolkit is the best
| available no?
| trulyme wrote:
| Meh. It's ok to assume low number of items and code
| accordingly. What is not ok is for the company to ignore such
| a problem for years, instead if detecting and fixing it.
| masklinn wrote:
| > The fact that this problem is caused by Rockstar's excessive
| microtransaction policy (the 10MB of JSON causing this
| bottleneck are all available microtransaction items) is the
| cherry on top.
|
| For what it's worth, 10MB of JSON is not much. Duplicating the
| example entry from the article 63000 times (replacing `key` by
| a uuid4 for unicity) yields 11.5MB JSON.
|
| Deserialising that JSON then inserting each entry in a dict
| (indexed by key) takes 450ms in Python.
|
| But as Bruce Dawson oft notes, quadratic behaviour is the sweet
| spot because it's "fast enough to go into production, and slow
| enough to fall over once it gets there". Here odds are there
| were only dozens or hundreds of items during dev so nobody
| noticed it would become slow as balls beyond a few thousand
| items.
|
| Plus load times are usually the one thing you start ignoring
| early on, just start the session, go take a coffee or a piss,
| and by the time you're back it's loaded. Especially after QA
| has notified of slow load times half a dozen times, the devs
| (with fast machines and possibly smaller development dataset)
| go "works fine", and QA just gives up.
| hnick wrote:
| > Plus load times are usually the one thing you start
| ignoring early on, just start the session, go take a coffee
| or a piss, and by the time you're back it's loaded.
|
| In GTA V, when I tried to enjoy multiplayer with my friends
| the abysmal load times were what killed it for me.
|
| You actually have to load into the game world - which takes
| forever - before having a friend invite you to their
| multiplayer world - which takes forever, again.
|
| So both a coffee, and a piss. Maybe they fixed that now?
| thaumasiotes wrote:
| > You actually have to load into the game world - which
| takes forever - before having a friend invite you to their
| multiplayer world - which takes forever, again.
|
| Is that... the same problem? Is microtransaction data
| different in your friend's multiplayer world than it is in
| the normal online world?
| hnick wrote:
| The article mentions story mode loading as well as online
| loading, but as I mentioned in another comment the story
| time shown there is much lower than what I experienced,
| probably because SSDs are now standard and were rarer in
| 2013 (I could not justify 50GB+ on this one game at the
| time). So it may be a mixture of factors.
| agret wrote:
| Then when you want to actually do an activity like a
| deathmatch you have to wait for matchmaking and then the
| loading - takes forever. Once you are finally in a match
| it's okay but as soon as the match ends you have to wait
| for the world to load again and then queue again which
| takes bloody forever. Spend 2hrs playing the game and have
| only a few matches, more time spent looking at loading
| screens than actually playing anything.
| ZuLuuuuuu wrote:
| Judging from your word choice "deathmatch" and your
| experience with long loading/matchmaking times I guess
| you might be a fellow Quake Champions player. Even if you
| are not, I agree that long loading times are a mood
| killer when you just want to play a couple of quick
| matches after work in your limited free time. It is even
| worse when you know the game's development is abandoned
| and it will never get fixed, even though you enjoy the
| game itself.
| franga2000 wrote:
| GTA V has a deathmatch mode and the parent comment sounds
| like it's talking about that. Especially the "once it's
| over, you need to load into the primary session, then
| wait for matchmaking, then wait for loading again" sounds
| exactly like GTA V.
| ZuLuuuuuu wrote:
| Ah I see, thanks for the clarification.
| yholio wrote:
| > more time spent looking at loading screens than
| actually playing anything.
|
| This could easily compete for the most expensive bug in
| history, up there with the Pentium Bug. It might have
| halved the revenue potential of a billion dollar
| franchise.
| simias wrote:
| I agree. I played GTA online for a bit and quite enjoyed it
| but I haven't touched it in a while and the insane loading
| times are a big reason why.
|
| It kind of baffles me that they haven't bothered to fix
| this trivial issue when the result is to cut 4 entire
| minutes of loading time.
| jjoonathan wrote:
| Back in dialup/DSL days I discovered a texture
| compression issue in America's Army (the free US Army
| game) that doubled its download/install size. Typical
| download times were about a day and the resume mechanism
| was poor, so this had the potential to save a lot of
| grief, not to mention hosting costs. I emailed them, they
| banned me for hacking, and the next version still had the
| same issue. Shrug.
| jack_riminton wrote:
| That's hilarious. Out of interest who in the company did
| you email?
| jjoonathan wrote:
| I was only able to find contact information for one
| person who I knew was probably technical (from their
| posts), so I sent it to them.
|
| I never learned the "other side" of this story, but a few
| years later the same dev team tried to recruit me at a CS
| contest, to which I politely declined.
|
| More details: I was young, without credit card, and
| gaming on a mac. AA was free and mac compatible. For a
| while -- apparently mac ports of unreal engine games were
| approximately all done by a single very productive
| contractor and from what I understand the US Army, uhh,
| stopped paying him at some point. So he stopped releasing
| the mac ports. From my point of view, this meant that I
| could only play with other mac users and couldn't use any
| of the fancy new maps. Logs indicated that the
| compatibility problems with the new maps were not
| particularly deep, so I got to parsing the unreal map
| files and was able to restore compatibility by deleting
| the offending objects. I implemented texture
| decoding/encoding mostly for curiosity and because
| textures were well documented in the reverse engineering
| "literature." I imagined a workflow where someone would
| bulk export and re-import textures and aside from the
| texture names the one piece of metadata I needed was the
| format: RGBA (uncompressed) or DXT (compressed)? I
| realized that I could easily identify DXT compression
| from the image histogram, so I didn't need to store
| separate metadata. Nifty! But it didn't work. Lots of
| textures stored in uncompressed RGBA8888 "erroneously"
| round-tripped to DXT. After poring over my own code, I
| eventually realized that this was because on many of the
| textures someone had enabled DXT compression and then
| disabled it, dropping the texture quality to that of DXT
| while bloating the texture size to that of RGBA8888
| (other textures were still stored as DXT, so compression
| itself was still working). I wrote a quick tool to add up
| the wasted space from storing DXT compressed textures in
| uncompressed RGB format and it came out to about half the
| total disk space, both before and after the top level
| installer's lossless compression. They could have re-
| enabled compression on most of the textures where they
| had disabled it without loss in quality, and if they had
| wanted a list of such textures I would have been able to
| provide it, but it didn't go down that way. When I shared
| what happened with my father, who had served, his
| reaction was "Now that's the Army I know!"
| zmix wrote:
| > So both a coffee, and a piss.
|
| Reminds me on loading "G.I. Joe" (from Epyx) on the C64
| with a 1541 floppy disk. However, the long loads came after
| every time you died and meant you also had to swap 4 disks.
| hnick wrote:
| I remember as a kid I went to someone's birthday party in
| the 80s and we wanted to play a karate themed game on
| something that used a cassette tape. It took so long to
| load we went and played outside!
|
| To be fair to GTA V, I don't think my installation was on
| a SSD because it was 50GB or something at the time (now
| it's 95GB?), but that said when it released SSDs were not
| as cheap or widespread as they are now so that's their
| problem. The linked article shows the initial load to be
| much shorter which did not match my experience.
| mst wrote:
| Oh gods, now I'm reminded of a cassette based game where
| there was one enemy where if he got a solid hit on you,
| you got bumped back to the beginning of the level.
|
| Which meant the game displayed "rewind to mark 500 and
| then hit play" and you had to do that to restart
| _lolsob_.
| jcims wrote:
| Not to one up but we didn't have any storage for our c64
| for the first year or so. We would team up to enter a
| game from Byte or whatever (one reader, one typer, one
| proofreader) and then protect that thing with our lives
| all weekend to keep people from unplugging it. The
| machine code games were the easiest to type but if they
| didn't work you were kind of hosed lol.
| vagrantJin wrote:
| That we can say the install bundle is 50 gigs with a
| straight face though? I remember not long ago games that
| were 8 gigs caused mayhem.
|
| I suppose devs dont care about that as long as their QA
| allows it.
| hnick wrote:
| I read that after patches GTA V is now around 95GB.
|
| Call of Duty: Black Ops Cold War is around 200GB fully
| installed with all features (HD textures etc).
|
| Some people have insinuated this is intentional to crowd
| out other games on console hard disks and make moving
| away from CoD have an opportunity cost. It's probably
| just laziness.
|
| I haven't looked into it in the past but some prior
| offenders had a lot of space wasted from uncompressed
| multi-lingual audio. Instead of letting us choose a
| language it installs them all so you can switch in game,
| and uncompressed for saving the CPU for game logic. For
| CoD the optional HD texture pack is 38GB so that's still
| a lot unaccounted for.
| rasz wrote:
| C64 tape with turbo was actually faster (~500 Bytes/s)
| than non turbo Floppy (~400 Bytes/s).
|
| Many 8bit Atari owners will have horror memories of
| aborted Tape loads. After over 20 years someone finally
| discovered a bug in original ATARI Tape loading ROM
| routine resulting in randomly corrupted loads, no amount
| of sitting motionless while the game loads would help in
| that case :)
| ajacksified wrote:
| They didn't fix it. I tried a few days ago, because it's a
| really fun game... except for these seemingly easy to fix
| issues that are huge barriers.
| hinkley wrote:
| And because a merchandising push in many games may be another
| 10-50 items, the first couple times the % increase is high
| but the magnitude is low (.5s to 1s) and by the time you're
| up to 1000, the % increase is too small to notice. Oh it took
| 30 seconds last week and now it's 33.
|
| Boiling the frog, as it were. This class of problems is why I
| want way more charts on the projects I work on, especially
| after we hit production. I may not notice an extra 500ms a
| week, but I'm for damn sure going to notice the slope of a
| line on a 6 month chart.
| Agentlien wrote:
| For the online games I worked on (a few of the recent NFS
| games) the items database was similar to the final set quite
| early in production and we kept an ongoing discussion about
| load times.
|
| I really liked this article, but I am a bit surprised that
| this made it into production. I have seen a few instances of
| this type of slowdowns live for very long, but they tend to
| be in compile times or development workflow, not in the
| product itself.
| hanniabu wrote:
| That QA bit is too true, "it works for me!" _shrug_.
| nicoburns wrote:
| You mention quadratic behaviours and there's probably some
| truth to that, but it seems to me that it's partly a C++
| problem. In any other langauge nobody would even consider
| hacking up JSON parsing using a string function. They'd use
| the stdlib functional if available or import a library, and
| this problem wouldn't exist.
| nitwit005 wrote:
| A lot of other languages make use of the c standard library
| functions to parse floats (and to do various trigonometric
| functions), so they may be more similar than you imagine.
| oblio wrote:
| Not super relevant, though. The average standard library
| from another language is a lot more hardened than the
| function written by Joe Sixpack in C last night.
| secondcoming wrote:
| Rapidjson
| raverbashing wrote:
| But C++ had at least a hash_set/hash_map since forever (or
| just set/map which are still better than this)
|
| I'm sure there are libraries to parse json in C++ or at
| least they should have built something internally if it's
| critical, instead they have someone less experienced build
| it and not stress test it?
| nicoburns wrote:
| >I'm sure there are libraries to parse json in C++
|
| There certainly are, but adding a library is much more
| difficult in C++ than pretty much any other language
| which seems to tempt people into hacky self-built parsing
| when they really ought to know better.
| thefz wrote:
| > Here odds are there were only dozens or hundreds of items
| during dev so nobody noticed it would become slow as balls
| beyond a few thousand items.
|
| Might be, but this particular issue has been raised by
| thousands of players and ignored for *years*.
| tekromancr wrote:
| Yea, given how easy it was for the author of the post to
| find it, I would guess that literally nobody in the last
| decade bothered to run a profiler to see where they were
| spending time.
|
| The only possible explanation is that management never made
| it a priority.
|
| I could see this happening. A project/product manager
| thinking "We could spend $unknown hours looking for
| potential speedups, or we could spend $known hours
| implementing new features directly tied to revenue"
|
| Which is kind of ironic since this fix would keep players
| playing for more time, increasing the chances that they
| spend more money.
| john_minsk wrote:
| I heard that Chrome team had this KPI from very early on -
| how much time it takes for Chrome to load and it stayed the
| same to date. i.e. they can't make any changes that will
| increase this parameter. Very clever if you ask me
| rasz wrote:
| Google lately "optimized" Chrome "time for the first page
| to load" by no longer waiting for extensions to initialize
| properly. First website you load bypasses all privacy/ad
| blocking extensions.
| Cthulhu_ wrote:
| Yeah I think that's the kind of odd behaviour that those
| KPI's end up causing; they 'cheat' the benchmark by
| avoiding certain behaviour, like loading extensions
| later.
|
| I mean I can understand it, a lot of extensions don't
| need to be on the critical path.
|
| But at the same time, I feel like Chrome could do things
| a lot better with extensions, such as better review
| policy and compiling them to wasm from the extensions
| store.
| saruken wrote:
| Wow, had no idea about this! Can you link me to a writeup
| or something?
| rasz wrote:
| https://github.com/Tampermonkey/tampermonkey/issues/1083
|
| confirmed by Tampermonkey dev.
| raf42 wrote:
| Thank you for confirming this, I thought I was going
| crazy seeing it happen a bunch recently. I assumed my
| system was just on the fritz.
| beached_whale wrote:
| It would be interesting to see what JSON library they used
| that uses scanf for parsing numbers. Nothing like a painters
| algorithm type scenario to really slow things down, but also
| JSON numbers are super simple and don't need all that work.
| That is hundreds of MB's of unneeded searching for 0s
| dijit wrote:
| Unlikely to be a library, either it's libc or it's
| homegrown.
|
| The only thing most game companies do when it comes to
| external libraries is to copy the source code of it into
| their repo and never update it, ever.
|
| OpenSSL is this way, it's a required installation for
| Playstation but debugging it is seriously hard, and
| perforce (the games industries version control of choice)
| can't handle external dependencies. Not to mention visual
| studio (the game industries IDE of choice..) can't handle
| debugging external libraries well either.
|
| So, most game studios throw up the hands, say "fuck it" and
| practice a heavy amount of NIH.
| thw0rted wrote:
| In another decade, there's going to be a story here about
| somebody getting their hands on the original source for
| this game, and the JSON parser will be a 10-line function
| with "//TODO: optimize later" at the top.
| WorldMaker wrote:
| Visual Studio keeps toying with the idea of a "NuGet for
| C++" and it is amazing that it still hasn't happened yet.
| It may seem to indicate that it isn't necessarily the IDE
| that can fix it, but the user's attitude. How much of the
| NIH and "just copy that dependency into the tree" is
| still encouraged for "security" [0] and
| "control"/"proprietary source"/"management" reasons?
|
| [0] Despite it being an obvious anti-pattern that you
| aren't going to update dependencies that require
| copy/paste and manual merge reviews, so security problems
| should be obviously more rampant than in systems where
| updating a dependency to the latest security patch is a
| single install command line (or update button in a GUI),
| there still seems to be so many C++ devs that love to
| chime in to every HN thread on a package manager
| vulnerability that they don't have those vulnerabilities.
| They don't "have" dependencies to manage, no matter how
| many stale 0-Days they copied and pasted from outside
| projects, they don't count as "dependencies" because they
| are hidden who knows where in the source tree.
| beached_whale wrote:
| I suspect vcpkg is the choice they made, it will/does
| have support for private and binary repo's too
| WorldMaker wrote:
| That certainly is the most recent attempt. They've had
| projects of one sort or another going back at least as
| far as 2013 from mentions in public blog posts but so far
| none of them seem to have got much traction with the
| community. Here's hoping it works this time?
| nullserver wrote:
| Was new guy at a startup. Soon noticed that chuck Norris was
| in our compiled JavaScript. Turned out Someone had included
| the entire test suite in production deploy.
|
| Had been like that for nearly a year. A few minutes of work
| brought out client js file from 12MB to less then 1mb.
| rsj_hn wrote:
| related: Guy Fieri in node https://nodesource.com/blog/is-
| guy-fieri-in-your-node-js-pac...
| [deleted]
| gordaco wrote:
| Ha, this is one of the reasons why I also include
| outlandish and wrong-looking stuff in unit tests. If we see
| where it doesn't belong, then we know for sure that we are
| doing something wrong.
|
| Most often I use unicode strings in unexpected alphabets
| (i.e. from languages that are not supported by our
| application and that are not used by the mother tongue of
| any developer from our team). This includes Chinese,
| Malayalam, Arabic and a few more. There was a time when I
| wanted to test the "wrong data" cases for some
| deserialising function, and I was part annoyed and part
| amusingly surprised to discover that doing
| Integer.parseInt("43046721") in Java does parse the arabic
| digits correctly even without specifying any locale.
| notretarded wrote:
| That just seems pointless
| jogjayr wrote:
| > Soon noticed that chuck Norris was in our compiled
| JavaScript
|
| Is that a library? Or the string "Chuck Norris"?
| jeffgreco wrote:
| Or the actor/martial artist?
| finnh wrote:
| bravo! i'll take the downvotes that this content-free
| comment will garner, but you just made my morning.
| nullserver wrote:
| String. Used as factory test for populating objects in
| tests.
|
| It certainly caught my attention.
| kevincox wrote:
| I was assuming that someone used the string as a name in
| the test?
| hnick wrote:
| Yes I think they meant they saw a weird name that stood
| out (I've seen Donald Duck at work) and they investigated
| more, finding it was test data.
| wiz21c wrote:
| I've seen "Carmen Electra" myself :-) And other funny
| (more gross) names in the dark corners of huge
| databases...
|
| I also had a customer discover a playboy center page at
| the end of a test we sent to them. One of the dev thought
| it'd be a nice reward. Things went very bad from him
| right after the phone call...
| nullserver wrote:
| Always assume others have the sense of humor of a 99 year
| old stick of tnt.
|
| Had someone at work mention they needed to drop off a dog
| at grooming on the way to pick up Chinese.
|
| I had to walk away for a bit, as I couldn't hold it in,
| but didn't know if that sort of humor would fly with that
| crowd.
| milesward wrote:
| hmm.. the entire pricing table for Google Cloud (nearly 100k
| skus and piles of weirdness) was only ~2mb... seems pretty
| big.
| ldng wrote:
| But is quadratic the real issue ? Isn't that a developer
| answer ?
|
| The best algorithm for small, medium or a large size are not
| the same and generally behave poorly in the other cases. And
| what is small? Medium? Large?
|
| The truth is that there is no one size fits all and
| assumptions _need_ to be reviewed periodically and adapted
| accordingly. And they never are... Ask a DBA.
| oivey wrote:
| In the small case here, there is no meaningful difference
| in speed between parsers. Using a quadratic algorithm has
| no advantage and is just an incorrect design.
| [deleted]
| gridspy wrote:
| quadratic is a fancy way of saying "this code is super fast
| with no data, super slow once you have a decent amount"
|
| The problem is that when you double the amount of stuff in
| the JSON document, you quadruple (or more) the scanning
| penalty in both the string and the list.
|
| Why quadruple? Because you end up scanning a list which is
| twice as long. You have to scan that list twice as many
| times. 2x2 = 4. The larger list no longer fits in the fast
| (cache) memory, among other issues. The cache issue alone
| can add another 10x (or more!) penalty.
| ldng wrote:
| > quadratic is a fancy way of saying "this code is super
| fast with no data, super slow once you have a decent
| amount"
|
| Well, that is an abuse of the term, by people that
| sometimes don't actually know what that really means. Up
| to a point, quadratic _IS_ faster than linear after all
| for example. Too many developer love too abuse the word
| blindly.
|
| If it is badly tested with no data, it is badly tested
| with no data. Period. Not "quadratic".
|
| > The problem is that when you double the amount of stuff
| in the JSON document, you quadruple (or more) the
| scanning penalty in both the string and the list.
|
| My point was precisely it depends on the data and initial
| assumption are to be routinely revised. I was making a
| general point.
|
| Maybe the guy was pinky-sworn that the JSON would hardly
| change and that the items were supposed to be ordered,
| sequential and no more than 101. For all you know it is
| even documented and nobody cared/remembered/checked when
| updating the JSON. But we don't know, obfuscated code
| don't comes with comments and context ...
|
| Or, it is actually a real rookie mistake. It probably
| was, but we don't have all the facts.
| mytherin wrote:
| > Well, that is an abuse of the term, by people that
| sometimes don't actually know what that really means. Up
| to a point, quadratic IS faster than linear after all for
| example. Too many developer love too abuse the word
| blindly.
|
| There is absolutely no guarantee that a quadratic
| algorithm has to be faster than a linear algorithm for
| small N. It _can_ be, in some situations for some
| algorithms, but the complexity class of the algorithm has
| nothing to do with that. A quadratic algorithm may well
| be slower than a linear algorithm for _any_ N.
|
| The only thing the complexity class tells us is that
| _starting from some N_ the linear algorithm is faster
| than the quadratic algorithm. That N could be 0, it could
| be 100, or it could be 1 billion.
|
| In my experience it's usually between 0~1000, but again,
| that depends. The complexity class makes no such
| guarantees. The complexity class tells us the general
| shape of the performance graph, but not exactly where the
| graphs will intersect: this depends on the constants.
|
| > If it is badly tested with no data, it is badly tested
| with no data. Period. Not "quadratic".
|
| It is both. The _problem_ is that the algorithm has
| quadratic complexity. The fact that it was badly tested
| caused this fact to remain hidden while writing the code,
| and turn into a real problem later on.
| jack_riminton wrote:
| Great explanation. Thanks
| masklinn wrote:
| > But is quadratic the real issue ?
|
| Yes. That is literally the entirety of the issue: online
| loading takes 5mn because there are two accidentally
| quadratic loops which spin their wheel.
|
| > The best algorithm for small, medium or a large size are
| not the same and generally behave poorly in the other
| cases.
|
| "Behaves poorly" tends to have very different consequences:
| an algorithm for large sizes tends to have significant set
| up and thus constant overhead for small sizes. This is easy
| to notice and remediate.
|
| A naive quadratic algorithm will blow up your production
| unless you dev with production data, and possibly even then
| (if production data keeps growing long after the initial
| development).
| gameswithgo wrote:
| What is complicated about it is that an online modern 3d game
| is huge, and there are 95,000 places where a dumb mistake could
| hurt performance a lot for some customers. You catch 94,999 of
| them and then "unforgiveable"
| Plutoberth wrote:
| If it was that way for a few months and then fixed... still
| pretty shoddy but sure. However, it has been that way for
| YEARS, and is one of the most common complaints among
| players. I wonder how much of the remaining loading time
| could actually be shaved off if someone with the source code
| took a crack at it.
| neatze wrote:
| I am not saying it is the case, nor understand details of
| solution in depth to comment on it, but in analogy, this reads
| to me, like an yelling at person who figure out how to solve
| rubix cube puzzle steps, because once steps are known solution
| is simple.
| burlesona wrote:
| No, other people have pointed this out, this should have been
| very easy to recognize as inefficient in the source code.
| More likely the code was written hastily and only tested
| against very small inputs, and then nobody ever actually
| tried to improve the famously long load times.
| wnoise wrote:
| The sscanf issue was not obvious: it looks linear. And
| should be, on a better sscanf implementation.
|
| The duplicate checking on the other hand is a classic
| "accidentally quadratic" case that is obvious.
| db48x wrote:
| In my experience most engineers have never used a profiler even
| once. They write the code, and if you're lucky they get it
| working correctly.
| mikevm wrote:
| Let's call them "code technicians" instead of engineers, ok?
| (that's a euphemism for "code monkeys")
| lucideer wrote:
| > _It is absolutely unbelievable [...] that a cash cow [...]
| has a problem like this_
|
| Likely it wasn't fixed precisely because it's such a cash cow.
| "It's making money, don't fuck with it".
| hibikir wrote:
| It's not just believable, but it's normal. I have spent quite a
| bit of my career maintaining software, and I don't recall one
| employers where low hanging fruit like this wasn't available
| everywhere.
|
| The problem is not that developers can't optimize things: you
| will find some developers capable of figuring this problem out
| anywhere. What makes this low hanging fruit so popular is the
| fact that we aren't measuring enough, and even when we do, we
| aren't necessarily prioritizing looking into things that are
| suspiciously slow.
|
| In the case of this example, the issue is also client-side, so
| it's not as if it's costing CPU time to Rockstar, so it's
| unlikely you'll have someone who can claim their job
| description includes wondering if the load times are worth
| optimizing. When problems like this one get solved is because
| someone who is very annoyed by the problem and either convinces
| a developer to even look into the problem. Most of the time,
| the people that suffer, the people that get to decide how to
| allocate the time, and the people that are in a position to
| evaluate the root cause of the problem never even get to talk
| to each other. It's the price we often pay for specialization
| and organizations with poor communication.
|
| Organizations where the people deciding what has to be done
| next, and where the company culture dictates that the way
| forward is to either complete more tickets faster, or find ways
| to be liked by your manager, are not going to be fostering the
| kind of thinking that solves a problem like this one, but
| that's a lot of what you find in many places. A developer with
| a full plate that is just working on the next feature isn't
| going to spend their time wondering about load times.
|
| But instead we end up blaming the developers themselves,
| instead of the culture that they swim in.
| smaudet wrote:
| Hear hear, we should make a punch a dummy manager/BA/Code
| standards 'lead' day...
|
| This code looks like someone with almost no experience hacked
| it together but because they were an intern and likely
| Rockstar is a toxic place to work, it never gets prioritized
| to be fixed.
|
| I think if managers prioritized cycle time, metri s more,
| they'd find that they are encouraging a lot of practices
| which lead to horrible efficiencies - "measure twice cut
| once" is a positive mantra which leads to more solid designs
| with less bugs.
|
| Agile sort of addressed this problem but unfortunately only
| at small size scales. Iteration and story capacity got
| overprioritized over quality, customer engagement, and self-
| leading teams.
|
| Plus things such as scaled agile suffer from the oxymoron of
| planning fast iteration - if you have a master plan you lose
| the ability to respond to change and iterate, or you allow
| iteration and you must accept if any team iterates the whole
| team must discard the plan...which at some point means you
| either accept high cycle times or you figure out a way to
| decouple functionality to the extent the planning becomes the
| standard fallacy of waterfall - wasting meeting time to go
| over a plan that isn't based on anything.
| moonchild wrote:
| > salty because their parent company unjustly DMCA'd re3
|
| Unjustly, but legally. The people you should be salty at are
| the lawmakers.
| bscphil wrote:
| Why can't I be salty at both? People have responsibility for
| their actions even if those actions are legal.
| breakingcups wrote:
| That still remains to be seen. A DMCA is not a court order.
| Anyone can file one and take a repository offline for two
| weeks.
| moonchild wrote:
| Yes but the code was clearly derived directly from a
| decompiled binary; not 'clean room' reverse engineering.
| Hence, illegal, regardless of whether a dmca takedown
| notice is filed.
| xyzzy_plugh wrote:
| That doesn't make it illegal, there are plenty of
| jurisdictions where non-clean-room reverse engineering is
| perfectly legal.
| charcircuit wrote:
| But distributing it without a license from the copyright
| holder is.
| jagger27 wrote:
| Distributing what, exactly? It required you already owned
| a copy of the game to install/build it.
| charcircuit wrote:
| Distributing whatever was in the repo. Requiring a copy
| of the game doesn't magically make it legal. A modded
| game is a derivative work.
| nl wrote:
| Deriving something from a decompiled binary isn't illegal
| in itself.
| ThatPlayer wrote:
| That still leaves it as a derivative work, which is
| protected from copying and distribution by copyright.
| Scoundreller wrote:
| > obfuscated executable loaded with anti-cheat measures
|
| I'm impressed that gamecopyworld.com is still online, updated,
| and has the same UI that it did in 2003
| lethologica wrote:
| Whoa, what a (refreshing) blast from the past.
| uhhhhhhhhhhhhhh wrote:
| Can you quantify the additional profit they would have made if
| this were fixed N years ago?
| oh_sigh wrote:
| Maybe long load times are advantageous? Because it creates
| longer user sessions on average? If you devote 10 minutes to
| loading the game you will probably want to play for at least 30
| minutes.
| erik_seaberg wrote:
| Wouldn't the most impatient customers be more likely to pay
| for items, rather than earn them in game?
| thrwyoilarticle wrote:
| The popular view is that companies who write software know how
| to prioritise, so if a problem like this isn't fixed, it's
| because they've done the calculations and decided it's not
| worthwhile.
|
| I disagree. If there are no internal incentives for the people
| who know how to fix this to fix it, or if there's no path from
| them thinking fixing it could improve revenues to being
| assigned the ticket, things like this won't get fixed. I can
| fully believe the load times will result in fewer users and
| lower expenditure.
|
| I think we'll see this happen with Facebook Messenger. Both the
| apps and the website have become slow and painful to use and
| get worse every month. I think we'll start to see engagement
| numbers dropping because of this.
| mywittyname wrote:
| I would think this would be one of the biggest revenue /
| developer_time changes in company history, considering how
| incredibly profitable online is.
| rsj_hn wrote:
| That the process breaks down in some cases doesn't mean they
| don't know how to prioritize. They clearly know how to
| prioritize well enough to make a wildly successful and
| enjoyable game. That doesn't mean no bad decisions were made
| over the last decade or so of development.
|
| Like anything else, things will be as bad as the market
| allows. So I'd expect monopolies to do a worse and worse job
| of making good decisions and companies in competitive fields
| to do a better and better job over time. Thus the difference
| between TakeTwo and Facebook, and the need for lower cost of
| entry and greater competition in all economic endaevors where
| efficiency or good decision making is important.
| AshWolfy wrote:
| Does it? Just because something is good or successful
| doesnt mean it was made well. Thats why we are seeing
| stories of crunch, failures of management compensated by
| extreme overwork.
| rsj_hn wrote:
| Now you are moving the goalposts to a philosophical
| discussion as to what types of business you like or don't
| like, and I'm not sure any progress can be made with that
| approach. Some will share your values, others will find
| them repugnant, and in any case it doesn't have much
| bearing on rockstar load times.
| tw04 wrote:
| You have just described why I laugh anytime someone complains
| that government is inefficient. ANY organization of
| sufficient size is "inefficient" because what a large
| organization optimizes for (for reasons I cannot explain)
| cannot align with what that organization's customers want
| optimized.
| buran77 wrote:
| With the added difference that governments also _have_ to
| be far more procedural by virtue of the way they are set
| up. Regardless of size they are accountable and responsible
| to a far higher degree in the eyes of the population they
| represent so there is a legitimate reason to be "slow".
|
| In games the added reason to be slow is that game code is
| by definition some of the least mission critical code one
| could find (competes with 90% of the internet web code).
| Your Linux or Windows code might run a hospital's
| infrastructure or a rover on another planet. A game on the
| other hand can launch with bugs the size of your
| windshield, and can stay like that forever as long as
| people still pay. And people will pay because games are not
| unlike a drug for many people.
|
| As such most game coding teams and coders are "trained" to
| cut every corner and skimp on every precaution. They're not
| needed beyond a very low baseline as far as software is
| concerned.
|
| Look at the amount of bugs or cheats incredibly popular
| games like GTA or CoD have. These are billion dollar a year
| franchises that leave all this crap on the table despite
| all the money they make. They have all the resources
| needed, it's a conscious call to proceed like this, to hire
| teams that will never be qualified enough to deliver a high
| quality product and will be encouraged to cut corners on
| top of that.
|
| Source: a long time ago I worked for a major game developer
| in a senior management role (unrelated to the dev activity)
| and left after feeling like "facepalm" for too long in
| every single SM meeting.
| com2kid wrote:
| Remember this story from a few days ago?
|
| https://randomascii.wordpress.com/2021/02/16/arranging-
| invis...
|
| I've seen plenty of interesting bugs, best I found
| personally was a compiler that was outputting files 1
| byte at a time.
|
| Games are riff with these sorts of bugs, but the volume
| of released games vs other types of software makes any
| sort of comparison unfair.
| toyg wrote:
| _> for reasons I cannot explain_
|
| Any sufficiently large institution, over time, will
| prioritise self-preservation over achieving their core
| mission. This is sociology 101. Once a company has enough
| users to make it hard or impossible to measure immediate
| performance, self-preservation is achieved with internal
| manoeuvering and selling to execs.
| jack_riminton wrote:
| Whats the remedy? apart from reducing the size of the
| organisation
| tehjoker wrote:
| It's not a perfect remedy, but you have to loop in the
| people affected by decisions as part of the decision
| making structure. That is, for example, customers and
| workers have to be part of the management structure.
|
| This doesn't happen because it would reduce the power of
| top decision makers and potentially impact profits. e.g.
| a customer might ask for a chronologically ordered
| timeline on Facebook, but that would harsh engagement
| metrics, revenue, etc. If stuff like this did happen more
| often though, you'd get products and services that more
| often achieve their stated aims.
| ayewo wrote:
| There was a fantastic discussion some years ago on ways
| to design an organization to minimize the tendency to
| drift towards self-preservation instead of remaining
| customer-focused.
|
| The HN discussion[1] was started by an article that
| provided numbers that seemed to suggest that Wikipedia's
| spending was slowly spiraling out of control.
|
| 1: https://news.ycombinator.com/item?id=14287235
| SilasX wrote:
| Competition/choice, which means that self-preservation
| _requires_ that they care about efficiency. It obviously
| that wasn 't enough here, but definitely tames some
| inefficiencies.
| pdimitar wrote:
| Many people would say "more accountability" but I've seen
| that used successfully to deflect lightning strikes to
| innocent people who were then fired so... I'd like to
| know as well.
| buran77 wrote:
| Even if you take engineering you see that adding more
| links on a chain increases latency and demands more
| auxiliary circuitry. But at least in engineering you can
| design each part to do what you want it to do close to
| perfection. And it will scale better because you can
| build everything to spec and bin, which is why we
| automate a lot of tasks. With humans that can't happen.
| Human "binning" is a constantly moving target.
|
| After tangentially working on this for a long time I'd
| say that the core issue is so deeply ingrained in human
| psyche that it may not even be a matter of education
| (starts early), let alone organization (starts happening
| when everything else is "set in stone"). There's no
| organizational structure that fits the types of
| activities we humans tend to do these days and that can
| deliver low latency, consistent results at scale. We
| mitigate issues in one area by accentuating them in
| others.
|
| You can have one large flat structure but the load on the
| single coordinating circuit (the manager) will compromise
| quality. You can split the thing in multiple,
| individually coordinated units but the added layer of
| coordination and latency will compromise performance.
|
| Maybe some form of general purpose but not quite full AI,
| something that combines human like intelligence and
| engineering like consistency, might be able to do what
| humans are supposed to but without the variability of
| humans (which is both good and bad).
| toyg wrote:
| Introducing AI into work relations is how you turn every
| org into Uber or Amazon delivery: platforms where the
| worker has no real agency on his work, the apotheosis of
| alienation. I have no doubt that someone will try it
| (already we see it creeping in for hiring), I just think
| it will be Fundamentally Bad.
| buran77 wrote:
| > Introducing AI into work relations is
|
| We don't really know what it is simply because we haven't
| introduced any "real" AI anywhere, let alone:
|
| >> some form of general purpose but not quite full AI
|
| Talking about efficiency as you scale up large
| organizations, it's inevitable that humans will introduce
| delays and variability in the work which cannot be
| eliminated because it's human nature, biologically and
| psychologically. Since humans can't change on a timescale
| that makes this discussion relevant, the only way for
| very large organizations (like large companies or
| governments) to operate just as efficiently as small ones
| is if they rely (quasi)exclusively on some AI that's as
| capable as top individuals at delivering results but with
| fewer of the drawbacks. Not only would it not operate as
| 100.000 distinct entities but as one or a few, it would
| also run consistently and predictably.
| donkeyd wrote:
| I doubt there is a remedy and personally accept this as a
| fact of life.
| [deleted]
| pcr910303 wrote:
| > I think we'll see this happen with Facebook Messenger. Both
| the apps and the website have become slow and painful to use
| and get worse every month. I think we'll start to see
| engagement numbers dropping because of this.
|
| In fact, I think the iOS app for FB Messenger did get a
| redesign due to problems and it's rewritten from scratch? I
| remember being pleasantly surprised after the big update...
| It became lightweight, integrates well with iOS and supports
| platform features.
|
| On the other hand, the desktop app or the website is a
| shitshow :-(
| Tsiklon wrote:
| The iOS app is really much more performant now than it was
| a couple of years ago. Significantly smaller, and quicker
| to start up
| alkonaut wrote:
| > if a problem like this isn't fixed, it's because they've
| done the calculations and decided it's not worthwhile.
|
| If it ever reached the point where it had to be an item in a
| priority list, it's already a failure. Some developer should
| have seen the quadratic behavior and fixed it. It's not the
| type of thing that should ever even be part of a prioritized
| backlog. It's a showstopper bug and it's visible for every
| developer.
| saghm wrote:
| > I think we'll see this happen with Facebook Messenger. Both
| the apps and the website have become slow and painful to use
| and get worse every month.
|
| The messenger website has been atrocious for me lately. On my
| high-powered desktop, it often lags a bit, and on my fairly
| high-end laptop, it's virtually unusable. I thought it must
| be something I changed in my setup, but it's oddly comforting
| to hear that I'm not the only one with such issues.
| ianlevesque wrote:
| Try https://www.messenger.com/desktop
| lethologica wrote:
| [tinfoil hat] A part of me believes that this was their
| intention in slowing down their browser site to the
| extent it has (and there is absolutely zero reason as to
| why it should load as slow as it does) in order to drive
| downloads for this application. [/tinfoil hat]
|
| It's funny. That page says "A simple app that lets you
| text, video chat and stay close to people you care
| about." If it's so simple, then why does the browser
| version of the site take forever to load?
| gocartStatue wrote:
| Yeah, it used to work well in mobile browser, then only
| in desktop mode in mobile browser, then not at all. Now
| it only "kinda sorta" works in desktop browser on
| ridiculously overpowered PC. Gotta keep the walls tight
| in that private garden.
| saghm wrote:
| Unfortunately I'm on Linux, and they don't seem to
| provide a Linux client. I did try out
| https://github.com/sindresorhus/caprine/, which worked
| really well, but I'm one of those weirdos who prefers
| websites to desktop apps.
| baby wrote:
| For me it's google maps, it has gotten so freaking slow
| both on mobile and on desktop. Actually google docs and
| sheet are the same.
| amluto wrote:
| Gmail. Typing in gmail is painfully laggy.
| stefs wrote:
| 1. people experiencing this issue have already bought the
| game, so there's little incentive here.
|
| 2. we can be reasonably sure people will buy newer GTA
| installments regardless of whether this bug is fixed or not.
|
| but:
|
| 3. if there's still money to be made from microtransactions
| this is a huge issue and would absolutely be worthwhile, imo.
| karmasimida wrote:
| > I can fully believe the load times will result in fewer
| users and lower expenditure.
|
| Is GTA online still attracts new users in droves? I doubt.
|
| If the old users live with the loading time for years, they
| are likely to continue living with it. It would be nice if
| Rockstar fixes it, but I doubt it would be anything except a
| PR win.
| thrwyoilarticle wrote:
| Before GTA online they entertained themselves in other
| ways. Eventually, they'll move on. The more friction there
| is to continue playing GTA online, the easier it is for
| there to be something to pull them away. Rockstar are now
| competing to be a use of someone's time, not for them to
| buy the game.
| formerly_proven wrote:
| Also: Rockstar being too cheap to implement anti-cheat on the
| _by far_ most successful online shooter on the planet.
|
| Also
|
| > I don't think there's any easier way out.
|
| lmfao
| buzzerbetrayed wrote:
| This may be obvious, but is GTAV the most successful online
| shooter on the planet? (Never played it)
| kingosticks wrote:
| I don't think it's obvious. In terms of player count (which
| is how I would personally rank success), it is not the most
| successful by a long way:
| https://en.wikipedia.org/wiki/List_of_most-
| played_video_game...
|
| However, games like PUBG and Fortnite are free-to-play
| (PUBG is only free on mobile?) so in terms of actual
| _sales_ , you could say GTA is more successful. Still not
| sure I'd class it as an "online shooter", though.
| formerly_proven wrote:
| GTA is many things for many different people. For some
| people, it's a racing game, for others it is about
| socializing and modding cars, for some it's a dogfighting
| game, some spend thousands of hours grinding the same set
| of PvE missions (for some reason), and for many it's a
| third and first person shooter / warzone simulator.
| kingosticks wrote:
| I totally agree, the same way many games are. But in the
| context of saying "x is by far most successful y on the
| planet" I think we have to stop somewhere. Otherwise an
| open game like GTA V becomes the most successful game (in
| terms of copies sold) in pretty much every category you
| can't reasonably stick Minecraft in. I don't think
| there's much value in that.
| sickofparadox wrote:
| Its the second best selling game of all time[1], and
| because Minecraft doesn't have guns, I suppose that would
| qualify it as the most successful online shooter (even if I
| think another genre would be more applicable).
|
| [1]https://en.wikipedia.org/wiki/List_of_best-
| selling_video_gam...
| charcircuit wrote:
| That list is incomplete. CSGO is the most played game on
| Steam and is now free to play. I would not be surprised
| in the slightest if more people have CSGO.
| RugnirViking wrote:
| interesting. Steamspy no longer works, but its last
| report in 2016 said it had 25 million sales (at the time
| it was roughly on par with minecraft sales). Since then,
| the concurrent player count has more than doubled, but
| its very difficult to get information about sales.
| kingosticks wrote:
| Wikipedia has it with 46 million owners on Steam, with
| the following footnote:
|
| > Not official; the numbers were estimated following a
| Steam API data leak in July 2018, which gave player
| ownership estimates for games on the platform that have
| achievements.
| gambiting wrote:
| Well but that's a bit like saying that since there are
| technically races in GTA it's also the most successful
| racing game. It will also be the most successful flying
| game and the most successful sailing simulator....etc
| etc.
|
| I don't know, even though there is shooting in GTA I
| don't think I'd call it a shooter.
| sickofparadox wrote:
| I agree that there is probably some other genre that
| would describe it better, but I think it wouldn't be
| unfair by any means to describe GTA as a third-person
| shooter.
| omgJustTest wrote:
| Focus is on the money, not on the technicals, on the production
| side. It is a game, to entertain and ultimately "waste time" on
| the consumer side. Also on the topic of parsing, +1-11gbyte/sec
| is possible if you go binary. Point is: this isn't a technical
| problem, it's a choice.
| grishka wrote:
| > the reverse engineered version of GTA III and Vice City
|
| Ohhh. Thank you for telling me about this. I just found a
| mirror and successfully built it for macOS. Runs so much better
| than the wine version. But I guess I'll never finish that RC
| helicopter mission anyway lol
| matheusmoreira wrote:
| I used to play this game a lot on PS4. I actually dropped it
| due to the ridiculous loading times... I still assumed it was
| doing something useful though. I can't believe they wasted so
| much of my time and electricity because of this. Even cheaply-
| made mobile games don't have bugs like this.
|
| > their parent company unjustly DMCA'd re3
|
| Wow, this is EA games level scumbaggery... I don't think I'm
| gonna buy games from them again.
| zionic wrote:
| I stopped playing GTAV online a few years back because of the
| crazy load times, not only that but you have to go through 6+
| minute load screens multiple times in many sessions.
|
| This oversight has cost them millions of dollars easy.
| wyaeld wrote:
| If it made over $1b in a year previously, and had such insane
| load times, its very plausible this bad coding has cost them
| north of another $1b.
|
| Probably ranks pretty highly up there in terms of damage to
| company financials, due to a lack of care.
| anoncake wrote:
| I find it absolutely believable that a for-profit company does
| not prioritize fixing a game that is already a cash cow anyway.
| pilif wrote:
| _> This online gamemode alone made $1 billion in 2017 alone._
|
| which of course goes to show that at least from a business
| side, this issue is completely inconsequential and all
| resources should be used to push for more monetization (and
| thus adding to the problem by adding more items to the JSON
| file) rather than fixing this issue, because, clearly, people
| don't seem to mind 6 minutes loading time.
|
| I'm being snarky here, yes, but honestly: once you make $1
| billion per year with that issue present, do you really think
| this issue matters at all in reality? Do you think they could
| make $1+n billion a year with this fixed?
| hderms wrote:
| The bigger the scale the bigger a few percentage point
| improvement would be worth. I would generally think if you're
| at 1bn in revenue you should devote 1%+ percentage points of
| your workforce towards finding low hanging fruit like this.
| If 1% of employees deployed to find issues that, when fixed,
| yield 2% improvement in revenue thats likely a winning
| scenario
| Ansil849 wrote:
| > It is absolutely unbelievable (and unforgivable) that a cash
| cow such as GTA V has a problem like this present for over 6
| years and it turns out to be something so absolutely simple.
|
| It is both believable and - by virtue of the fact that, as you
| said, the series continues to be a cash cow - is apparently
| forgivable.
|
| Here's the thing: the company has zero reasons to fix this, or
| other ostensibly egregious affronts like DRM, because gamers
| keep buying the product. There is literally no economic
| incentive to 'fix' it.
| _AzMoo wrote:
| How many players have they lost to excessive load times?
| astrange wrote:
| I've lost interest in a lot of online games because the
| clients always do their mandatory updates on launch, which
| is exactly the time you want to play the game.
|
| (Same thing for websites - they show you all the annoying
| popup signup sheets/survey questions the instant you load
| the page.)
| _AzMoo wrote:
| For sure. I fit my gaming into about 2 hours a week. If
| it takes 6 minutes to load there's no way I'm going to
| play it. I know I'm not the target market but I'm still a
| missed opportunity.
| Ansil849 wrote:
| > How many players have they lost to excessive load times?
|
| Judging by the number of successful sequels the franchise
| has spawned, the answer is 'an insignificant number'.
|
| The fact of the matter, and the point my comment was trying
| to make, is that the overwhelming majority of players do
| not sufficiently care about load times or other complaints
| to deter them from paying for the game. That is the reality
| of the situation.
| simias wrote:
| That doesn't make sense, saying that the game still
| managed to be profitable doesn't show that the loss was
| insignificant. Those loading times are really annoying
| and detract from the game, and although I can't quantify
| the loss either I'd be very surprised if it was
| insignificant.
|
| I'd say especially since the ones who are most likely to
| be affected by these issues are working adults with
| limited playtime who won't want to sit in front of their
| monitor for 5 minutes waiting for the game to load and
| also happen to be people with disposable income to pour
| into a game.
| 0xy wrote:
| Something I've noticed in highly successful companies is that
| problems never get fixed because the sound of the money printer
| from the core business is deafening.
|
| Our customer portal loads 2 versions of React, Angular,
| Knockout and jQuery on the same page? Doesn't matter, it's
| printing billions of dollars.
|
| Rockstar's money printer is so loud that they don't care about
| problems.
|
| Same thing for Valve, their money printer is so loud that they
| barely bother to make games anymore and let the Steam client
| languish for years (how did they let Discord/Twitch happen?).
| tester756 wrote:
| What does Valve / Steam have to Discord?
| pityJuke wrote:
| In addition to other comments, Steam Chat had significant
| in-roads with the gaming audience that would eventually
| form the foundation of Discord. It is quite plausible, had
| Steam improved chat earlier, that Discord might have never
| gotten the traction it got.
|
| Nowadays, I find Steam Chat is a ghost town.
| meibo wrote:
| Valve only recently implemented a semi Discord-clone(with
| way better quality voice chat, give it a try some time if
| you haven't yet).
|
| Their chat system has been famously bad and mostly
| unchanged since the early 2010's, and only very recently
| was reworked into this.
| tester756 wrote:
| >(with way better quality voice chat, give it a try some
| time if you haven't yet).
|
| Well, I'm using VoIPs basically for >decade everyday and
| voice quality was never something I cared about (I mean
| that all soft that I've used was somewhat decent in that
| matter)
|
| the most important thing is - how to get all people on
| the same program? and I'm finding it not so realistic to
| get all friends to Steam
| charcircuit wrote:
| I think the parent was implying that steam groups could
| have improved to the point where Discord would not be
| necessary.
| LukvonStrom wrote:
| Look at sharepoint. It's a total nightmare of a platform to
| develop on, but the people just adapted and built their
| businesses on it
| passivate wrote:
| What makes it a nightmare? Can you share a few issues
| you've run into?
| nl wrote:
| I've never seen search work properly (where "properly" =
| 90+% of searches return desired document in first 2
| results)
| passivate wrote:
| I see. We have some internal facing stuff that our IT has
| done using sharepoint. Its not pretty/modern by any
| stretch, but it works just fine. Was curious what the
| world looks like from the other side.
|
| Isn't the search issue more of a indexing engine problem
| though? Can you plug in other engines?
| Sohcahtoa82 wrote:
| In my experience, Sharepoint is stupidly slow, and it
| feels like it's trying to be everything.
|
| Like...I don't need it to notify me that I have new
| e-mail. I already have either Outlook or an Outlook tab
| running.
|
| Sharepoint is a shining example of feature creep run
| wild.
| the_only_law wrote:
| I still have no idea what Sharepoint even is. The way I've
| always seen it used is a way to host sites with file
| hosting tied to them. It feels like an over engineered CMS.
| ip26 wrote:
| I suspect sharepoint is the platform version of excel &
| VBA. You & I might hate it, but it gets powerful
| capabilities into the hands of regular people.
|
| sharepoint is probably many non-engineer's very first
| exposure to actual version control with checkouts,
| checkins, version history, and merge.
| ant6n wrote:
| Sharepoint has version control? I thought it's mostly a
| poorly functioning Dropbox copy.
| easton wrote:
| This comment is absolutely hilarious to me, because the
| main selling point for SharePoint (for the longest time)
| was that you could version control Microsoft Office
| documents. Then intranets happened, someone wanted to
| build a Turing machine with it, and now it's the
| monstrosity we know now.
| Nexxxeh wrote:
| >Same thing for Valve, their money printer is so loud that
| they barely bother to make games anymore and let the Steam
| client languish for years (how did they let Discord/Twitch
| happen?).
|
| Not sure that's a fair criticism.
|
| Alyx was widely praised. Artifact... Wasn't. I don't know
| about Dota Overlords. And that's just the last couple of
| years.
|
| They've also developed hardware like SteamLink, Steam
| Controller, some high-end VR gear...
|
| They develop a LOT. They just don't release a while lot.
|
| I agree there should be a lot more work and effort in the
| client. And they constantly fuck up handling their esports.
|
| But I don't think "barely bother to make games anymore" isn't
| one of them.
| djmips wrote:
| It's fair. They are not a games developer anymore. Alyx was
| good but a boutique game made by a behemoth games
| marketplace company and before it came out, almost a decade
| had passed since Portal 2...
| woobilicious wrote:
| Valve released Dota 2, and then ported it to a brand new
| engine, they've been consistently adding updates to that
| game for 10 years, they've been working on CS:GO, with
| similar effect, I think you lot forget that the majority
| of Valves developers were working primarily on team-
| fortress and counter-strike as mods before they hired
| them, they're primarily a multiplayer house.
|
| Valve unlike most companies maintains their games for
| more than 10 years.
|
| Just because they haven't released anything obvious to
| the casual observer like new single player titles that is
| easily marketed is only showing your ignorance of Valves
| entire catalog of games, and attitude to development.
| Scaevolus wrote:
| I suspect that the core engine programmers moved onto other
| projects long ago, leaving GTA:O running with mostly artists
| and scenario designers to produce more DLC.
|
| This bug wouldn't present in the first couple years with the
| limited amount of DLC, so by the time it got ridiculous there
| wasn't anyone left with the confidence to profile the game and
| optimize it. A junior dev could fix this, but would probably
| assume that slow loads are a deep complex engine problem that
| they won't be able to fix.
|
| Alternatively, management would declare that there's too much
| risk doing more technical engine work, and not sign off on any
| proposed "minor" optimizations because it's too risky.
| jorams wrote:
| > This bug wouldn't present in the first couple years with
| the limited amount of DLC
|
| GTA Online loading times have been infamous for a _very_ long
| time. They were already unjustifiably bad when they released
| the game for PC, and at that point engine programmers would
| surely be involved.
| Aeolun wrote:
| The 1.5 minute default load time is also ridiculous.
| anotherfish wrote:
| This is very much the likely scenario. The money is in
| further DLC. The existing GTAO engine is "done" from their
| perspective.
|
| I'd guess also that the next version of the base engine is in
| RDR2 or later and doesn't have these issues. But at the same
| time they likely wouldn't backport the changes for fear of
| cost overruns.
| umanwizard wrote:
| "worth their salt" is doing a lot of work here. No true
| Scotsman fallacy?
|
| I think you might be surprised by how few programmers even know
| what a profiler is, let alone how to run one.
| thrwyoilarticle wrote:
| That seems like a misapplication of the fallacy. If we assume
| 'worth their salt' is a synonym for 'good', then saying any
| good developer can operate a profiler is entirely reasonable.
| xwolfi wrote:
| I work in a large multi billion company and we have people
| staring at a slow problem for a decade before a noob comes with
| a profiler and find they browse every key of a Map instead of
| calling get and such. Or do 1 million db queries on a GUI
| startup...
|
| Not surprised they didn't bother for 6 minutes when it takes us
| 10 years to fix a 30minutes locked startup.
| randomNumber7 wrote:
| > It is absolutely unbelievable (and unforgivable) that a cash
| cow such as GTA V has a problem like this present for over 6
| years
|
| Agree. I found the slow behavior of sscanf while writing one of
| my first C programs during an internship^^ You literally just
| have to google "scanf slow" and find lots of information.
| karmasimida wrote:
| It could be that at the time GTA online first publishes, the
| list as hashmap isn't too much of an issue, due to limited
| catalog, but get progressively worse as the inventory grows.
|
| Ofc this is just a hypothesis, but I see the hesitation to
| change legacy code if it ain't broken as a wide spread
| mentality.
| sellyme wrote:
| > I see the hesitation to change legacy code if it ain't
| broken as a wide spread mentality.
|
| Load times measured in double digit minutes on a significant
| number of machines meets absolutely every reasonable
| definition of "broken".
| AnonsLadder wrote:
| Does anyone have a link to a copy of re3? Iirc, there was a
| gitrepo that kept a copy of all DMCA'd repos
| djmips wrote:
| try the hacker news search (bottom of the page) and you'll
| find stories on the takedown where there are links to backups
| posted in the comments.
| xyst wrote:
| is it really unbelievable? companies this big tend to
| prioritize hiring a shit ton of middlemen (VPs, project
| managers, developer managers, offshore managers) in order to
| avoid paying out for talent to build and constantly maintain
| the project. I guess paying a shit ton of money to 1 person to
| manage 10+ poorly paid contractors works out for them,
| accounting wise.
|
| If one really examined the accounting for GTAO, I would bet
| that most of the billions of dollars that were earned in micro
| transactions went to marketing, product research, and to middle
| management in the form of bonuses.
| spuz wrote:
| Even if you view this as a business decision rather than a
| technical one, any smart project manager would realise a 6
| minute loading time literally costs the company millions per
| year in lost revenue. (How many times have you felt like
| firing up GTA Online only to reconsider due to the agonising
| load time). I would guess this was simply a case of business
| folk failing to understand that such a technical issue could
| be so easily solved plus developers never being allowed the
| opportunity to understand and fix the issue in their spare
| time.
| somehnguy wrote:
| The insane loading times are literally the exact reason I
| haven't played in years. Every time I played I just ended
| up frustrated and got distracted doing something else while
| waiting, so I just quit playing altogether. I don't know
| how people stand the loading times.
| hnick wrote:
| Maybe the load times inadvertently work like the
| intentional spelling mistakes in a Nigerian scammer's
| email.
|
| It's bait for the non-discerning customer who is more
| likely to empty their wallet for microtransactions
| because they have less experience with games so don't
| know what is normal :)
| jamiek88 wrote:
| The fact that this scenario is not immediately ludicrous
| to me is saddening.
| jonwinstanley wrote:
| Totally agree, I loved the story mode and never got into
| online due to the amount of time spent loading and
| finding interesting stuff to do.
| alien_ wrote:
| The people who observe the slow loading time already paid
| for the game, so I guess R* won't lose much revenue because
| of this nasty bug.
| the_gipsy wrote:
| IIRC R* makes orders of magnitudes more from
| microtransactions than from the game box
| bcrosby95 wrote:
| The game has microtransactions. Coincidentally, also a
| large reason why load times were so slow.
| [deleted]
| delecti wrote:
| The online mode has microtransactions. People not playing
| anymore aren't paying for microtransactions either.
| [deleted]
| [deleted]
| AuryGlenz wrote:
| I didn't buy the game to play with my friends because I
| heard of how terrible the loading situation was.
| IshKebab wrote:
| It's kind of hard to believe. GTA5's online mode is their
| cash cow, and _6 minute load times_ are common?! It 's kind
| of amazing people even play it with those load times. It's
| such a simple problem that one dev could have found and fixed
| it within a day.
| toast0 wrote:
| It's not at all hard to believe if you've been playing
| video games for a while.
|
| Everything is getting slower and slower, and nobody cares.
|
| When I played the Atari 2600, I had to wait for the TV to
| warm up, but otherwise there were no games with anything
| approaching load times (with 128 bytes of RAM in the
| console, who would know). The NES didn't have much in the
| way of load times either, but you did have to fiddle with
| the cartridge slot. SNES and Genesis didn't usually load
| (Air Buster being a noticeable exception). CD based systems
| sure did like to load, but that's somewhat understandable.
| In the mean time, more and longer boot screens. The Switch
| uses a cartridge (or system flash/SD cards), but it likes
| to load forever too.
|
| PC Gaming has had loading for longer, but it's been getting
| longer and longer.
|
| Some arcade games have lengthy loading sequences, but only
| when you turn them on while they read their entire storage
| into RAM so they can be fast for the rest of the time
| they're on (which in arcades is usually all day).
| yoz-y wrote:
| > Everything is getting slower and slower, and nobody
| cares.
|
| It really depends. The latest crop of games I've played
| (Doom Eternal, Cyberpunk) loads way faster than games
| from a few years back (aforementioned GTA-V, Shadow
| Warrior 2...).
|
| This is also on the same machine, so it's not the
| hardware that makes it faster.
| hnick wrote:
| Doom Eternal's load times were so good I didn't even
| bother moving it to my SSD (I junction to a larger HDD by
| default).
| ubercow13 wrote:
| Shorter loading times were one of the main selling points
| of this console generation.
| iknowstuff wrote:
| Direct Storage will allow for hardware-accelerated
| decompression straight from an NVMe into GPU memory,
| without involving the CPU and system RAM.
|
| https://devblogs.microsoft.com/directx/directstorage-is-
| comi...
| wtallis wrote:
| The more important thing about DirectStorage is probably
| that it will encourage games to use multithreaded async
| IO rather than serializing all their IO requests even
| when the underlying storage device requires dozens of
| simultaneous requests to deliver its full throughput.
| astrange wrote:
| Does it need "multithreaded async IO" or just "async IO"?
| It's usually async _or_ multithreaded; the native multi-
| request I/O APIs are single threaded, but if you have
| multithreaded I/O using simpler APIs, the system is
| batching them into one request at the cost of a little
| latency.
| johncolanduoni wrote:
| Kernel-mediated async disk IO is still a mess on all
| major platforms except for newer Linux kernels with
| io_uring. There's no way to call the APIs in a way that
| won't block sometimes, and to even have a snowball's
| chance in hell of not blocking requires giving up on the
| kernel's disk cache.
|
| Also you're probably going to want to do multithreaded
| decompression anyway, and it'll be more efficient if you
| have the threads completing the reads do the
| decompression themselves. So in any case you probably
| want multiple threads handling the completion events.
| wtallis wrote:
| NVMe is natively a multi-queue storage protocol, so
| there's no reason for the application or OS to collect IO
| requests into a single thread before issuing them to the
| lower layers. The normal configuration is for each CPU
| core to be allocated its own IO queue for the drive. But
| multithreaded synchronous (blocking) IO often isn't
| enough to keep a high-end NVMe SSD properly busy; you run
| out of cores and get bogged down in context switching
| overhead at a few hundred thousand IOPS even with a
| storage benchmark program that doesn't need any CPU time
| left over for productive work.
|
| With a sufficiently low overhead async API (ie. io_uring)
| you _can_ saturate a very fast SSD with a single thread,
| but I 'm not sure it would actually make sense for a game
| engine to do this when it could just as easily have
| multiple threads independently performing IO with most of
| it requiring no synchronization between cores/threads.
| amluto wrote:
| I'm not entirely convinced that DirectStorage _can_ do
| DMA directly from the device to the GPU. I suspect that
| even current NVMe devices aren't quite fast enough for
| this to be a huge deal yet.
|
| I think, but I'm not entirely sure, that Linux can do the
| peer to peer DMA trick. One nasty bit on any OS is that,
| if a portion of the data being read is cached, then some
| bookkeeping is needed to maintain coherence, and this
| adds overhead to IO. I wouldn't be surprised if consoles
| had a specific hack to avoid this overhead for read-only
| game asset volumes.
| smaudet wrote:
| Lol
|
| "Hey we have this great new tech that makes things even
| faaster!!"
|
| 2 years later
|
| "GTA 6 found to have double online load times, denies
| claims that game performs worse than GTA 5, tells people
| to upgrade their systems"
|
| 4 years later:
|
| "Tech blogger reverses code, realizes someone managed to
| loop access between hard drive and gpu despite extremely
| common modern tech, gets 10x boost after spending a day
| fixing junk product"
|
| Better technology just hasn't met its match from dumber
| management and more bureaucratic dev shops...
| coredog64 wrote:
| There were a few systems for the 2600 that used a
| cassette tape to load larger games than would fit on a
| ROM cassette.
|
| I can't recall the name, but I had the hack and slash
| adventure game variant. The connector on the custom
| cartridge was fiddly and required a stout rubber band to
| reliably work.
| xyzelement wrote:
| I am always amused by comments like this. You have no idea
| what development practices they follow (neither do I) but
| it's hilarious to read your tone.
|
| GTA has achieved tremendous success both as an entertaining
| game and as a business. It's enjoyed by millions of people
| and generates billions in revenue. As per this article, it
| has startup problems (which don't seem to actually really
| hurt the overall product but I agree sound annoying) but the
| bigger picture is: it's a huge success.
|
| So - Rockstar has nailed it. What exactly is your platform
| for analyzing/criticizing their processes or even having a
| shot of understanding what they are? What have you build that
| anyone uses? (not saying you haven't, but.. have you been
| involved with anything remotely close to that scale?)
|
| And if not, whence he high horse?
| Aeolun wrote:
| I don't need to be successful to have a platform to be
| outraged. It doesn't matter that it's Rockstar, if
| anything, the fact that they're so successful and couldn't
| be bothered to save so many people literal hours of their
| lives in loading time makes it worse.
| saagarjha wrote:
| You can be right in many places and still wrong in some,
| and enjoy enormous success as a result of all you have done
| well. That does not mean nobody can criticize you for
| something that you have clearly done wrong.
| ant6n wrote:
| GTA is fine ... but the storytelling is meh. Missions keep
| repeating, and there's little to draw you in. You drive
| somewhere, somebody gets whacked, you drive back. Rinse and
| repeat. The makers try compensate with shocking and crass
| violence and humor, but at some point it just feels kind of
| juvenile.
|
| Maybe it got better in recent releases, I kind of stopped
| following after GTA4.
| djmips wrote:
| Why is this getting voted down, are there that many cynical
| people out there?
| Sohcahtoa82 wrote:
| They're getting voted down because they're making a
| ridiculous argument.
|
| They're basically saying that GTAV's massive commercial
| success should grant it immunity to criticism.
| brokencode wrote:
| Successful people and businesses can be wrong. You are not
| making a case for why those development practices are okay,
| but are simply appealing to authority.
|
| I and most other customers would argue that 6 minute
| loading times are atrocious, and if there is an easy fix
| like this, it makes me lose a lot of respect for the
| developer who doesn't fix it. It maybe would even make me
| avoid them in the future.
|
| A reputation is built over years, but can be lost pretty
| much instantly. Companies have to continue serving their
| customers to enjoy ongoing success.
| hombre_fatal wrote:
| They didn't make an appeal to authority but an appeal to
| commercial success, and they're right on.
|
| The fact that GTAO is so popular should make most HNers
| rethink what they know about the commercial necessity of
| optimization vs building a compelling product.
| incrudible wrote:
| The loading times were not initially that long and then
| the slow CPU makes a big difference.
|
| This really only goes to show how much you can get away
| with if you have an outstandingly popular product that
| has no direct competition. Chances are that _your_
| product is not that compelling, if it performs poorly,
| that will hurt adoption. It will never become
| outstandingly popular in the first place.
| sillysaurusx wrote:
| Attitudes like yours are why gamedevs keep to themselves.
|
| "Unbelievable" and "unforgivable" eh? It's a greedy attitude.
| Instead of viewing GTA5 as a success that's brought a lot of
| people happiness, you view it as a money cow designed to
| extract every last bit of profit - and time, since this bug
| caused 70% longer loading times.
|
| Perhaps it's both. But you, sitting here behind a keyboard with
| (correct me if I'm wrong) no gamedev experience, have no idea
| what it's like on a triple-A gamedev team with various
| priorities. The fact that the game works _at all_ is a minor
| miracle, given the sheer complexity of the entire codebase.
|
| The fact that someone was able to optimize the obfuscated
| executable is a wonderful thing. But they weren't a part of the
| team that shipped GTA 5. If they were, they certainly wouldn't
| have been able to spend their time on this.
| daodedickinson wrote:
| I did mean to reply to you (72 days ago plus in another
| thread) I just can't make a Twitter, like I've tried and
| doesn't like my IP or email addresses or something, not sure
| sillysaurusx wrote:
| Hmm, that's unfortunate. Well, I hope you find a way to get
| on Twitter. Your thoughts are always welcome, and there are
| a lot of interesting people in the ML scene there.
| ryandrake wrote:
| This kind of excuse making is one of the reasons I got out of
| software development. It's not just gamedev. Priorities are
| way out of wack when you have time to put in binary
| obfuscation, but no time to fix such a huge performance
| bottleneck. The idea that "it's a miracle software works at
| all" demonstrates the chronic prioritization and project
| management competence problem in the industry.
|
| It's ok to recognize a thing as a business success but a
| technical failure. In fact many software projects are
| business successes _despite_ awful and unforgivable quality
| compromises. You don't get to whitewash it just because the
| thing prints money.
| systemvoltage wrote:
| How do we then address chronic incompetence? Never complain
| about it?
|
| This is not small. This kind of incompetency if employed in
| a different sector such as security would lead to losing
| personal data of millions.
|
| > "it's a miracle software works at all"
|
| This is not the case here. Please re-evaluate your
| calibration on this topic.
| sillysaurusx wrote:
| 1. you replied to the wrong person.
|
| 2. this kind of incompetence exists in all other sectors.
| That's why pentests are so crucial, and why they guard
| the security of millions.
|
| 3. we'll have to agree to disagree that it's a minor
| miracle. Having seen the complexity firsthand, it's quite
| amazing.
| systemvoltage wrote:
| Agree that this kind of incompetence exists in all
| sectors and I think we don't talk about it, it becomes
| _acceptable_. We 're not trying to blame a single
| developer, that'd be inappropriate. But, the management
| and QA culture in a AAA game studio that rakes billions
| ought to be better.
|
| The complexity is in reverse engineering the binary. The
| developer has access to the full source code and the
| profiling tools I presume.
|
| Another one is in Microsoft Flight Simulator, instead of
| downloading multiple archives, it downloads one, unzips
| it using a single CPU core and then downloads another
| one. MSFS 2020 takes a few hours to install and that's
| not just because of the internet connection, but this
| shitty installation code.
| sillysaurusx wrote:
| If loading times were prioritized, features would be cut.
| Which features would you cut out of the game in order to
| have fast loading times?
|
| This is what you'd need to decide. And then afterwards, it
| might not print as much money as you think it will.
|
| It's easy looking at it from the outside. Not so easy from
| the inside.
| ianlevesque wrote:
| Did you read the article? Zero features needed to be cut,
| this is a 30 minute fix.
| sillysaurusx wrote:
| Building the engine alone takes 30 minutes after each
| change. You're not going to get anything done in 30
| minutes. And the more you work on this, the less you're
| working on shippable features that make money.
| ianlevesque wrote:
| In the mobile space, an app opening faster is so valuable
| you can make a career out of just that. I haven't worked
| desktop/console games, but having load times be 6
| _minutes_ longer than it needs to be, when you 're trying
| to make money on ongoing microtransactions, has got to be
| losing you so much more money than the time spent fixing.
| sillysaurusx wrote:
| That's a fine argument, and I'm sure that if management
| knew they could get a 70% decrease in load times in
| exchange for focusing on this one area, they would have
| done so. But nobody knew. And discovering that would have
| been expensive.
|
| I'll meet you halfway though: they should have had
| profiling sessions that pointed to the JSON parsing code
| as the issue. I imagine that all of their profiling
| efforts were focused on the runtime performance, not the
| load time. Simply put, no one did that profiling, and I
| don't fault them for focusing on runtime performance
| (which is where the real money is, as Cyberpunk 2077
| demonstrated by not having it, and subsequently having
| their PS4 orders yanked and refunded).
| rasz wrote:
| You do realize Cyberpunk 2077 shipped running better on
| base PS4 than "critically acclaimed" Control by Remedy?
| sillysaurusx wrote:
| Cyberpunk 2077 ran so horribly on PS4 that Sony had to
| refund everyone who purchased it.
| rasz wrote:
| No, Sony pulled Cyberpunk 2077 after CD Project promised
| refunds and then forced Sony to do it on their end. Sony
| didnt like that.
|
| Once again - Control by Remedy ran on base PS4 at 10
| frames per second, TEN frames. Critically acclaimed,
| reviewers loved it and didnt tend to mention TEN frames
| per second on base consoles, not pulled from the store.
| ryandrake wrote:
| I'd have to have been there, seen the list of features
| with eng estimates and trade offs, but yes I would have
| happily made the call to chuck one of them if it meant a
| measurably higher quality product, like this massive load
| time improvement. Hell, that zoom-out-zoom-in animation
| when you switch characters probably took as much time to
| code as it would have to fix this bug. I think anyone
| with good product sense and the balls to make a priority
| call that might get someone upset could make the right
| call here.
| scaramanga wrote:
| But after it's already made record-breaking profits and
| is a huge cash cow with recurring revenue. You could just
| say "hey, I'll hire one single contract developer to do
| these kinds of quality of life things" and make a
| fraction less profit.
| ryandrake wrote:
| > make a fraction less profit
|
| Unfortunately this part often kills quality initiatives.
| Why fix bugs for your existing customers when you can
| deploy the engineering resources on a DLC or a sequel
| which will milk those customers for more? There is no
| more craftsmanship or pride in good work left in
| software.
|
| "When you're a carpenter making a beautiful chest of
| drawers, you're not going to use a piece of plywood on
| the back, even though it faces the wall and nobody will
| see it. You'll know it's there, so you're going to use a
| beautiful piece of wood on the back. For you to sleep
| well at night, the aesthetic, the quality, has to be
| carried all the way through." -Steve Jobs
| sakarisson wrote:
| > Which features would you cut out of the game in order
| to have fast loading times?
|
| If this is a serious question, I'd say cut any of the new
| vehicles introduced in the last 2 years. None of them are
| nearly as impactful as this optimization. In fact, I am
| having issues imagine any individual feature at all
| that's as important as this fix.
| scrose wrote:
| I don't think the OP was specifically calling out any game
| devs. Any engineer who has worked on any software projects
| knows that you usually can only do as well as your deadlines
| and managements priorities allow for.. Unless the stars line
| up and you have the resources and autonomy to fix the issue
| yourself, and love working for free on your own time.
| sillysaurusx wrote:
| Unfortunately, they were calling out gamedevs:
|
| _Tweaking two functions to go from a load time of 6
| minutes to less than two minutes is something any developer
| worth their salt should be able to do in a codebase like
| this equipped with a good profiler._
|
| But, I fully agree with your assessment, for what it's
| worth.
| scaramanga wrote:
| I read that to mean "If a manager would say to one of his
| developers, who are probably all worth their salt, take a
| profiler and go figure out why our load times suck, then
| this would be fixed."
|
| But a manager shouldn't even have to do that, because in
| a well functioning team, if the dev leads come back with
| such a fix, they won't get punished for going off the
| reservation and would probably be doing this of their own
| initiative.
|
| But if things like that get met with "why have you wasted
| time on this? we gave you the list of priorities and it
| does not include load times" then dev leads will make
| sure all developers time is filled with things which are
| prioritized.
|
| Edit: grammar
| scaramanga wrote:
| GTA-5 broke even within 48 hours of it's release. Nearly a
| decade later, it still costs $60 for a digital copy with
| (practically) zero distribution costs. It has made over $6Bn
| in revenue, and is said to be the most profitable
| entertainment product of all time.
|
| How much would it have cost to fix this issue?
|
| Is anyone saying that it is a game developers fault? I mean,
| what is that you think would prevent a game developer from
| fixing this?
|
| Because I think, anyone even vaguely familiar with the
| software industry in general is going to come up with answers
| like:
|
| 1. It would not cost very much 2. No it isn't a developers
| fault, because it's clear that even an intern could fix this
| 3. Management isn't interested, or is too disorganized, or
| focussed on cynical extraction of every last bit of profit.
|
| And from that perspective, it certainly does make it seem
| like a cynical cash cow.
|
| I don't know many game developers, but I do know people in
| other parts of the software industry and professionals in
| general. And I think that they keep to themselves because
| they have first hand experience of how the industry works and
| understand it better than anyone. The probably sympathise
| with the right of the public to feel ripped off.
|
| That said, I still paid for the game, I think it's fun.
| Apparently there is "no alternative" to this state of
| affairs.
| dragonwriter wrote:
| > GTA-5 broke even within 48 hours of it's release. Nearly
| a decade later, it still costs $60 for a digital copy with
| (practically) zero distribution costs
|
| Well, that's the nominal full retail price against which
| the various discounts are measured, sure, but I doubt
| that's what most people buying it these days pay except if
| they are getting something else with it. I'm pretty sure
| it's in the stack of things I've gotten free this year on
| Epic that's in my "I might check it out sometime" queue,
| it's $29.98 right now from Humble Bundle, etc.
| Sohcahtoa82 wrote:
| > Nearly a decade later, it still costs $60 for a digital
| copy
|
| It's actually only $30 and frequently goes on sale for $15.
| It hasn't been $60 (on Steam at least) since June 2018.
| serf wrote:
| >It is absolutely unbelievable (and unforgivable) that a cash
| cow such as GTA V has a problem like this present for over 6
| years and it turns out to be something so absolutely simple.
|
| Having played the game, it's not surprising to me in the least.
|
| I have never yet encountered another such 'wild-west' online
| experience.
|
| It's the only game that is so un-modereated that i've ever
| played where the common reaction to meeting a hacker that is
| interested in griefing you is to call your hacker friend white-
| knight and ask him to boot the griefer-hacker from the lobby.
|
| Reports do next to nothing -- and 'modders' have some very real
| power in-game, with most fights between 'modders' ending in one
| of them being booted to desktop by the other exploiting a CTD
| bug (which are usually chat text-parser based..)
|
| On top of all this, Rockstar attempts to have an in-game
| economy , even selling money outright to players in the form of
| 'Shark Cards' for real-life currency , while 'modders'
| (hackers) easily dupe gold for anyone that may ask in a public
| lobby.
|
| This isn't just all coincidence; the game lacks any kind of
| realistic checks/balances with the server for the sake of
| latency and interoperability -- but this results in every 13
| year old passing around the Cheat Engine structs on game-
| cheating forums and acting like virtual gods while tossing
| legitimate players around lobbies like ragdolls -- meanwhile
| Rockstar continues releasing GTA Online content while ignoring
| playerbase pleas for supervision.
|
| It's truly unique though -- an online battlefield where one can
| literally watch battles between the metaphorical white hat and
| black hat hackers; but it's a definite indicator of a poorly
| ran business when vigilante customers need to replace customer
| service.
|
| Also, an aside, most 'mod-menus' -- the small applets put
| together using publicly available memory structs for game
| exploit -- most all have a 'quick connect' feature that allows
| hackers to join lobbies much faster than the GTA V client
| usually allows for. This feature has existed for years and
| years, and I believe it performs tricks similar to those listed
| in the article.
| nikanj wrote:
| The old maxim of "Premature optimization is the root of all
| evil" has over time evolved to "If you care one iota about
| performance, you are not a good programmer".
| 29athrowaway wrote:
| I am not sure I would call that "evolution".
| GhostVII wrote:
| The problem here isn't a lack of optimization, it's a lack of
| profiling. Premature optimization is a problem because you
| will waste time and create more complex code optimizing in
| places that don't actually need it, since it's not always
| intuitive what your biggest contributors to inefficiency are.
| Instead of optimizing right away, you should profile your
| code and figure out where you need to optimize. The problem
| is that they didn't do that.
| Cthulhu_ wrote:
| I'd like to add, while GTA is running I'm really impressed
| by its performance, even / especially when it was on the
| PS3; you could drive or fly at high speed through the whole
| level, see for miles and never see a loading screen. It is
| a really optimized game, and that same level is continued
| in RDR2.
|
| Which makes the persisting loading issue all the weirder.
| bborud wrote:
| I went back and had a look at that maxim a few years ago and
| found that it actually doesn't say what many people claims it
| says. And definitively not as the blanket excuse for slow
| code that it has always to some degree been used as.
|
| The reason for the misunderstanding is that the kinds of
| practices it actually talks about are uncommon today. People
| would often take stuff written in higher level languages and
| reimplement them in assembler or machine code. Which makes
| them more time-consuming to change/evolve.
|
| It also isn't like it is hard to figure out which part of a
| piece of software that is taking up your runtime these days.
| All worthwhile languages have profilers, these are mostly
| free, so there is zero excuse for not knowing what to
| optimize. Heck, it isn't all that uncommon for people to run
| profiling in production.
|
| Also, it isn't like you can't know ahead of time which bits
| need to be fast. Usually you have some idea so you will know
| what to benchmark. Long startup times probably won't kill
| you, but when they are so long that it becomes an UX issue,
| it wouldn't have killed them to have a look.
| branko_d wrote:
| I think this part of the Knuth's quote is central:
|
| > Programmers waste enormous amounts of time thinking about,
| or worrying about, the speed of noncritical parts of their
| programs, and these attempts at efficiency actually have a
| strong negative impact when debugging and maintenance are
| considered.
|
| And he is explicitly advocating for optimizing the critical
| part:
|
| > Yet we should not pass up our opportunities in that
| critical 3%.
|
| And somehow, people have latched onto the catchphrase about
| "early optimization" that was taken out of context.
| [deleted]
| vendiddy wrote:
| And fwiw, the full quote is:
|
| We should forget about small efficiencies, say about 97% of
| the time: premature optimization is the root of all evil.
|
| Yet we should not pass up our opportunities in that critical
| 3%.
| nicbou wrote:
| Then again, using the correct data structure is not really
| optimisation. I usually think of premature optimisation as
| unnecessary effort, but using a hash map isn't it.
| rodgerd wrote:
| This is the key point: premature optimization would be
| e.g. denormalising a database because you think you might
| have a performance problem at some point, and breaking
| the data model for no good reason.
|
| Here the wrong data structure has been used in the first
| place.
| CogitoCogito wrote:
| My belief is that your first goal should be cognitive
| optimization. I.e. make it simple and clear. That
| includes using hash maps when that is the proper data
| structure since that's is what is called for at a base
| design level.
|
| The next step is to optimize away from the cognitively
| optimal, but only when necessary. So yeah it's really
| crazy this was ever a problem at all.
| yxhuvud wrote:
| OTOH, at some point what is good from a cognitive
| viewpoint depends on what idioms you use. So it can be
| helpful to actively chose which idioms you use and make
| certain to not use those that tend to blow up performance
| wise.
| CogitoCogito wrote:
| Given the current code example, I don't even thing idioms
| come into it. Here a hash map data structure was called
| for regardless of whatever idioms come into other parts
| of the design. This is just fundamental and has nothing
| to do with functional programming, OOP, etc.
| marcosdumay wrote:
| If it doesn't change semantics, it's optimization.
|
| It's just a zero cost one, so if anybody appears
| complaining that you are caring to choose the correct
| data structure and that's premature (yeah, it once
| happened to me too), that person is just stupid. But not
| calling it an optimization will just add confusion and
| distrust.
| bborud wrote:
| Exactly. And not only being able to pick the correct data
| structure for the problem, (and possibly the correct
| implementation), but being able to know what is going to
| need attention even before a single line of code is
| written.
|
| Most of my optimization work is still done with pencil,
| paper and pocket calculator. Well, actually, most of the
| time you won't even need a calculator.
| blowski wrote:
| Also, it's not "never optimise". It's "only optimise once
| you've identified a bottleneck". I guess in a profit-making
| business you only care about bottlenecks that are costing
| money. Perhaps this one isn't costing money.
| astrange wrote:
| Not all performance problems have obvious "bottlenecks",
| some are entirely second-order effects.
|
| For instance, everyone tells you not to optimize CPU time
| outside of hotspots, but with memory usage even a brief
| peak of memory usage in cold code is really bad for
| system performance because it'll kick everything else out
| of memory.
| skzo wrote:
| ding ding ding ding
| saagarjha wrote:
| This one isn't _measured_ to be costing money. Trying to
| figure out how much money you're losing as a result of
| performance problems is extremely difficult to do.
| nmfisher wrote:
| Precisely. Hasn't GTA Online done over a billion in
| revenue?
|
| Given how incredibly successful it's been, it's
| conceivable the suits decided the opportunity cost of
| investing man-hours to fix the issue was too high, and
| that effort would be better spent elsewhere.
| Dylan16807 wrote:
| They're making lots of money but the ridiculous load
| times _absolutely_ cost them money. It 's not worth an
| unlimited amount of dev time to fix, but they definitely
| should have assigned a dev to spend one day estimating
| how hard fixes would be.
| danlugo92 wrote:
| N=1 but one of the reasons I don't partake in modern
| gaming is ridiculous loading times and mandatory update
| sizes.
| rjmunro wrote:
| It's costing them money just in the time wasted by their
| own internal QAs and devs loading the game to test it.
| nmfisher wrote:
| That's also possible. I haven't played it myself, so I
| really can't comment.
| blowski wrote:
| Arguably, it could actually make them money, since it
| provides a window of free advertising. I have no data
| either way, but I wouldn't assume long load screens are
| necessarily bad for business.
| Dylan16807 wrote:
| They don't need more than a full minute of advertising
| every load.
| CogitoCogito wrote:
| Perhaps this one is costing them a lot of money.
| smolder wrote:
| That belief is getting a bit outdated now that computing
| efficiency is hitting walls. Even when compute is cheaper
| than development, you're still making a morally suspect
| choice to pollute the environment over doing useful work if
| you spend $100k/yr on servers instead of $120k/yr on coding.
| When time and energy saved are insignificant compared to
| development expense is of course when you shouldn't be
| fussing with performance.
|
| I don't think the anti-code-optimization dogma will go away,
| but good devs already know optimality is multi-dimensional
| and problem specific, and performance implications are always
| worth considering. Picking your battles is important, never
| fighting them nor knowing how is not the trick.
| djmips wrote:
| I agree 100% - the whole cheery lack of care around
| optimization to the point of it becoming 'wisdom' could
| only have happened in the artifice of the huge gains in
| computing power year on year.
|
| Still, people applying optimizations that sacrifice
| maintainability for very little gain or increase bugs are
| still doing a disservice. People who understand data flow
| and design systems from the get-go that are optimal are
| where it's at.
| ehsankia wrote:
| That doesn't really apply here. I don't even play GTA V but
| the #1 complain I've always heard for the past 6 years is
| that the load times are the worst thing about the game. Once
| something is known to be the biggest bottleneck in the
| enjoyment of your game, it's no longer "premature
| optimization". The whole point of that saying is that you
| should first make things, then optimize things that bring the
| bring the most value. The load time is one of the highest
| value things you can cut down on. And the fact that these two
| low hanging fruit made such a big difference tells me they
| never gave it a single try in the past 6 years.
| anotherfish wrote:
| We used to start up Quake while we waited then we'd forget
| about GTAO. Later we'd discover GTA had kicked us out for
| being idle too long. Then we'd just close it.
|
| That should be embarrassing for Rockstar but I don't think
| they would even notice.
| djmips wrote:
| Sure it does apply. These complaints come out after the
| game has been released. They should have optimized this
| before they released, while they even designed the system.
| However that's considered premature optimization, when in
| fact it's just bad design.
| ehsankia wrote:
| > while they even designed the system
|
| See, that's exactly why you're wrong. This wasn't a bad
| "design". If the fix required to rebuild the whole thing
| from scratch, then you would have a point and thinking
| about it "prematurely" would've been a good idea. In this
| case, both the fixes were bugs that could've been fixed
| after the game was finished without having to undo much
| of the work done.
|
| The whole point of the saying is that you don't know
| what's gonna be a bottleneck until after. Yes by
| optimizing prematurely, you would've caught those two
| bugs early, but you would've also spent time analyzing a
| bunch of other things that didn't need to be analyzed.
| Whereas if you analyze it at the end once people complain
| about it being slow, you're only spending the minimum
| amount of time necessary on fixing the things that
| matter.
| jcelerier wrote:
| > Whereas if you analyze it at the end once people
| complain about it being slow
|
| I think that we should also stop doing crash tests in
| cars. Just release the car to the public and analyze
| human crashes afterwards.
| handoflixue wrote:
| You do understand that "my entertainment takes 6 minutes
| to load" is a very different problem from "my essential
| transportation kills people"? And therefore call for
| different approaches?
| [deleted]
| hinkley wrote:
| I've worked a number of places where the engineering culture
| discourages any sort of fishing expeditions at all, and if I
| weren't so stubborn everything would take 2-4x as long to run
| as it does. I say engineering culture, because at 2 of these
| places everyone was frustrated with engineering because the
| customers wanted something better, but the engineers would
| point at flat flame charts, shrug, and say there's nothing that
| can be done.
|
| Bull. Shit.
|
| There's plenty that can be done because there are parts of a
| process that don't _deserve_ 15% of the overall budget. The
| fact that they are taking 1 /6 of the time like 5 other things
| is a failure, not a hallmark of success. Finding 30% worth of
| improvements with this perspective is easy. 50% often just
| takes work, but post-discovery much of it is straightforward,
| if tedious.
|
| My peers are lying with charts to get out of doing "grunt work"
| when there's a new feature they could be implementing. But
| performance _is_ a feature.
| tamrix wrote:
| Well this sprint we didn't release any new features but we
| reduced the load.... Dammit hackernews!
| luckystarr wrote:
| Well, that's embarrassing. I can't even imagine the level of
| shame I would feel if I had written the offending code.
|
| But, you know, premature optimization yadda yadda.
| Cakez0r wrote:
| It's not a premature optimisation to use a hashset instead of a
| list though!
| ufo wrote:
| The bug is more devious than that. The code looks linear at a
| glance and the culprit is that sscanf is actually O(N) on the
| length of the string. How many people would expect that?
| nine_k wrote:
| As they say, a lot of classified stuff and closed-source code
| remains classified and closed not because it contains important
| secrets, but because those who hold the lock feel too ashamed
| and embarrassed to show the contents.
| whatever1 wrote:
| Probably this was written under a very strict release deadline
| and it worked ok at the time (less items for
| microtransactions). The problem lies with the management that
| never picked up the issue once it became a big problem. Pretty
| sure that any developer in R* is capable of optimizing this
| parser.
| gridspy wrote:
| I imagine there is a senior programmer working for another game
| company. They are currently kicking themselves about the poorly
| performing and rushed loading code they wrote while still
| working at R*. But there is nothing they can do about it now,
| since they have moved on.
| rcxdude wrote:
| It's the kind of thing that's very easy to accidentally write,
| it's not that shameful. What's shameful is not investigating
| the load times at all, since the problem is so easy to see when
| any measurement is done.
| luckystarr wrote:
| If 63k entries with 10MB of actual data takes minutes to
| process on a current computer I'd consider that shameful.
|
| 10MB is less than the cache in modern CPUs. How can this take
| minutes(!).
| mekkkkkk wrote:
| As others have pointed out, this is a good illustration of why
| you need accurate data during development. I bet that the data
| set used during development was dirty with duplicates and way too
| small. It was faster and more convenient to just code defensive
| garbage than to be the annoying one nagging to another team about
| the state of data. So this was written, probably along with a
| TODO comment, and then forgotten about, and ultimately shipped.
| I've done this same thing. Not with any real consequences, but
| still. It's what happens when time is of the essence.
|
| How it remained undetected for so long is really weird though.
| Surely they must've had a massive amount of complaints about the
| loading times. I completely stopped playing because of them. How
| could they not have investigated the root cause, atleast once, in
| six years?
| chihuahua wrote:
| It seems that TODO comments just don't cut it. Either you
| create a bug/task for it, or you just forget about it.
| comboy wrote:
| Holy cow, I'm a very casual gamer, I was excited about the game
| but when it came out I decided I don't want to wait that long and
| I'll wait until they sort it out. 2 years later it still sucked.
| So I abandoned it. But.. this... ?! This is unbelievable. I'm
| certain that many people left this game because of the waiting
| time. Then there are man-years wasted (in a way different than
| desired).
|
| Parsing JSON?! I thought it was some network game logic finding
| session magic. If this is true that's the biggest WTF I saw in
| the last few years and we've just finished 2020.
|
| Stunning work just having binary at hand. But how could R* not do
| this? GTAV is so full of great engineering. But if it was a CPU
| bottleneck then who works there that wouldn't just be irked to
| try to nail it? I mean it seems like a natural thing to try to
| understand what's going on inside when time is much higher than
| expected even in the case where performance is not crucial. It
| was crucial here. Almost directly translates to profits.
| Unbelievable.
| lumberingjack wrote:
| It gets worse they're brand new game Red Dead online does the
| same thing as soon as it did it the first time I logged out and
| charged back
| jiggawatts wrote:
| > Parsing JSON?!
|
| Many developers I have spoken to out there in the wild in my
| role as a consultant have wildly distorted mental models of
| performance, often _many_ orders of magnitude incorrect.
|
| They hear somewhere that "JSON is slow", which it is, but you
| and I know that it's not _this_ slow. But "slow" can encompass
| something like 10 orders of magnitude, depending on context. Is
| it slow relative to a non-validating linear binary format? Yes.
| Is it _minutes_ slow for a trivial amount of data? No. But in
| their mind... it is, and there 's "nothing" that can be done
| about it.
|
| Speaking of which: A HTTPS REST API call using JSON encoding
| between two PaaS web servers in Azure is about 3-10ms. A local
| function call is 3-10ns. In other words, a lightweight REST
| call is _one million times_ slower than a local function call,
| yet many people assume that a distributed mesh microservices
| architecture has only "small overheads"! Nothing could be
| further from the truth.
|
| Similarly, a disk read on a mechanical drive is hundreds of
| thousands of times slower than local memory, which is a
| thousand times slower than L1 cache.
|
| With ratios like that being involved on a regular basis, it's
| no wonder that programmers make mistakes like this...
| salawat wrote:
| Tbe funny thing is, as a long time SDET, I had to give up
| trying to get people to write or architect in a more "local
| first" manner.
|
| Everyone thinks the network is free... Until it isn't. Every
| bit move in a computer has a time cost, and yes, it's
| small... But... When you have processors as fast as what
| exist today, it seems a sin that we delegate so much
| functionality out to some other machine across a network
| boundary when the same work could be done locally. The reason
| why though?
|
| Monetizability and trust. All trivial computation must be
| done on my services so they can be metered and charged for.
|
| We're hamstringing the programs we run for the sole reason
| that we don't want to make tools. We want to make invoices.
| gridspy wrote:
| And like so many things, we're blind to how our economic
| systems are throwing sand in the gears of our technical
| ones.
|
| I love your point that shipping a library (of code to
| locally execute) with a good API would outperform an online
| HTTPS API for almost all tasks.
| dan-robertson wrote:
| I don't think the lesson here is "be careful when parsing json"
| so much as it's "stop writing quadratic code." The json
| quadratic algorithm was subtle. I think most people's mental
| model of sscanf is that it would be linear in the number of
| bytes it scans, not that it would be linear in the length of
| the input. With smaller test data this may have been harder to
| catch. The linear search was also an example of bad quadratic
| code that works fine for small inputs.
|
| Some useful lessons might be:
|
| - try to make test more like prod.
|
| - actually measure performance and try to improve it
|
| - it's very easy to write accidentally quadratic code and the
| canonical example is this sort of triangular computation where
| you do some linear amount of work processing all the finished
| or remaining items on each item you process.
|
| As I read the article, my guess was that it was some terrible
| synchronisation bug (eg download a bit of data -> hand off to
| two sub tasks in parallel -> each tries to take out the same
| lock on something (eg some shared data or worse, a hash bucket
| but your hash function is really bad so collisions are
| frequent) -> one process takes a while doing something, the
| other doesn't take long but more data can't be downloaded until
| it's done -> the slow process consistently wins the race on
| some machines -> downloads get blocked and only 1 cpu is being
| used)
| woko wrote:
| > actually measure performance and try to improve it
|
| This reminds me that I used to do that all the time when
| programming with Matlab. I have stopped investigating
| performance bottlenecks after switching to Python. It is as
| if I traded performance profiling with unit testing in my
| switch from Matlab to Python.
|
| I wonder if there are performance profilers which I could
| easily plug into PyCharm to do what I used to do with
| Matlab's default IDE (with a built-in profiler) and catch up
| with good programming practices. Or maybe PyCharm does that
| already and I was not curious enough to investigate.
| petters wrote:
| Is there some reason sscanf can not be implemented without
| calling strlen?
| ddulaney wrote:
| It could be, and the article acknowledges that possibility.
| For example, a cursory check of the musl sscanf [0]
| suggests that it does not (though I may have missed
| something). However, whichever implementation Rockstar used
| apparently does.
|
| [0]: https://git.musl-
| libc.org/cgit/musl/tree/src/stdio/vfscanf.c
| pja wrote:
| A lot of libc implementations seem to implement sscanf()
| this way: As well as the windows libc ones mentioned
| above, I checked the OpenBSD & glibc implemtations & they
| worked the same way.
| JdeBP wrote:
| Slightly less cursory:
| https://news.ycombinator.com/item?id=26298300
| acdha wrote:
| > actually measure performance and try to improve it
|
| This really rings truest to me: I find it hard to believe
| nobody ever plays their own game but I'd easily believe that
| the internal culture doesn't encourage anyone to do something
| about it. It's pretty easy to imagine a hostile dev-QA
| relationship or management keeping everyone busy enough that
| it's been in the backlog since it's not causing crashes.
| After all, if you cut "overhead" enough you might turn a $1B
| game into a $1.5B one, right?
| Jach wrote:
| Lots of possibilities. Another one I imagined is that "only
| the senior devs know how to use a profiler, and they're
| stuck in meetings all the time."
| acdha wrote:
| I could easily imagine variations of that where this is
| in maintenance mode with a couple of junior programmers
| because the senior ones either burnt out or moved on to
| another project. I've definitely gotten the impression
| that most games studios have roughly the same attitude
| towards their employees as a strip-miner has towards an
| Appalachian hilltop.
| disgruntledphd2 wrote:
| If this were anyone else but Rockstar, I'd agree with
| you.
|
| But Rockstar essentially only have GTA and Red Dead to
| take care of, it's not like they're making an annual
| title or something :)
| acdha wrote:
| True, but they could still be understaffing and have
| their senior people working on the next big version
| rather than maintenance. It's definitely penny wise,
| pound foolish no matter the exact details.
| simias wrote:
| The JSON parsing is forgivable (I actually didn't know that
| scanf computed the length of the string for every call) but
| the deduplication code is a lot less so, especially in C++
| where maps are available in the STL.
|
| It also comforts me into my decision of never using scanf,
| instead preferring manual parsing with strtok_r and strtol
| and friends. It's just not robust and flexible enough.
| ant6n wrote:
| I thought the lesson is "listen to your customers and fix the
| issues they complain about".
| Nitramp wrote:
| - do not implement your own JSON parser (I mean, really?).
|
| - if you do write a parser, do not use scanf (which is
| complex and subtle) for parsing, write a plain loop that
| dispatches on characters in a switch. But really, don't.
| thw0rted wrote:
| This is probably good advice but not even relevant. It's
| down one level from the real problem: when your game spends
| 6 minutes on a loading screen, *profile* the process first.
| You can't optimize what you haven't measured. Now, if
| you've identified that JSON parsing is slow, you can start
| worrying about how to fix that (which, obviously, should be
| "find and use a performant and well-tested library".)
| dan-robertson wrote:
| I think sscanf is subtle because when you think about what
| it does (for a given format string), it's reasonably
| straightforward. The code in question did sscanf("%d",
| ...), which you read as "parse the digits at the start of
| the string into a number," which is obviously linear. The
| subtlety is that sscanf doesn't do what you expect. I think
| that "don't use library functions that don't do what you
| expect" is impossible advice.
|
| I don't use my own json parser but I nearly do. If this
| were some custom format rather than json and the parser
| still used sscanf, the bug would still happen. So I think
| json is somewhat orthogonal to the matter.
| Nitramp wrote:
| > The code in question did sscanf("%d", ...), which you
| read as "parse the digits at the start of the string into
| a number," which is obviously linear.
|
| I think part of the problem is that scanf has a very
| broad API and many features via its format string
| argument. I assume that's where the slowdown comes from
| here - scanf needs to implement a ton of features, some
| of which need the input length, and the implementor
| expected it to be run on short strings.
|
| > The subtlety is that sscanf doesn't do what you expect.
| I think that "don't use library functions that don't do
| what you expect" is impossible advice.
|
| I don't know, at face value it seems reasonable to expect
| programmers to carefully check whether the library
| function they use does what they want it to do? How would
| you otherwise ever be sure what your program does?
|
| There might be an issue that scanf doesn't document it's
| performance well. But using a more appropriate and
| tighter function (atoi?) would have avoided the issue as
| well.
|
| Or, you know, don't implement your own parser. JSON is
| deceptively simple, but there's still enough subtlety to
| screw things up, qed.
| dan-robertson wrote:
| But sscanf _does_ do what they want it to do by parsing
| numbers. The problem is that it also calls strlen. I'm
| still not convinced that it's realistically possible to
| have people very carefully understand the performance
| characteristics of every function they use.
|
| Every programmer I know thinks about performance of
| functions either by thinking about what the function is
| doing and guessing linear/constant, or by knowing what
| the data structure is and guessing (eg if you know you're
| doing some insert operation on a binary tree, guess that
| it's logarithmic), or by knowing that the performance is
| subtle (eg "you would guess that this is log but it needs
| to update some data on every node so it's linear"). When
| you write your own library you can hopefully avoid having
| functions with subtle performance and make sure things
| are documented well (but then you also don't think they
| should be writing their own library). When you use the C
| stdlib you're a bit stuck. Maybe most of the functions
| there should just be banned from the codebase, but I
| would guess that would be hard.
| thaumasiotes wrote:
| > I assume that's where the slowdown comes from here -
| scanf needs to implement a ton of features, some of which
| need the input length, and the implementor expected it to
| be run on short strings.
|
| I didn't get that impression. It sounded like the
| slowdown comes from the fact that someone expected sscanf
| to terminate when all directives were successfully
| matched, whereas it actually terminates when either (1)
| the input is exhausted; or (2) a directive fails. There
| is no expectation that you run sscanf on short strings;
| it works just as well on long ones. The expectation is
| that you're intentionally trying to read all of the input
| you have. (This expectation makes a little more sense for
| scanf than it does for sscanf.)
|
| The scanf man page isn't very clear, but it looks to me
| like replacing `sscanf("%d", ...)` with `sscanf("%d\0",
| ...)` would solve the problem. "%d" will parse an integer
| and then dutifully read and discard the rest of the
| input. "%d\0" will parse an integer and immediately fail
| to match '\0', forcing a termination.
|
| EDIT: on my xubuntu install, scanf("%d") does not clear
| STDIN when it's called, which conflicts with my
| interpretation here.
| JdeBP wrote:
| No it would not. Think about what the function would see
| as its format string in both cases.
|
| The root cause here isn't formatting or scanned items. It
| is C library implementations that implement the "s"
| versions of these functions by turning the input string
| into a nonce FILE object on every call, which requires an
| initial call to strlen() to set up the end of read buffer
| point. (C libraries do not have to work this way. Neither
| P.J. Plauger's Standard C library nor mine implement
| sscanf() this way. I haven't checked Borland's or
| Watcom's.)
|
| See https://news.ycombinator.com/item?id=26298300 and
| indeed Roger Leigh six months ago at
| https://news.ycombinator.com/item?id=24460852 .
| pja wrote:
| Yes, it looks that way. On the unix/linux side of things,
| glibc also implements scanf() by converting to a FILE*
| object, as does the OpenBSD implementation.
|
| It looks like this approach is taken by the majority of
| sscanf() implementations!
|
| I honestly would not personally have expected sscanf() to
| implicitly call strlen() on every call.
| XCSme wrote:
| > But how could R* not do this? GTAV is so full of great
| engineering
|
| I assume there were different people working on the core game
| engine and mechanics VS loading. It could as well be some
| modular system, where someone just implemented the task "load
| items during online mode loading screen screen".
| mdoms wrote:
| Me and my twice-a-week gaming group enjoyed GTA V but abandoned
| it years ago simply because of the load times. We have 2x short
| slots (90-120 minutes) each week to play and don't want to
| waste them in loading screens.
|
| We all would have picked this game back up in a second if the
| load times were reduced. Although I must say even with the same
| results as this author, 2 minutes is still too long. But I'll
| bet that, given the source code, there are other opportunities
| to improve.
| thw0rted wrote:
| I wonder if a paid subscription would have fixed this? If you
| left a paid MMO, they'd probably ask you to fill out an exit
| survey, and you could say "I'm canceling because load times
| are terrible", which would (hopefully) raise the priority of
| reducing load times. But since GTA online is "free", there's
| not a single exit point where they can ask "why did you stop
| playing".
| sova wrote:
| Really great work. Here's to hoping someone at Rockstar notices.
| You didn't even need access to the original sourcecode to debug
| it! Quite impressive my friend. Thanks for the great write-up!
| singhrac wrote:
| Honestly, while this horrible code is mildly offensive to me, I'm
| pretty impressed by this person's persistence. It's one thing to
| identify a bug in a compiled program, but it's another to patch
| it without fully understanding what's going on. Caching strlen
| was a particularly clever trick that sidestepped a bunch of more
| complicated solutions.
| eptcyka wrote:
| BTC mining is using a significant amount of global energy
| production every year now. The more cynical might say that
| ultimately it's a waste of human effort, but I have to say that
| this bug (and it's most definitely a bug) should be patched (and
| should've been patched) to save the kilowatts of power wasted
| parsing mostly the same data.
| high_byte wrote:
| exactly my thinking
| sim_card_map wrote:
| over the course of 7 years and millions of players
|
| lots and lots of megawatts, not kilowatts
| seer wrote:
| A lot of comments here kind of brush off the fact that this has
| been going on for _years_! I would expect quadratic behaviour to
| slip here and there in non real time code places. We make
| mistakes after all.
|
| I've had to revisit production code from time to time to discover
| my or my colleague assumptions about library functions were not
| actually true.
|
| But for someone in the possession of the source code this looks
| like _minutes_ worth of profiling time! With a stellar technical
| accomplishment that GTA5 truly is, you would expect tons of
| really talented devs that could have traced this one down on
| their lunch break.
|
| The fact that they didn't speaks volumes about organisational
| structure within R*. I think some higher up managers _really_
| need some stern talking to!
| 4cao wrote:
| Excellent investigation, and an elegant solution.
|
| There's a "but" though: you might end up getting banned from GTA
| Online altogether if this DLL injection is detected by some anti-
| cheat routine. The developers should really fix it on their end.
| away_throw wrote:
| It's highly unlikely you are going to get banned on GTA even
| with cheats. The anti-cheat is a joke. The game is filled to
| the brim with cheaters. If me and my friends play, we play with
| cheats just to protect ourselves from other cheaters.
| half-kh-hacker wrote:
| Game cheat dev here: Just to provide some context, the GTA
| Online client is woefully horrible at doing client-side
| validation on the packets it receives from other peers.
| (there isn't an authoritative server)
|
| This means that anyone in your session can send you a
| weirdly-formed packet to crash your game. Most cheats have
| protections against this by just doing Rockstar's job and
| adding better validation around packet interpretation
| routines.
|
| Using "cheats just to protect [your]selves" actually makes a
| lot of sense.
| kuroguro wrote:
| Yeah, there's some disclaimers in the PoC repo. Definetly use
| at your own risk.
| AJRF wrote:
| There's one thing that made me curious. The author said the load
| times when the game came out were awful as well.
|
| So has that JSON always been 10mb or has it grown a lot over
| time?
|
| Has the load time crept up since launch or has it gone down?
| Shadonototro wrote:
| think about it, how much electricity wasted
|
| millions of players, billions of minutes wasted
|
| when you scale things, every bit of optimizations IS MENDATORY!
|
| #saveOurPlanet
| oatz wrote:
| Love performance issues like this, good read.
| DarkByte wrote:
| This is quite impressive though it is important to remember that
| this came out for machines like the Xbox360 and the PlayStation
| 3. I am not convinced these modifications would "fit" on those
| machines without significant thrashing/paging and significant
| memory usage to the point where other negative effects come into
| play. Great read and effort regardless.
| captain_price7 wrote:
| I loved so much reading it, I was thinking that if someone were
| to write _fictional_ , sherlock holmes like fantasy story where
| our sherlock would take some (maybe fictional) widely used
| software at each episode, investigate it like this, and reveal
| some (fictional) grand bug in the end- I'd totally read it.
|
| Yeah I know it sounds stupid, but I suspect real Sherlock Holmes
| was inspired by a true story like this one too, and at least some
| contemporary detectives started to enjoy reading them.
| saagarjha wrote:
| There's no need for the examples to be fictional, there are
| more than enough real world cases to share. Sadly, many of my
| personal ones end in "I filed a bug with an annotated
| screenshot of decompiled code indicating where they should fix
| it but nothing happened".
| [deleted]
| TamDenholm wrote:
| The Github link is dead, DMCA or incorrect link?
| sbierwagen wrote:
| This link? https://github.com/tostercx/GTAO_Booster_PoC
| Avery3R wrote:
| probably just forgot to be unprivated. The blog post was
| published today
| kuroguro wrote:
| yep, my bad
| flumpcakes wrote:
| Do you have any resources or advice to building up the
| skills required to do something like this?
|
| I like the idea of fixing up some older, unsupported, games
| I have issues with but would like to upskill using some
| structured resources first (books/videos/guided projects).
|
| Great work! This is the kind of wizard-magic I wish came
| naturally to me.
| bluedino wrote:
| Learn C and Assembly, learn how to debug your own
| programs.
|
| Then try debugging other programs.
| kuroguro wrote:
| ^ yeah that. I started with C so I always had decent low
| level understanding. I've found I pick things up faster
| if I'm doing stuff, not reading about it but YMMV.
|
| I probably I picked up most of my reverse engineering
| skills practicing on crackmes and reading/watching other
| people's solutions on crackmes.de (now dead) but you can
| try your luck in the successor https://crackmes.one/
| arisudesu wrote:
| It doesn't explain some things to me. Maybe I read the article
| inattentively, but which stage of the loading hangs for a very
| long time? Loading screen, or city overview when joining a
| session? In my experience, on modern hardware, the loading screen
| goes very quickly (a minute or two).
|
| What's worth noting is that the city overview can hang for 20-30
| minutes during connecting. At the same time, suspending the
| GTAV.exe for 20-30 seconds allows you to immediately throw the
| player into an empty public session. It is unlikely caused by
| slow parsing of json, more like the suspend causes udp timeout
| when connecting to the slow session hosts.
| Taylor_OD wrote:
| I wonder how long Rockstar will take to respond/fix.
| xyst wrote:
| not surprising - the game industry is absolutely notorious for
| cutting corners. didn't know they cut corners this much though.
|
| will r* fix it? maybe, especially since some person literally did
| half of the work for them. But given R* is a large company, this
| probably wont be fixed for a long time, and GTAO is probably
| being maintained by the lowest bid contractor group.
|
| They probably have all of their full time assets working on the
| next iteration of GTA.
| fctorial wrote:
| > especially since some person literally did half of the work
| for them
|
| All of the work.
| jjulius wrote:
| >But given R* is a large company, this probably wont be fixed
| for a long time, and GTAO is probably being maintained by the
| lowest bid contractor group.
|
| They've also made just an absolute assload of money from GTA:O
| in spite of the godawful load times. Why bother spending the
| money to fix it when people are happy to deal with it and keep
| giving you their own cash?
| CTDOCodebases wrote:
| Imagine all the time that people are wasting if they are sitting
| there waiting for it to load. I wonder how many lifespans it adds
| up to?
| samhh wrote:
| I wonder about the environmental impact of CPU cores churning
| away uselessly on so many machines so many times. Probably
| enough to power a home for a few years?
| gfodor wrote:
| Hacker News
| mattbillenstein wrote:
| Christ.
| jonasfj wrote:
| I bought a PlayStation a few years go to play GTA..
|
| Never tried consoles before, but figured the point was that it
| would be fast and hassle free.
|
| GTA was fun, but after completing it I never load it, just to
| play for fun, because PS load times + GTA load times are too
| much.
|
| Now I just play 20 year old games under wine on Linux.. that's
| less of a hassle, and I don't have to touch windows.
| holyknight wrote:
| Thank god. I always suspected that those loading times were cause
| by some retarded implementation detail. GTA5 is not that complex
| to justify that kind of loading times. Even the hardware has
| scaled massively since their launch and it doesn't even matter.
| josephg wrote:
| It so often is. This aspect of modern computing annoys me so
| much - modern computers, networks and cdns are so fast nowadays
| that most actions should be instant. My OS should boot
| basically instantly. Applications should launch instantly.
| Websites should load almost instantly. I'll give a pass for 3D
| modelling, video editing, AAA video games and maybe optimized
| release builds of code. But everything else should happen
| faster than I can blink.
|
| But most programs are somehow still really slow! And when you
| look into why, the reason is always something like this. The
| code was either written by juniors and never optimized because
| they don't know how, or written by mids at the limit of their
| intelligence. And full of enough complex abstractions that
| nobody on the team can reason holistically about how the whole
| program works. Then things get slow at a macro level because
| fixing it feels hard.
|
| Either way it's all avoidable. The only thing that makes your
| old computer feel sluggish for everyday computing is that
| programmers got faster computers, and then got lazy, and then
| shipped you crappy software.
| saagarjha wrote:
| The most infuriating response to this is "the programs are
| doing _so much_ these days!" Well, yes, a chat app might do
| emoji and stuff now. But it's certainly not doing 1000x the
| number of things...
| XCSme wrote:
| Well, if the chat app is implemented in Electron it has to
| load a fully-fledged browser before even starting to load.
| saagarjha wrote:
| Which is clearly not a "feature" I want.
| XCSme wrote:
| Why not? You can now have all browser vulnerabilities
| directly in your chat app!
| josephg wrote:
| Yes! And a lot of those things are pointless, buggy
| busywork for the project managers involved. I'd rather well
| designed, minimal, fast, intuitive software with not many
| features over something like Xcode - packed full of buggy
| features that lag or crash the whole ide regularly. Polish
| is unsexy, but usually way more important than we give it
| credit for.
|
| As they said at Zynga, move slow and fix your shit.
| progx wrote:
| I can't believe what Rockstar did here. I and 3 friends stop
| playing GTA V cause of the unbelievable stupid loading times.
| jokoon wrote:
| Parsing text is always expensive on the CPU, that's why it's
| better to often prefer binary files when it's possible.
|
| It's also the reason the DOM is so slow, in my view.
|
| I remember spotting a lot of fast JSON parsers around, but again,
| there doesn't seem to be any popular, open, flexible, binary file
| format out there.
|
| Meanwhile, games are always larger and larger, machine learning
| requires more and more data.
|
| There is ALWAYS money, battery and silicon to be saved by
| improving performance.
| baby wrote:
| Flamegraphs ffs!
| c7DJTLrn wrote:
| Sad that the source
| (https://github.com/tostercx/GTAO_Booster_PoC) has been taken
| down, I wanted to give this a try myself. I imagine a lot of GTA
| Online players would.
| [deleted]
| kuroguro wrote:
| Author here, my bad, forgot to set it to public ^^;;
| c7DJTLrn wrote:
| Many thanks!
| scrollaway wrote:
| You should have a way to contact you somewhere on your site.
| That blog post is an excellent stand in for a resume :)
| polote wrote:
| I have absolute respect for people who are able to do that kind
| of things. A few years back I've tried to play with reverse
| engineering but I never managed to get the skills.
| pid_0 wrote:
| Very cool analysis. Its unbelievable that this is still unfixed.
| I don't play GTA anymore mostly because of the load times for
| every single action.
| segmondy wrote:
| This is why I come to HN, I was going to skip this because I
| thought it was about video games, but really glad to have read
| it, and loved every line of the article.
|
| So much to get from this.
|
| Even if you don't have the source, you can make a change if you
| are annoyed enough.
|
| If you don't like something, and the source code is out there,
| really go contribute.
|
| Performance matters, know how to profile and if using an external
| dependency, then figure out their implementation details.
|
| Algorithms & Data structures matter, often I see devs talking
| about how it doesn't matter much but the difference between using
| a hashmap vs array is evident.
|
| Attentive code reviews matter, chances are they gave this to a
| junior dev/intern, and it worked with a small dataset and no one
| noticed.
| madeofpalk wrote:
| > Even if you don't have the source, you can make a change if
| you are annoyed enough.
|
| Well, until you get flagged by the anti cheat and get your
| account and motherboard banned...
| zionic wrote:
| Imagine getting banned for fixing their insane load times lol
| rocqua wrote:
| Getting banned for DLL injection seems very likely to me.
| It certainly is a risk.
|
| Heck, it might be against the EULA, which probably doesn't
| hold op legally, but is decent grounds for a ban.
| madeofpalk wrote:
| Getting banned for modifying the game process seems _very_
| commonplace and likely? It 'll be a part of any anti-cheat
| system, it's basically table stakes.
| bluedino wrote:
| I always tell a story about an application we ran, it generated
| its own interface based on whatever was in inventory. Someone
| did something really stupid and duplicated each inventory item
| for each main unit we sold...so you had recursive mess.
| Assigning 100,000 items when previously it was 100-ish
|
| Anyway, everyone just rolled their eyes and blamed the fact
| that the app was written in Java.
|
| It ended up generating an XML file during that minute long
| startup....so we just saved the file to the network and loaded
| it on startup. If inventory changed, we'd re-generate the file
| once and be done with it.
| indeedmug wrote:
| It's a lot easier to blame an language for being slow because
| it's obvious. Blaming algorithms requiresputying in the time
| to figure things out.
| acdha wrote:
| There's also a confound between a language and its
| communities. I've seen so many cases where a "slow"
| language like Python or (older) Perl smoked Java or C++
| because the latter developers were trying to follow
| cultural norms which said that Real Decelopers(tm) don't
| write simple code and they had huge memory churn with dense
| object hierarchies and indirection so performance ended up
| being limited by O(n) XML property lookups for a config
| setting which nobody had ever changed whereas the "slow"
| language developer had just implemented a simple algorithm
| directly and most of the runtime was in highly-optimized
| stdlib native code, a fast regex instead of a naive
| textbook parser which devolved into an object churn
| benchmark, etc.
|
| Languages like Java get a lot of bad reputation for that
| because of popularity: not just that many people are hired
| into broken-by-design environments (or ones where they're
| using some framework from a big consultancy or a vendor who
| makes most of their revenue from consulting services) but
| also because many people learn the language as their first
| language and often are deeply influenced by framework code
| without realizing the difference between widely used long-
| term reusable code and what most projects actually need and
| are staffed for. It's easy to see the style of the Java
| standard frameworks or one of the major Apache projects and
| think that everyone is supposed to write code like that,
| forgetting that they have to support a greater number of
| far more diverse projects over a longer timeframe than your
| in-house business app nobody else works on. Broader
| experience helps moderate this but many places choose poor
| metrics and neglect career development.
| closeparen wrote:
| I think this is a perfect example of "algorithms and data
| structures emphasis is overblown." Real world performance
| problems don't look like LeetCode Hard, they look like doing
| obviously stupid, wasteful work in tight loops.
| nikanj wrote:
| And trying to optimize them gets you stink eye at code review
| time. Someone quotes Knuth, they replace your fast 200 lines
| with slow-as-molasses 10 lines and head to the bar.
| gridspy wrote:
| Unfortunately this. Or they will say "don't optimize it
| until it proves to be slow in production" - at which point
| it is too dangerous to change it.
| rictic wrote:
| True that it's rare that you need to pull out obscure
| algorithms or data structures, but in many projects you'll be
| _constantly_ constructing, composing, and walking data
| structures, and it only takes one or two places that are
| accidentally quadratic to make something that should take
| milliseconds take minutes.
|
| The mindset of constantly considering the big-O category of
| the code you're writing and reviewing pays off big. And
| neglecting it costs big as well.
| ilaksh wrote:
| Except that you need to test your software and if you see
| performance problems, profile them to identify the cause.
| It's not like you have one single chance to get everything
| right.
| wnoise wrote:
| ... that's the exact opposite of what I took from this.
|
| The obviously stupid, wasteful work is at heart an
| algorithmic problem. And it cropped up even in the simplest
| of data structures. A constant amount of wasteful work often
| isn't a problem even in tight loops. A linear amount of
| wasted work, per loop, absolutely is.
| lmm wrote:
| It's not something that requires deep algorithms/data
| structures knowledge, is the point. Knowing how to invert a
| binary tree won't move the needle on whether you can spot
| this kind of problem. Knowing how to operate a profiler is
| a lot more useful.
| baby wrote:
| And here what matters is not your programming skills, it's
| your profiling skills. Every dev writes code that's not the
| most optimized piece from the start, hell we even say "don't
| optimise prematurely". But good devs know how to profile and
| flamegraph their app, not leetcode their app.
| segmondy wrote:
| actually, "don't optimize prematurely" is a poor advice.
| just recently I was doing a good review that had the same
| issue where they were counting the size of an array in a
| loop, when stuff was being added to the array in the loop
| too. obvious solution was to track the length and
| arr = [] while ...: if something:
| arr.append(foo) ... if count(arr) == x:
| stuff ...
|
| changed to arr=[] arr_size = 0
| while ...: if something:
| arr.append(foo) arr_size++; ...
| if arr_size == x: stuff ...
|
| This is clearly optimization, but it's not premature. The
| original might just pass code review, but when it wrecks
| havoc, the amount of time it will cost will not be worth
| it, jira tickets, figuring out why the damn thing is
| slower, then having to recreate it in dev, fixing it,
| reopening another pull request, review, deploy, etc.
| Sometimes "optimizing prematurely" is the right thing to do
| if it doesn't cost much time to do or overly completely the
| initial solution. Of course, this depends on the language,
| some languages will track the length of the array so
| checking the size is o(1), but not all languages do, so
| checking the length can be expensive, knowing the
| implementation detail matters.
| dthul wrote:
| I'm not sure I would prefer the second version in a code
| review. I find the first version is conceptually nicer
| because it's easy to see that you will always get the
| correct count. In the second version you have to enforce
| that invariant yourself and future code changes could
| break it. If this is premature optimization or not
| depends on the size of the array, number of loop
| iterations and how often that procedure is called. If
| that's an optimization you decide to do, I think it would
| be nice to extract this into an "ArrayWithLength" data
| structure that encapsulates the invariant.
| thaumasiotes wrote:
| > In the second version you have to enforce that
| invariant yourself and future code changes could break
| it.
|
| Yes, that's a real issue. But we've been given two
| options:
|
| - Does the correct thing, and will continue to do the
| correct thing regardless of future changes to the code.
| Will break if the use case changes, even if the code
| never does.
|
| - Does the correct thing, but will probably break if
| changes are made to the code. Will work on any input.
|
| It actually seems a lot more likely to me that the input
| given to the code might change than that the code itself
| might change. (That's particularly the case for the
| original post, where the code serves to read a
| configuration file, but it's true in general.)
| dthul wrote:
| Yes, I absolutely see the reasoning and I think if one
| does go the route of encapsulating the more efficient
| array logic one can have the best of both options.
| thaumasiotes wrote:
| > if one does go the route of encapsulating the more
| efficient array logic one can have the best of both
| options.
|
| Do you see a way to do this that doesn't involve rolling
| your own array-like or list-like data type and replacing
| all uses of ordinary types with the new one? (This is
| actually already the implementation of standard Python
| types, but if you're encountering the problem, it isn't
| the implementation of your types.)
| dthul wrote:
| I guess it depends on the language and library you are
| using but I have the feeling that in most cases one would
| probably need to replace the usage of the old data type
| with the new one.
| rocqua wrote:
| With these things, I have always had the hope that an
| optimizing compiler would catch this. I think it is an
| allowed optimization if the count function is considered
| `const` in C or C++ at least.
| johnfn wrote:
| Exactly - though to add a little nuance to your post, it's
| about having a million loops in a 10M line code base and
| exactly one of them is operating maximally slowly. So
| preventing the loop from entering the code base is tough -
| finding it is key.
| segmondy wrote:
| leetcode style thinking will allow you to spot obviously
| stupid wasteful work in tight loops.
| fctorial wrote:
| This was probably a compiler bug. I don't think the programmers
| coding the business logic were using 'strlen' and 'sscanf'
| directly.
| Amadhoun9 wrote:
| Wowwwww
| T3RMINATED wrote:
| Hi this is Roman from Rockstar.
| ameyv wrote:
| This should be tweeted to their official channel or someone
| inside company (on twitter) who has ability let relevant Rockstar
| dev know about it...
| 1watts wrote:
| I bet this feature, relating to in game purchasing options, was
| retro-fitted by some non-core dev team who delivered it to spec
| and fixed cost
| [deleted]
| TinkersW wrote:
| Pretty strange that nobody at Rockstar bothered to run the games
| loading phase in a profiler, this problem would have been super
| easy to observe if they had.
| estomagordo wrote:
| Hey OP, what would it take to persuade you to have a look at
| Hearthstone next?
| coold wrote:
| [joke]so, they say that c++ is faster than java[/joke]
| ce1lingisr00f wrote:
| I like this page design. It looks familiar. Does anyone know
| where this design comes from? Or if it's original?
| kuroguro wrote:
| It's a stripped down version of hexo's dark cactus theme:
| https://github.com/probberechts/hexo-theme-cactus
|
| Don't think it's too popular, but the inspiration may be from
| elsewhere if you've seen it around.
| bagacrap wrote:
| scary to think of the carbon emissions this bug has generated
| over the years.
| avar wrote:
| GTA:O shows advertisements for in-game purchases on the loading
| screen. How many advertisements you see is a function of how long
| the loading screen takes.
|
| Something tells me this "problem" was discovered long ago at
| Rockstar HQ, and quietly deemed not to be a problem worth
| solving.
| softwhale wrote:
| Could be, but I can imagine people giving up on GTA: Online
| altogether because it takes too much time to load.
| bb123 wrote:
| Yep this was me about 4 years ago. I distinctly remember
| sitting through the loading screen and being absolutely
| astonished at how long it took. I never fired it up again
| because I just couldn't be bothered to wait that long.
| mxfh wrote:
| But for the ones that don't, it equals a virtual commute, so
| it might be a good filter and the audience beyond that
| watershed is more serious about spending money there. ;)
| anonymousab wrote:
| > Could be, but I can imagine people giving up on GTA: Online
| altogether because it takes too much time to load.
|
| Luckily for them, churn is usually a different problem solved
| by a different part of the team/org with different priorities
| and insights.
| tumetab1 wrote:
| I'm one that had pre-order bonus on GTA Online but gave up in
| part due to the loading times. If I have one hour to play I
| don't want to spend 10 minutes on loading screens.
|
| It's funny to think that I would probably pay some amount to
| have GTA Online without these absurd loading times and
| without modders/hackers :D
| vmception wrote:
| They either:
|
| Found that they get more purchases BECAUSE of the long loading
| time, despite bouncing other players (the ad theory and happy
| coincidence for them to have the ad placement slots from shitty
| engineering)
|
| The engagement team was told bullshit by the engineering team
| about how impossible it is to fix that issue
|
| Or they are just making enough not to care
| the_gipsy wrote:
| Or
|
| the "engagement team" got fooled by longer session times
| vmception wrote:
| hahaha true, the a/b test showed them what they wanted to
| see and they never stopped to ask users if its what they
| wanted
|
| "Roll that out to everyone!"
| cmeacham98 wrote:
| I was going to say surely this has extremely diminishing or
| even negative return past 30-60 seconds, but then I remembered
| lots of people are willing to sit through 10 minutes of
| commercials to watch 20-30 minutes of TV. So I guess for the
| right type of customer it works?
| crazygringo wrote:
| But on network TV, 8 minutes of commercials are
| _interspersed_ among 22 minutes of content.
|
| Virtually nobody's willing to sit through 10 minutes of
| commercials straight.
| thitcanh wrote:
| It depends on the channel, really. I remember once I was
| sitting at an eatery in the Philippines (basically a
| cafeteria with immediately-available food) and the TV just
| played commercials from the time I sat down until I left.
| [deleted]
| Taylor_OD wrote:
| I remember realizing I was 10-20 minutes into an
| infomercial instead of regular commercials late at night
| when I was young. I was probably distracted at the time but
| man did I feel stupid when I realized that it wasnt just a
| really long P90X commercial.
| SilasX wrote:
| Especially if you have to watch the full ten minutes
| _first_.
|
| Then again, that's what they do at movie theaters...
| thaumasiotes wrote:
| The GTA thing is even worse than that; television doesn't
| air commercials _before the beginning of the show_!
| azhenley wrote:
| Someone hire this person!
___________________________________________________________________
(page generated 2021-03-01 23:02 UTC)