[HN Gopher] How Netflix uses Java
       ___________________________________________________________________
        
       How Netflix uses Java
        
       Author : ivanche
       Score  : 205 points
       Date   : 2023-10-10 07:25 UTC (15 hours ago)
        
 (HTM) web link (www.infoq.com)
 (TXT) w3m dump (www.infoq.com)
        
       | jarym wrote:
       | Interesting the article jumps straight from REST to GraphQL and
       | forgets Falcor[0] - Netflix's alternative vision for federated
       | services. For a while it looked like it might be a contender to
       | GraphQL but it never really seemed to take off despite being
       | simpler to adopt.
       | 
       | [0] https://netflix.github.io/falcor/
        
         | parthdesai wrote:
         | Netflix themselves have moved off falcor though
         | 
         | https://netflixtechblog.com/migrating-netflix-to-graphql-saf...
        
         | paulbakker wrote:
         | Falcor is actually part of the "old" architecture described in
         | the talk. Because it's mostly unknown and no longer used I
         | didn't go into the details of it.
         | 
         | Falcor was developed at the time Facebook was developing
         | GraphQL in-house. It has similar concepts, but never took off
         | the way GraphQL did.
        
         | [deleted]
        
         | lfkdev wrote:
         | `Sad Prime noises`
        
         | dustingetz wrote:
         | iirc falcor predated graphql
        
           | ppseafield wrote:
           | I was at the React Rally conference where Falcon was publcly
           | announced in August of 2015. I recall that Facebook gave a
           | GraphQL presentation right before.
           | 
           | It seems GraphQL was first announced publicly in February
           | 2015.
        
         | baby wrote:
         | Probably because most people don't want to work with Java
        
       | dlhavema wrote:
       | Most of the postings for backend positions at Netflix I've seen
       | call out nodejs. Can I assume they do both? Is one legacy and the
       | other newer stuff, or are they more complimentary?
       | 
       | Anyone on in the inside know?
        
         | nameless912 wrote:
         | Things are certainly more of a blend now than what's presented
         | in this presentation, but the presenter is a big Java platform
         | guy here. I would say ~70% of the services I interact with on a
         | day to day basis are Java, another 20% in Node, and then the
         | last 10% is a hodgepodge of Python, Go, and more esoteric
         | stuff.
         | 
         | It varies from team to team; the "Studio" organization that
         | supports creating Netflix content does lots of nodeJS due to
         | the perception that it's faster to iterate on a UI and API
         | together if they're both in the same language. On my team,
         | we're very close to 50/50 due to managing a bunch of backend,
         | business process type systems (Java), and a very complex UI
         | (with a NodeJS backing service to provide a graphql query
         | layer). Regardless, the tooling is really quite good, so
         | interacting with a Node service is roughly identical to
         | interacting with a Java service is roughly identical to
         | interacting with anything else. We lean into code generation
         | for clients pretty heavily, so graphQL is a good fit, but gRPC
         | and Swagger are still used pretty frequently.
        
           | dlhavema wrote:
           | Thanks for responding. That's good insight
        
       | yayitswei wrote:
       | I heard Clojure is fairly popular at Netflix as well.
        
         | jvican wrote:
         | Not true. Clojure use is very rare.
        
       | geodel wrote:
       | This seems entirely unsurprising/standard Java setup. Perhaps it
       | is proximity to Hollywood that some glamor is rubbed off on bog
       | standard enterprise tech stack of Netflix.
        
       | edejong wrote:
       | Interesting, no mention of Scala at all. Did Netflix say goodbye
       | to Scala altogether?
        
         | rickette wrote:
         | You're probably confusing twitter with netflix, the former
         | is/was a scala shop.
        
         | paulbakker wrote:
         | We never really used it, aside from some niche use cases. It's
         | always been Java primarily.
        
       | inparen wrote:
       | Spring Boot and Spring cloud for backend & graphql for the win.
       | ;-)
        
         | RamblingCTO wrote:
         | No, just no. Performance and debugging are just plain horrible.
         | The spring team loves to force you into their automagic shit
         | and this bean stuff is so annoying. You almost got no compile
         | time safety in this stack. It's the bane of my existence. I'd
         | like to know that a compiled program will run. That seems
         | virtually impossible with java/spring boot.
        
           | Fabricio20 wrote:
           | It's funny to see this perspective! I used to work in a few
           | companies locally who had adopted the early java-ee style for
           | their applications and my experience is exactly the opposite.
           | When going to spring I'm usually diagnosing issues on the
           | application layer (ie: business issues, not framework
           | issues), while on the java-ee applications I was often having
           | to fix issues down at the custom persistence layer each
           | company had, etc.. I see where you come from having looked at
           | the "old" spring stack (non -boot), and I can see people
           | getting mad over the configuration hell and how stuff is
           | hidden behind xml.. Much like how java-ee is!
        
             | chasd00 wrote:
             | Iirc spring was invented as a response to the j2ee madness.
        
             | bedobi wrote:
             | Spring and Java EE are not the only options out there.
             | These days DropWizard, Ktor and others are MUCH better
             | alternatives.
        
           | wing-_-nuts wrote:
           | What do you propose as an alternative? Something like
           | Micronaut trades more compile time for stricter checks and
           | faster runtime. Do you use something like that?
        
             | RamblingCTO wrote:
             | I implemented a microservice with micronaut starting 3,5
             | years back. The fragility of krush/exposed is not very fun
             | tho. we introduce another hurdle because we only use
             | Kotlin. I'm open for ideas though.
             | 
             | PS: we replaced/will replace all microservices with Go.
        
               | wing-_-nuts wrote:
               | Hah, yeah, we have a number of microservices implemented
               | with golang. As 'clunky' as the language can feel
               | sometimes the tooling is first class.
        
               | RamblingCTO wrote:
               | Absolutely! I love how fast it is and how easy it is to
               | write stuff real fast. Not a lot of footguns, except the
               | pointer-for-loop thing afaik. It does feel clunky and
               | quite manual sometimes!
        
               | kaba0 wrote:
               | The whole language is a footgun..
        
           | pylua wrote:
           | Spring is basically a standard in itself and it is easier to
           | hire people in it. It also normalizes large pieces of the
           | backend application so even though they are written by
           | different people they are similar.
           | 
           | Once you learn the annotation based configuration it also
           | saves a lot of time.
           | 
           | The performance is valid but it will only keep improving.
        
             | RamblingCTO wrote:
             | To be honest, over the almost 5 years we're into using it,
             | I spent more time debugging funky stuff due to that design
             | that it would've cost me to just do it by hand. By a big
             | margin.
        
               | loginatnine wrote:
               | As a Spring power user, you sound like you're ranting
               | against something you don't/didn't understand.
        
           | didntcheck wrote:
           | It's not quite as good as compile-time or type-based
           | guarantees, but IME configuration errors with Spring are
           | almost always flagged up immediately on application startup.
           | As long as you have at least one test that initializes the
           | application context (I.e. @SpringBootTest) then this should
           | be caught easily
        
           | bedobi wrote:
           | 100% agree, Java and Spring are a mess and there's no
           | justifiable reason to use them in 2023 (and no, "that's what
           | we've always used" isn't a good justification)
           | 
           | Like srsly even DropWizard is better than Spring lol, let
           | alone other even simpler frameworks like Ktor which is built
           | on a much improved language over Java
        
           | throwaway1492 wrote:
           | [flagged]
        
             | krooj wrote:
             | This is absolutely spot-on. The concepts that Johnson & co
             | coalesced around, mainly IoC, DI, interfaces, and layers,
             | are bedrock foundations for how most people write any
             | service-oriented system. Say what you want about the things
             | that Boot itself does, like auto configuration, or the Data
             | persistence abstractions, but the foundations of the
             | library are bedrock software engineering, so if you're
             | gonna dump on something that's proven over 20+ years, you
             | better come correct.
             | 
             | This "lingua franca" of Java development also allows you to
             | hire, onboard, and get people productive - rapidly.
        
             | bedobi wrote:
             | > It's plain this person lacks the experience and
             | perspective to appreciate the problems a stack such as
             | Spring solves
             | 
             | lol, having worked in Spring codebases for over a decade at
             | this point, I assure you those of us who dislike Spring
             | know exactly what problems it "solves" (which, btw, it
             | doesn't)
             | 
             | if you can't seriously acknowledge and understand the
             | mountains of criticism that has been leveled against
             | Spring, by experienced and novices alike, you're the one
             | who is arrogant and needs a reality check
             | 
             | but yeah it's typical of Spring fans to portray it as if
             | Spring is The Only Professional Option when in reality
             | there are much, much better, easier and simpler solutions
             | for the same problems out there, including eg DropWizard
             | and Ktor
        
             | geodel wrote:
             | "This person" makes rather relevant comment on Spring
             | boot's severe shortcomings.
             | 
             | Whereas your comment from throwaway account is blabbering
             | nonsense.
        
               | krooj wrote:
               | Their comments on the shortcomings boil down to, "I
               | didn't spend any time learning how this works."
        
               | throwaway1492 wrote:
               | Well that supports my point actually, that these "short
               | comings" aren't short comings at all if one takes the
               | time to learn the framework. Blabbering? People often say
               | that who have no valid technical or intellectual
               | response.
        
               | geodel wrote:
               | Why would framework be correct if the premise of heavy
               | runtime reflection based approach itself is bad?
               | 
               | You can spend your lifetime learning this _framework_ ,
               | others can simply reject on first principles' basis.
        
               | wiseowise wrote:
               | > Why would framework be correct if the premise of heavy
               | runtime reflection based approach itself is bad?
               | 
               | Because the framework allows zero reflection approach.
        
               | RamblingCTO wrote:
               | I don't see anything beyond ad hominem. That reveals more
               | about yourself than you probably think. I'll move on
               | then.
        
           | smrtinsert wrote:
           | Performance and debugging simple, and compile time safety is
           | Javas core domain. I think you're over focusing on proxying
           | or enhancement of beans, but if you look at a documentation
           | for a reasonable amount of time there's really nothing to it.
        
           | nameless912 wrote:
           | That has not been my experience on the inside - I spend most
           | of my days working on a Spring Boot based service at Netflix
           | and frankly it's one of the most effortless environments I've
           | ever worked in. Granted, there's a lot of ecosystem support
           | from the rest of the company, but things are very low effort,
           | and generally very predictable. I can usually drop a
           | breakpoint in a debugger in exactly the right spot and find a
           | problem immediately.
        
           | StevePerkins wrote:
           | I'm not sure what "no compile time safety in this stack" even
           | means in the context of a strongly-typed compiled language.
           | 
           | If you are referring to the dependency injection container
           | making use of reflection, then Spring Native graduated from
           | experimental add-on to part of the core framework some years
           | ago. You can now opt for Quarkus/Micronaut-style static
           | build-time dependency injection, and even AOT compilation to
           | Go-style native executables, if you're willing to trade off
           | the flexibility that comes with avoiding reflection. For
           | example, not being able to use any of the "@ConditionalOnXXX"
           | annotations to make your DI more dynamic.
           | 
           | (Personally, I don't believe that those trade-offs are worth
           | it in most cases. And I believe that all the Spring magic in
           | the universe doesn't amount to 10% of what Python brings to
           | the table in a minimal Django/Flask/FastAPI microservice. But
           | the option is there if your use case truly calls for it.)
           | 
           | Honestly, I've never run into anyone who considers Spring to
           | be "the bane of their existence", where the real issue wasn't
           | simply that the bulk of their experience was in something
           | else. Where they weren't thrown into someone else's project,
           | and resent working with decisions made by other people, but
           | don't want to either dig in and learn the tech or else search
           | for a new job where they get to make the choices on a
           | greenfield project.
        
             | RamblingCTO wrote:
             | > I'm not sure what "no compile time safety in this stack"
             | even means in the context of a strongly-typed compiled
             | language.
             | 
             | Easy: as I said down below, you can actually get wildly
             | different classes because of things like Hibernate proxies.
             | Also, you guessed it, the dependency injection part. I just
             | hate it. Nothing works together. We have so many hacks and
             | weird work-arounds because something doesn't work.
             | (Websockets not working with tomcat for example, or the
             | many funky troubles with using two modes of authentication
             | at once).
             | 
             | > Honestly, I've never run into anyone who considers Spring
             | to be "the bane of their existence", where the real issue
             | wasn't simply that the bulk of their experience was in
             | something else. Where they weren't thrown into someone
             | else's project, and resent working with decisions made by
             | other people, but don't want to either dig in and learn the
             | tech or else search for a new job where they get to make
             | the choices on a greenfield project.
             | 
             | In my previous startup, we used python and flask. Something
             | I don't deem scalable for bigger teams/apps. We love Kotlin
             | and Gradle (especially multi-modules). But there are so
             | many drawbacks that just suck time. I have a bunch of
             | private projects, all in Go. Fast and efficient as heck.
             | Nothing I'd like to scale beyond maybe 5 people or 20k loc
             | tho (no idea if bazel or something could help with that, no
             | experience). You get a lot of good stuff, but you gain in
             | fragility with Java/Kotlin.
             | 
             | Another point that just comes to mind is: how unsecure is
             | this thing even? Dependencies are ages old, requiring you
             | to litter your gradle build files with work arounds and
             | overwrites so you're secure from some (often critica)
             | vulns.
        
               | sublimefire wrote:
               | This sounds a bit negative but at the same time not
               | purely related to Spring but rather to the choices of
               | external dependencies. I've worked using Spring for years
               | and it makes me smile reading it.
               | 
               | Hibernate looks easy but the abstractions have a cost
               | associated with using and maintaining it. There are a lot
               | of settings you need to get right. There were dedicated
               | DBAs that would optimize it in the past. You could just
               | use JPA to make the life simpler.
               | 
               | Tomcat.. I mean why? It was great but I'd say it maybe
               | went out of pace compared to everything else. Why not
               | embedded Jetty. At this point I'm starting to have doubts
               | about how you deploy the services to begin with.
               | 
               | Dependency injection is actually great albeit the usual
               | problem is that you need to read books to understand it
               | as there is more than one way to do everything. My pain
               | point was usually related to the differences you need to
               | do among Java, Groovy, Kotlin but otherwise it is
               | awesome.
               | 
               | Flask is shite, basic, Python has a hard time solving its
               | mess with dependencies and its multithreading support is
               | meh. Go is great I love it. But if you want to create
               | enterprise software, Java ecosystem has you covered and
               | the engineers are cheaper to hire.
        
               | RamblingCTO wrote:
               | Yeah, I'm also heavily ranting. My nick is meant as a
               | hint haha. In the end and after quite a few years we're
               | almost at a point where we love to work with our stack,
               | minus the occasional weird funky thing happening because
               | of the ecosystem (which would never happen in something
               | like Go for example!). Hexagonal + Kotlin + Gradle multi-
               | modules is a beast!
               | 
               | > My pain point was usually related to the differences
               | you need to do among Java, Groovy, Kotlin but otherwise
               | it is awesome.
               | 
               | Yup, agreed! We got to run kotlin pretty nice and got to
               | play it together nicely with gradle convention plugins.
               | 
               | Agreed on Python. Was my first properly learned language
               | back in the days and my first backend lang. Absolutely
               | annoying. I also love go, but it lacks something like
               | gradle, especially multi-module support. I love it but
               | don't see it on a scale as our Kotlin codebase.
               | 
               | > engineers are cheaper to hire
               | 
               | I think there's also the benefit of an exotic language.
               | You might only get 10 CVs, but you could probably hire
               | half of them because only passionate people bother to
               | look into exotic things. With java you have a bigger
               | bandwidth ... which causes a lot of work.
        
               | StevePerkins wrote:
               | 1. Hibernate isn't even part of Spring. You're free to
               | use a newer ORM like jOOQ if you like, or Spring Data
               | JDBC if you want to get closer to the raw SQL. This gets
               | to what I was saying earlier... usually when I encounter
               | someone with strong opinions on Spring, they haven't
               | really dug in to learn much about it. They don't know
               | where "Java" ends and "Spring" begins, they're just
               | winging it and don't like that this doesn't work out well
               | with larger team projects (arguably with ANY stack).
               | 
               | 2. I don't care which programming language or framework
               | you are using. If you hate dependency injection as a
               | general pattern, then every alternative I've ever seen
               | boils down to either: (1) monkeypatching all over the
               | place to achieve the same goals, or else (2) just static
               | hardcoding everything and not writing unit tests with any
               | mocks. I mean, plenty of people utilize one of those
               | approaches. They just usually don't do so while
               | discussing safety and security with a straight face.
               | 
               | 2. You cite Python and Go as alternatives, yet
               | immediately acknowledge that they're unsuitable beyond
               | small teams or apps (my God, I'll take dependency issues
               | with Maven Central over PyPI any day of the week!).
               | Honestly, this whole sub-thread seems to boil down to you
               | just preferring to work on small codebases over large
               | codebases. And that's perfectly fine! I just don't think
               | that's language or framework-dependent.
        
               | RamblingCTO wrote:
               | You seem to read what you want to read. I was agreeing
               | with you on the benefits of Java/Spring while still
               | disliking it, citing my experience with go and python.
               | 
               | Your only argument is that if this stack makes problems,
               | you just haven't read enough and must be an idiot winging
               | it. The truth is that this whole thing, especially the
               | standard spring stack including hibernate, is just one
               | big footgun. You can't isolate any of these, as this
               | stack intertwines to achieve this level of footgun
               | concentration. Some faults are hibernate, some spring's,
               | some is due to DI, and some are due to java. And I'm fed
               | up with the level of trickery we need to have to work
               | with this without issues. I'd like to focus on the bugs
               | and shitty architecture I introduce instead of others.
               | 
               | > (1) monkeypatching all over the place to achieve the
               | same goals, or else (2) just static hardcoding everything
               | and not writing unit tests with any mocks. I mean, plenty
               | of people utilize one of those approaches. They just
               | usually don't do so while discussing safety and security
               | with a straight face.
               | 
               | These are both stupid things to do in general. If you do
               | proper hexagonal architecture you need neither of these
               | hacks.
        
               | StevePerkins wrote:
               | > _" I was agreeing with you on the benefits of
               | Java/Spring while still disliking it..."_
               | 
               | Very well. You seemed to saying that Spring was far less
               | attractive than some unnamed alternative, and I was
               | trying to say that I don't see any such alternative in
               | the same space. If you're basically saying, _" Yeah,
               | that's probably true, but I want to vent a little
               | anyway_", then fair enough.
               | 
               | I will however point out that Spring and Hibernate, etc
               | are NOT tightly-coupled. You absolutely do not have to
               | use any particular database persistence library with
               | Spring, or any particular message bus framework or
               | anything else. If you want to use jOOQ or MyBatis or raw
               | JDBC or anything else, go nuts. The fact that most people
               | gravitate toward certain de facto defaults doesn't make
               | those requirements whatsoever.
               | 
               | > _" If you do proper hexagonal architecture you need
               | neither of these hacks."_
               | 
               | I remember seeing that buzzword 15 or so years ago. From
               | a glance at Wikipedia, this seems to be from the "Agile
               | Manifesto" guy. It _looks_ like most or all of this has
               | been absorbed into 12-factor or microservices
               | architecture or what have you. Regardless, I 'd be
               | surprised if it truly obviates the need for unit tests.
        
               | RamblingCTO wrote:
               | > Yeah, that's probably true, but I want to vent a little
               | anyway
               | 
               | You got me!
               | 
               | > I will however point out that Spring and Hibernate, etc
               | are NOT tightly-coupled. You absolutely do not have to
               | use any particular database persistence library with
               | Spring, or any particular message bus framework or
               | anything else. If you want to use jOOQ or MyBatis or raw
               | JDBC or anything else, go nuts. The fact that most people
               | gravitate toward certain de facto defaults doesn't make
               | those requirements whatsoever.
               | 
               | With tight coupling I actually meant tight integration
               | and each bringing their own set of weirdness.
               | 
               | > Regardless, I'd be surprised if it truly obviates the
               | need for unit tests.
               | 
               | Nah, absolutely not. But you get proper separation of
               | concerns. You are almost doing it if you do spring. You
               | just need a lot of interfaces and separate those concerns
               | along specific lines. This gives you plain java domain
               | code, isolated code for each type of side effects (be it
               | web or files/database stuff) which you can test. You can
               | also replace components very fast and easy without big
               | refactorings.
               | 
               | My biggest pain with Spring/Hibernate/whatever is that it
               | permeates every layer, even if you don't want it to. I
               | brought up Hibernate because it took me ages, that
               | Hibernate proxies everything and spring security doesn't
               | understand that, doing funky things if you check proxied
               | objects (don't get me started on string SpEL ... worst
               | idea ever. Can't test shit and doesn't protect you from
               | typos). And it just happened sometimes. That sometimes
               | was when there was any lazy loading involved. Fun times!
               | 
               | With hexagonal (ports and adapters) you force yourself to
               | separate the concerns. We also enforce it via gradle
               | multi-modules and architecture tests. It works quite well
               | so far. No damn compromises because hibernate forces you
               | to have nullability in domain models where it wouldn't be
               | possible from a domain point. It's all in different
               | modules, hidden from everyone else. It also forces you to
               | write domain code in plain java (or in our case kotlin).
               | No funny side effects because of lazy loading, hibernate
               | proxying, jackson funny business or anything else. It's
               | been a bliss so far. We had to replace a lot of spring
               | boot magic auto stuff (like anything hikari, hibernate,
               | repos, models, DDLs, we need to do that manuall). But we
               | can actually properly unit test code knowing that it
               | can't fail a few layers up because of something missing.
               | We can properly use Kotlin with all of its niftyness with
               | sealed classes and so on. Really nice! Now we just need
               | to solve the bean issues with additional tests and the
               | typical gradle/java annoyances of throwing hurdles at you
               | just because and then we have a very nice stack we love
               | to use. And no footguns because we enforce a lot via
               | compile time safety checks. We try to design everything
               | so that it can't be used wrong. And best of all: a test
               | suite without every test using something with
               | transactions, a full application context or anything.
               | Plain domain code, plain Kotlin tests. It's a big step
               | up.
               | 
               | I can only recommend having a look into that. It's not
               | that far away from a Spring Boot app with a lot of
               | interfaces and Beans implementing that.
        
             | Tainnor wrote:
             | I've never used any other webstack (and I've used several,
             | in several languages) where it was so hard to figure out
             | why different tests pollute each other, causing subtle
             | failures depending on the order of test execution. Sure,
             | you could say that it's all a matter of learning the right
             | way, but there are just a terrible amount of footguns.
             | 
             | Debugging why something doesn't work in Spring can also be
             | a nightmare.
             | 
             | And the reference point for "compile-time safe framework"
             | is of course not Django, which is written in a dynamically
             | typed language...
        
               | RamblingCTO wrote:
               | > I've never used any other webstack (and I've used
               | several, in several languages) where it was so hard to
               | figure out why different tests pollute each other,
               | causing subtle failures depending on the order of test
               | execution. Sure, you could say that it's all a matter of
               | learning the right way, but there are just a terrible
               | amount of footguns.
               | 
               | > Debugging why something doesn't work in Spring can also
               | be a nightmare.
               | 
               | Making these points you can see if somebody actually used
               | Spring in production ;)
               | 
               | How much time we've lost with caching in tests and other
               | shennanigans. I've never had a stack causing more
               | problems by just being how it is than with Spring Boot.
               | Don't even get me started on the absolute clusterfuck of
               | hibernate. I'd love to move on from anything from this
               | stack and we're working on it! Absolute dumpster fire.
        
               | wiseowise wrote:
               | Where are you migrating?
        
               | RamblingCTO wrote:
               | No idea yet. But the first step is to have totally
               | isolated domain logic in plain Kotlin. That's a nice
               | start. Next we'll look into Graalvm, but no high hopes
               | that everything we need is compatible.
               | 
               | /edit: it's not meant to replace spring, I know what
               | these things do. It's to remediate some of the
               | performance issues and compile time safety, which using
               | graalvm inevitably brings ...
        
               | nvm0n2 wrote:
               | GraalVM and Spring and Hibernate are all totally
               | different things. You can't replace Spring with GraalVM,
               | that makes no sense.
               | 
               | You could look at Quarkus, Helidon, Micronaut or KTor if
               | you want a Kotlin framework.
        
               | StevePerkins wrote:
               | > _"...failures depending on the order of test execution.
               | "_
               | 
               | Every time I've seen issues where the order of test
               | execution matters, it's either been: (1) someone one
               | writing integration tests and calling them unit tests, or
               | (2) transaction management issues with H2 or some other
               | embedded test database.
               | 
               | For #1, you're going to have a bad time with any stack.
               | For #2, transaction management with Spring Data JPA is
               | admittedly a tricky subject to learn. However, most of
               | the time you can simply slap a "@Transactional"
               | annotation on unit test methods that mutate the database,
               | and that will cure what ails you.
               | 
               | > _" Debugging why something doesn't work in Spring can
               | also be a nightmare."_
               | 
               | I don't know what could make Spring more challenging to
               | troubleshoot than any other Java framework, _other than_
               | diagnosing issues with complex dependency injection
               | config. And that is often grossly overblown. Most of the
               | time, you don 't HAVE to use reflection-heavy tools like
               | "@Profile" or "@ConditionalOnXXX". Keep it simple and
               | there's really not much magic there. And IntelliJ or any
               | other professional-grade IDE can help to manage the magic
               | you do choose to employ.
               | 
               | > _" the reference point for "compile-time safe
               | framework" is of course not Django"_
               | 
               | I'm simply saying that in any "X Stack Considered
               | Harmful" discussion, eventually you have to put your
               | cards on the table and disclose which stack you are
               | comparing X to.
               | 
               | Everything is written in Rust here in the imaginary
               | fantasy world of HN. But back in the land of the real,
               | almost all line-of-business server side API development
               | is written in either Java/JVM, Python, .NET, Node, or
               | rarely some PHP or Ruby holdovers. Roughly in that order.
               | Occasional oddballs here or there using Go or something
               | else, but not common enough to make a dent and typically
               | very difficult to evangelise.
               | 
               | So with that palette available to me, I'm going paint
               | server-side microservices with Java and Spring virtually
               | every time. My default comparison is to Python simply
               | because it seems the greatest rival today in terms of
               | adoption. But if we're looking at strongly-typed compiled
               | languages only, in the business world in practice that
               | limits you to Java/JVM or C#/CLR. Which is like saying
               | that Coke is the bane of your existence and your company
               | should be drinking Pepsi.
        
               | [deleted]
        
               | RamblingCTO wrote:
               | > Every time I've seen issues where the order of test
               | execution matters, it's either been: (1) someone one
               | writing integration tests and calling them unit tests, or
               | (2) transaction management issues with H2 or some other
               | embedded test database.
               | 
               | Yeah, h2 is a very very bad idea. Hibernate and jpa are
               | also garbage (proxies, lazy loading and transactions are
               | just a big footgun). Have fun trying to use Sprint
               | security and proxied hibernate objects. Somehow you have
               | no guarantee that the class you receive and are checking
               | and is guaranteed by code is what you expect it to be
               | because of proxies. Absolute nightmare. We're using
               | testcontainers, which has their issues on its own.
               | 
               | > I don't know what could make Spring more challenging to
               | troubleshoot than any other Java framework > "@Profile"
               | or "@ConditionalOnXXX"
               | 
               | We have one configuration with @Profile to prevent it
               | from configuring stuff in tests. Our biggest issue is
               | that code that compiles is not guaranteed to run or even
               | crash when running (@Lazy anyone?). I hate this. It
               | caused us so much pain. A bigger issue really is the
               | automagic I was quoting. They just configure shit
               | together because they think it's nice. Also causes a lot
               | of headache. You have to enable/disable a random bunch of
               | shit until it somehow works. And after they released a
               | version fixing vulns that were open for > 1 year because
               | of old dependencies this vodoo breaks. (Especially the
               | garbage that is spring security. Still no migration guide
               | for ACl in v6 huh?)
               | 
               | Fun fact: not a fan of Rust. It's totally overblown for
               | web backends. I love me some Go, but having the ability
               | to use gradle multi-modules in a ports and adpaters
               | architecture is just plain awesome. Not sure where the
               | journey will take us, but Kotlin (which we use
               | exclusively) is a lot of fun. I'd love to have native
               | Kotlin at one point exclusively. No more failing builds
               | because of a wrong jdk or anything.
               | 
               | Just everything about this ecosystem is fragile af. Not
               | sure how we got to a point where shipping a simple Go bin
               | is easier than shipping java. Don't get me started on the
               | resource usage and performance.
        
               | wiseowise wrote:
               | > I'd love to have native Kotlin at one point
               | exclusively. No more failing builds because of a wrong
               | jdk or anything.
               | 
               | I'm sorry to disappoint you, but Kotlin compiler is
               | written in mix of Kotlin and Java which is being called
               | from Gradle.
               | 
               | > Just everything about this ecosystem is fragile af. Not
               | sure how we got to a point where shipping a simple Go bin
               | is easier than shipping java.
               | 
               | ./gradlew distZip and you're done.
               | 
               | > Don't get me started on the resource usage and
               | performance.
               | 
               | What about it?
        
             | ljm wrote:
             | I never hated Spring, I got used to working with it. I
             | found the annotation-based approach to configuring classes
             | really opaque though, and it was hard to understand how to
             | configure things because those were also magic annotations,
             | and then some things were beans and others weren't and...
             | ugh.
             | 
             | I don't fault Spring for it, and we were on Java 11 back
             | then with just a little bit of new hotness. Java itself
             | just didn't lend itself to the best ergonomics.
             | 
             | And you could fault Ruby or any dynamic language for the
             | same, but they usually save you a little bit of overhead or
             | boilerplate.
        
               | ackfoobar wrote:
               | I was once dropped into a Spring codebase. I consider
               | myself a decent programmer, but I couldn't navigate it at
               | all. Other than that time I've been able to avoid magical
               | frameworks in my career.
               | 
               | But I do want to learn Spring. What material would you
               | recommend?
        
               | krooj wrote:
               | Spring In Action. Not Spring Boot in Action - Spring.
               | https://www.manning.com/books/spring-in-action-fifth-
               | edition
        
               | cgh wrote:
               | Not to be glib or state the obvious but the Spring
               | project has excellent docs:
               | https://docs.spring.io/spring-
               | framework/reference/index.html
        
               | StevePerkins wrote:
               | I'm always interested in hearing people's alternatives.
               | Modern C# and ASP.NET is likewise annotation based. Most
               | contemporary Python frameworks heavily leverage
               | decorators, which is just another name for annotations.
               | Decorators have made their way into Typescript. Rust code
               | is chock full of attributes, and I've even seen Golang
               | libraries simulate annotations with the `reflect`
               | package.
               | 
               | This never really sounds like a Spring or Java thing. It
               | always sounds like a "not liking dependency-injection as
               | a general pattern" thing. My issue with that is two-fold:
               | (1) people should just say that, and not tie it
               | specifically to any one stack, and (2) the alternatives
               | seem to be either monkeypatching or else writing
               | untestable code, and those alternatives are hardly any
               | better.
        
               | ljm wrote:
               | I'm not saying annotations are bad though, I'm saying
               | Spring's over-reliance on them makes the code harder to
               | follow. Some of those annotations are actually done at
               | compile-time via Lombok, which makes it even harder to
               | know what the hell is happening.
               | 
               | Decorators and annotations are essentialy higher-order
               | functions and higher-order classes, but there is nothing
               | about them that makes it intuitive. They actually become
               | quite difficult to write and much harder to debug, OOP's
               | shit version of a monad.
        
             | deepsun wrote:
             | I like middle-ground dependency injection: do it, but only
             | during build time.
             | 
             | Dagger library does that (generates all the glue code
             | during pre-compile phase), so all the dependency-injection
             | already happened before runtime. Also, JVM has an easier
             | time reasoning/optimizing the code.
             | 
             | But I also found that writing the glue code by hand, same
             | way as Dagger would do, is not that hard IMO.
        
             | dkarl wrote:
             | I assume they mean problems with dependency injection.
             | 
             | If that's what they mean, I agree. I've seen dependency
             | injection frameworks used in a bunch of different
             | companies, and there are always people who consider it an
             | essential lifesaver, like they just can't imagine working
             | without it, and it always baffles me, because I've worked
             | on equally large codebases that didn't use it, and it was
             | occasionally a significant annoyance to pass dependencies
             | by hand, but never equal to the annoyance of dealing with
             | an automatic dependency injection framework.
             | 
             | This repeated experience of working with and without
             | dependency injection, finding dependency injection to be at
             | least as much hassle as it saves, and seeing that the
             | people I've worked with who choose dependency injection
             | have massively warped impressions of what it's like to work
             | without it (they often think it's, like, _impossible_ ) has
             | led me to see it as a tool that is driven mostly by FUD, at
             | least at the scale of code that I have worked with.
             | 
             | And that's without even considering the deleterious effect
             | that dependency injection has on design. In my cynical
             | moments I think this is the real reason people love
             | dependency injection. It's not that people hate the five
             | minutes it takes to figure out how to manually pass a
             | dependency to a module; it's that they hate the
             | subconscious thinking that happens in those five minutes,
             | as they see how the change affects the code, and it dawns
             | on them that it's a code smell for every module to depend
             | on everything else. Sometimes a dependency is a code smell,
             | and dependency injection means you barely get a whiff, so
             | faint you can pretend it's just your imagination. Doing it
             | by hand means you get a few minutes to bask in the stench.
             | You can't pretend you didn't notice.
             | 
             | Getting people to care about modularity, coupling, and
             | cohesiveness in an application with dependency injection is
             | markedly harder, just like it's harder in a language with
             | global variables, just like it's harder to get people to
             | think about APIs and modularity in a monolith than in
             | microservices. And for me that's the worst part of working
             | on codebases with pervasive dependency injection!
             | Dependency injection is a massive liberating force for
             | people who want to work without thinking about design.
             | Instead of thinking about it, they just add another
             | spaghetti dependency and keep on going, and do the same
             | thing tomorrow, and the next day, and the next day. It's
             | impossible to stop them! Adding a dependency is so
             | immediate, so easy, there's no moment where they have to
             | stop and think, "Hmmm, why am I using a dependency in
             | module A that was only ever used in module B before? I'll
             | need to instantiate it at the application level instead.
             | But that means I need to pull some internal logic out of B
             | so it can be run before B exists. Should I really be doing
             | this? I'd better think/ask about this before I do it."
             | Instead they just add an annotation and see if the
             | dependency injection framework can figure it out. That's
             | the only kind of problem I can think of that dependency
             | injection excels at solving: problems that should never
             | have been solved in the first place.
        
               | StevePerkins wrote:
               | > _"...it was occasionally a significant annoyance to
               | pass dependencies by hand... "_
               | 
               | Your comments are confusing on whether you eschew a
               | dependency injection framework, or eschew dependency
               | injection as a pattern in general.
               | 
               | If it's the former, then you are certainly free to roll
               | your own DI rather than leveraging a library such as
               | Spring, Quarkus, Micronaut, Guice, Dagger, or any of the
               | others out there. I strongly doubt that you'll implement
               | something better than any of those. However, having
               | written the code yourself you'll be more likely to
               | understand it all, and that can be attractive for many
               | people.
               | 
               | If it's the latter, then it's harder to take the position
               | seriously. Look, the Java ecosystem certainly suffered
               | from extreme OO over-engineering in the late-1990's and
               | early-2000's, no doubt about it. A lot of people went too
               | far overboard with enterprise-y design patterns, for
               | sure. But that pendulum has been swinging back for over a
               | decade now... and in the 2020's, the point of using DI is
               | not to go crazy with the enterprise design patterns. The
               | point is write code that can support an actual unit test
               | suite!
               | 
               | You use DI so that your test suites can inject mocks or
               | stubs, to isolate external integrations from the code
               | under test. I don't care which language or framework you
               | use, or whether you're using a 3rd-party DI library or
               | implementing the approach manually yourself. If you're
               | not using DI to accomplish this in your tests, then
               | you're either: (1) using monkeypatching to achieve the
               | same goal in a more dangerous manner, or (2) just writing
               | un-testable code, and perhaps writing some integration
               | tests that you pretend are unit tests.
        
               | dkarl wrote:
               | I think you're blurring some lines when you present a
               | choice between using a DI framework, rolling your own
               | framework, or not using "dependency injection as a
               | pattern in general." "Dependency injection as a pattern
               | in general" is unavoidable in complex software and
               | undoubtedly a necessary thing in many contexts, whereas
               | dependency injection _frameworks_ are tools that are not
               | necessary for using  "dependency injection as a pattern."
               | 
               | My preferred approach to dependency injection is using
               | the plain mechanisms of the language I'm using to pass
               | dependencies as constructor and function parameters,
               | without any framework.
               | 
               | I pass test fakes the same way. When doing things this
               | way feels burdensome due to overly complex method
               | signatures or overly complex initialization code, I look
               | for ways to improve my design.
               | 
               | This works at every scale of code I've worked at. It's
               | possible there's some size of project that it doesn't
               | scale to, but for every project I've worked on where
               | another engineer swore that dependency injection was
               | absolutely necessary on the project, I've worked on a
               | larger project where nobody ever suggested dependency
               | injection and everything was fine.
               | 
               | My experience doesn't prove that dependency injection
               | isn't necessary at some larger scale that I haven't
               | experienced, of course, but it does convince me that the
               | industry is rife with programmers who think they can't
               | get by without it at scales where it is not only
               | unnecessary but probably harmful.
        
           | dimgl wrote:
           | I completely agree with this. Spring was an absolute
           | nightmare during the short period of time where I had the
           | misfortune of using it. It also didn't help that the codebase
           | was a monstrosity... classes following no design patterns and
           | having 40k lines. But still...
        
             | RamblingCTO wrote:
             | Agreed! Have fun reading spring source code or
             | documentation to find anything. An obscure stackoverflow
             | answer from 10 years back will hold the answer though. Fun
             | times
        
           | vmaurin wrote:
           | The issue with Spring ecosystem is that people use it without
           | knowing why or which problem it solves but because almost
           | everyone is using it. And most of the time, they don't need
           | Spring (maybe a company like Netflix did, but it didn't prove
           | to be the right choice at the end)
        
           | twh270 wrote:
           | We've adopted Quarkus and it's been a breath of fresh air.
           | Excellent all around, DX, performance, features, it's all
           | been good.
        
             | RamblingCTO wrote:
             | Have you had the chance to compare it to Micronaut?
        
             | m_0x wrote:
             | In Quarkus, can you inject different beans depending on
             | configuration?
        
               | michens wrote:
               | Yes, either @IfBuildProfile/@DefaultBean if your logic
               | depends on build profile, or
               | @LookupIfProperty/@LookupUnlessProperty if it depends on
               | property value
        
           | krooj wrote:
           | This is just... ignorance; your argument is basically, "I
           | don't understand/want to learn how X works; therefore, X must
           | be garbage"
        
             | RamblingCTO wrote:
             | If you read more of my responses you'll see that I've been
             | a Spring user for quite some time and know how it works. I
             | just dislike how it works. Do you have any argumentation or
             | did you just want to call me ignorant?
        
           | misja111 wrote:
           | FYI, you can still use XML based configuration in Spring. The
           | choice is yours. See https://docs.spring.io/spring-
           | framework/docs/4.2.x/spring-fr...
           | 
           | I agree it is not common to do it, most teams follow the
           | autoconfiguration madness.
        
         | krooj wrote:
         | The stack is tremendously productive, but history has taught me
         | a few things when dealing with Spring:
         | 
         | 1. It's always best to start people off with plain old spring,
         | even with an XML context, such that they understand the
         | concepts at play with higher level abstractions like Boot.
         | Hell, I even start with a servlet and singletons to elucidate
         | the shortcomings of rolling your own. 2. Don't fall prey to
         | hype around new projects in the Spring ecosystem, such as their
         | OAuth2 implementation, since they often become abandonware.
         | It's always best to take a wait and see approach 3. Spring
         | Security is/was terrible to read, understand, and extend ;)
        
           | inparen wrote:
           | Ha ha, spring security is tricky and high chance may surprise
           | some one while "boot"strapping a new project. But once done,
           | it is out of way.
           | 
           | I did not like much of the XML, because it always seemed lot
           | of duplication. All you doing is copying bean definitions and
           | changing bean id and class/interface most of the time. But it
           | became non issue over time. Now spring boot made it really
           | easy with all those annotations.
        
         | Cthulhu_ wrote:
         | Spring is a safe and reliable choice I'd say; not the most
         | exciting, but neither code nor frameworks _should_ be exciting,
         | they 're used to solve a problem, they shouldn't become the
         | problem itself.
         | 
         | GraphQL is interesting to me, I thought the clients were pretty
         | similar across all platforms, meaning their API usage should
         | also be similar enough to not need the flexible nature of
         | GraphQL. But then, it allows for a lot more flexibility and
         | decoupling - if a client needs an extra field, the API contract
         | does not need to be updated, and not all clients need to be
         | updated at once. Not all clients _will_ be updated either, they
         | will need to support 5-10+ year old clients that haven 't
         | updated yet for whichever reason.
        
           | robertlagrant wrote:
           | Well, if the field is not available then new backend code
           | will need to be written, resolvers, integrations, etc. But it
           | does allow UIs to take less info over the wire, and
           | eitherfewer joins need to be done or fewer performance-
           | oriented APIs need building, as you say.
        
           | m_0x wrote:
           | > not the most exciting
           | 
           | It was exciting when J2EE was dominating.
        
         | olavgg wrote:
         | I am a big fan of Spring Boot, its one of the few frameworks
         | that just works and let me focus 100% on solving business
         | problems. I've tried Micronaut, Quarkus, Dropwizard, but they
         | slow me down too much compared to just using Spring Boot.
         | 
         | For me delivering business value is the most important metric
         | when I am comparing frameworks. Spring Boot wins every time.
        
         | baby wrote:
         | I had to review a Spring application once and that convinced me
         | never to work with Java ever again
        
       | yafetn wrote:
       | Netflix's DGS framework for GraphQL is nice to work with but
       | we've been frustrated with some prioritization choices by the
       | team. For instance, if you're using Kotlin, it's impossible to
       | define and pass scalars to the latest version of the client.
       | There's a year-old issue highlighting this problem that's been
       | ignored it seems.
       | 
       | https://github.com/Netflix/dgs-codegen/issues/455
        
       | oldnet wrote:
       | [flagged]
        
         | stanac wrote:
         | How is java being used on server side affecting your client?
        
           | oldnet wrote:
           | It's constantly sending requests from BE to FE - checking IP,
           | checking watch time etc.
        
             | robertlagrant wrote:
             | That is nothing to do with Java.
        
             | williamdclt wrote:
             | That's entirely unrelated to the backend language (and
             | mostly unrelated to CPU usage for that matter).
        
               | oldnet wrote:
               | do you know javascript? it can eat CPU
        
               | 0x6c6f6c wrote:
               | Java and JavaScript have no relation at all.
        
               | oldnet wrote:
               | How you will send request to FE and back if you have java
               | in BE?
        
               | Capricorn2481 wrote:
               | What did you mean when you wrote this? You can send a
               | request to any backend server, regardless of language.
        
               | jkern wrote:
               | I'm not trying to be harsh, but it genuinely sounds like
               | you have no idea what you're talking about
        
               | kaba0 wrote:
               | You can be harsh -- it is pretty dumb to come here
               | shitting on a whole platform, when they don't even know
               | the basics.
        
               | oldnet wrote:
               | Tell me ... why it's only netflix what makes CPU go on
               | max? Why netflix is giving me info message when I turn on
               | VPN in the middle of movie? Why it's only netflix who
               | stop working another second when my internet connection
               | has quick failure?
               | 
               | Sorry but it seems that you have no idea how netflix
               | really works.
        
               | kaba0 wrote:
               | Do you even know what a frontend/client is vs a backend?
               | 
               | The frontend is anything but java (the website uses
               | javascript, they have mobile apps, not sure what they are
               | written in, but besides android having a few necessary
               | wrappers, these are also not java applications).
               | 
               | You don't see the backend and they do different things
               | between different services so not much point in comparing
               | them - that's a huge, complex, partially cached-at-your-
               | ISP even network infrastructure.
               | 
               | Besides, Java itself is very fast and is itself no reason
               | for an applications' (especially network-related) bad
               | performance. It literally runs like half of the internet,
               | with basically every other top 100 IT companies having
               | business-critical infrastructure dependent on it.
        
               | oldnet wrote:
               | BE doesn't need constant info about user if BE would be
               | correctly written.
               | 
               | I never saw BE which sends so much requests to FE for
               | user info. Someone just has no idea how to use Java
               | without making CPU wild.
        
               | frederikb wrote:
               | The point that the others are trying to make is that
               | these questions are not relevant in the context of the
               | backend technology choices. The REST or GraphQL endpoints
               | could be handled by monkeys writing bits via typewriters
               | and it wouldn't affect the CPU usage in your browser.
        
             | Cthulhu_ wrote:
             | That shouldn't take that much CPU tbh, but if it does, it
             | sounds like they have some optimization to do.
             | 
             | I wonder if it's like DRM in video games, causing a lot of
             | overhead.
        
               | HeckFeck wrote:
               | Makes sense. The most gifted coders wouldn't work in DRM
               | and telemetry. At least I hope that's what is happening.
        
               | oldnet wrote:
               | Maybe.
               | 
               | I watch streams on youtube, twitch etc and everything is
               | fine.
               | 
               | However whole CPU is flying to the moon if I will watch
               | something on netflix.
               | 
               | Btw Chrome was pure a nightmare for CPU but Brave is
               | better.
        
         | ejlxsh wrote:
         | What's your CPU usage for Netflix? What device/OS are you
         | using?
        
       | [deleted]
        
       | talent_deprived wrote:
       | When they say "applications" do they mean stuff with more meat
       | than microservices? If they're mostly microservices, 2800 seems
       | low to me for someone with the recognition factor of Netflix.
        
       | ValtteriL wrote:
       | >Netflix observed a 20% increase of CPU usage on JDK 17 compared
       | to JDK 8. This was mostly due to the improvements in the G1
       | garbage collector.
       | 
       | Help me here, why do GC improvements cause CPU increase?
        
         | znpy wrote:
         | > Help me here, why do GC improvements cause CPU increase?
         | 
         | In Java 8 (afaik) there were pretty much no generational or
         | concurrent garbage collectors, so garbage collector would
         | happen in a stop-the-world manner: all work gets put on a halt,
         | garbage collection happens, then the work can resume.
         | 
         | If you have a better GC, you have shorter and less frequent
         | needs to do a stop the world pause.
         | 
         | Hence the code can run on cpu for more time, getting you higher
         | cpu usage.
         | 
         | Higher cpu usage is often actually good in situations like
         | this: it means you're getting more work done with the same
         | cpu/memory configuration.
        
           | dboreham wrote:
           | Java8 was at least a decade into generational and concurrent
           | GC. It does STW once in a while though which may be what you
           | meant.
        
             | _old_dude_ wrote:
             | You have two kind of concurrent GCs, the ones where the
             | marking phase in concurrent with the application and the
             | ones where the evacuation phase is also concurrent with the
             | application.
             | 
             | G1 only does the marking concurrently, the evacuation is
             | done in small pauses. A decade ago, there was only one
             | concurrent evacuation GC available in Java, C4 from Azul.
             | Now, we have Shenandoah and ZGC.
        
         | edpichler wrote:
         | To free memory. Also, 20% increase is not 20% in total. It's
         | 20% when you go from 10 to 12 cpu usage, or from 50 to 60, for
         | instance.
        
           | _the_inflator wrote:
           | Well done.
           | 
           | I always appreciate numbers and the differentiation between
           | relative and absolute numbers in this case.
           | 
           | "We doubled our workforce in one week!" - CEO's first hire...
           | ;)
        
         | jjtheblunt wrote:
         | I haven't seen the specific profiling data, but it's possible
         | that the garbage collector is running a collection thread,
         | concurrently with regular processing threads, and thereby
         | preventing entire world synchronization points which would idle
         | processor cores.
        
         | radomir_cernoch wrote:
         | Also interested! We saw basically the exact opposite. :-)
        
         | tpm wrote:
         | I read it as a good thing: GC improvements -> more available
         | memory -> more work done by the CPU. But still would be
         | interested in more detail.
        
         | pyeri wrote:
         | It's like hiring more workers to accomplish the exact same
         | output as before. "See, I achieved 20% growth in my targets!",
         | some recruiter will say!
        
           | groestl wrote:
           | No, it's like improving a form to minimize the need for
           | follow-up questions to the customer, and now seeing your
           | workers (the same you had before) processing 20% more forms
           | instead of waiting for responses.
        
         | paulbakker wrote:
         | It's a 20% improvement. So less time spent on GC.
        
           | [deleted]
        
         | ahoka wrote:
         | Higher CPU usage paradoxically means better performance. When I
         | last did OPS we used to watch total CPU usage of all services
         | and if it was not 100%, then we started to look for a
         | bottleneck to fix.
        
         | blackoil wrote:
         | I think this is a 20% improved utilization of CPU, earlier app
         | was memory-bound or/and GC was consuming CPU. Now app has 20%
         | more CPU available. It should be doing correspondingly more
         | work. This could definitely be written clearly.
        
           | moffkalast wrote:
           | > Bakker provided a retrospective of their JDK 17 upgrade
           | that provided performance benefits, especially since they
           | were running JDK 8 as recently as this year. Netflix observed
           | a 20% increase of CPU usage
           | 
           | Seems like it's exactly that, OP cropped out the relevant bit
           | where they list it having an overall performance benefit for
           | that extra CPU time. Otherwise it could be assumed that it
           | just hogs more CPU to get the same result.
        
           | [deleted]
        
           | bunderbunder wrote:
           | I haven't dealt with this side of Java in a while, but it
           | reflects my experience poking at Java 8 performance. At some
           | (surprisingly early) point you'd hit a performance wall due
           | to saturating the memory bus.
           | 
           | A new GC could alleviate this by either going easier on the
           | memory itself, or by doing allocations in a way that achieves
           | better locality of reference.
        
         | matsemann wrote:
         | The CPU can do more tasks without being limited by memory
         | pressure, perhaps?
         | 
         | I guess it depends on if they mean "we used 20% more CPU for
         | the same output", or "we could utilize the CPUs 20% more".
        
         | groestl wrote:
         | Because the memory / I/O is not the bottleneck anymore, and the
         | CPU can now run optimally.
        
         | Macha wrote:
         | A somewhat common problem is to be limited by the throughput of
         | CPU heavy tasks while the OS reports lower than expected CPU
         | usage. A lot of companies/teams just kind of handwave it away
         | as "hyperthreading is weird", and allocate more machines.
         | Actual causes might be poor cache usage causing programs to
         | wait on data to be loaded from memory, which depending on the
         | CPU metrics you use, may not show as CPU busy time.
         | 
         | For companies at much smaller scale than netflix where employee
         | time is relatively more costly than computer time, this might
         | even be the right decision. So you might end up with 20 servers
         | at 50% usage, but using 10 servers will take twice as long but
         | still appear to be at 50% usage.
         | 
         | If the bottlenecks and overhead are reduced such that it's able
         | to make more full use of the CPU, you might be able to reduce
         | to e.g. 15 machines at 75% CPU usage. Consequently the
         | increased CPU usage represents more efficient use of resources.
        
           | CraigJPerry wrote:
           | >> while the OS reports lower than expected CPU usage
           | 
           | >> which depending on the CPU metrics you use, may not show
           | as CPU busy time
           | 
           | If your userspace process is waiting on memory (be that
           | cache, or RAM) then you'll show as CPU busy when you look in
           | top or whatever - even though if you look under the covers
           | such as via perf counters, you'll see a lack of instructions
           | executed.
           | 
           | The CPU is busy in this case and the OS won't context switch
           | to another task, your stalled process will be treated as
           | running by the OS. At the hardware thread level then it will
           | hopefully use the opportunity to run another thread thanks to
           | hyper threading but at the OS level your process will show
           | user space cpu bound. You'll have to look at perf counters to
           | see what's actually happening.
           | 
           | >> you might end up with 20 servers at 50% usage, but using
           | 10 servers will take twice as long but still appear to be at
           | 50% usage.
           | 
           | Queue theory is fascinating, the latency change when dropping
           | to half the servers may not be just a doubling. It depends on
           | queue arrival rate and processing time but the results can be
           | wild, like 10x worse.
        
           | xorcist wrote:
           | When you put it like that, yes. Hardware is cheap and all
           | that. In practice I think that an organization that doesn't
           | understand the software it is developing has a people
           | problem. And people problems generally can't be solved with
           | hardware.
           | 
           | If somebody knows how to make that insight actionable, let me
           | know. No, hiring new people is not the answer. In all
           | likelihood that swaps one hard problem for an even harder.
        
             | toast0 wrote:
             | IMHO, Usually the people problem is that there are too many
             | people working on the same machine. Sometimes that's
             | unavoidable.
             | 
             | Sometimes, honestly, understanding the software its
             | developing isn't an important business goal. It makes me
             | personally angry, but most businesses do right by not
             | picking business goals to placate me.
             | 
             | Sometimes you just have too many people.
             | 
             | Sometimes you can restructure your software and systems so
             | that fewer people are working on a system and they can
             | understand it better. Sometimes that would also involve
             | restructuring your organization, which has pluses and
             | minuses.
             | 
             | If you can ensure the smaller teams run similar stacks,
             | there can be some good knowledge transfer when one team
             | figures out an underlying truth about the platform that
             | could apply elsewhere. And sometimes you get a platform
             | expert team that can help with understanding and problem
             | solving throughout the teams.
        
         | jillesvangurp wrote:
         | Most modern GCs trade off CPU usage and latency. Less latency
         | means the CPU has to do more work on e.g. a separate thread to
         | figure out what can be garbage collected. JDK 8 wouldn't have
         | had the G1 collector (I think, or at least a really old version
         | of that) and they would have probably been using one of the now
         | deprecated garbage collectors that would be collecting less
         | often but have a more open ended stop the world phase. It used
         | to be that this would require careful tuning and could get out
         | of hand and start taking seconds.
         | 
         | The new ZGC uses more CPU but it provides some hard guarantees
         | that it won't block for more than a certain amount of
         | milliseconds. And it supports much larger heap sizes. More CPU
         | sounds worse than it is because you wouldn't want to run your
         | application servers anywhere near 100% CPU typically anyway.
         | So, there is a bit of wiggle room. Also, if your garbage
         | collector is struggling, it's probably because you are nearly
         | running out of memory. So, more memory is the solution in that
         | case.
        
           | ahoka wrote:
           | I don't think he meant that.
        
           | BinaryRage wrote:
           | The figure is about the overall _improvement_ , not sure why
           | that reads increase.
           | 
           | On JDK 8 we are using G1 for our modern application stack,
           | and we saw a reduction in CPU utilisation with the upgrade
           | with few exceptions (saw what I believe is our first
           | regression today: a busy wait in ForkJoinPool with parallel
           | streams; fixed in 19 and later it seems).
           | 
           | G1 has seen the greatest improvement from 8 to 17 compared to
           | its counterparts, and you also see reduced allocation rates
           | due to compact strings (20-30%), so that reduces GC total
           | time.
           | 
           | It's a virtuous cycle for the GRPC services doing the heavy
           | lifting: reduced pauses means reduced tail latencies, fewer
           | server cancellations and client hedging and retries. So
           | improvements to application throughput reduce RPS, and
           | further reduce required capacity over and above the CPU
           | utilisation reduction due to efficiency improvements.
           | 
           | JDK 21 is a much more modest improvement upgrading from 17,
           | perhaps 3%. Virtual threads are incredibly impressive work,
           | and despite having an already highly asynchronous/non-
           | blocking stack, expect to see many benefits. Generational ZGC
           | is fantastic, but losing compressed oops (it requires 64-bit
           | pointers) is about a 20% memory penalty. Haven't yet done a
           | head to head with Genshen. We already have some JDK 21 in
           | production, including a very large DGS service.
        
             | algo_trader wrote:
             | > Virtual threads are incredibly impressive work,
             | 
             | Do you have an (un)informed opinion on minimum task sizes
             | for the green threads?
             | 
             | My interest is re-factoring java code to reduce total wall
             | clock time, on large compute with plenty of memory/cache.
        
             | pron wrote:
             | > G1 has seen the greatest improvement from 8 to 17
             | 
             | Yep. G1 in newer JDKs is very different from G1 in JDK 8,
             | but Parallel GC has also seen very significant
             | improvements: https://kstefanj.github.io/2021/11/24/gc-
             | progress-8-17.html
        
       | coding123 wrote:
       | Every company that went down the grpc route will be doing hacks
       | for the next 10 years until they eventually get rid of it.
        
         | kuratkull wrote:
         | Please provide a comprehensive substantiation for you comment.
        
       | talent_deprived wrote:
       | It's too bad they are using Gradle and Intellij, used both
       | before, went back to Maven and Eclipse. Personal preference I
       | guess.
        
         | ryanianian wrote:
         | IntelliJ is far and away above any other IDE and is well worth
         | the paid license imho. It's a professional tool written by and
         | for professionals. VSCode and Eclipse have some inertia in very
         | particular workflows/tools and a few different (valid) ways of
         | operating, but nothing is as polished and cohesive as the
         | JetBrains.
         | 
         | Gradle, however, is a dumpster-fire of footguns and obtuse and
         | non-debug-able DSLs.
        
       | zeruch wrote:
       | Early on (15+ years ago) I spent a few weeks there on contract
       | and I noticed they used Java EVERYWHERE, and not always well.
       | They had a CS app named after a key Star Wars character that was
       | in all likelihood a breach of the Geneva Convention. A code
       | atrocity with the performance of a sloth on its 8th bong rip with
       | a UX from hell.
        
         | sillywalk wrote:
         | "A code atrocity with the performance of a sloth on its 8th
         | bong rip with a UX from hell."
         | 
         | Sounds like Apple Music.
        
           | fmntf wrote:
           | I was thinking of Jira
        
             | favorited wrote:
             | ServiceNow, for me.
        
             | MenhirMike wrote:
             | Gotta be SharePoint.
        
           | hbn wrote:
           | For some reason whenever I'm on my work's VPN, Apple Music
           | lets me play 1 album and then the next time I try to start a
           | song it will tell me I'm not logged in and I'll have to force
           | quit and relaunch (frequently a few times) before it will let
           | me play another album.
           | 
           | Apple Music is the only app that has this problem.
        
             | m-s wrote:
             | Apple uses certificate pinning for some of its domains,
             | which doesn't play nicely with corporate firewalls.
             | 
             | https://support.apple.com/en-us/HT210060
        
         | bruh2 wrote:
         | What does CS stand for here? I guess it's not computer science?
         | 
         | Also, that description made me lmao, thanks
        
           | jedberg wrote:
           | Customer service.
        
           | khalilravanna wrote:
           | Another reminder that acronyms are pretty terrible for
           | communication. Every time I onboard with a new org there's a
           | whole new set of acronyms to learn that's barely faster than
           | typing out the unabbreviated version. Nice to save a couple
           | seconds when the cost is only a bunch of people not able to
           | follow along when people are communicating.
           | 
           | To be clear: not ragging on OP in particular at all but more
           | at the widespread practice at a company level.
        
             | d35007 wrote:
             | > Another reminder that acronyms are pretty terrible for
             | communication. Every time I onboard with a new org there's
             | a whole new set of acronyms to learn that's barely faster
             | than typing out the unabbreviated version
             | 
             | My (cynical) take on the matter is that software engineers
             | want to prove how smart they are to other software
             | engineers, and they think using obscure jargon and
             | abbreviations is a good way to do it.
        
               | khalilravanna wrote:
               | I'm sure that's some part of it. I think another part,
               | kind of similar, is acronyms being used as a form of in-
               | group/out-group behavior. "I know the lingo so I'm in the
               | club. You don't so you aren't"
        
           | zeruch wrote:
           | CS = Customer Support/Care in that regard.
        
         | jedberg wrote:
         | If it helps that CS app was rewritten about 10 years ago (when
         | I worked there, but not on that app) in part due to the
         | complaints you mention. It's totally true that most resources
         | were spent on customer facing apps. Internal apps were
         | definitely not of the same quality, because they didn't need to
         | be.
        
           | zeruch wrote:
           | Good to hear. The fact that in 2005 you had an app that
           | required seemingly petabytes of memory to operate, and put on
           | machines barely powerful enough to play minesweeper, was in
           | and of itself a series of bad decisions...but the app itself,
           | and it's layout were just maddening. It's like MC Escher was
           | the UX lead.
        
         | myvoiceismypass wrote:
         | When I was there a decade ago, it started becoming more
         | polyglot friendly (node apps had to use a jvm sidecar to do
         | internal communications originally!)
        
           | jedberg wrote:
           | My team wrote some of the Python libraries for internal
           | services just so we could avoid that Java sidecar! It took 10
           | times longer to boot the sidecar than the Python app.
        
         | Arrath wrote:
         | Evocative description there, bravo
        
           | civilitty wrote:
           | It's so evocative that I don't even care about Netflix or
           | Java anymore.
           | 
           | I just want to know where I can buy a bong ripping sloth [1]
           | and whether they're legal in California.
           | 
           | [1] https://imgur.com/a/S3NVS16
        
       | smrtinsert wrote:
       | Not surprised about Rx. Rx is great at the UI layer imho, or
       | anything with streams. For microservices, I don't see how it
       | would have ever fit, since microservices should be as simple as
       | possible doing just one thing.
        
         | yCombLinks wrote:
         | Netflix created RXjava
        
           | smrtinsert wrote:
           | created or ported? I thought it was created elsewhere.
        
             | leosanchez wrote:
             | RX.Net was the original I think for .NET ?
             | 
             | I might be wrong
        
             | krooj wrote:
             | The concepts are ported from other actor-based systems,
             | like Akka.
        
             | yCombLinks wrote:
             | Right, RX was first created by Microsoft. Netflix
             | implemented it in java.
        
       | dewey wrote:
       | In case you are wondering what LOLOMO stands for, it's "List of
       | List of Movies".
        
       | madaxe_again wrote:
       | Ah, the way they break out artwork calls explains the weird
       | behaviour I see with my U.K. Netflix account in Portugal -
       | English titles, Portuguese posters, regardless of language
       | preferences.
        
         | yurishimo wrote:
         | Yes because you're being served the Portugal catalog. English
         | is simply a localization setting that can be applied to any
         | region.
        
           | dewey wrote:
           | Then this must be an edge case with UK, as within the EU you
           | would get your "home catalog":
           | https://europa.eu/youreurope/citizens/consumers/internet-
           | tel...
        
         | dewey wrote:
         | How does the artwork explain that? Wouldn't they just need to
         | call the artwork service with your current language preference
         | instead of the default language of your current geolocation?
        
           | giraffe_lady wrote:
           | Internationalization vs localization in the wild.
        
       | [deleted]
        
       | agilob wrote:
       | Is this the talk? Looks like this is it
       | https://www.youtube.com/watch?v=5dpLVvRpPPs
        
         | paulbakker wrote:
         | The an older version of the same talk. Things have moved a bit
         | since, Java 21 and such, but mostly the same.
        
         | edpichler wrote:
         | Apparently, it is.
        
       ___________________________________________________________________
       (page generated 2023-10-10 23:02 UTC)