[HN Gopher] Why Discord is switching from Go to Rust (2020)
___________________________________________________________________
Why Discord is switching from Go to Rust (2020)
Author : harporoeder
Score : 181 points
Date : 2021-02-22 17:32 UTC (5 hours ago)
(HTM) web link (blog.discord.com)
(TXT) w3m dump (blog.discord.com)
| jeffbee wrote:
| Feels like there might have been a middle ground short of a
| scratch rewrite. LRU caches are always suspect. LRU is pretty
| much the worst replacement policy you can choose. I wonder if a
| smaller cache with a better replacement policy (2Q, ARC, ...)
| would have solved this specific problem.
|
| Another approach I've used with caches in Go is to just use a
| huge byte slice as the cache without any pointers, so the GC can
| ignore it.
| Jtsummers wrote:
| > Another approach I've used with caches in Go is to just use a
| huge byte slice as the cache without any pointers, so the GC
| can ignore it.
|
| Just to check that I understand what you're suggesting here,
| this is comparable to how we embedded systems folks might pre-
| allocate (either statically or at the very start of a program
| execution) a large chunk of memory and use it instead of using
| traditional malloc/free (or language equivalent), correct? So
| you bypass the GC and avoid any potential performance penalties
| introduced by having a smaller number of massive allocations
| that never get moved, freed, or reallocated.
| jeffbee wrote:
| Yes, the idea is similar. If you make a huge map in Go with
| millions of items, it will stress the GC. If you take the
| same information and just put it into a big byte slice, and
| index into it in a way that the GC doesn't think you're
| pointing to addresses, then it will be much faster. Of course
| now you are writing a little more code.
| tschellenbach wrote:
| 2021 will be about them switching back. All jokes aside, it's
| really common for engineers to randomly switch between solutions
| incurring huge costs for the companies they work for. Often as it
| is in this case with no gain for the company, other than some
| resume building for the involved team members. We use Go here at
| Stream and power activity feeds and chat for a billion end users.
| Works perfectly.
|
| Cockroach also wrote about GC optimization:
| https://www.cockroachlabs.com/blog/how-to-optimize-garbage-c...
|
| For caching specifically there is this library as well:
| https://dgraph.io/blog/post/introducing-ristretto-high-perf-...
|
| Nothing wrong with Rust, exciting project, but personally think
| Go is better for building performant micro services like this.
| Rust seems more suited for building browsers, drivers etc. Rust's
| performance is slightly better, but it's not as refined around go
| routines and developer productivity.
|
| Edit: On the other hand, Discord could have made the very valid
| decision that they have an easier time recruiting for Rust than
| for Go. Which is in fact a good reason to switch.
| jhgg wrote:
| Not a chance. We are going hard on rust in 2021 after some very
| successful projects in 2019 and 2020. our engineers have ramped
| up on the language - and we have good support internally (both
| in terms of tools, but also knowledge) to see its success.
|
| Once you've passed the learning curve - imo, rust is far easier
| and more productive to write than go - especially if you know
| how to leverage the type system to build idiomatic code and
| apis that are very hard to use incorrectly. Every piece of rust
| code we have shipped to production so far has gone perfectly
| thanks to the really powerful compile time checks and
| guarantees of the language. I can't say the same for our
| experiences with go.
|
| Our reasons go well beyond "oh the gc in go has given us
| problems" but more like "go as a language has willingly ignored
| and has rejected advances in programming languages". You can
| pry those algebraic data types, enums, borrow checker, and
| compile time memory management/safety, etc... from my cold dead
| hands.
|
| And for network programming - I've been incredibly delighted
| with the power and flexibility of rusts futures combined with
| tokio. The fact that you can just drop a future to cancel it,
| as opposed to having to thread around a ctx which may or may
| not work has been amazing.
| tschellenbach wrote:
| Those reasons make much more sense than the GC focus of the
| blogpost. Definitely a lot of nice features in Rust.
| phabora wrote:
| The GC was apparently the reason behind rewriting that
| particular service. No wonder there are more reasons when
| you look at all the services across the company.
|
| But you couldn't help yourself even after being corrected
| by someone who knows what they're talking about.
| jimbokun wrote:
| When I read descriptions like yours, Rust sounds like the
| language delivering on the promises of Haskell in the Real
| World.
| marcosdumay wrote:
| Some promises of Haskell.
|
| Rust is still a very low level language, but if you like
| Haskell, you won't go into anything else when writing some
| low level program.
| xiphias2 wrote:
| Haskell overpromised a lot.
|
| When I looked at it, the favourite quicksort example was
| so elegant, I wanted to learn the language. Then after I
| started looking at the run time (after wasting a lot of
| time going through learning Haskell), I realized that it
| was all a lie, ant that sort wasn't quick at all.
|
| I don't see the same overpromising problems with Rust:
| you usually get the performance that you think you will
| get when you're writing the code.
| travv0 wrote:
| So let me get this straight: You don't like Haskell
| because someone used it to write an inefficient quicksort
| example online?
| xiphias2 wrote:
| Not somebody, it was on the home page as a use case of
| how much better Haskell is than other languages.
|
| The default libraries are extremely inefficient.
|
| Generally if I use Python/Ruby I can get a 50x slow down
| that is accepted, but with Haskell it can be more, as
| it's very hard to think about what the compiler does
| (memory usage can blow up easily).
| ngrilly wrote:
| > rust is far easier
|
| That's an unusual claim :) Could you elaborate? How is the
| learning curve, especially for fresh graduates who have been
| trained only on languages like C, Java and Python?
|
| > The fact that you can just drop a future to cancel it, as
| opposed to having to thread around a ctx which may or may not
| work has been amazing.
|
| Agreed. I like Go, but that's big drawback of goroutines and
| threads.
| chayleaf wrote:
| In my experience, Rust puts many costs upfront, which makes
| it harder to do stuff when you aren't used to doing it the
| Rust way, but makes it easier to _get it right_ if you are
| used to Rust. It basically gives you a relatively strong
| guarantee that if your code compiles and has no warnings,
| it 's mostly collect - yes, that includes logic errors to
| some extent thanks to the type system. That also makes
| refactoring easier - chances are, the bugs you possibly
| added by refactoring will get caught by the compiler.
| Obviously, that's not a complete replacement for testing,
| but it does reduce the amount of tests you need to write to
| ensure correctness.
|
| Frankly, I don't consider myself a particularly good
| programmer - but when I use Rust, I feel confident because
| of being backed by the compiler. That's something I can't
| say about any other language, and I only have less than a
| year of Rust experience compared to 5 years of total
| experience programming.
|
| Imo Rust is _easier to master_ than other languages, but it
| 's definitely harder to do basic stuff in it because of the
| upfront costs I mentioned.
| xiphias2 wrote:
| How would a fresh graduate be able to write safe C code
| when even experienced people are unable to?
|
| I have worked with multi-threaded C++ code long enough to
| see that it's impossible to work in a team while keeping
| the code safe. Usually servers crash and canaries catch it
| in production, but if we have type system improvements that
| solve these problem, why not take advantage of it?
| marcosdumay wrote:
| A language being easy to write has no relation at all with
| it being easy to learn.
| infogulch wrote:
| I've noticed this kind of reply on rust threads often.
| Someone says something like:
|
| > Rust moves many potential program problems from runtime
| to compile time; it has a high bar to meet, but once you
| meet it the program is rock solid
|
| And then somebody replies with:
|
| > Oh my favorite language doesn't make me do all that work
| up front, therefore easier (TM). Why Rust so hard?
|
| But the reply misses the point: Make no mistake, you're
| still paying the cost (with FASTCASHNOW-interest rates),
| but the cost is just smeared out over days or weeks as you
| fix, debug, and tune it during use. You may choose whether
| to do that work up front or while the program is in active
| use, but avoiding the work is not a choice.
| Thaxll wrote:
| So there is also no point keep using Elixir then.
| jhgg wrote:
| We will continue to use beam/otp where it makes sense. In
| fact some rust runs in BEAM now thanks to Rustler. We wrote
| a blog about this!
|
| https://blog.discord.com/using-rust-to-scale-elixir-
| for-11-m...
| Thaxll wrote:
| You said that Go doesn't have: algebraic data types,
| enums, borrow checker, and compile time memory
| management/safety which Erlang / Elixir is even worse in
| some of those because it's a dynamic typed language so
| it's a bit confusing.
| jhgg wrote:
| I think you're missing the point. We use Elixir to solve
| one class of problems which its excellent in doing
| (notably: real time distributed systems.) In this area,
| there is nothing even remotely close in terms of
| capability. So, we choose the runtime over the language
| (elixir is a pretty cool dynamic language though!). Also,
| due to the unique nature of Erlang/Elixir, a lot of
| problems that exist in C/C++/Rust/Go don't exist in that
| language. Of course this has meant writing a lot more
| tests to test things that a typechecker should be able to
| catch. But it is what it is. We're following Gleam though
| to see where that ends up.
|
| BTW, here's some places I've spoken about Discord's
| Elixir usage:
|
| - a podcast: https://smartlogic.io/podcast/elixir-
| wizards/s5e8-nowack-hei...
|
| - interview with Jose Valim (elixir's creator):
| https://elixir-lang.org/blog/2020/10/08/real-time-
| communicat...
| c-cube wrote:
| I think a difference is that Elixir is deliberately
| dynamically typed, so you don't expect type safety. You
| write the happy path and rely on "fail fast" and the
| actor hierarchies to write robust code.
|
| Besides, Elixir/Erlang has the best dynamically-typed
| equivalent of algebraic types (tuples + atoms + pattern
| matching) :-)
| [deleted]
| aaomidi wrote:
| I agree, rust is so much easier once you've passed that
| learning curve.
|
| Not having to worry if this random reference you're using has
| been overwritten somewhere else in the code is absolutely
| refreshing.
|
| It allows me to bring back the mental map of my code and the
| variables to the exact scope I'm writing in.
|
| This is an underrated feature of rust. Especially for newer
| devs who do not have that mental map of the code yet, and are
| worried that any small change they make might cause cascading
| failures. That's just something you don't think about in
| Rust.
| phabora wrote:
| > Edit: On the other hand, Discord could have made the very
| valid decision that they have an easier time recruiting for
| Rust than for Go. Which is in fact a good reason to switch.
|
| In other words, discard the technical arguments laid out in the
| article and just speculate that it was an HR/management
| decision.
| 5evOX5hTZ9mYa9E wrote:
| > resume building
|
| We call this project farming where I am. I'm not a dev and
| don't work in IT industry, but I do work for a large company
| and every few months some new project is rolled out to
| 'improve' things and most of the time we all collectively groan
| and wait for the person to get promoted.
|
| The most frustrating part is when once in a blue moon the
| project turns out to be useful, and the person gets promoted
| off of it and abandons the project. Nobody then wants to pick
| it up since they won't get any credit for running it and it
| ends up getting abandoned.
| franklampard wrote:
| > Often as it is in this case with no gain for the company,
| other than some resume building for the involved team members
|
| This is too cynical. A large re-write will not be approved at
| any large companies without millions of meetings and
| documentations filled with justifications.
| berkes wrote:
| ... All of which are researched with proof of concepts.
| Followed by some small trials. And finalized with one or two
| actual rewrites of non-critical software.
|
| I'm not saying that all companies do it this way, and that
| there is no "resume building" or "magpie-ing" by some
| engineers in many companies. But it is perfectly possible to
| state a thesis "Rust is better for our X and Y and probably
| for Z". and to then research and prove that thesis.
| Sikul wrote:
| Original author here. Cool to see it posted again.
|
| We're still using and investing in rust at Discord. For example,
| in the past year we successfully pulled part of the core
| messaging system out of python and into a rust service. Rust has
| been a boon for developer productivity and correctness of code.
|
| We've also been very happy with the way the rust ecosystem is
| evolving and the amount of adoption the language has seen.
|
| I'm happy to answer questions here.
|
| P.S. We're still growing and hiring (including remotely). You can
| find my email in my profile.
| lifeplusplus wrote:
| is it true the GC problems in GO have been fixed since then?
| Sikul wrote:
| rsc said they made some big improvements for this particular
| type of problem. I'm not sure which version of go the
| improvements landed in.
| xtracto wrote:
| What happened with your Elixir stuff? (
| https://blog.discord.com/scaling-elixir-f9b8e1e7c29b ). Is it
| still there or are you using something else now?
| Sikul wrote:
| Elixir is without a doubt still one of the core languages. We
| try to use the right tool for the job and Elixir is the right
| tool for parts of our system.
|
| We also combo Elixir NIFs and rust to speed up parts of the
| Elixir services https://blog.discord.com/using-rust-to-scale-
| elixir-for-11-m...
| xtracto wrote:
| Interesting. We are researching Elixir for an online
| transaction ledger (thousands of payments per second) along
| with kafka. So far thats the technology that seems more
| suitable. That discord article was really appreciated to
| understand some of the intricacies of using Elixir.
| fbernier wrote:
| I certainly know of a few Adtech companies using Erlang
| to handle > 1M bid requests per seconds :). OTP is pretty
| great for that.
| ufmace wrote:
| This post is getting a bit long in the tooth, at least regarding
| relatively new and fast-moving ecosystems like Go and Rust. I
| wonder if the Go team has determined any way to mitigate these
| long-running GC pauses for apps that don't allocate often since
| this article was published.
|
| I also wonder how well Rust has continued to work for Discord, if
| it's been a problem to keep up with the rapid movement
| particularly in the async ecosystem in the last year or so.
| aw1621107 wrote:
| > I wonder if the Go team has determined any way to mitigate
| these long-running GC pauses for apps that don't allocate often
| since this article was published.
|
| Go 1.12 might have addressed this particular issue [0, 1], but
| it's possible that that came out too late for Discord to try
| before deciding to rewrite the service.
|
| [0]: From Go 1.12 patchnotes: "Go 1.12 significantly improves
| the performance of sweeping when a large fraction of the heap
| remains live."
|
| [1]:
| https://old.reddit.com/r/golang/comments/eywx4q/why_discord_...
| cfors wrote:
| Sounds like Discord was running gigantic in memory Go caches.
|
| Not that it's the wrong approach, but this is a very real-time
| system-y problem that Rust is designed for. So I understand why
| this made sense.
|
| Obviously rewriting in Rust worked for them, but I'm curious if
| they tried the Ballast[0] approach, along with smaller cache's
| per service. That would make the GC scan on the cache smaller
| while still avoiding some of the Go startup heap issues. I'm not
| sure if the target heap size ever was accepted by the Go team,
| but it may have been easier than a full rewrite.
|
| [0] https://blog.twitch.tv/en/2019/04/10/go-memory-ballast-
| how-i...
| [deleted]
| masklinn wrote:
| > I'm curious if they tried the Ballast[0] approach
|
| From the essay, the issue discord had was that they were _not
| triggering enough GCs_. The ballast approach serves to tune the
| GC frequency down.
|
| In Discord's case they were getting screwed by a periodic
| mandatory GC trigger.
|
| > along with smaller cache's per service
|
| They did use a smaller cache, but that increased their 99%
| since... there was now less stuff in the cache.
| cfors wrote:
| Makes me wonder what their partitioning strategy was. I can't
| imagine a cache that is over 4GB+ that couldn't be
| partitioned at least a bit further. But good call out, thank
| you for the correction.
| kyrra wrote:
| There's a proposal to put in official support for minimum heap
| size: https://github.com/golang/go/issues/44309
| vonsydov wrote:
| tl;dr because of garbage collection
| rufusroflpunch wrote:
| These threads are always full of people criticizing these moves
| by companies. But nothing is gained of nothing is risked, right?
| Sometimes these things pay off, and we would never know without
| early adopters.
| dvfjsdhgfv wrote:
| In other words, they solved the problem they wouldn't have if
| they had stuck to C in the first place.
| [deleted]
| Twisol wrote:
| We may never know for sure, but I bet they avoided a whole
| bunch of problems by using Go instead of C. Probably wise to go
| lower only when needs must.
| strangelove026 wrote:
| I really enjoyed reading this article last time it was posted. I
| particularly like it because it shows evidence that current
| tooling wasn't doing the job. I believe I remember seeing some
| comments on how Discord was on an old version of go that wasn't
| receiving the benefit around gc collections, but we'll never know
| if that would've been a solution.
|
| That all said, I've noticed that people here are really off
| golang in favor of rust. I believe I sort of understand. I've
| recently switched from python to go at work, and I frequently
| bring up why I like it so much (caveat I'm overly enthusiastic
| about the things I like related to my work). But again, that all
| said, at the end of the day, both Python and golang will
| definitely get the job done.
| kbar13 wrote:
| i'm curious if allocating a memory ballast would have helped
| discord in this case?
|
| a one line patch in main() with something like
|
| ballast := make([]byte, 10<<30)
|
| might help their GC spikes, since garbage collection would only
| trigger when the heap size reaches 20GB, at which point the heap
| will be reduced back down to 10GB or so.
|
| https://blog.twitch.tv/en/2019/04/10/go-memory-ballast-how-i...
| aw1621107 wrote:
| The GC cycles were triggered not due to heap size but due to
| the forced 2 minute max time between GC cycles [0], so a
| ballast wouldn't have helped here.
|
| [0]:
| https://github.com/golang/go/blob/0f66fb7b856b02497cf801ce72...
| ttymck wrote:
| This is really interesting. Are there not alternative
| runtimes, or configurable gc periods?
|
| My understanding of the JVM GC is limited, but it seems
| highly configurable: does it (or other languages/VMs) have an
| analogous construct, and is it (usually) configurable?
| masklinn wrote:
| The Go core team intensely dislikes tunables, so Go has
| very few of these (really the opposite of the JVM).
|
| There is a central issue for GC tunables, but it's mostly
| where the various concern GC have gone to die so far:
| https://github.com/golang/go/issues/42430
| masklinn wrote:
| > i'm curious if allocating a memory ballast would have helped
| discord in this case?
|
| No. Twitch's ballast is because
|
| > At steady state, our application was triggering ~8-10 garbage
| collection (GC) cycles a second (400-600 per minute).
|
| Go's GC triggers a collection every time the heap size doubles
| (until that has no return on investment and it just grows the
| heap). By creating a very large heap up-front they'd give
| themselves breathing room to avoid being just on the edge of
| (but remaining below) a resize, leading to continuous
| collections keeping them below.
|
| But Discord had the opposite issue: a large cache means a large
| heap size, in a steady state that would be extremely stable
| (since the heap size is a function of cache size, once the
| cache is full the only changes are replacing old keys for
| news). As a result, the only GC would be the mandatory periodic
| run, which would have to go through the entire (very large)
| heap, causing STW latency spikes every 2mn.
| mraza007 wrote:
| I'm just curious why can't they use C or C++ instead of Rust
| nilkn wrote:
| Introducing C or C++ into an org with no expertise in them
| feels much riskier than introducing Rust. All three are
| difficult languages but at least with Rust you have Cargo and
| memory safety by default. I've seen smart folks burn days just
| setting up dependency resolution and builds for C++ stacks.
| Experts don't have that problem but a team that has never used
| it professionally very well might.
| unethical_ban wrote:
| I think you're being downvoted for pushing a narrative (somehow
| C langs are better than Rust) without providing your opinion or
| evidence. It baits people into defending Rust without you doing
| any work to defend your position.
| DC1350 wrote:
| It doesn't look as good on a resume. There's thousands of
| people with decades of experience in C or C++ that you need to
| compete with for clout. It's easier to invent your own problems
| so you get to be the best man for the job.
| cletus wrote:
| Compile time memory safety?
|
| Unless you have an existing code base or a necessary
| integration or requirement that demands it, I honestly wouldn't
| start a C/C++ project in 2021 all other things being equal.
| jeffbee wrote:
| Because nobody would upvote that on HN. The HN zeitgeist of
| programming languages is absurdly biased. In actual industrial
| practice the dominant languages are Java, C++, C, C#, Python,
| Javascript, dozens of things people rarely discuss like matlab,
| VB, delphi, a bunch of other shit, and way, way down at the
| bottom, Rust, and then about fifty other languages, and finally
| Erlang. It's ten times easier to find a job writing MIPS
| assembly code than to find one programming in Rust, but nobody
| wants to blog about that.
| jacksnipe wrote:
| In actual industrial practice, most YOUNG C++ developers I
| know would love to be able to work with Rust at work.
| jeffbee wrote:
| There's a lot of territory between what one wishes to
| happen at work, and what actually happens at work.
| jacksnipe wrote:
| Right, I'm just saying that the HN zeitgeist is -- at
| least in this case -- reflecting a real belief. It's what
| people want to work with, not what currently is run
| everywhere in production; and what developers want to
| work with definitely matters.
|
| Discord rewrote this service in Rust because they had
| already had success using Rust for some performance
| critical services; it seems likely that they chose to
| invest in Rust and not C++ for those services because
| their developers pushed harder for or were more
| interested in Rust.
| jeffbee wrote:
| Exactly, which is why my original responses to this year-
| old blog post was "Why did we switch to Rust: because we
| wanted to". The blog post should focus on those social
| aspects, not invent tenuous technical back-
| justifications.
| jacksnipe wrote:
| I don't think C++ is a compelling choice over Rust if you don't
| already have a lot of C++ experience in your organization. C++
| is way harder to learn than Rust, IMO, not because of any set
| of language features or footguns, but because the ecosystem,
| tooling, etc... is just so different from what most "modern"
| devs have come to expect.
|
| C just doesn't offer the same feature set at all.
| mraza007 wrote:
| I see but C++ is widely used and you can easily find C++ devs
| compared to Rust.
|
| This brings me to my next questions C++ has been there for a
| long time compared to Rust but still people decide to go with
| Rust
|
| Just curious to know
| jacksnipe wrote:
| It seems like Discord wasn't about to hire new developers
| for this, they wanted to use their existing talent. In that
| case, what matters is how easy a new language is to learn,
| not hire for.
|
| Also, _good_ C++ devs are decidedly _not_ easy to come by.
| Especially when you get into performance sensitive work,
| they can command huge salaries; and a bad C++ dev can do
| more damage to a codebase than any other language I know
| of.
| mraza007 wrote:
| Got it, You explained it really well !!
| [deleted]
| sgt wrote:
| I wouldn't mind looking into Rust to eventually replace our Java
| and Python based stacks, but every time I look at it, I tend to
| give some thought to the difficulty of writing Rust as opposed to
| reading/writing/maintaining Java for an entire team. Frankly I'm
| worried that not all team members will become good Rust
| developers, and the codebase will develop serious rot over time.
| Any experiences with this?
| l2cluster wrote:
| We switched Ruby -> Scala, and it took considerable training
| effort and some years before everyone was comfortable. I would
| estimate 15% of the developers still can't write moderately
| good Scala code.
| johnnycerberus wrote:
| I think you should be worried instead of the amount of
| libraries that you give up from the JVM and Python ecosystems
| before anything else. Also, the applications of Rust and
| Java/Python generally do not overlap, I would be cautious in
| approaching this transition. Scala would be a better choice
| since you have expertise on the JVM and it has a familiar
| syntax to Python devs.
| ris wrote:
| > our Java and Python based stacks
|
| I always find it bizarre when python (and to some extent java)
| are lumped in as languages that you found Go or Rust a suitable
| replacement for. They're just such different ends of the
| spectrum - Go/Rust will just _never_ be able to match the
| expressivity and rapid development ease of python because they
| have such different design goals.
|
| If you find porting a production application from python to
| Go/Rust a satisfactory outcome, then I suspect you should never
| have been using python for it in the first place or you've been
| using python All Wrong.
| geodel wrote:
| Ah Discord, that provides electron based chat client. I am sure
| they can write efficient software when its their money.
| kemenaran wrote:
| To be fair, they probably ship one of the best and most
| optimized Electron app out there.
|
| I still don't like Electron, but Discord is doing a good job
| with it. Even their React-Native mobile apps feel decently
| snappy (where usually large React-Native apps are buggy and
| slow as fuck).
| saagarjha wrote:
| That's pretty debatable.
| KuhlMensch wrote:
| Note: this is an account of using Rust in a single service
|
| > The service we switched from Go to Rust is the "Read States"
| service.
|
| And the conclusion doesn't indicate "switching", but rather Rust
| is an "option" for Discord, and already used in the stack:
|
| > When starting a new project or software component, we consider
| using Rust. Of course, we only use it where it makes sense.
| AlchemistCamp wrote:
| In its early days, this forum was full of excited young devs who
| had read PG's old essays and we're actively _looking_ for
| technologies that would give them a competitive edge.
|
| Now, it's completely the opposite.
|
| Now, when a company successfully leverages technologies as a
| competitive edge and becomes a multi-billion dollar success, the
| thread is dominated by stodgy criticism for not picking the
| absolute most mainstream technologies possible.
| dang wrote:
| Discussed at the time:
|
| _Why Discord is switching from Go to Rust_ -
| https://news.ycombinator.com/item?id=22238335 - Feb 2020 (640
| comments)
|
| also https://news.ycombinator.com/item?id=23874336 - July 2020 (7
| comments)
| lrae wrote:
| (Feb 4, 2020)
| reikonomusha wrote:
| Meta-comment, and I'm _not_ claiming this is what's true about
| Discord.
|
| Every time I see a company post about how they're changing
| programming languages, I tend to believe it _actually_ happened
| because engineers with enough political clout decided they wanted
| everybody to change to their preferred language, not truly
| because of technical reasons, but because they just wanted to be
| paid to write in a cool language. But, of course, you'd be
| lambasted by your colleagues if you _didn't_ provide technical
| reasons, so you go and hunt for technical reasons that seem
| insurmountable and build a case for it, turning a blind eye to
| perhaps less exciting solutions that don't require a programming
| language change.
|
| (It's sort of like how a cop in America can find you doing
| something illegal in any circumstance if they look hard enough.
| Except in this case, it's a matter of finding technical
| deficiencies.)
|
| This strategy highly depends on clout though, which is why I
| think technical arguments are frequently really a facade. If,
| say, a junior programmer doesn't have clout, even presenting an
| equivalent technical argument won't lead to a change in
| programming language, especially if it's the senior's favorite.
| I'm sorry to say, but most tech organizations aren't meritocratic
| or egalitarian in this way (and perhaps they shouldn't be).
|
| As it pertains to the article--which again I don't claim is all
| "fake" politicking (I don't know anything about or anyone at
| Discord)--they mostly ripped on garbage collection. They
| specifically talked about how GC was bad because their LRU cache
| had to be scanned, which was part of why they observed spikes in
| their metrics. I wonder why they didn't try doing things like
| putting the LRU cache into foreign, unmanaged memory to avoid
| garbage collecting it. Easier said than done, but why not?
|
| I think using _any_ language with any serious and large project
| will find deficiencies in the language. At some point in a
| product lifecycle it's more profitable to just work with and
| ameliorate those deficiencies than do rewrites where the _new_
| deficiencies will be completely unknown.
| silentsea90 wrote:
| +1 going from meme language to meme language seems highly
| suspicious. Go and Rust are fine languages, but given these are
| relatively new and immature, I find it startling that Discord
| had to do writes/rewrites so often. I'd wager that if Discord
| had started sooner than 2015 (say, 2013ish) - we'd have seen
| Scala thrown into the fray as well.
| sacado2 wrote:
| Go is starting to get a bit old though. Version 1.0 dates
| back from 2012, 9 years ago. It's as old and mature as Java
| was in 2005.
| johnnycerberus wrote:
| > we'd have seen Scala thrown into the fray as well.
|
| Scala has already found its niche in Big Data, streaming and
| distributed systems. Usually, the stack that you can find in
| my area, consist mostly of Java codebases, sprinkled with
| some Scala services that are purely data-oriented. There's a
| high probability that Discord is already using Scala.
| mraza007 wrote:
| Yup this makes me wonder why do companies go for relatively
| newer languages compared to old ones such as Java,C++ or C as
| they have been matured and has been widely used
| Laremere wrote:
| Go was announced in 2009, and went 1.0 in 2012. Sure, it's
| "newer" than languages from the 90s, but calling it "new"
| when it's nearly a decade old doesn't pass the smell test.
| Serious software has been written in both Go and Rust, and
| real issues with Java, C++, and C that these newer
| languages address are well known.
|
| Compare Go and C++, and I bet C++ has had more language
| changes since Go's 1.0 than Go has. That makes large
| portions of C++ "relatively new and immature" compared to
| Go.
| mraza007 wrote:
| I agree with you on that but this still makes me question
| the problem they are trying to solve can't they solve in
| Go instead of switching to Rust.
|
| Furthermore, A lot of companies to this date have there
| performance critical code written in C++ , C or Java
|
| Such as you can take HFT for example they use C++ for
| most of their trading applications instead of newer
| languages
|
| I'm definitely not saying C++,C or Java is better than
| Rust or Golang each language has its own pros and cons
| devmor wrote:
| Not all problems are specifically technical in nature.
|
| One of the big issues with Go is that it's got a steep
| learning curve. (There was an article about this posted
| on HN yesterday, in fact)
|
| Rust, by contrast, is an extremely easy language to
| learn. Of all the languages I've picked up at least
| partially in my ~12 years of professional programming,
| Rust was probably second only to to PHP in "get up and
| start running with it".
|
| If Discord is trying to pick up new developers, or cross
| discipline seniors to improve their talent pool, Rust is
| definitely a better choice than Go.
| snth wrote:
| This is the reverse of the characterization of the two
| languages I usually see. I usually see "Go is easy to
| learn and use" and "Rust is hard to learn but has
| performance advantages". I often see comments about how
| you "really need to go read the Rust book" and not just
| try to pick up Rust on the fly.
| steveklabnik wrote:
| Not to discount your experience, because I think that
| this topic has a lot of nuance and is very different on
| an individual level, but most people regard Go as easy to
| learn, and Rust as difficult to learn.
|
| I wish we had a good way to evaluate this question in a
| quantitative way, but I haven't seen one yet. So
| reasonable people are going to disagree on the answers
| here.
| domano wrote:
| This is the first time i have seen this argument. Just
| understanding ownership seems more complex to me than
| writing a simple web app in go or PHP for that matter.
|
| I am sure rust is great but it is at the very least more
| difficult to learn then go.
| tjalfi wrote:
| > I agree with you on that but this still makes me
| question the problem they are trying to solve can't they
| solve in Go instead of switching to Rust.
|
| High performance systems in GC languages often
| preallocate or use object pools to reduce time spent in
| the garbage collector. LMAX's Disruptor, for example,
| allocates all memory at startup; no additional memory is
| allocated so the garbage collector never runs.
| Generational GCs make this strategy more effective
| because long-lived or large objects are traced rarely or
| not at all. Another option for trading systems is to run
| garbage collection after hours when the market has
| closed.
|
| Go's GC is conservative, not generational, and has a hard
| coded collection interval of 2 minutes; you pay for a GC
| even if no memory has been allocated. There are only a
| few configuration options for Go's GC; if it doesn't work
| for your use case then you can either fork or use a
| different language.
| jimbokun wrote:
| They did solve it in Go, then got performance
| improvements re-implementing a small piece in Rust.
|
| This is more about scale and saving computing costs, than
| proving out a business idea or implementing new features.
| bgorman wrote:
| Have you ever considered that programming language design
| is an evolving field, and programming languages directly
| influence programmer productivity?
|
| C++, C and Java are old, so some mistakes can be forgiven.
| However there are problems with these languages that new
| languages have addressed.
|
| Do you think a car from 1985 is better than one from 2015?
| The same applies to programming languages. Given the
| explosion of programming, it is absurd that the best ideas
| from PLT are not in mainstream languages.
| doodpants wrote:
| The simplest answer is that these newer languages were
| created specifically to address problems with existing
| languages. If a new language has no compelling reason for
| anyone to switch to it, why bother to create it in the
| first place?
| johannes1234321 wrote:
| It's not that easy. A new language starts because a set
| of developers has a problem, however that doesn't mean
| that others share that problem or agree to the choices
| and solutions made. Rust seems to be in a good situation
| where this worked for large parts, but many languages
| came and got no or only short attention.
| chc wrote:
| There are actually very good reasons to go with Rust over
| those languages, which were the motivating reasons Rust was
| developed in the first place. Rust supports lower-level
| development than Java, offers much stronger safety and
| correctness guarantees than C or C++. This is why Google's
| security team also supports reimplementing C libraries in
| Rust (https://security.googleblog.com/2021/02/mitigating-
| memory-sa...).
| wrenky wrote:
| Easier to hire people working in the new hot languages, and
| get younger programmers out of college.
| johannes1234321 wrote:
| I don't think so. There is a full load of people with
| experience in C++ or Java, but people with experience in
| performance critical (whatever that means) Rust code is
| more limited for being a newer language. You however win
| on the hipness scale and attention in places like hacker
| news.
|
| As others have said, however, a big factor certainly is
| that an internal lead developer lobbied for Rust, which
| they experimented with.
| ModernMech wrote:
| At the same time, in college we are teaching our students
| Java and C++ because that's what companies expect our
| students to know. Half our CS curriculum is in Java. The
| students I teach loathe C and C++ and want to learn Rust,
| but we won't teach it to them except in an experimental
| elective that won't count toward their graduation. So
| it's hard to say how many young programmers out of
| college actually know these "new hot languages".
| mraza007 wrote:
| I totally agree with you on this comment since as I
| college graduate I learned Java , C++ ,C and python but I
| was exposed to Golang and Rust through online discussion
| forums.
|
| Plus most of the organizations still write code in
| matured languages such as C,C++ and Java.
| aaomidi wrote:
| Because excited developers generally means more
| satisfaction from work. And one way to keep developers
| excited is to give them new toys to play with.
|
| Also mature languages, even though they're more stable, are
| just more of a pain to work with since they don't have a
| lot of the new shortcuts.
|
| For something like Discord, Rust is perfect.
| mraza007 wrote:
| I agree with you, developers do get excited when are
| using newer technologies but the question is what if a
| new problem arises and they might not be able to find a
| to it solution.
|
| Just like how they did in this case completely switching
| from Go to Rust
| aaomidi wrote:
| Then they'll have to cross that bridge when they come to
| it. Nothing of this article gave me the impression that
| the second they saw this bug they're like WELL TIME TO
| SWITCH.
|
| I'm sure we're glossing over probably hundreds of hours
| of investigations by dozens of people.
| jhgg wrote:
| We kept dealing with GC pauses in go, and eventually
| after investing a bunch of time trying to fix this - we
| decided to research what it'd be like to rewrite it in
| another language.
|
| The service itself is pretty simple, so it wasn't a
| gargantuan task. So it was either, rewrite this again in
| Go (using off-heap data structures to avoid GC pauses,
| which makes the software both harder to maintain, and
| also less idiomatic) or try a new language where we won't
| have these issues - where we don't have to resort to
| clever hacks to work around the runtime system.
|
| Since then we've continued our investment in Rust, after
| success of this project (and many others we've yet to
| blog about as well.)
| [deleted]
| collyw wrote:
| Crap developers get excited by new tech. Good devs want
| to get problems solved. If a new tech is the best
| solution then use shiny new tech. Usually it's not the
| best choice.
| aaomidi wrote:
| What a ridiculous statement. You can be excited by new
| tech and realize how it can solve your problems.
| collyw wrote:
| And you can be maintaining a system built by developers
| who got excited by shiny new tech and made a load of
| inappropriate technology choices because they were flavor
| of the month and looked good on their resumes.
| aaomidi wrote:
| It sounds like you've had bad experiences with some devs
| and you're applying that to everyone.
|
| Maintaining a codebase written in C is far far harder
| than one written in Java and that's far harder than one
| written in Go.
| flippinburgers wrote:
| Excitement < the countless hours required to rewrite a
| complex system in a non-trivial language.
| ericbarrett wrote:
| In Discord's defense, their product's scope and userbase has
| increased orders of magnitude over just a few years. Setting
| language choice aside, one would expect several rewrites of
| core code and infra over this time.
| silentsea90 wrote:
| My intention was to not set the language choice aside,
| especially where rewrites are squarely targeted at
| migrating to a different language.
|
| Rewrites of services to decomp into different services,
| migrate to different datastores to scale up etc. is
| completely justified and is a good problem to have.
| phabora wrote:
| This is a fine suspicion to have. It's a fine theory, even. But
| why raise this suspicion for this particular neo-language case?
| Why not raise it across the board?
|
| Such a theory is more robust if it predicts how the
| organization will work in general. Thus one would also be
| partial to suspecting that neo-languages are declined for
| political reasons _as well_ -- that a few powerful voices
| simply prefer to find other solutions, and then find post-hoc
| technical arguments.
|
| Such a theory works less well if it is only used to explain
| some particular (ostensibly) technical decisions.
| hedgehog wrote:
| Agree, Rust seems like a good choice but I'm curious why they
| needed to replace Go. GC pause time is mostly due to the amount
| of pointer-containing heap memory. Moving the the cache storage
| to slabs of pointer-free "dumb" slices should pretty much avoid
| the problem so long as you force GC frequently (large heap will
| confuse the heuristic). Shard the slabs across a handful of
| goroutines, one channel each to handle locking, and that should
| perform fine compared to an equivalently complex Rust
| implementation.
| ctvo wrote:
| The article lays out the points as if they exhausted all
| their Go options and ran into some deep language problem. I
| agree with you that this isn't a huge deal in Go, and could
| have been solved without a language, tooling, etc. change.
|
| It reads like a tool (Rust) looking for a problem.
| agumonkey wrote:
| I skimmed through this
| https://blog.kraken.com/post/7964/oxidizing-kraken/ yesterday
|
| now its php->rust so a widely different migration but rust was
| measured for its latency cutting value, not just fad factor
|
| I'd believe discord and others are on rust for this and not
| just the new shiney (that would be to their loss)
| [deleted]
| xtracto wrote:
| Right, specially because last thing I read Discord was using
| Elixir: https://blog.discord.com/scaling-elixir-f9b8e1e7c29b
| adamnemecek wrote:
| They are using both https://blog.discord.com/using-rust-to-
| scale-elixir-for-11-m...
| zn44 wrote:
| i believe this also often happens when original designers of
| the systems are out and not involved anymore, the new team
| wants to rewrite everything from scratch but that's hard to
| sell. So they find few technical reasons and pick some latest
| fun tech to play with.
| FalconSensei wrote:
| Yeah, unfortunately in some cases is easier to sell rewriting
| to a new language, than just refactoring/rewriting in the
| current stack.
| aaomidi wrote:
| Well, on the positive side a company like Discord moving to
| Rust means better tooling than what is currently available. So
| that's exciting.
| reikonomusha wrote:
| Better tooling for the programmers you mean? I don't
| regularly program in either language (Go or Rust), but I
| thought Google ran a pretty tight ship on good Go tooling,
| except with a bit of module SNAFU in its past.
| aaomidi wrote:
| Go's tooling is pretty nice. I was talking about better
| tooling for Rust. Maybe JetBrains will make a Rust-specific
| IDE.
|
| So far though rust-analyzer >>>>> JetBrains' plugin
| alephu5 wrote:
| This is my experience too. I've seen a lot of decisions made
| due to personal preference, at considerable technical and
| financial expense.
|
| The most recent example was a replacement for a complex YAML
| DSL, served by spring config sever and interpreted by a python
| program on every instantiation. This thing was literally using
| 20% of our CPU cycles over the whole compute cluster and
| estimated to cost around EUR6k per month. Long story short I
| provided a fully working pre-processing solution with
| effectively no runtime cost, no dependency on the spring config
| server and a more powerful & type-safe DSL. It was dhall with a
| bit of bash and python, although you could also have used
| jsonnet or any configuration language. Another developer with
| more clout pushed me out in favour of their own solution, which
| I believe was to keep me away from his political pedestal. A
| few months later I left and from what I've heard the company is
| still burning 20% of it's compute budget on this...
| spoiler wrote:
| > I tend to believe it actually happened because engineers with
| enough political clout decided they wanted everybody to change
| to their preferred language, not truly because of technical
| reasons, but because they just wanted to be paid to write in a
| cool language.
|
| I would say it's cool _because_ of the technical reasons,
| though! You make it seem like people are inventing technical
| reasons to make rust seem cool. I could make a long "technical
| reasons" list why I would personally prefer to work in Rust,
| but it boils down that it makes me more productive, and the
| code more performant with fewer gotchas and less
| sacrifices/compromises (compared to say C++ or C). Do I think
| Rust is cool? I guess... But that which attracted me to Rust
| was there already; I didn't have to make it up[1].
|
| I understand not every language is for everyone. I've had
| colleagues dismiss rust simply because they prefer C++ (and we
| work with C++). Some will even agree Rust is better when
| presented with technical reasons, but still not like Rust "just
| because."
|
| > I wonder why they didn't try doing things like putting the
| LRU cache into foreign, unmanaged memory to avoid garbage
| collecting it. Easier said than done, but why not?
|
| I write Go intermittently since I participate a little in the
| Kubernetes ecosystem, and haven't dwelled that deeply into it.
| How would this be done without reaching for C, though? Or would
| it be fitting to swap languages in this "subset" of the
| implementation?
|
| [1]: I feel like "make up" is the wrong phrase here, but I
| can't think of a better one right now. I guess what I want to
| say that Rust's "cool" reputation is well deserved. I don't
| think the technical reasons are made-up to serve as office-
| politics cannon fodder.
| unanswered wrote:
| > I wonder why they didn't try doing things like putting the
| LRU cache into foreign, unmanaged memory to avoid garbage
| collecting it
|
| Ah yes, because what I _really_ want out of an ostensibly
| memory-safe language like go is to have to resort to C, or what
| amounts to C, to get anything done. Or I could use a language
| which doesn 't have this garbage issue. But that would just be
| political nonsense apparently.
| reikonomusha wrote:
| I'm not saying that it's a perfect solution, and yes, using C
| memory defeats the purpose of memory safety. But it's a
| trade-off, as most engineering things are. Rewriting in Rust
| is also a gigantic trade-off along _many many_ more axes, and
| Rust isn't immune to people using `unsafe` in practice
| either. In both Go and Rust, it's possible to carefully
| engineer so as to isolate where sources of memory unsafety
| are, either through the language or through principled policy
| and software design.
| jzoch wrote:
| You make an excellent point i just wanted to correct your
| usage of "unsafe" in rust: It does not signify "im writing
| c" now as many invariants are still upheld in Rust even
| with the usage of unsafe. Id say its still much "safer"
| than using unmanaged memory in go as you have far more
| facilities you may use and more invariants are upheld.
| unanswered wrote:
| The problem with go here is (a) garbage collection, such
| that rust likely wouldn't have needed any unsafe outside of
| well-known libraries; and worse (b) when you need to do
| stupid C shit in go, there's no `unsafe` to contain it; it
| just spews over the rest of your code.
|
| "Principled policy and software design" is _exactly_ what
| 's needed to get rid of memory bugs in C too! So we can
| conclude that these things are impossible, or else we would
| not need memory safe languages in the first place. Since
| it's increasingly apparent that go requires these mythical
| safeguards after all, it's safe to say that it is _not_ a
| memory safe language in the useful sense.
| reikonomusha wrote:
| In general, I think this all-or-nothing attitude paired
| with FUD language is detrimental to good engineering.
| Memory unsafety is generally bad, I get it. But it's
| exaggerating to say that these things "spew" all over
| codebases, or that you're entering some inescapable
| memory hell that has no way to be treated properly. The
| original proposal was to manage some memory yourself
| within Go to solve one very specific problem with a very
| specific module of code. That's not a lot of surface area
| and has a clear API. It doesn't seem unfathomable that
| qualified, highly paid engineers can't do a good job
| solving that problem comprehensively without rewriting
| everything else.
| aaomidi wrote:
| I really am glad Discord did not go with this route. It's a
| chatting application, which means you're dealing with a TON
| of user generated content. Unmanaged memory and user
| generated content is an absolutely terrible idea.
| ctvo wrote:
| ...?
|
| The article is about a very specific service handling
| state for if you've seen a message in a channel. It has
| nothing to do with processing user content.
| ufo wrote:
| I think the question is why not rewrite just the LRU cache in
| C/C++/Rust (whatever floats your boat), as opposed to
| rewriting everything in Rust.
| stryan wrote:
| That sounds like basically what they did; they rewrote the
| ReadStates service in Rust and it sounds like that's
| basically a wrapper around an LRU cache. The rest of
| Discord is still Go/Elixer/whatever else they use.
| masklinn wrote:
| One reason would be that doing that would require switching
| over to cgo, at which point you deal with the PITA that is
| an FFI, and the PITA that is cgo (it's easy to enable but
| it drastically alters your "game plan").
|
| And here the cache is essentially the entirety of the
| micro-service:
|
| > [The service's] sole purpose is to keep track of which
| channels and messages you have read.
|
| so it's a giant map of (user, object) -> read flag or
| something along those lines. The DB commit and service
| input would be very light, and not really your main
| concern.
| fiddlerwoaroof wrote:
| The question I'd have is "why use a custom cache at all",
| this sounds like a problem that has a whole pile of
| battle-tested off-the-shelf solutions.
| jhgg wrote:
| The original implementation of this "service" was in
| Redis using lua scripts.
|
| We hit the performance ceiling of redis, and rather than
| deciding to go with redis-cluster, and dealing with the
| headache of managing persistent redis, we went this way,
| so we could consolidate data storage to
| scylladb/cassandra (it was cassandra at the time of
| rewrite, scylla now since we've migrated our data stores
| over).
| MengerSponge wrote:
| Ham Sandwich Development
| swiley wrote:
| The main thing that bothers me about rust is that there is one
| complete implementation of the language (and one or two very
| incomplete implementations) and the singular complete one isn't
| even self hosting.
|
| Coming from scheme where everyone and their dog has an R5RS
| compatible REPL that's pretty disturbing.
| steveklabnik wrote:
| What makes rustc not self hosting?
| masklinn wrote:
| > The main thing that bothers me about rust is that there is
| one complete implementation of the language
|
| The essay is about Go which is pretty much in the same
| position: gccgo is a non-entity.
|
| > and the singular complete one isn't even self hosting.
|
| What are you talking about? Rust was self-hosting before it was
| even stable (a common source of problems as the language can
| then get optimised for implementing the language, thankfully it
| doesn't seem to have harmed it too badly).
| sandGorgon wrote:
| How timely!
|
| https://stackoverflow.blog/2021/02/22/choosing-java-instead-...
|
| I do believe that there is nothing better tuned for performance
| vs developer productivity than the JVM.
|
| The language semantics may be a question for personal
| distaste..but then you do have Kotlin (https://jooby.io/ ) which
| ranks fairly near the top on techempower benchmarks
| https://www.techempower.com/benchmarks/#section=data-r20&hw=...
| swyx wrote:
| > Discord has never been afraid of embracing new technologies
| that look promising. For example, we were early adopters of
| Elixir, React, React Native, and Scylla. If a piece of technology
| is promising and gives us an advantage, we do not mind dealing
| with the inherent difficulties and instability of the bleeding
| edge. This is one of the ways we've quickly reached 250+ million
| users with less than 50 engineers.
|
| this sounds like a recipe for disaster in every other company i
| have worked with. are Discord's engineers just THAT good or is
| there something else going on here?
| steveklabnik wrote:
| This was covered a year ago, and is literally the most popular
| post for "Rust" ever on this site
| https://news.ycombinator.com/item?id=22238335
| [deleted]
| jjjeii3 wrote:
| GO promised to be very fast, but if you check the actual
| benchmarks:
|
| https://www.techempower.com/benchmarks/#section=test&runid=8...
|
| High-level C# (asp.net) is almost twice as fast as GO in the
| benchmark... Rust is also in the top 3.
|
| So, why should I use GO instead of C#, if ASP.NET/C# is so
| blazing fast and requires a lot less lines of code to achieve the
| same? -> It makes absolutely sense to switch to C# or Rust.
| [deleted]
| anewaccount2021 wrote:
| Go is slower than Rust but three times as fast as you'll ever
| need it to be
|
| Everything in Techempower that is in the top half of
| performance is likely to be fast enough that your own
| architectural issues will be gating your performance, not the
| language/framework
| cletus wrote:
| Why? Well, for one, Go produces a statically linked binary and
| C# requires a runtime be installed.
|
| If we've learned nothing from Java and Python, it's that
| managing runtimes and environment can be a huge pain point.
|
| EDIT: swipe keyboard put ringtone instead of runtime and I
| missed it. Corrected.
| carlosrg wrote:
| > Go produces a statically linked binary and C# requires a
| runtime be installed
|
| Not true since .NET Core 3.
|
| $ dotnet publish --self-contained --runtime osx.11.0-x64
| -p:PublishSingleFile=true
| doodpants wrote:
| Not disagreeing with you, but just to be pedantic:
|
| In .NET Core 3, you can build and distribute a single-file
| executable which, when run, will actually unpack an
| executable and a ton of DLLs into a temp directory, and run
| it from there.
|
| In .NET 5 they changed it, so that when you build a
| "single-file" executable, you actually get an executable
| plus a few DLLs, which must be distributed together.
| meepmorp wrote:
| >C# requires a ringtone be installed.
|
| This confused the hell out of me until I realized it's
| supposed to be runtime.
| tantalic wrote:
| On the other hand, would you be surprised to learn the .NET
| runtime includes a ringtone?
| meepmorp wrote:
| That's probably why it confused me so much; I couldn't
| rule out that interpretation reliably enough to interpret
| it as a typo.
| thaumasiotes wrote:
| Autocorrect: turning simple, obvious mistakes into opaque,
| irrecoverable mistakes.
|
| It's never been clear to me why someone would want this.
| philwelch wrote:
| Thanks. I assumed it was an allusion to the .Net runtime
| having a bunch of unnecessary crap built into it, including
| a ringtone for some arcane reason.
| tonyedgecombe wrote:
| You can do that with C#: https://docs.microsoft.com/en-
| us/dotnet/core/deploying/singl...
| juskrey wrote:
| With C# and Java, we have learned that time and battle tested
| VMs can remove terrible pain points in proper hands.
| cletus wrote:
| To be fair, I have found Java's JRE to be relatively
| painless. It's fairly simple to have multiple installed and
| choose which one to use as a command line option.
|
| Python OTOH...
| PragmaticPulp wrote:
| If performance is the top priority and time to market can take
| a back seat, I skip Go and choose Rust (or other known-fast
| languages).
|
| However, time to market is almost _always_ a priority. Unless
| my margins are razor-thin, spending twice as much on servers
| for shorter development cycles is a net win in most cases.
| Shadonototro wrote:
| Don't look at that kind of benchmark, most cheat, and notice
| how latency is bad, because C# is not native code, so the JIT
| needs warmup just like java, wich introduces hicups and non-
| deterministic performance
|
| So these perfect situations where you only do the same thing
| and call same code over and over never happen; never!
|
| Anyways, GO isn't popular for its RAW perf, GO is popular
| because:
|
| - Simple language
|
| - Native code
|
| - GC
|
| - Cross compilation
|
| - Fast iteration (build/rebuild/deploy)
|
| - Relatively tiny, statically compiled executable
| MaxBarraclough wrote:
| > the JIT needs warmup just like java
|
| That doesn't sound right. .Net has good support for caching
| of generated native code. It's well ahead of Java in this
| regard.
| Kipters wrote:
| > the JIT needs warmup just like java
|
| You can use ReadyToRun[0] to pre-JIT portions of the code for
| faster startup. Full AOT compilation is in the works too[1]
| albeit it's still experimental.
|
| [0]: https://docs.microsoft.com/en-
| us/dotnet/core/deploying/ready...
|
| [1]:
| https://github.com/dotnet/runtimelab/tree/feature/NativeAOT
| felipellrocha wrote:
| I think you're missing the point. The point is that you can
| get all those properties in Rust without the performance hit.
| (Except for, well, GC, obviously)
| Shadonototro wrote:
| The problem with rust is the language is kind of scary to
| learn, once you get past the learning curve, it is okay i
| guess
|
| And on top of that iteration time is hurt because of both
| the language, the compiler perf, and the lifetime problems
| that you need to constantly think about
|
| The benefits of using rust on other hand is non-negligible,
| i wish the team would focus on solving that kind of
| immediate problems
| unanswered wrote:
| Lack of garbage is a pro, not a con.
| jmaygarden wrote:
| Rust does not build fast. It's more comparable to C++ if
| not worse.
| drenvuk wrote:
| that depends on your dependencies I believe. If the
| people who keep including serde into their crates would
| stop everyone would have faster compile times.
|
| Stop using serde. please.
| saagarjha wrote:
| Dependencies make this much worse, but Rust itself is not
| really very fast at being compiled.
| madushan1000 wrote:
| I mean those benchmarks are mostly web server related. And
| it's pretty common in web servers to trigger the same code
| path over and over again all the time. There's a reason why
| people are not rewriting their java web server application in
| something native to get more performance out of it.
| atraac wrote:
| These benchmarks also in no way represent actual webapps.
| Have you seen this code? Have you ever written a webapp
| where not using hardcoded, static headers like Content-
| Length was a bottleneck?
| merb wrote:
| the plaintext c# one actually only sets a content length
| manually since without it it would be a streaming
| response (Transfer-Encoding: chunked), which would not
| make the server slower, but the client, because of
| buffers (also as far as I know the Content-Length needs
| to be set in this benchmark). btw. go does not do that
| when calling w.Write without wrapping the responsewriter
| inside a buffer (which is probably also the case why it
| is slower, since it needs to know the size of the written
| response before it goes over the network and btw. only
| go-std is slow because it's written for simple use cases,
| there are others like fasthttp who do stuff a little bit
| more like c# and thus are faster (even inside ;-) go-std
| never cared about being the fastest.)
| aaomidi wrote:
| > - Simple language
|
| So simple it can't do Object.keys()
|
| Don't get me wrong, I enjoy writing code in Go, Rust, and TS.
| But my god, Go literally is a pain in the ass every time I
| want to do something that isn't a map reduce problem. And
| even for map reduce problems it's annoying due to lack of
| generics.
|
| Maybe generics will finally give Go something that every
| other language has. A strong, useful, standard library for
| collections.
| Shadonototro wrote:
| I agree with you, i'm not using Go myself, i tried it many
| times but i always hit a roadblock, it's interesting
| because i hit the same kind of issues with ZIG
|
| Simple on paper, but what's the experience as a result?
| some are okay with it, some love the restrictions and
| limitations, but i personally don't enjoy them
|
| But again, i can understand the choices and the benefits
| phabora wrote:
| Neither Go nor Zig are spelled ALLCAPS
| Thaxll wrote:
| Go is as fast as dot net in the same benchmark you posted:
| https://www.techempower.com/benchmarks/#hw=ph&test=plaintext
|
| aspcore 7,016,966
|
| gnet 7,010,982
|
| People share results of benchmark where some languages are
| missing.
| xtacy wrote:
| They key takeaway really is that for high performance, the way
| you think about memory layout, data structures, memory
| management, etc. all start to really matter. It is possible to
| write programs with memory pools to reuse data structures as
| much as possible without having to defer the work to the GC,
| which can then introduce performance spikes.
|
| That said, C# and JVM are both mature and have decades of GC
| improvements, which could explain why golang is falling behind.
| Time will tell.
| stiray wrote:
| I am not saying Go is faster or anything I am just saying that
| take the grain of salt with reading the benchmarks as they are
| on same level as googling for "best microwave oven in 2021"
| where you know it is better not to even start reading the
| results as they will be full of paid "fake" reviews. Not to
| even start about microwave oven brand fanboys.
|
| You have to understand that the programming language is a
| product in same way as anything else and there are influencers,
| marketing bribes, paid advertisements, forum/hn/fb/... paid
| spammers, etc.
|
| Just remember what spam spike the Rust (I am again not talking
| about rust as a language... just about the noise everyone made)
| had here when it emerged, they threw the shadow on the Nigerian
| spam, oh the amount of noise from Rust-Distributed-Evangelist-
| Task-Force (they will probably downvote me to hell for
| mentioning this :D), I had a lot of fun reading HN at that
| time, the most used word was 'and', the second was 'rust', it
| was so exaggerated that it was like watching Monty Python.
|
| In today world the only review, or if you want, benchmark, is
| the one you make. Nothing else counts and if you feel that C#
| is faster then enjoy using it.
|
| And quite frankly, who cares about speed today (except some
| very strange people like me), people are using very strange
| languages and want to run them on backend. And they actually
| do. Ignore the benchmarks, no one cares (except those that
| would like to advertise themself like "faster than C" :D). Just
| threw in some more vapor on the cloud and you have "fixed" the
| speed issues. /s
| d1str0 wrote:
| Performance is rarely the top priority for a software
| engineering project. If you task a whole team to switch from Go
| to Rust or C# purely because "Rust goes brrr" you're going to
| have a bad time.
|
| I use Go because its decently fast, I like the syntax, has a
| great stdlib, and I feel like it's very easy to share a Go
| codebase with other engineers.
| steve_adams_86 wrote:
| One of the key things I like about Go, despite shortcomings,
| is your last point. Our team effortlessly shares Go code, and
| its tooling makes it so any of us at any time can easily
| compile and run projects without hassles or hang ups.
|
| The other day we reviewed some oldish and complex code and it
| was so plain to see what it was doing and get back into the
| flow of that logic. I don't find this happens with other
| languages as much.
|
| I'm not saying Go is the right tool everywhere for everyone.
| I do love that about it though.
| collyw wrote:
| This sounds great but thinking of my system at work, it's
| not the language that is the problem. It's all the moving
| parts that you need to get to play together nicely before
| you can even start to develop. Front end, back end,
| database, nginx proxies, email platforms.
| steve_adams_86 wrote:
| Go has won here for us as well because it forces you to
| write very dumb code, resulting in exposing the logic of
| how systems interface in a very dumb but easy to
| understand way.
|
| It may take some scanning to go through all the dumb
| code, but you typically won't get fooled by abstractions.
| Go offers so little in terms of abstractions that you can
| usually see right through them.
|
| This won't be everyone's experience, but it has been mine
| very consistently across internal and 3rd party code.
| jariel wrote:
| We might want to highlight this more often that not.
|
| Code legibility is starting to become a primary if not the
| primary factor for a lot of software these days, because
| that's what allows it to be maintained.
|
| I think more analysis should be done on why this is for go.
| Java and C# should be similarly legible, they are not
| complex languages either. It's possible that go encourages
| certain paradigms.
| shakezula wrote:
| Definitely agree with this. Go is one of the only languages
| I've worked in where I can jump into _any_ project and it
| feels familiar, I just have to dig around to get into the
| specifics.
|
| I have never had that same familiarity from Python, Ruby,
| JavaScript. I think there's something to be said about that
| in Go's favor.
| jokethrowaway wrote:
| While I agree with you on the familiarity across
| different projects (and I applaud ideas like gofmt) -
| it's also very verbose code (due to the simplicity of the
| language).
|
| That makes it very hard to read go code and build a
| mental model in your head.
| ctvo wrote:
| > it's also very verbose code (due to the simplicity of
| the language). That makes it very hard to read go code
| and build a mental model in your head.
|
| Verbose does not equal complex usually.
|
| Something that's a single method call on a built-in type
| may be 3 lines in Go. This doesn't make it hard to
| understand or keep in your head. "This iterates over a
| map" is the same regardless of the lines of code. You can
| very quickly skim verbose code, especially in a language
| like Go, where idioms and conventions are identical
| across many code bases.
| make3 wrote:
| I think that the argument here is that the c# environment is
| much bigger and that the programming level is at a higher
| level. That these are things that are often made at a
| sacrifice for performance but that that's not the case here,
| so why use Go at all
| [deleted]
| _wldu wrote:
| I like Go for all these reasons too, and the fact that vim
| and a Makefile are all we need to be very productive.
| dleslie wrote:
| Performance is a cost concern; if I can pay for less server
| resources then I can either pay my programmers more or hire
| more programmers.
| dilyevsky wrote:
| Discord also runs in cloud. They could save $$$ by just
| going to the colo instead
| merb wrote:
| I doubt that, with that many users they would probably
| need to build their own stuff, around the globe. their
| servers have such a low latency, that this would be
| impossible by just going "colo".
|
| peope who say that they can just save money in their colo
| are probably never even working in the scale of discord,
| which is fine.
| nielsbjerg wrote:
| Databases and traffic is going to have your lunch anyway.
| Good developers cost a good deal more than extra
| infrastructure. Adding people to help you cut cloud cost is
| easier and more predictable than re-implementing.
| collyw wrote:
| Bad developers will cost you more.
| hu3 wrote:
| Have you seen the kind of C# code they write to compete in Tech
| Empower? It's nowhere near what one would write on a daily
| basis:
| https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast...
|
| Perhaps Go folks should just use inline assembly to demonstrate
| how useless these benchmarks are.
| thelazydogsback wrote:
| Most code is not CPU-bound, and for code that is, a hot spot
| is all the usually needs to be optimized. Today .Net Core is
| remarkably performant, and the fact that just by writing code
| carefully, or using Span<> etc., you can get close to bare-
| metal perf with w/o resorting to FFI to unsafe code is a good
| thing. This is much easier than trying to keep your friendly
| Rust borrow-checker happy even for moderately complex data-
| structures with varying lifetimes.
| hu3 wrote:
| When you put it that way .NET Core runtime does seem to
| offer great performance at low cost.
| merb wrote:
| btw. these are the platform benchmarks of c# (only kestrel +
| manually writing the response, basically no framework code)
| the real ones are in benchmarks (aspcore-mw), which just use
| a middleware and are also really really fast. btw. golang's
| std http server is just really really slow. fasthttp solves
| that and is basically 6x faster in some real code cases.
| infensus wrote:
| This 1000x times. Should be posted every time someone links
| to TechEmpower benchmark results. And especially for the most
| contrived benchmarks like "Plaintext" and "JSON
| serialization", which are basically echo server competitions
| that have nothing to do with real applications.
| DennisP wrote:
| That's nowhere near as bad as inline assembly. It might not
| be the usual way to write a web app, but it's normal C# with
| common libraries. Any C# dev could use that as an example if
| they ran into a major performance issue.
|
| These benchmarks aren't intended to show the performance of
| everyday code. They show what experts can achieve on each
| platform. That's why they take pull requests instead of
| writing the code themselves from tutorials.
| hu3 wrote:
| That's true. .NET Core high performance optimization looks
| more approachable.
|
| I'd still rather see the popular way of doing things be the
| benchmark.
| jamra wrote:
| One of the things to consider when comparing C# and Go is how
| it does parallelism. In C#, the new async/await mechanism can
| go horribly wrong if the programmer doesn't use it correctly.
| This can cause threads to start up, which is in fact slower
| than older .NET tech that had larger starting thread pools. You
| can view this with PerfView to see if it's your load issue.
|
| In Go, the scheduler is smart enough to stop something even
| when it's not blocking on IO. This makes it more robust.
|
| Of course, use what you need for when you need it. Rust
| definitely shines in some circumstances.
| chrisoverzero wrote:
| > In C#, the new async/await mechanism [...]
|
| C# grew the `await` operator in C# 5.0, released in August of
| 2012. Go altogether hit 1.0 on 2012-03-28. If the one is new,
| the other is new.
| CobsterLock wrote:
| Off-topic: who is in charge of choosing colors for graphs? Being
| red-green colorblind, choosing blue and purple is makes it
| impossible for me to read the charts. I know its stylish, but
| every time i make a graph i choose very high contrast colors.
| Another point: is it just being color blind or do other people
| struggle in charts/maps that have shades of a color? like
| different shades of blue represent different rates of covid
| deaths across a map. just curious if anyone else struggles with
| this
| chungy wrote:
| I am not colorblind, and the chosen colors for this graph are
| similar enough to make it difficult to read.
|
| High-contrast helps the rest of us, too.
| jlouis wrote:
| Brewer colour schemes ftw.
| ihumanable wrote:
| These are just the default colors for Datadog graphs, so I
| guess the answer to your question is Datadog.
| silentsea90 wrote:
| Incredibly brave of them to go with nightly build of Rust for
| async functionality. Not as in the weeds, but I would have been
| afraid of release being delayed forever and being stuck with the
| nightly release for a long time.
| cheaprentalyeti wrote:
| I didn't know Discord was written in Go to begin with. I thought
| it was written using electron or something like that.
| TheChaplain wrote:
| The client is electron, I believe Go/Rust is used in the
| backend systems.
| mewpmewp2 wrote:
| I thought backend was elixir.
| notamy wrote:
| As I remember,
|
| - Real-time stuff (websocket gateway, etc.) is Elixir with
| some Rust NIFs
|
| - REST API is Python
|
| - Other stuff is Rust or Python afaik? Probably wrong here
|
| Obligatory "I don't work there"
| dnautics wrote:
| Rust in hot paths
| dnautics wrote:
| The backend is still quite a bit of elixir (I imagine, I don't
| know as I don't work there) - as of October 2020 blog post
| jhgg wrote:
| We were experimenting with Go as a "fast" backend language to
| implement stuff like data services and high throughput systems
| in.
|
| Go has proven itself disappointing in many ways. Rust however
| has only continued to evolve and get better as the community
| matures. 2021 is definitely the year of "rust" for us - and we
| will be using it in many more places. Both on the backend but
| also on the client - where we embed native modules written in
| rust.
| throwawaysea wrote:
| Does Discord use the cloud for infrastructure or are they big
| enough that they are going their own way?
| jhgg wrote:
| We are hybrid. We use bare metal for our RTC workload
| (voice video streaming) and GCP for the rest.
| throwawaysea wrote:
| I'm curious if you have any thoughts on power
| savings/sustainability from using Rust, or from using
| Rust with ARM-based processors on bare metal, for
| example. Is it meaningfully different at your scale?
| ngrilly wrote:
| How Go disappointed you beyond the case in the linked
| article? Is there a few cons in using Rust compared to Go? In
| particular, what about language complexity/learning curve and
| compilation times?
| jhgg wrote:
| The only major cons to rust imo are the slow compile times,
| the steep learning curve and the immature ecosystem.
|
| You can solve the first hurdle with better hardware (for
| the same price as our usual specced macbook pro, I opted to
| custom build a beefy workstation with a 3790X).
|
| The second, you just solve with time. I remember when I
| first started with rust I hated it. I was working on a
| native project, and felt like it would have been easier to
| do in C++. But internally now we have more mindshare and
| knowledgeable folk who understand Rust - this has helped
| tremendously in onboarding engineers to the language. Also,
| having a working editor (via rust-analyzer) and the quality
| of compiler error messages have improved significantly in
| the past few years.
|
| In terms of immature ecosystem, I think the rate at which
| the ecosystem is advancing is incredibly promising. Tokio
| hit 1.0 in Dec 22 of last year, which will finally put an
| end to the dependency upgrade thrash we've had to deal
| with. Going from tokio 0.1 -> 0.2 -> 0.3 -> 1.0 has been a
| journey. But with 1.0, finally, libraries will have a
| stable API to target, so we won't have to worry about this
| much for a while. Additionally, in the following months we
| intend to start funding critical open source rust projects
| that we use to ensure their success.
| ngrilly wrote:
| Amazing feedback. Very much appreciated.
| anticensor wrote:
| Have you considered also switching the client to Rust?
| jhgg wrote:
| No. React/TypeScript is too productive for our project
| engineers.
|
| But we do write our perf critical stuff in the client in
| C++/Rust! For example, our entire video capture pipeline is
| in C++/Rust (the meaty bits in rust, some glue in C++ to
| interact with WebRTC.)
| politician wrote:
| Are there any apps that you know of that use Servo as a
| rendering engine? I'm curious and would like to look at
| some examples; I thought Servo was intrinsically tied to
| Firefox.
| anticensor wrote:
| No, not yet. But Servo can be used headless:
|
| https://github.com/paulrouget/servo-embedding-example
| rvz wrote:
| > When starting a new project or software component, we consider
| using Rust. Of course, we only use it where it makes sense.
|
| Big takeaway here for those who are married to their tools. Rust
| happens to be just another tool for this company, and it works
| well for them; where it makes sense of course.
| im3w1l wrote:
| The impression I get when I hear they use Go, Elixir _and_ Rust
| on the backend is that they are chasing every shiny trend. I
| doubt it makes sense from a purely tecnical perspective and is
| more a way of indulging and investing in the skills of their
| engineers.
| nilkn wrote:
| I don't work at Discord, but Elixir seems like a perfect fit
| for the sort of infrastructure they build. BEAM is a marvel
| and there are huge success stories for it when it comes to
| chat apps (e.g., WhatsApp). Elixir is not necessarily a
| _fast_ language, though, so it 's easy to see how they'd run
| into a need for something like Go or Rust for the occasional
| service where you want to push single-core performance to the
| limit.
| jhgg wrote:
| Elixir was not us chasing a trend - but rather betting on a
| language and runtime system we knew would be amazing if it
| worked.
|
| Discord would not exist in the form it does today if it was
| not for BEAM/OTP. The majority of the chat system was built
| and maintained by a handful of people. Fundamentally our
| architecture has not dramatically changed since it's
| inception in 2015. Can you say the same for other real time
| distributed systems as they have scaled from thousands to
| hundreds and of millions of users?
|
| Rust was a similar bet - the language looked promising and we
| were right. It's been amazing to use - and it's only gotten
| better since we started. rust-analyzer is amazing!
| masklinn wrote:
| It's not really the subject of the discussion, but since
| you're here and interacting with people: would it be
| possible to ask the UI/UX folks if they could finally make
| the built-in shortcuts configurable (and deactivable)?
|
| It's really weird that such a game adjacent piece of
| software completely hardcodes shortcuts, especially when so
| many of them either conflict with or are near-misses for
| common chords, especially for very disruptive actions like
| _starting voice calls_. Even more so as it _already_ has
| configurable keychords, just not for most of the useful
| stuff.
|
| Also they should fix the labelling around these, either
| it's just not working or it doesn't work reliably, but
| trying to override built-in shortcuts via the configuration
| (in order to disable them) has never worked for me.
| [deleted]
| bobthebuilders wrote:
| Rust will never replace C++. It is terrible langauge driven by
| terrible programmers.
| unethical_ban wrote:
| @bobthebuilders, it is one thing to give productive, insightful
| criticism of a technology, even if it is scathing. It is
| another thing to insult the creators of a technology. Please
| adjust your tone.
| Smaug123 wrote:
| Pretty sure this comment has wandered astray?
| Jtsummers wrote:
| bobthebuilders' comment was marked dead, GP responded to
| the same parent comment as the dead comment. If you turn on
| showdead you'll be able to see what they're talking about.
| unethical_ban wrote:
| Exactly. As my name implies, I'm not a fan of
| shadowbanning unless the account is clearly vicious or
| spam.
| [deleted]
| ibraheemdev wrote:
| previous discussion:
| https://news.ycombinator.com/item?id=22238335
___________________________________________________________________
(page generated 2021-02-22 23:01 UTC)