[HN Gopher] WhatsApp scaled to 1B users with only 50 engineers
       ___________________________________________________________________
        
       WhatsApp scaled to 1B users with only 50 engineers
        
       Author : unripe_syntax
       Score  : 545 points
       Date   : 2021-10-25 07:07 UTC (15 hours ago)
        
 (HTM) web link (www.quastor.org)
 (TXT) w3m dump (www.quastor.org)
        
       | jokethrowaway wrote:
       | While what you says about focus is important, I've seen plenty of
       | focused startups die.
       | 
       | WhatsApp had success because they executed well enough at the
       | right time.
       | 
       | I hate WhatsApp with passion because of its limits and that
       | horrible web interface that needs your phone connected in order
       | to work.
       | 
       | I know plenty of people (me included) who stopped talking with
       | some groups of people because they didn't tolerate WhatsApp.
       | 
       | It's mediocre software at best, but because of the network effect
       | and first mover advantage, that was enough.
        
       | LudwigNagasena wrote:
       | Well, what number would one expect? Surely, the workforce doesn't
       | scale linearly with users.
        
       | Traster wrote:
       | I think one thing to consider is that WhatsApp is probably by
       | nature a scalable application. It is in the end, simply a client
       | app that sends data to other client apps. So by nature, you scale
       | in parallel very well, there are no 1-all communications. So
       | unlike something like facebook where the data all needs to go to
       | a central server and be polled by an unknown (but potentially
       | arbitrary large) number of clients, in this case you have a huge
       | number of point to point communications. It also means your
       | server application can be largely stateless.
       | 
       | I'm not saying that it's an easy problem to solve, but it is a
       | much easier problem to solve than a lot of other applications.
        
         | lampe3 wrote:
         | I fully agree here.
         | 
         | Sending messages to a known (small) number of people is a
         | solved problem for a long time. Email for example.
         | 
         | With WhatsApp, it is even easier than email because WhatsApp is
         | a closed system, and you can optimize.
         | 
         | They used an uncommon setup? Okay cool. How does this help me?
         | or any other company?
         | 
         | I'm not sure what I should learn from this story?
         | 
         | The only thing is: Feature creep is bad.
        
         | Tepix wrote:
         | Why do you assume that WhatsApp uses point-to-point
         | connections? My assumption is that most devices cannot talk to
         | each other directly and require central servers.
        
           | hbn wrote:
           | I could be wrong but I don't think he was saying "point-to-
           | point communication" in the technical aspect of the term
           | (i.e. peer-to-peer). Rather, just making the point that
           | they're not storing tons of data on their server that has to
           | be polled on demand by an unknown amount of clients forever.
           | Even if there's a server between the users, once they receive
           | a message and pass it onto whoever necessary, their job is
           | done and they don't need to worry about that data any more.
        
         | namdnay wrote:
         | > It is in the end, simply a client app that sends data to
         | other client apps
         | 
         | That's not the case. Consider the "one tick" messages - they
         | must be on a server somewhere. One tick means they've left your
         | phone but aren't yet on (all) the recipients' phones
        
           | Drew_ wrote:
           | Servers proxy the message yes but once all recipients (maybe
           | a dozen clients in the worst cases) receive the message, the
           | server can drop the data once and for all. No need to
           | optimize for harder problems like search, caching, or
           | recommender systems.
        
         | crate_barre wrote:
         | To add to this, not only is it a simpler problem, but they
         | stood on the shoulders of giants that already solved it. They
         | used ejabberd, a full-featured xmpp server written in Erlang.
         | Imagine half your job already done for you. Granted, they
         | heavily customized it by this point, but they did a great job
         | picking the right off the shelf solution to begin with.
        
       | amelius wrote:
       | Chat is the number 1 toy problem that is explained in every
       | Erlang book out there.
       | 
       | Therefore, honestly I'm not really surprised to see 50 engineers
       | capable of building a product around it.
        
       | rdsubhas wrote:
       | My experience is, # of engineers is directly related to # of
       | product people, which is in turn related to _quantity_ (not
       | quality) of ideas that the founders /leaders are desperate to try
       | out.
       | 
       | Higher _quantity_ of ideas == More engineers. Higher _quality_ of
       | ideas == Fewer engineers.
       | 
       | WhatsApp success is not about what WhatsApp did, it's about
       | everything they _didn 't_ do. No desktop, no browser (until after
       | reaaaaaaallly long), no account creation (your phone number is
       | enough, thanks), no hidden ad engine, no hundreds of stupid
       | features that product people stuff down in the name of
       | experimentation.
       | 
       | There are some leaders who want anything and everything: Throw
       | 100 darts in the wall. Leave what sticks (don't touch or nurture
       | it). Go back to throwing 100 more. The other guy does that? We
       | want that too. Mobile? Yes. Desktop? Yes. Browser? Yes. Different
       | affiliate channels? Yes. Custom features for B2B partners? Yes.
       | Merchandizing? Yes. Build a mini Ad engine? Yes. Multi user
       | profile management systems? Yes. Yes, yes, yes, every requirement
       | is a go.
       | 
       | The crucial difference is: What happens when you have a
       | restricted pool of people, but insist on doing 100s of things?
       | Some leaders simply treat this as an "org structure" problem, not
       | a problem of focus. They slice and dice the org until the 30
       | people become split into teams of 2-3 with a manager for each,
       | running into coordination hell on each step - just so that more
       | quantity of features can be seen on paper. This is how most
       | startups are, except for those rare few which don't "talk" about
       | focusing, but actually focus.
        
         | pards wrote:
         | > _What happens when you have a restricted pool of people, but
         | insist on doing 100s of things? Some leaders simply treat this
         | as an "org structure" problem, not a problem of focus. They
         | slice and dice the org until the 30 people become split into
         | teams of 2-3 with a manager for each, running into coordination
         | hell on each step_
         | 
         | 1,000 times this.
        
         | a_c wrote:
         | There was an article about when do startup brings in their
         | first "product manager". Before bringing in "product people",
         | the founders are the product people. I see it as the inflection
         | point when the founder start delegating the product making to
         | others. This is when the organization started scaling.
         | 
         | Doing one thing good is a thing in the past. Venture capital
         | has pushed basically all start-ups into 100-feature shops
        
         | tinus_hn wrote:
         | Ultimately though WhatsApp ended up with a billion users and 0
         | revenue.
        
           | bryans wrote:
           | Except for that $19 billion exit. Companies don't need to be
           | turning a profit anymore -- they just need to prove they can
           | scale, and someone else will figure out how to monetize it
           | after a buyout. Hell, you don't even need to be a startup
           | aiming for a buyout anymore to get away with being
           | financially incompetent. Reddit is about to file their IPO at
           | a valuation of $10 billion, even though they've been losing
           | money for nearly two decades and are noticeably (and
           | unsuccessfully) scrambling to find any presentable evidence
           | that they're not complete buffoons.
        
           | hermanradtke wrote:
           | They charged $1 per year before Facebook removed the fee.
        
           | rglullis wrote:
           | Not true. They were charging $1/year from users in the richer
           | markets. They could easily reach hundreds of millions per
           | year in revenue if they followed this model.
        
             | dntrkv wrote:
             | ~$30M revenue in 2014 from 600M users.
             | 
             | The vast majority of people will leave your service if it
             | goes from $0 to even $1. So no, I don't think they could
             | "easily" scale that up.
        
               | rglullis wrote:
               | It didn't go from $0 to $1. It was always "$0 to most,
               | _$1 for those that really didn 't care about paying
               | because they knew that anything else would have much
               | higher real costs_"
               | 
               | Hell, If WhatsApp ever got to say "Facebook wants to buy
               | us, but we are also considering changing to a pay-what-
               | you-want model to avoid selling out", I'd easily give
               | them $10/year without batting an eye.
        
               | vladoski wrote:
               | Yes, but when I was younger, everyone that I knew, even
               | digital illiterate people, paid that fee. Why? Because
               | WhatsApp was (and is) everywhere.
        
           | thruway516 wrote:
           | A billion users IS a goldmine. Ask Google, facebook, twitter,
           | reddit. None of these companies had any significant revenue
           | for the first couple years.
        
         | HeavyStorm wrote:
         | you're perfectly right
         | 
         | I get scared every time that senior leadership pushes a new
         | member on the team, because I know that overall quality will
         | drop; their view is completely related to how much things they
         | want to done on every cycle, and they don't care about quality
         | (until it blows up in their face)
        
         | papito wrote:
         | They also famously had no meetings, like at all.
        
           | Scoundreller wrote:
           | Like, 5 engineers never huddled together to figure out how to
           | solve a problem?
        
             | vb6sp6 wrote:
             | Getting together with your peers to discuss a problem is
             | entirely different than being called into a meeting with
             | someone who can only use a iPad to evaluate progress on
             | things they cant understand while several other people sit
             | idle waiting for their turn.
        
             | papito wrote:
             | There is no need to turn this into an argument by
             | suggesting that somehow WhatsApp banned its engineers from
             | getting together. That never happens, ever. We are
             | obviously discussing scheduled meetings and scrums.
             | 
             | By the way, this is mentioned here:
             | https://www.wired.com/2015/09/whatsapp-serves-900-million-
             | us...
        
         | didibus wrote:
         | I think there is something else that I find hard to explain,
         | but it stems from the same vein.
         | 
         | The best way I can call it is the impedence mismatch between
         | what business people imagine and envision, and what the real
         | products, their code, their systems are actually doing and are
         | actually designed to do.
         | 
         | It creates a kind of cost explosion in the velocity needed and
         | it is often solved by throwing more developers at it instead of
         | rethinking the actual design of the products so that they can
         | better grow to support additional use cases and scale.
         | 
         | You can't totally call it a hack, but it kind of is. A system
         | was designed in a way that doesn't scale to the user demand and
         | to the additional features or the new abstractions that it
         | wants to grow into. Instead of rethinking holistically about
         | how that system should be re-designed to now encompass all of
         | this new stuff, and then put the work into changing it from its
         | current model to that new one. What people do is they break up
         | the current system in two, so now you have two service where
         | you had one before. And now they hire a new team of engineers,
         | manager, PM, etc. to start working on this new half. This
         | repeats again where each of those two half are split in half
         | again, hiring a new team, now there are four teams and four
         | service, etc.
         | 
         | As that happens, you can no longer redesign the way the systems
         | fundamentally work, so each team accrues more and more
         | complicated workarounds on their individual pieces and all new
         | feature becomes cross team projects where you now need someone
         | to coordinate the work between all teams for each little
         | project.
         | 
         | Once you start down this path, there is no way out, the only
         | solution becomes throwing even more teams at it, breaking
         | things down again and again into smaller pieces with a whole
         | team behind it as each piece becomes more and more complex to
         | all work around the inherent holistic limitations that were
         | never reconsidered.
         | 
         | It's a vicious cycle, the piece cannot grow in it's current
         | state because it is too complex, break it up, have two teams
         | tackle each half, now they can take the complexity until they
         | themselves add more stuff to it which makes them too complex
         | once more, break it again, add another team, rinse and repeat.
         | 
         | The issue is often that complexity is accidental, due to the
         | design in place and the tech dept accrued till now, as opposed
         | to being inherent to the domain problem. But instead of
         | addressing the accidental complexity, it is simply broken down
         | in two so we can throw more people behind taming it.
        
         | ryanSrich wrote:
         | This is why, after being both an engineer and a product manager
         | (and VP of Product), and now a CTO of a seed stage startup, I'm
         | fairly convinced PdMs are unnecessary. At least for the the
         | type of company I'd like to build (WhatsApp as a primary
         | example of that).
         | 
         | Off loading the founders vision to someone else is a failure of
         | both communication and org structure (at the seed-SB stage).
         | People may not think hiring their first PdM is offloading their
         | vision, but it is. That person is going to come up with their
         | own ideas, validate their own assumptions, and push for product
         | updates that directly serve their own self interests. To think
         | that this won't happen is naive.
         | 
         | What I've found works the best is hiring competent engineers,
         | giving them adequate business context, a decent design spec,
         | and letting them figure it out.
        
           | nickthemagicman wrote:
           | >> adequate business context, a decent design spec, and
           | letting them figure it out.
           | 
           | I wish more PM's thought like you.
        
             | toomanyrichies wrote:
             | Ditto. As an engineer, it feels like product managers often
             | feel a need to keep cranking out new features as a way to
             | justify their seat at the table. I would love to work in an
             | engineering culture which found a way around that problem.
             | If that means the job falls on me instead, so be it.
        
               | nickthemagicman wrote:
               | They had me at adequate design spec.
        
           | 015a wrote:
           | This is always a controversial opinion, at least in some
           | circles, but: I've long said that the PM role was created
           | solely because companies can't find enough good engineering
           | talent.
           | 
           | That isn't to assert that PMs don't do fantastic, necessary
           | work. More-so that there's a more ideal reality where the
           | engineers should be doing that work; the outcomes, in my
           | experience, are better in orgs where engineering is not just
           | influential in, but responsible for, customers, business
           | outcomes, product planning, etc.
           | 
           | This isn't always possible. Most orgs don't resemble
           | Whatsapp; they resemble Google. A hundred product divisions
           | building a thousand different things. Its already difficult
           | to hire engineering talent to fill an org like this, let
           | alone the additional number you'd have to hire in order to
           | account for the fact that a double-digit percentage of an
           | engineer's role _should be PM-like stuff_.
           | 
           | So, throw away idealism; outsource that part of the engineer
           | role to another specialty which is easier to hire for: PM.
           | Its reasonable.
           | 
           | But the issue has arisen in that industry has forgotten than
           | this was a necessary sacrifice; its not desirable. We've now
           | got university and bootcamp programs dedicated to training
           | for PM roles. Many companies have entrenched the idea that
           | every team/org needs a PM. Tons of people are now adding
           | financial momentum to this idea that their skillset is
           | valuable and necessary; the ship can't be turned so easily.
           | 
           | At the end of the day, this is a disservice to many
           | engineers, as their people, product, and business skills fall
           | out of practice and they're further relegated into code-
           | monkey positions. Its actually a disservice to PMs as well;
           | they could have been trained in engineering and unlocked far
           | more professional mobility. In the worst cases, it leads to
           | PMs resembling the Office Space "what is it, exactly, you do
           | here" guy ("I bring the specs from the client, to the
           | engineers!" "Then, why don't the engineers just talk to the
           | clients?" "Well, they're not people people!" mhm)
        
             | jimbokun wrote:
             | > Most orgs don't resemble Whatsapp; they resemble Google.
             | 
             | When they were a startup, Google didn't resemble Google,
             | either.
             | 
             | For large organizations, I'm a fan of the "internal
             | startup" management structure. This just means each product
             | team should be largely self contained, but has the
             | resources of the larger organization to fall back on.
             | 
             | So the leader of each product team can act like a mini-
             | founder, defining and driving the product without being
             | tightly coupled to any other product teams release cycle.
        
           | [deleted]
        
           | twotwotwo03 wrote:
           | I can attest Product Managers were the downfall of one
           | startup I worked at. The company had dozens of millions in
           | funding and managed to piss it all away with ill-founded
           | escapades driven by a handful of Product Managers. If
           | anything, the Product Managers seemed to be managing their
           | careers, not the product. They babied engineers until most of
           | the engineers stopped caring about the product and just gave
           | up working hard at all. All the brilliant ideas were trite
           | insights you could read on two year old Gartner reports --
           | "release a cloud version".
           | 
           | CEO and CTO beware, do not let Product Managers happen to
           | your company.
        
             | AussieWog93 wrote:
             | On the flip side, my old PM had an incredibly deep
             | understanding of both the product and the customers buying
             | it (an area in which us SWEs were lacking), in addition to
             | a semi-hands-on understanding of the engineering process
             | too.
             | 
             | He wasn't a professional software engineer, but he took the
             | time to actually learn how to program through some side
             | projects and through this gained a real understanding of
             | what would be trivial for us to implement and what
             | wouldn't.
             | 
             | To top it all off, he had good rapport with the CEO and was
             | adept at managing upwards.
             | 
             | The product wouldn't be half as good as it was if not for
             | him.
             | 
             | Rather than tarring an entire profession with the same
             | brush, I think a much fairer "warning" to founders is to
             | make sure the people you're hiring are competent and that
             | the job they are doing is something that actually adds
             | value to the organisation. PMs, when deployed
             | appropriately, can be worth their weight in gold.
        
         | tomc1985 wrote:
         | I really don't like working for those 100 darts people. They
         | are a royal pain in the ass.
         | 
         | Much better are the folks with a solid vision who can drive
         | folks to execute on it, successfully.
        
         | wpietri wrote:
         | This is a great point, but I'd add that the problem isn't
         | experimentation. Neither is it having a lot of ideas. It's that
         | they aren't using good experiments to kill off most of the
         | ideas.
         | 
         | I think of it in terms of divergence and convergence. Divergent
         | thinking is great. If you think of an army on the march,
         | leaders will be thinking about possible paths and sending out
         | scouts. But when the scouts (experiments) come back with
         | reports (results) they don't just send some soldiers along
         | every path that looks promising. They winnow the options down,
         | converging their thinking.
         | 
         | Bad leaders don't have the discipline for that. And I think a
         | lot of them pick up that habit from sales and/or seeking
         | investment. "What will it do? What _won 't_ it do!" That's
         | great for manipulating the feelings of the person you're
         | currently talking to so they'll sign the deal. But it's
         | terrible for running an effective, focused product development
         | organization.
         | 
         | I know I've seen companies die like that. And it's probably way
         | more than we know, because as WhatsApp demonstrates, focusing
         | hard on what's working is a great way to succeed well enough
         | that people study and talk about you later.
        
         | SavantIdiot wrote:
         | > Higher quantity of ideas == More engineers. Higher quality of
         | ideas == Fewer engineers.
         | 
         | Part 1: By definition: more engineers = more ideas.
         | 
         | Part 2: Not so much.
         | 
         | On the one hand, mediocre engineers are sometimes sequestered
         | into small teams where they can't interfere with the important
         | projects that all the smart people and other resources are
         | being thrown at. They just sit there and churn out crappy ideas
         | and crappy non-vital tools. Like daily indicators.
         | 
         | On the other hand, I can see that as more of a tautology: ideas
         | come from one person and are grown by a team. So be definition
         | a quality idea comes from one person, and turns into a team.
        
         | def_true_false wrote:
         | That's all nice and well, but some of your examples are not
         | things that save effort -- using phone numbers is surely better
         | for network effects, but not in any way simpler then having
         | regular account registration, unless you just completely ignore
         | security.
         | 
         | WA apps are not better than the competition. Basic stuff like a
         | way to disable automatic downloading of media for groups you
         | are in is still missing. No desktop app. The web frontend is
         | still crap. Outsourcing account registration to mobile phone
         | companies is ridiculous (same as Signal) -- in most of the
         | world, it makes it pretty much impossible to preserve your
         | privacy while using WA. Endless nagging about plain text cloud
         | backups makes WA E2E encryption completely worthless, since
         | even if you opt-out, 99% of your contacts will have them
         | enabled.
         | 
         | The impressive part, considering the size of their team, was
         | their backend stuff and scale.
         | 
         | Discord, with similar team size, was able to build something
         | that succeeded in displacing IRC. (Their Android app was
         | laughably bad for the longest time as well.)
        
           | pessimizer wrote:
           | Whatsapp:Erlang as Discord:Elixir. The effectiveness of the
           | teams might more be the effectiveness of OTP/BEAM for network
           | applications.
        
             | dnautics wrote:
             | almost certainly part of it but not the whole story.
        
           | Zababa wrote:
           | > Discord, with similar team size, was able to build
           | something that succeeded in displacing IRC
           | 
           | I think they displaced Skype, mumble and teamspeak too.
        
             | rexreed wrote:
             | Discord displaced Slack in our organization and we're never
             | looking back at Slack ever again.
        
         | austincheney wrote:
         | This confuses the mistakes of a mature company. Different
         | dynamics.
         | 
         | For example Google had thousands of engineers back when they
         | had only 3 products: search, auction, and AdWords. I remember
         | Travelocity having over a thousand developers and not being
         | able to redesign a webpage.
         | 
         | Its deeper than a product problem. Its an execution problem.
         | It's the ability to do what works with great criticality
         | opposed to what's expected, standard, normal, or comforting.
        
         | that_guy_iain wrote:
         | WhatsApp was originally a super simple product. Send and
         | receive messages. It was fundamentally a solved problem. It was
         | all about the fact you could send free messages in a world
         | where sms cost 10 cents a message. It wasn't that their
         | messages were faster, their messages were anything other than
         | free.
         | 
         | They had a year fee of $1. Which I never paid yet I used their
         | service anyways. The reason they were able to scale so large
         | with so few devs is simple, their product was super simple. 50
         | tech people to build a messaging system is actually a lot.
        
           | stiltzkin wrote:
           | How things would have work around if Microsoft released MSN
           | Messenger for Android and iOS. They had the userbase and the
           | brand already.
        
             | billypilgrim wrote:
             | Why didn't MSN become what WhatsApp is today? I always
             | wondered how they could screw this up. Smartphones are
             | perfect for chat apps, that's basically what they are
             | supposed to be used for, and MSN / ICQ already covered a
             | gigantic portion of the instant messaging market. Did they
             | just assume people would stay on desktop for chatting
             | forever?
        
               | math-dev wrote:
               | Yeah Microsoft was too dysfunctional before, but they've
               | improved a lot recently and are killing it in many areas
        
               | ratww wrote:
               | Good question. I remember Microsoft had an app around
               | 2010, but it just failed to gain critical momentum, maybe
               | because people were already moving en masse to Facebook
               | and WhatsApp itself.
               | 
               | The app was only ok-ish. There were some alternative apps
               | in the AppStore. But people just preferred talking trough
               | WhatsApp and Facebook Messenger (which was already two
               | years old at this point). I guess people gravitated to
               | WhatsApp because it was less cluttered and snappier, and
               | to Facebook because it also had a proper social network
               | in it, but I have no idea. Two years later Microsoft
               | would kill MSN and integrate it with Skype.
        
             | RF_Savage wrote:
             | Yep. They ware the dominant instant messenger here. They
             | would have had a massive user base from day one.
        
             | Scoundreller wrote:
             | Or AOL, but maybe it died out too soon?
        
           | daanlo wrote:
           | In my recollection it wasn't about saving 10 cents in the
           | beginning, but about saving 1-2$ per message, since one of
           | the first use cases it enabled was free international
           | texting. That was a HUGE benefit. All the first evangelists I
           | knew were in long distance relationships ;)
        
           | Jensson wrote:
           | > The reason they were able to scale so large with so few
           | devs is simple, their product was super simple. 50 tech
           | people to build a messaging system is actually a lot.
           | 
           | Tell that to Twitter. Whatsapp staying at 50 instead of
           | growing to 5000 is impressive.
        
             | that_guy_iain wrote:
             | Different product. One is a social media network and one is
             | a messaging app.
        
               | relaxing wrote:
               | You say that now, but Twitter also began as a way to send
               | a quick message to your friends.
        
               | recursive wrote:
               | You used to be able to use it as a way to create a free
               | self-managing SMS distribution list. Pretty cool, before
               | I had phone with data capabilities.
               | 
               | That was even a use case I understood. Now I feel too old
               | to understand how to use twitter. But that can't be it.
               | Much older people seem to have no problem.
        
             | jokethrowaway wrote:
             | That's unfair, Twitter had to wrestle scaling with Rails
             | while WhatsApp used Erlang which is a perfect match for
             | scalable communications.
        
         | chpmrc wrote:
         | > no hundreds of stupid features that product people stuff down
         | in the name of experimentation.
         | 
         | Amen to that!
        
         | ksec wrote:
         | >WhatsApp success is not about what WhatsApp did, it's about
         | everything they didn't do.
         | 
         | I wrote something similar [1] about its success not long ago on
         | HN,
         | 
         | >WhatApp grew because it was the only IM that worked on
         | Smartphone. Thanks to Erlang. Yes. ICQ didn't bother. Microsoft
         | refuse to accept Smartphone is the future of computing hence
         | there never was MSN on Smartphone. Other contender tried but
         | none of them managed to Scale. I still remember all my
         | colleagues and friends used to download instant messenger on
         | our phone to test them out, creating a group and basically DDoS
         | each other until the system or network crash. No IM was able to
         | handle the traffic load, not only WhatsApp did it many times
         | better than their competitors, they also had almost all
         | platform support, from Symbian to Blackberry.
         | 
         | There is another part of the story that dont get much
         | appreciation, is how they managed 2 million connection per box
         | on a what is now a relatively slow server. The latest M1 Max
         | would have been a lot faster. I often wonder if they could have
         | pushed to 10 Million connection per box with same latency under
         | current CPU and Memory. ( Not saying it is a good idea. )
         | 
         | [1] https://news.ycombinator.com/item?id=25984924
        
         | numair wrote:
         | It's comical that all of the replies to what you've written
         | claim you're wrong, because you're actually 100% right. And
         | this is even more true when you remember that the company had
         | fewer than 20 people for a long time, and that there was only
         | _one real goal_ , that seems to be forgotten: they needed to
         | clone BlackBerry Messenger and get it working on every platform
         | possible. That was it. That's all they needed to do, and that's
         | really all that they did.
         | 
         | I tried to convince the CEO of Research In Motion, who I worked
         | for, that WhatsApp would end up destroying his business. He
         | thought I was insane for thinking something so small and simple
         | could have such a big effect on his big, important company.
         | Yeah, well, whatever you say, man.
        
           | dleslie wrote:
           | Also can't overlook that for many SMS was ludicrously
           | overpriced, but the early basic smartphones (palm and
           | symbian) often had wifi capability; WhatsApp provided secure,
           | ludicrously cheap instant messaging across all meaningful
           | handsets.
           | 
           | And that's all it had to do; the telecoms weren't interested
           | in making SMS cheap or secure, RIM didn't want BBM on
           | anything but their devices, ICQ/AIM/MSN/etc were similarly
           | unavailable.
        
             | Griffinsauce wrote:
             | > WhatsApp provided secure, ludicrously cheap instant
             | messaging across all meaningful handsets.
             | 
             | You can scrap secure from this, the vast majority of users
             | don't care or even know what that means.
             | 
             | If SMS was more secure I 100% would still expect whatsapp
             | to win. Convenience and cost are the biggest drivers here.
        
               | Scoundreller wrote:
               | TBH, I was incredibly disappointed when I discovered that
               | BB shared the same encryption key for all users unless
               | you were on their enterprise system. While I'm just one
               | user, I also discouraged others from buying into them
               | too.
        
               | jimbokun wrote:
               | > You can scrap secure from this, the vast majority of
               | users don't care or even know what that means.
               | 
               | Until you get sued or called in front of legislative
               | bodies to testify about why your customer data was
               | breached.
               | 
               | But maybe you're big enough at that point to ride it out.
               | 
               | On the other hand, it's very very hard to add in robust
               | security after reaching critical mass.
        
               | Griffinsauce wrote:
               | I have no idea how your point connects to mine. We're
               | talking about consumers choices here.
        
               | dleslie wrote:
               | Ensuring user privacy _from you_ can be an effective
               | defensive measure for small companies who cannot afford
               | to mount a meaningful legal defense.
               | 
               | "Here's all the data we have, officers. We don't have the
               | keys. Good luck."
        
               | Griffinsauce wrote:
               | Again, how does this connect with why Whatsapp won as an
               | app?
               | 
               | Consumers didn't follow this logic. At all. They went for
               | the convenient and free option.
               | 
               | I'm so confused by this thread. It's like you guys are
               | pasting the output of an ML model with only the input
               | "secure" at me.
        
               | AussieWog93 wrote:
               | >I'm so confused by this thread. It's like you guys are
               | pasting the output of an ML model with only the input
               | "secure" at me.
               | 
               | You talked about encryption on HN. What do you expect?
               | Might as well have brought up gun control at a Republican
               | rally.
               | 
               | (Not that you were anything but 100% right. The general
               | public didn't care about encryption at all until the
               | Snowden leaks.)
        
               | HeyLaughingBoy wrote:
               | They still don't!
        
               | AussieWog93 wrote:
               | I think that depends on how you define "general public".
               | My mum doesn't, but my (far more digitally savvy) teenage
               | cousin does.
               | 
               | Back in 2009, though, this was a very different story.
               | Even tech-savvy people didn't care.
        
           | math-dev wrote:
           | Nice story. What are you doing now?
        
             | numair wrote:
             | Post-FAANG digital infrastructure. This is that moment,
             | when the cycle starts all over again.
        
               | EGreg wrote:
               | What specifically?
        
               | angelzen wrote:
               | Thanks for sharing your insights. With all kindness, this
               | sounds like post-UnionPacific. The neo robber barons have
               | won. To go on with the metaphor, not sure the future
               | holds a car to fulfill point-to-point transportation
               | demand, as our digital information needs are vastly
               | oversupplied.
               | 
               | What kind of unmet demand you see out there?
        
               | dleslie wrote:
               | Anything of note we humble readers should take a look at?
        
               | numair wrote:
               | I'm having one of those "why can't there be 36 hours in a
               | day" moments in life, but I've got to publish more about
               | this very soon, so, it's on the way.
               | 
               | When you combine the story of WhatsApp's small team /
               | high impact development with the remote work shift of the
               | past few years, it seems that the next cycle (regardless
               | of when/what it is) will be the first that isn't centered
               | on SF/SV/California. It's interesting to think of how
               | that'll affect the psychology of the area.
        
               | 908B64B197 wrote:
               | > will be the first that isn't centered on
               | SF/SV/California.
               | 
               | I've heard this every 5 years for the last... 25 I think?
               | 
               | Still hasn't happened.
        
               | math-dev wrote:
               | Looking forward to reading it, plz let us know once done
        
           | Scoundreller wrote:
           | Explains how they doubled down on hardware and lost.
           | 
           | BB's market cap is currently a bit over 1/3rd of whatsapp's
           | acquisition price. And BB has minimal long term debt.
        
           | 908B64B197 wrote:
           | > I tried to convince the CEO of Research In Motion, who I
           | worked for, that WhatsApp would end up destroying his
           | business. He thought I was insane for thinking something so
           | small and simple could have such a big effect on his big,
           | important company. Yeah, well, whatever you say, man.
           | 
           | I remember selling every BlackBerry stock I had when I got my
           | hands on their touchscreen phone (the Storm was it?).
           | 
           | 3 years behind the original iPhone, while Apple had just
           | launched the 3G with the App Store. Guess which phone (and
           | stock) I purchased then?
           | 
           | Was there really no-one at BlackBerry following what was
           | going on in the valley at that time?
        
             | giantrobot wrote:
             | There's a number of stories about the insane level of
             | hubris at RIM. Before the iPhone smartphones were seen as
             | "business" phones. RIM reportedly felt like hot shit
             | because they had managed to just _not fuck up_ while Palm,
             | Microsoft, and Nokia tripped over themselves to try to make
             | a BlackBerry But Better.
             | 
             | RIM was so focused on the "business" market they really
             | couldn't conceive of a real consumer device. Their upper
             | management apparently looked at the _billions_ of feature
             | phones sold (IIRC Motorola had sold a quarter billion RAZRs
             | by then) and said  "nah let's double down on the order of
             | magnitude smaller business market". They dipped their toe
             | in feature phones with the Pearl but it was a joke compared
             | to the original iPhone with its original software that
             | didn't even support MMS.
             | 
             | By starting with more consumer friendly features (Web,
             | media player, phone features) Apple had an extremely
             | competent baseline of functionality. As soon as they added
             | workable Exchange integration (iOS 3 or 4?) the iPhone
             | (including older models) instantly became workable business
             | devices. RIM had far more ground to make up for going from
             | business to consumer than Apple had going consumer to
             | business.
             | 
             | https://archiveprod.canadianbusiness.com/technology-
             | news/how...
             | 
             | https://techland.time.com/2013/09/29/the-inside-story-of-
             | the...
             | 
             | https://sites.psu.edu/global/2018/02/09/blackberry-and-
             | the-i...
        
           | i_have_an_idea wrote:
           | > I tried to convince the CEO of Research In Motion, who I
           | worked for, that WhatsApp would end up destroying his
           | business. He thought I was insane for thinking something so
           | small and simple could have such a big effect on his big,
           | important company. Yeah, well, whatever you say, man.
           | 
           | Well, he had a point. What killed BB was the iPhone, not that
           | their chat app was going to lose out to the competition. They
           | simply had a product that was inferior in every single
           | important way.
        
             | hello_moto wrote:
             | It's attack from both directions:
             | 
             | BB and BBM are network effect. iPhone alone is not enough
             | to disrupt the social-media aspect of BBM (which was very
             | sticky at that time in quite a few countries).
             | 
             | WhatsApp + Android actually killed BB+BBM in those
             | countries and not the expensive iPhone
        
             | cinntaile wrote:
             | People owned a BB because of its built-in message service.
        
               | aeoanx wrote:
               | A lot of people liked the keyboard
        
         | jeltz wrote:
         | As an end user I think WhatsApp is a terrible product for some
         | the reasons you mentioned (tied to phone number, no desktop
         | client) but I cannot deny that it was the smart thing to do
         | from a business perspective.
        
         | toast0 wrote:
         | > They slice and dice the org until the 30 people become split
         | into teams of 2-3 with a manager for each, running into
         | coordination hell on each step
         | 
         | This is the key mistake. Having teams of 2-3 with a manager
         | each is way too many managers. _If_ you have a clear focus,
         | small teams are great because they can work autonomously, with
         | little management and good results. You can 't hire your way
         | out of coordination hell.
         | 
         | I was at WhatsApp between 2011 and 2019; for the most part
         | early on, we had essentially the CEO who does a mix of product
         | management and engineering management and engineering, an
         | actively engineering server manager (who also did some client
         | work), and a client engineering manager. Android team grew
         | enough to have an engineering manager and we also ended up with
         | one product manager. Post aquisition, staff grew and so did
         | management.
        
           | MomoXenosaga wrote:
           | Post acquisition WhatsApp had to start making money.
           | 
           | For all the doom and gloom Facebook is still too scared of
           | putting ads in there.
        
             | laurent92 wrote:
             | At least they can use it to harvest people's networks and
             | interests. Even more efficiently than Facebook itself.
        
             | Scoundreller wrote:
             | Not letting a Facebook competitor get ahold of it has value
             | in itself.
        
         | ehPReth wrote:
         | no iPad app either :/
        
         | ryandrake wrote:
         | I've always called this Feature Cram, and it's standard
         | procedure at basically every company I've ever worked, small
         | and large, B2B or B2C. Nobody is willing to say "Our software
         | is a success and it's _done_. Let 's fix the major bugs, put it
         | into maintenance mode and start a new, separate, great idea!"
         | Nope, instead it's always "OK the MVP is going well. Let's jam
         | ten other unrelated things into it and see what takes off and
         | what are the dogs (and be stuck with the dog features as
         | technical debt forever). Then, when none of them reach the
         | success of the original actually solid idea, they hire moar
         | people and jam 40 more feature ideas in. Eventually, the
         | original great thing that brought the company success is so
         | hard to find and use because of the weight of the other crap,
         | that a new upstart that Just Does One Thing Well comes in and
         | eats their lunch.
         | 
         | Then, at that new upstart, someone asks "What else can we bolt
         | on to this thing to make it really grow??"
        
           | bobthepanda wrote:
           | This may very well be unpopular, but this is what happens
           | when you pay someone in measures tied directly to share
           | price, or even more directly in shares; they have every
           | incentive to do whatever it takes to juice the share price
           | up. VC money has even more extreme growth expectations.
        
           | Closi wrote:
           | Yes, although there is a balance - lest we forget that you
           | couldn't use WhatsApp on your computer unless your phone was
           | turned on until a month ago, and there is still no iPad app.
           | 
           | I think WhatsApp is in the place where it is so ubiquitous in
           | some social circles that it faces less competitive pressures
           | than other apps and can afford some of these pretty large
           | gaps.
        
         | amelius wrote:
         | Uh yeah, but it was only possible to have such a simple product
         | because they reached mass adoption. Without network effects,
         | you've got to differentiate and thus add complexity to gain
         | users.
        
           | Spivak wrote:
           | But then how did they reach mass adoption in the first place?
           | It's not like they were a Swiss Army knife and before and
           | stripped features once they got big.
        
             | amelius wrote:
             | That's a totally different question. A larger company such
             | as Microsoft could have easily grabbed the market first,
             | but they didn't.
        
               | abduhl wrote:
               | This comment flies in the face of actual history. While
               | your point about Microsoft may be right (who knows with
               | how they've ruined their messaging platforms?) it
               | certainly doesn't hold true for iMessage which came out
               | in roughly the same timeframe as WhatsApp (2011 vs 2009).
        
               | saghm wrote:
               | iMessage still has yet to come to other platforms, and at
               | least for a few years after the iPhone's launch, many
               | people viewed it as a luxury product and were unlikely to
               | spend the money to get one compared to cheaper phones
               | (which WhatsApp supported).
        
             | et-al wrote:
             | They had fantastic product-market fit.
             | 
             | As dleslie and that_guy_iain have mentioned, it was
             | expensive to send an SMS text back then, especially in
             | Europe when your friends have different country codes. So
             | WhatsApp was a free alternative that tapped the unmet need
             | for millions.
             | 
             | And signing up with only a phone number seems like a
             | nobrainer nowadays, but the mindset of the time was still
             | email + password. Sure, Microsoft could have launched a
             | competitor, but they would most certainly require you to
             | use an MSN account, thus adding friction to adoption.
        
         | onlyrealcuzzo wrote:
         | So is SpaceX a low quality idea? Some problems are just not
         | feasibly solved by a handful of people. Does that make them low
         | quality ideas?
        
       | mkl95 wrote:
       | Ultimately, how many engineers a company has is not that relevant
       | when it comes to scaling.
       | 
       | A small team of product-oriented engineers with a devops
       | mentality can easily develop, maintain, and continuously deploy a
       | considerably complex application.
       | 
       | A similarly complex project in the hands of a larger sales driven
       | company often resembles a spaghetti ball, and is held together
       | with rotting shell scripts and unproductive manual processes.
        
       | jokoon wrote:
       | I remember reading they worked very hard to make their app
       | available on many, many types of phones. Which is weird because
       | they regularly announce they drop support for certain phone OS.
        
         | nnx wrote:
         | It made sense years ago... before the iOS/Android duopoly we
         | have now.
        
       | streamofdigits wrote:
       | Beware of survivorship bias.
       | 
       | Team does X, does not achieve scale, nobody knows about it
       | 
       | Team does X, achieves scale, X is the new sliced bread
        
         | papito wrote:
         | The first team does not achieve scale because they are still
         | building microservices. The word of the day is "lean".
        
           | streamofdigits wrote:
           | intuitively I would agree, but the point of my comment was
           | that intuition might be a poor guide if we want to understand
           | what kind of team and software architectures "succeed".
           | 
           | if you rewind some years microservices was also the word of
           | the day (and probably there were some iconic projects picked
           | to justify...)
        
       | kosolam wrote:
       | WhatsApp is a spy app of Facebook nowadays
        
         | cyberpsybin wrote:
         | And iMessage is compromised bloatware
        
         | oblio wrote:
         | That was its fate anyway.
         | 
         | Some things just can't really make money unless they do it in
         | scummy ways (or we haven't found a viable alternative or a way
         | to protect them).
         | 
         | Public instant messengers, public forums, public file hosting,
         | public image hosting.
        
           | globular-toast wrote:
           | Whatsapp is built on a massive, mature, global system
           | specifically designed for sending messages. It's sad that it
           | needs to exist and even more sad that it has to "make money"
           | for someone. Aren't we already paying for our internet
           | connexion/phone bill? Why do we need to be used by Whatsapp
           | just to utilise it for its most basic purpose?
        
             | dunefox wrote:
             | Because it needs to be built, supported, etc.?
        
               | globular-toast wrote:
               | Does it? Dumbphones from 30+ years ago can still send SMS
               | messages just fine with zero software updates. Maybe once
               | a decade or so that could have been enhanced with longer
               | messages, pictures etc. But overall the messaging problem
               | really should have been solved by now.
        
               | dunefox wrote:
               | Yes, it does. SMS aren't encrypted. And messaging is
               | closer to being solved than ever before. If you want to
               | use a "dumbphone" with SMS you can do that.
        
           | shawabawa3 wrote:
           | With 50 employees their "please pay $1/year" model might have
           | actually made enough revenue
        
             | jmrm wrote:
             | I paid totally happy with the service they gave me, and I
             | would pay monthly if that would mean having total privacy.
        
             | oblio wrote:
             | Nobody I know paid it even back then. That's why they used
             | it, because it was free.
             | 
             | It's like Microsoft Office these days. If your home license
             | expires they keep threatening you they'll disable it. For
             | years and years.
        
           | alanfranz wrote:
           | Whatsapp used to charge 1 eur or usd per year per user.
           | 
           | Assuming a 20% vat, that's $800M in yearly revenue. If you
           | spend $500k per engineer per year, you're still left with
           | $775M to handle infrastructure and all other expenses.
           | 
           | Whatsapp would have been sustainable without fb. That's the
           | regret.
        
             | mrweasel wrote:
             | They could also easily have had different pricing based on
             | location. Requiring EU and US users to pay EUR5 per year,
             | while have perhaps just EUR0,25 in Asia, and free in
             | Africa.
             | 
             | If you can't get someone to pay EUR5/$5 per year for a
             | service they use every single day, then there's something
             | really wrong in how we as users think about the service we
             | use every day.
        
               | dunefox wrote:
               | > If you can't get someone to pay EUR5/$5 per year for a
               | service they use every single day, then there's something
               | really wrong in how we as users think about the service
               | we use every day.
               | 
               | Which is absolutely the case with most people and
               | recurring payments, even with one time payments. If
               | Signal costed money (even 1-2$), I could not have
               | convinced even 10% of the people I got over to Signal -
               | recurring or one time.
        
             | harpratap wrote:
             | You are very naively assuming it would actually reach 1B
             | users if it was paid.
        
               | [deleted]
        
               | grardb wrote:
               | Technically, but GP is allocating over 95% of post-tax
               | revenue to everything other than engineer salaries, which
               | is probably extremely generous.
               | 
               | WhatsApp had 200M users in February 2013[1], which was a
               | year before they were acquired by Facebook. Perhaps they
               | wouldn't have gotten to 1B users without Facebook, but
               | I'm sure they still would've grown in the last ~8 years,
               | and fewer users likely results in lower costs. My guess
               | is they would still be doing very well.
               | 
               | [1] https://en.wikipedia.org/wiki/WhatsApp
        
               | YetAnotherNick wrote:
               | Even before the sale, most of the customers are non
               | paying. They are offering the service for free in
               | countries with largest userbase(India and Brazil I
               | think). I really doubt it had reached 50 million paying
               | at any point in time.
        
             | kosolam wrote:
             | Exactly. Facebook bought WhatsApp's users like cattle on
             | auction. WhatsApp guys just needed the right price offer.
             | Now with pseudo tech posts like the current they are trying
             | to mask it's public image behind some ancient tech
             | innovation from it's past...
        
               | foepys wrote:
               | WA Management was misled by Facebook about what FB wanted
               | to do with WA. I agree that you'd have to pretty naive to
               | believe FB but it's not like WA founders sold solely
               | because of money.
        
               | lotsofpulp wrote:
               | I think you would have to be pretty naive to think WA
               | founders were above having a price at which they would
               | sell solely for the money.
        
               | foepys wrote:
               | WhatsApp founder Brian Acton left 850 million USD on the
               | table when he left Facebook prematurely in protest of
               | Facebook's lies about the vision they had for WhatsApp
               | after its acquisition. After that he donated 50 million
               | USD to the Signal foundation.
        
             | mrich wrote:
             | Facebook to WhatsApp team in the acquisition negotiations:
             | "If you don't sell to us, we'll build our own and offer it
             | for free"
             | 
             | And they would have been out of business sooner or later.
             | There are enough people wanting to save that one dollar.
        
               | the_other wrote:
               | Facebook now have to maintain Messenger AND WhatsApp. And
               | Insta also has chat. So I disagree that FB could have
               | killed WA.
        
               | the_other wrote:
               | I'm tempted to go further. FB bought WA because it would
               | have killed their ad business.
        
               | hbn wrote:
               | > If you don't sell to us, we'll build our own and offer
               | it for free
               | 
               | Weren't they already doing that? Facebook Messenger
               | officially launched in 2011 (essentially an upgrade to
               | Facebook's built-in chat functionality which existed
               | before that) and was free, yet Whatsapp continued to
               | grow.
        
               | pydry wrote:
               | There are plenty of other ways they could have made money
               | that weren't scummy and didnt require hitting up every
               | user for $1.
               | 
               | Business accounts, facilitating p2p payments (minus a
               | scummy new coin), etc.
               | 
               | Sustainability would have been possible in all sorts of
               | ways even though massive levels of profits (in any
               | company) inevitably means being scummy.
        
             | dncornholio wrote:
             | I've used Whatsapp for a few years when they had this
             | pricing model. Use 1 year for free and pay after. I've
             | never paid anything and nobody who I know needed to pay to
             | keep using WhatsApp.
        
               | ducklingslicks wrote:
               | I think it was free on Android but not on iOS. But not
               | completely sure because I only had Android back then
        
               | robjan wrote:
               | It was a one off payment on iOS but a poorly enforced
               | annual subscription on Android
        
             | raverbashing wrote:
             | Much cheaper than what fb makes on ads per user per year,
             | I'm sure
        
           | KptMarchewa wrote:
           | 1) They won't succeed unless they were international from the
           | beginning.
           | 
           | 2) A lot of people would be rightly suspicious to use govt
           | systems for their own private data.
        
           | jfklsdfjsdf wrote:
           | Makes me wonder just what FB will do to Whatsapp to keep the
           | money flowing in, as FB's core app/website declines.
        
             | shapefrog wrote:
             | Plan A: Ads
             | 
             | Plan B: ummm
        
           | octodog wrote:
           | Whatsapp had 700m+ monthly users in early 2015 [1] before
           | they scrapped their subscription fee in early 2016 [2].
           | Whatsapp was free to use in some jurisdictions for the first
           | year, so we can't assume there was $700m in revenue, but
           | there was certainly enough to pay for 50 engineers you would
           | think.
           | 
           | [1] https://www.businessinsider.com/whatsapp-
           | passes-700-million-...
           | 
           | [2] https://blog.whatsapp.com/making-whats-app-free-and-more-
           | use...
        
             | vadfa wrote:
             | Whatsapp only had a subscription fee for android users, and
             | I don't know anybody who ever paid for it. They would keep
             | giving you more and more free time; free subscription time
             | would never run out. I imagine it ran out for some users,
             | but I don't know what the criteria was. Perhaps for users
             | in certain countries? Or for users with lots of iphone
             | friends?
             | 
             | On the other hand, iphone users had to pay for the app
             | upfront to download it off the app store.
        
       | thrower123 wrote:
       | Fifty engineers almost sounds like too many, if you aren't using
       | most of them to develop internal frameworks and reinvent wheels.
       | 
       | You'd have an Android team, an iOS team, a couple teams working
       | on core APIs and infrastructure. Ideal team size is about five.
        
       | 28194608 wrote:
       | Dear whatsapp devs please disable the feature which shows how
       | much time and when to EVERYONE in the world. TY.
        
       | unsungNovelty wrote:
       | They not only used FreeBSD for their servers, their CEO at the
       | time Jan Koum donated $1 million dollars to FreeBSD foundation
       | too. Talk about giving back!
       | 
       | The only thing they failed at was making money. I really wished
       | they had. They knew what Whatsapp meant to the world. Look at WA
       | now.
       | 
       | https://freebsdfoundation.blogspot.com/2014/11/freebsd-found...
        
         | paxys wrote:
         | WhatsApp was successful only because they had no intention of
         | ever making money. Simple lightweight app, no tracking, no ads,
         | no upsell. Yet they were funded by VCs who would want a return
         | on their investment at some point.
        
         | moralestapia wrote:
         | >The only thing they failed at was making money.
         | 
         | Huh? They're all million/billionaires now. That part of the
         | plan seems to have worked.
        
           | cerved wrote:
           | presumably they meant revenue
        
           | btbuildem wrote:
           | Because they sold to FB. I think parent meant "they failed to
           | make money on their own"
        
             | moralestapia wrote:
             | You probably don't know the story. Selling was part of the
             | plan all along. Whatsapp didn't even try to make money on
             | their own.
        
               | aliswe wrote:
               | source?
        
               | unsungNovelty wrote:
               | The founders hearts were in the right place. Brian Acton
               | created Signal Foundation and Jan Koum like I said also
               | have done good things.
               | 
               | They also tried the paid version for a while if you
               | remember. And they were also dead against ads on the app.
               | Unless I missed something or if I am wrong somewhere (At
               | which point I would love to be corrected), Selling was
               | the plan all along doesn't seem like a right observation.
        
               | moralestapia wrote:
               | >The founders hearts were in the right place. Brian Acton
               | created Signal Foundation and Jan Koum like I said also
               | have done good things.
               | 
               | I'm not saying they are bad people, and selling is not a
               | bad thing either. I'm just pointing out that selling was
               | the plan all along.
               | 
               | The WhatsApp story that gets told regularly (like in this
               | instance) is a story of how such few employees and such
               | small infrastructure is enough to build a massive
               | product. You know what that translates to? Low opex. If
               | they wanted to, WhatsApp would've been quite profitable
               | if they wanted but instead they chose to sell to the
               | highest bidder.
               | 
               | Again, that's not a bad thing, but also, I'm sure there
               | were PLENTY of people that were interested in buying a
               | billion user platform with 99% user retention. Why did
               | they sold to Facebook? Only they know, but back then
               | Facebook already had a tarnished reputation, so they
               | definitely didn't do it "because of their mission and
               | values".
        
               | _1 wrote:
               | Wasn't it $1 per year before Facebook bought them?
        
               | jonasdegendt wrote:
               | I believe so, but wasn't it much like the Winrar license
               | where nobody was actually forced to pay at the end of the
               | day?
        
               | Jensson wrote:
               | It was free for the first year iirc, and then a dollar a
               | year. A dollar a year is way more than needed to cover
               | the cost of hosting, and most would spend that to avoid
               | paying for SMS.
        
               | pradn wrote:
               | They ended up with a billion or two users. $1-2 billion
               | with the subscription is good for a company of a few
               | dozen employees. They could have branched into all sorts
               | of value-add stuff in the future if they wanted to, all
               | without tracking and such even. A simple payments or
               | shopping interface a la Instagram Stores could have done
               | the trick.
        
           | unsungNovelty wrote:
           | I meant they as in Whatsapp - the company. And yeah to be
           | more specific, I meant making revenue on their own.
        
       | seanhandley wrote:
       | I wonder if WhatsApp could be considered the most successful
       | user-facing application with an Erlang backend.
        
         | polyomino wrote:
         | What other candidates are there?
        
           | bradhe wrote:
           | Facebook Messenger, if I remember correctly, was also
           | originally authored in Erlang.
        
             | mewpmewp2 wrote:
             | Discord also if I remember correctly?
        
               | [deleted]
        
               | d3nj4l wrote:
               | Discord was written in Elixir!
               | https://blog.discord.com/scaling-elixir-f9b8e1e7c29b
        
               | mewpmewp2 wrote:
               | Does it mean it's technically also based on Erlang?
        
               | avianlyric wrote:
               | It runs on the Erlang BEAM virtual machine. But it's very
               | much it's own language, and looks and feels very
               | different to Erlang.
               | 
               | I would say the Elxir is based on Erlang about as much as
               | Scala is based on Java.
        
           | [deleted]
        
           | peoplefromibiza wrote:
           | while WhatsApp is the largest I am aware of (it's one of the
           | leading platforms Worldwide anyway), there are quite a few,
           | sometimes in the form of Elixir:
           | 
           | - Discord
           | 
           | - Pinterest
           | 
           | - Spotify
           | 
           | - Klarna
           | 
           | - Riot games (the messaging service)
           | 
           | - PepsiCo (for their billion making e-commerce)
           | 
           | - Toyota (for the connected platform)
        
         | a_humean wrote:
         | What about its original purpose, which was telecoms
         | infrastructure?
        
           | seanhandley wrote:
           | Oh, absolutely.
           | 
           | I framed it this way intentionally. Without a doubt the most
           | successful Erlang apps are telecoms related.
        
             | true_religion wrote:
             | WhatsApp is basically modern SMS. That's sort of a telecom
             | application.
        
             | ddalex wrote:
             | Almost like Erlang was designed by a telecom company ....
        
             | nodejs_rulez_1 wrote:
             | Telecoms probably funded Erlang way more than WhatsApp did.
        
           | feketegy wrote:
           | Probably that's why they opted for Erlang.
        
       | shmatt wrote:
       | There was a joke, or maybe anecdote, going around the Israeli
       | high tech industry when Google purchased Waze
       | 
       | When it came time to get an overview of the codebase being
       | purchased. The Google Maps car navigation iOS team had over 200
       | engineers. The Waze iOS team had 2
        
         | Jensson wrote:
         | > The Google Maps car navigation iOS team had over 200
         | engineers.
         | 
         | I don't really believe that. I worked at Google and the
         | frontend teams were usually just a couple of engineers per
         | target client, I doubt any projects had 200 entire engineers
         | just for a single target client. Not even big projects like
         | gmail has nearly that many.
         | 
         | Edit: If I were to guess Google maps would have 200 engineers
         | in total, or at least 200 figure would be the entire team of
         | frontend engineers for every target client, and then many more
         | working in other roles. 1 engineer per million users is a
         | pretty good rule of thumb for Google products, and a large
         | majority of those will work on backend systems.
        
         | lightning19 wrote:
         | off topic but I recently started working with an Israeli tech
         | company and I'm amazed by the culture they have, what about
         | Israel produces such high levels of entrepreneurship?
        
       | zibzab wrote:
       | Remember the fast-cheap-secure triangle of product development?
       | 
       | Now guess which one they skipped
        
         | southerntofu wrote:
         | The client is another story, but for the server they forked
         | ejabberd which is a reliable, established XMPP server. So they
         | probably had "fast-cheap-secure" on the server-side, because 6
         | years had been spent ironing out ejabberd bugs before the first
         | version of Whatsapp came out.
         | 
         | My understanding is that vulns in Whatsapp so far were client-
         | side, but i'm interested if i missed some on the server side.
        
           | Zash wrote:
           | They had some homebrew encryption and authentication before,
           | see eg https://blog.thijsalkema.de/blog/2013/10/08/piercing-
           | through...
           | https://blog.thijsalkema.de/blog/2013/10/08/piercing-
           | through...
        
             | southerntofu wrote:
             | > homebrew encryption
             | 
             | What could possibly go wrong? :)
             | 
             | Thanks for the links!
        
         | tim333 wrote:
         | I'm not sure the development was super fast. They also kept it
         | fairly simple.
        
       | bjarneh wrote:
       | This article does very little to explain how they did that. It
       | basically states the type of tech they used (Erlang, FreeBSD and
       | SoftLayer), and something about not trying to over engineer
       | stuff. The title also makes it sound like 500 engineers would
       | make it easier to scale up WhatsApp, which does not make too much
       | sense either.
        
         | LewisVerstappen wrote:
         | Hey,
         | 
         | Author of the article here.
         | 
         | Totally agreed. Apologies for that!
         | 
         | The article isn't really "an article" but it's an email
         | newsletter I send out. I'm quite surprised to see it on the
         | frontpage of HN lol.
         | 
         | So what you're reading is one of my emails.
         | 
         | It's meant to be more of a summary with additional links to
         | sources where people can read more if they're interested
         | (emailing people long-form articles doesn't work well as a
         | content format from what I've seen - even ignoring the length
         | limit that email has).
         | 
         | I'll add in some more links for additional reading (like the
         | high scalability post) to give some more detail for people
         | interested.
         | 
         | Thanks a lot for the feedback.
        
         | willejs wrote:
         | Theres a high scalability post from years ago that has a bit
         | more information.
         | http://highscalability.com/blog/2014/3/31/how-whatsapp-grew-...
        
           | bjarneh wrote:
           | Thanks, it does seem like choosing a language that was made
           | for doing things in parallel to begin with was a good idea.
           | To be honest, I've always felt that programming in functional
           | languages creates less bugs, but perhaps that's just my
           | imagination.
        
             | dunefox wrote:
             | It certainly creates fewer bugs of a specific nature,
             | depending on the language.
        
           | aledthemathguy wrote:
           | this is great. wondering if Telegram also have a post similar
           | to this? (unlike whatsapp, they do store the messages)
        
           | vadfa wrote:
           | >19B messages in & 40B out per day
           | 
           | whatsapp being a closed system I don't understand how this
           | disparity is possible.
        
             | the_other wrote:
             | Most people are in group chats.
        
         | Cthulhu_ wrote:
         | The article is honestly kinda low on details, depth or new
         | insights, but at least it links to some sources.
        
         | Closi wrote:
         | > The title also makes it sound like 500 engineers would make
         | it easier to scale up WhatsApp, which does not make too much
         | sense either.
         | 
         | It's something I think we have all seen a lot of times - that
         | by the time a company is serving 1 billion users it has quickly
         | expanded out it's engineering team hugely, and because of that
         | additional abstraction is required and the complexity/LOC
         | skyrockets.
        
           | bjarneh wrote:
           | > it has quickly expanded out it's engineering team hugely
           | 
           | But makes little sense (as a developer/engineer myself) to
           | think that growth in users, requires a ton of new
           | developers/engineers. We are not tattoo artists, our job can
           | scale indefinitely if set up properly, i.e. all you should
           | need to scale further (to 1BN or 7BN) should be enough money
           | to buy more hardware; which WhatsApp/Facebook clearly has.
        
             | macintux wrote:
             | You're assuming that their use case & technology stack &
             | application scale cleanly horizontally and they're not
             | spending all their time fighting fires.
             | 
             | True in this instance, but far from a universal truth.
        
             | Closi wrote:
             | I think that makes sense if the application doesn't need to
             | also grow much functionally during the expansion, but I
             | believe the more common pattern is more users = more
             | functionality to manage those users, support teams, global
             | legal compliance and accommodate more niche phone systems =
             | more engineers.
        
           | mnsc wrote:
           | "this team of 7 engineers is responsible for formatting the
           | text content of a chat, including alignment and sizes of
           | emojis and gifs"
           | 
           | "this team of 4 engineers is responsible for formatting the
           | date of a message"
           | 
           | "this team of 7 engineers is responsible for the overall
           | formatting of a chat"
           | 
           | "this team of 5 engineers is responsible for the formatting
           | of the non-chat pars of the application, settings, profile
           | page"
           | 
           | "this team of 4 ux persons is responsible for aligning the
           | non-technical parts of formatting and the user experience
           | over all parts of the application"
           | 
           | "this crossfunctional team of 5 is responsible for creating a
           | framework to let the configuration of formatting be
           | disconnected from the actual implementation of the
           | formatting"
           | 
           | "this team of 7 QA engineers will aid in manual verification
           | of changes and bugfixes but will also automate test cases for
           | formatting in the entire application"
           | 
           | "this supporting team of 4 will develop automation tools and
           | enable the formatting teams to collaborate in a high speed
           | agile context"
        
         | mrweasel wrote:
         | The technology stack isn't even that relevant. There have been
         | a few different articles about WhatsApp and their 1B users with
         | only 50 engineers. The key in previous articles was that
         | WhatsApp hired really smart people.
         | 
         | Having extremely talented engineers and a rather small scope is
         | what allowed them to grow til 1B users, with a minimum staff.
         | Yet people seem to focus on the technology, because that's more
         | easily replicated, and easier to accept, in my opinion.
        
           | Renaud wrote:
           | I think the technology stack they choose is highly relevant
           | to their success.
           | 
           | The choice of Erlang was crucial: it's basically built for
           | communications and has all pesky issues like version
           | updating, resilience to failure and scalability already taken
           | care of by nature of its architecture and framework.
           | 
           | Of course, these 50 engineers were smart, most in that
           | specialised space are, but I strongly doubt that they could
           | have achieved the reliability and scalability of what made
           | the success of WhatsApp with a PHP or Ruby back-end without
           | more complexity, more ressouces and more hardware (like what
           | FB and Twitter had to go through).
        
             | mrweasel wrote:
             | Well true, but my point is that you're not going to be able
             | to scale as successfully as WhatsApp, just by using the
             | same technology stack, if your problem is different.
             | 
             | You're right that Erlang was/is the right choice for
             | WhatsApp, and it was most likely picked as the language of
             | choice, because of the smart people working there. It's the
             | same with FreeBSD, a failing startup isn't going to be able
             | to layoff 50% of its engineers just by switching from
             | Linux.
        
               | crate_barre wrote:
               | Erlang was picked for a particular reason. WhatsApp uses
               | a prebuilt open-source chat solution, an XMPP server
               | written in Erlang, ejabberd. This thing was first
               | released in 2003 (old). Already a smart move to get
               | started.
               | 
               | I'm pretty sure they hired Erlang developers to dig into
               | ejabberd internals and optimize certain things. They
               | didn't just decide to become an Erlang shop out of the
               | blue.
               | 
               | It wasn't Erlang that was the initial right choice, it
               | was using xmpp and ejabberd that was the root reason.
               | Erlang just happened to be a consequence of that.
               | 
               | https://www.ejabberd.im/
               | 
               | I will contest your attribution of 'smart' with Erlang.
               | These types of correlations are generally bias fitting.
               | It justifies 'smart' being correlated with any and all
               | niche languages, eg 'so and so likes Haskell, so they
               | must be smart'. No good.
               | 
               | It's better to attribute 'smart' with the pragmatic
               | decision they made to simply use a pre-existing chat
               | server solution that already has the capability to scale.
               | Harder to assess this as smart since there's no
               | 'signaling' here, you have to objectively assess if it
               | was the right tech (which it seems like it was). Way less
               | vanity in this assessment as opposed to what I already
               | pointed out, how your Haskell or Rust devs must be
               | particularly smarter, as opposed to say PHP or JavaScript
               | devs who are considered _dumber_. I don't buy it, I need
               | to see more than just your affiliations.
               | 
               | So, I reject your initial post contending _'The
               | technology stack isn 't even that relevant.'_ It was
               | precisely the tech decisions that mattered, and the right
               | people to make such decisions. Chicken and egg scenario,
               | I'll concede that.
               | 
               | In any case, one does not simply pick a old chat server
               | written in Erlang out of the blue - this decision was
               | critical. How many over-funded tech teams would try to do
               | this from scratch in Go? Plenty, and that whole team
               | would easily be full of 'smart' people.
        
               | toast0 wrote:
               | > I'm pretty sure they hired Erlang developers to dig
               | into ejabberd internals and optimize certain things. They
               | didn't just decide to become an Erlang shop out of the
               | blue.
               | 
               | Eh, I was there since October 2011, and we didn't hire
               | any people who knew Erlang until much later. All of the
               | early server engineers (including me) learned it on the
               | job. By the time I left in 2019, I think we hired two
               | people to the server team with previous Erlang
               | experience; it's hard to find people with it, and while
               | it might have been nice, it's not important.
               | 
               | It's possible we had some consulting possibly before I
               | was hired, but I don't remember seeing any evidence of
               | that; OTOH, I do remember setting up and working with a
               | FreeBSD consultant and Moxie when he was consulting on
               | end to end. From my understanding, when things started
               | bottlenecking, Rick Reed was hired to fix bottlenecks;
               | which he had been doing at Yahoo! for many years and had
               | worked with Jan and Brian there.
               | 
               | FWIW; WhatsApp the service started as just a text status,
               | built on PHP and MySQL, but people were using it to chat,
               | so the founders went looking for a chat server to use
               | rather than building one from scratch. I don't know the
               | decision process, but ejabberd was then powering Facebook
               | chat at the time. (Of course, Facebook abandoned Erlang,
               | they said because they couldn't find people with Erlang
               | experience to hire)
               | 
               | Anyway, by the time I got there, I was told that the chat
               | server had been mostly refactored over time and while a
               | lot of names remained the same, and some of the basic
               | architecture was the same, it wasn't ejabberd anymore.
               | Mostly I worked on things that weren't chatd, and I don't
               | think I've seen ejabberd code, so I can't verify, but it
               | seems likely, as we customized the protocol, auth,
               | offline messaging, contacts, session handling, etc.
               | 
               | Erlang is a _tremendously_ right fit for a chat server,
               | and hot code loading is almost necessary when you have
               | hundreds of thousands or millions of connections per chat
               | machine and want to push small changes. Of course,
               | changes to BEAM itself, or the OS kernel take restarts,
               | so you still need to do those from time to time.
        
           | dgb23 wrote:
           | Absolutely. A relatively small, highly efficient team that
           | invests in their knowledge, skill and processes is the
           | _cause_ of correct technology choices and not the other way
           | around.
        
           | bjarneh wrote:
           | > Yet people seem to focus on the technology, because that's
           | more easily replicated, and easier to accept, in my opinion.
           | 
           | I think you are correct. It also works as some sort of
           | advertisement. Others feel like they made the right tech
           | choices, if they choose the same tech as WhatsApp or Spotify.
           | Completely forgetting that they probably need to be somewhat
           | proficient in Erlang to get the benefits from the language
           | etc.
           | 
           | I guess that "hire extremely good developers" is less sexy
           | than just choose Rust/Erlang etc...
        
             | WJW wrote:
             | It's not just less sexy, it leads to frustrated
             | readers/viewers of your article/blog post/conference talk
             | etc. If someone asked how to build a better system for
             | their company and the answer they got was "first spend 5-10
             | years becoming a better programmer", they will be less
             | happy than if you sell them on the idea that adopting
             | Rust/Erlang/etc will magically make the hard task easier.
        
             | emerongi wrote:
             | Language choice does act as a filter, though. If you choose
             | Erlang, do you get smarter developers by default?
        
               | dgb23 wrote:
               | Smarter? I think that's too broad.
               | 
               | But if you choose technology for its intrinsic merits you
               | attract engineers with taste, who have the drive and
               | luxury to care about their work on a different level.
               | 
               | There are plenty more people who can't be arsed to or
               | don't have the time or simply don't find it valuable
               | enough to adopt "niche" technology, despite technical
               | merits.
               | 
               | I don't think it has anything to do with smarts, but
               | rather with priorities and human nature. Most people
               | follow the mainstream because they favor stability and
               | convention, those who break out in different directions
               | favor autonomy and freedom.
        
               | bjarneh wrote:
               | From my experience it's always a good idea to choose a
               | language that people would have to learn by themselves,
               | i.e. none of the school languages (Java, C, C++, C#,
               | Python, Javascript etc).
               | 
               | In that way you only attract people who do sit down and
               | learn new languages by themselves in their spare time,
               | which filters out people with less interest in
               | programming at least.
        
       | varispeed wrote:
       | I bet these engineers got only a tiny tiny fraction of the
       | billions of revenue.
       | 
       | It always fascinates me how in the engineers' mind it reconciles
       | that their work brings corporations they work for billions and
       | yet they have to rent a tiny apartment for a substantial portion
       | of their salary and otherwise lead a pretty average life. Then
       | going to lavish offices witnessing how the money they could make
       | use of is wasted on vanity.
        
         | southerntofu wrote:
         | That's the founding principle of capitalism: private property.
         | 
         | People living someplace don't own it, and people working some
         | field don't own it. Some "owner" owns the land and means of
         | production and extracts value from people doing the actual
         | work.
         | 
         | Yes, it's a deeply broken system. A slightly better system
         | revolves around self-organized workers coop where everyone gets
         | equal pay and there are no shares to hold (or everyone owns the
         | same amount). An even better system abolishes money and private
         | property so that people can live meaningful/useful lives
         | without worrying about imaginary numbers ruining their entire
         | existence.
        
       | dopamean wrote:
       | I thought what allowed them to get to that many users was that
       | they were on so many platforms and that they used a large team of
       | remote engineers in Europe to do that.
        
         | EVa5I7bHFq9mnYK wrote:
         | What allowed them to get to that many users was
         | 
         | 1)mobile operators charged an arm and leg for text messages,
         | especially in developing countries
         | 
         | 2)whatsapp did not require registration with user name and
         | password, as their competitors (Skype, ICQ and others).
        
       | bradhe wrote:
       | Some of these points are a bit suspicious, and I bet don't tell
       | "the whole story" the way they're intended. For instance, the
       | statement about not investing in automation unless absolutely
       | necessary or the point about hot-swapping Erlang code.
       | 
       | Would be good to actually hear from one of these engineers for a
       | discussion on the paint points of working on WhatsApp back then.
        
         | tdrdt wrote:
         | Automation can be good, but will also be another system to
         | support.
         | 
         | I think this XKCD applies: https://xkcd.com/1205/
        
           | bradhe wrote:
           | I'd like to see what their criteria for "absolutely
           | necessary" is--I bet it doesn't fall on the right side of
           | that table.
        
           | bryanrasmussen wrote:
           | If you shave off 1 second I have a hard time believing that 1
           | second will not just be swallowed up by the entropy of coffee
           | breaks.
        
       | steve_adams_86 wrote:
       | It crossed my mind while considering this that a figure like this
       | doesn't mean the same as it would have pre-cloud infrastructure.
       | 
       | My team is fairly experienced but not so much in infrastructure.
       | Even so, we build and deploy on Google cloud with fairly
       | excellent scalability and ease. Ten years or longer ago I suspect
       | the story would be different - the infrastructure and scale
       | constraints could have pinned us at times. We would have needed
       | to hire more to support that effort.
       | 
       | And what I'm getting to is that yes, they had 50 engineers in
       | their pay roll, but many more supporting various SaaS products
       | and infrastructure they use and pay for.
       | 
       | Not to say this isn't impressive still, I think it's a great
       | thing and worth aspiring to. I love small teams.
        
         | Jensson wrote:
         | Whatsapp used bare metal servers. It isn't hard to do, just
         | that people no longer learn to do it.
         | 
         | > Ten years or longer ago I suspect the story would be
         | different
         | 
         | Whatsapp was built 12 years ago.
        
           | steve_adams_86 wrote:
           | I assumed they did their largest scale ups more recently,
           | though. I could be wrong!
        
         | toast0 wrote:
         | > And what I'm getting to is that yes, they had 50 engineers in
         | their pay roll, but many more supporting various SaaS products
         | and infrastructure they use and pay for.
         | 
         | I was at WhatsApp from 2011 to 2019.
         | 
         | Hardware was outsourced to SoftLayer, of course. I think they
         | had a sizable operation, but if you move that in house, you've
         | got a team of 4ish for 24x7 colo work (we were only in one
         | location for an embarassing amount of time), plus a networking
         | person, and maybe a manager. We certainly got better service
         | with SoftLayer's larger team including ease of getting more
         | servers, but we would have had better visibility in house, so
         | it's a tradeoff. Softlayer's staff was probably smaller than
         | the combined staff if all of their major customers in-sourced,
         | anyway.
         | 
         | Other than that, we used multiple providers for SMS and voice
         | verification; some of whom have a lot of staff. You need more
         | staff for this, IMHO, because in addition to 24x7 coverage, you
         | also need to arrange connectivity with carriers in all
         | timezones and all languages. But some of the companies we used
         | seemed pretty small, so I dunno.
         | 
         | We were using Zendesk in our customer service system after
         | growing out of Gmail and before in-sourcing it after
         | aquisition, and Dyn for DNS (which doesn't take much staff to
         | insource).
         | 
         | What other SaaS do you think we used? Do you want to count the
         | OS app stores and push services? Maybe payments while that
         | lasted? G Suite for corporate email when forwarding to personal
         | mail became too silly?
         | 
         | We self-hosted our code repository and bug tracker.
        
           | steve_adams_86 wrote:
           | I hope I didn't come across as though I was criticizing at
           | all. I meant to point out that having 50 engineers build
           | something today means something different than it would have
           | a decade ago, although in retrospect my age/career are
           | getting away from me and 15-20 years ago would have been a
           | much better example.
           | 
           | I should emphasize that the accomplishment is incredible,
           | regardless of how much we outsource things today. I know you
           | didn't outsource crucial talent or the ability to deliver a
           | good core product; you can't do that as far as I know.
           | 
           | I apologize if it seemed like I was putting down the
           | accomplishment. I'm only meaning to reflect on how much more
           | technical and infrastructure work happens outside of our
           | teams these days than it has in the past. Perhaps though it
           | isn't how much _more_ happens, but how _different_ what
           | happens is. Building these products has changed a lot in some
           | ways.
           | 
           | Edit: Also thanks for overview, that was a cool cursory look
           | at how you operated.
        
             | toast0 wrote:
             | No offense was taken.
             | 
             | I mean, honestly the way we built WhatsApp 10 years ago is
             | pretty similar to how we would have built it 20 years
             | ago... except that we'd probably have had to move towards
             | dedicated colo space, instead of 'managed hosting' bare
             | metal. I think we may have ended up going that way if we
             | didn't get acquired and move into FB datacenters.
             | 
             | SMS aggregators were certainly around in 2000, just as well
             | as in 2010 and 2020 ... although they've got fancier
             | webpages now.
             | 
             | I mean, you _can_ outsource a bunch of stuff, but we just
             | didn 't use very much. You can't outsource the core product
             | and deliver reliability, and we didn't have much that isn't
             | the core product. SMS verification was only reliable (ish)
             | because we used several providers with real time statistics
             | guiding the choice. (Of course, some of the providers use
             | the other providers, so it's very messy)
        
       | corobo wrote:
       | Is there even a correlation between scale and engineers, other
       | than scale = VC money = hire more cause we can afford it?
        
         | Ekaros wrote:
         | I would expect there to be certain scaling. Specially when you
         | move to more severs and more redundancy. Cloud solves part, but
         | still not everything. If single person can support 10k or 100k
         | users. You might still need more than that one to move to
         | million or hundred million.
         | 
         | Also some features like language support and such get more
         | complicated when you want to widen userbase from just English
         | speaking ones.
        
           | corobo wrote:
           | > If single person can support 10k or 100k users
           | 
           | I asked if there was a correlation here, using an if
           | statement as an answer doesn't answer it haha. You're now
           | asking the same question I was, just in-line
        
       | tiffanyh wrote:
       | DuckDuckGo does around 35B searches per year with 160 employees.
       | 
       | https://duckduckgo.com/traffic
       | 
       | Not the same efficiency but it goes to show how certain domains
       | and technology stacks can scale exceedingly well. Plus in DDG
       | case I believe they outsource a lot of their tech to 3rd parties.
        
       | heywherelogingo wrote:
       | This site is getting quite repetitive. I think the balance of
       | News to things of interest has gotten quite out of whack. This
       | article is not news, and is also in recent memory so not an
       | interesting re-visit. Is HN average age decreasing? The content
       | character has changed in recent years. Something's
       | changed/changing.
        
         | akkpxl wrote:
         | I thought I was the only one who felt this way. Almost every
         | day I see the same "style" of articles that isn't news or that
         | interesting. One thing I've noticed is almost every day I see
         | an article explaining why remote work is better and/or how tech
         | interviews are broken, both of which are popular opinions a lot
         | of people on this site share. Maybe it's just me though
        
       | mikewarot wrote:
       | The users are all handled through a common set of code, I don't
       | see why its remarkable that it takes 50 engineers to keep things
       | going. Customer support, I can see that eating manpower, but not
       | keeping the code running on a few racks of servers.
        
       | short12 wrote:
       | That sounds about right, even slim
       | 
       | I've honestly never understood how faang companies can have such
       | incredible numbers of devs when the products basically move at a
       | snails pace for years now
        
       | tdrdt wrote:
       | _" The simpler product makes it much easier to maintain and
       | scale."_
       | 
       | This is good news for the developers, the product owners and the
       | customers.
       | 
       | I still can't understand why today people choose a (Javascript)
       | stack of build systems, ton of dependencies, and all kinds of
       | exotic tech that is the latest and most hyped. As a developer you
       | need to support this in the future. It might be nice to build
       | today but it will be a nightmare later.
       | 
       | I have seen days go to waste because of Docker misconfigurations,
       | problems with dependency versions, outdated dependencies not
       | available anymore, bugs deep down in an unknown dependencies, and
       | so on. Personally I want to build systems. I don't want to spend
       | my day debugging all kinds of weird problems.
       | 
       | The (old) WhatsApp teams sounds like a great workplace.
        
         | atirip wrote:
         | It is called CV driven development. The cold truth is if you do
         | not have latest SV hype in your CV, you become pariah,
         | unhireable. It's that simple.
        
         | hnlmorg wrote:
         | Docker isn't itself the problem. It's people who insist on
         | `latest` et al.
         | 
         | Docker itself can provide a stable backbone that allows you to
         | run repeatable and predictable builds.
        
         | onion2k wrote:
         | _I still can 't understand why today people choose a
         | (Javascript) stack of build systems, ton of dependencies, and
         | all kinds of exotic tech that is the latest and most hyped. As
         | a developer you need to support this in the future. It might be
         | nice to build today but it will be a nightmare later._
         | 
         | You know that JavaScript developers don't see JavaScript as
         | weird or exotic, right?
        
         | freeall wrote:
         | Do we really think Javascript is exotic? Hasn't it proved
         | itself over many years now?
         | 
         | That doesn't mean it can't have issues like anything else. And
         | because of the massive scale of the language you'll see a lot
         | of shit. It's a language and ecosystem like anything else -
         | it's what you do with it that matters.
        
           | _Understated_ wrote:
           | I wouldn't say JavaScript is exotic, it's more to do with
           | what is being done with it.
           | 
           | Traditionally it was used to enhance HTML pages with some DOM
           | manipulation but it's now being used for a million other
           | things too.
           | 
           | So historically, if your JavaScript code broke your page
           | would still render as it was just html but now there are all
           | sorts of build processes and long chains that use JavaScript
           | to create the html in the first place... I'd consider that
           | the exotic bit.
           | 
           | Traditional JS running in the browser manipulating DOM
           | elements is very much a foundational aspect of the web and
           | won't deteriorate with age (with the exception of perhaps
           | deprecated functions in the far-off future) but all these js
           | libraries and tools and build processes with massive
           | dependency-chains are what's being referred to.
        
             | lhnz wrote:
             | It's been used as a general purpose programming language
             | for over a decade. It is a 'boring technology' nowadays and
             | hardly exotic.
             | 
             | Of course, you could decide to do something exotic with it
             | (edge rendering, data programming, whatever) but that's not
             | a problem with the language but people wanting to stretch
             | themselves...
        
               | pas wrote:
               | It still sucks by default.
               | 
               | https://i.redd.it/h7nt4keyd7oy.jpg
               | 
               | Yes, the ecosystem is pretty mature, we have things like
               | ESlint, TypeScript, Babel, TC39, and so on.
        
             | emerongi wrote:
             | JavaScript has matured massively over the past 10 years.
             | I'm not a Node developer myself, but JS + TS is a very
             | palatable combination to me. I'd pick it over many other
             | languages.
             | 
             | 10 years ago I would not have touched JS unless I was
             | forced.
        
               | brigandish wrote:
               | > I'd pick it over many other languages.
               | 
               | I'd be interested to know which ones, because I can't
               | think of one advantage that Javascript has - aside from
               | the ubiquity of browsers. If we're comparing any other
               | aspect of it though, I just don't know what advantage it
               | has (seriously).
        
               | sacado2 wrote:
               | Typescript has a pretty cool type system. A mixture of
               | dynamic and static typing. You can be as dynamic as with
               | js (or python), or stricter than Java (you can state a
               | reference cannot be null for instance), or anything in
               | between (and have dynamic parts and static parts in the
               | same program).
        
               | tolgerias wrote:
               | Functions are first class. Async i/o by default. Familiar
               | syntax (I'm sure Haskell or Clojure are better languages,
               | but they sure take some time to get used to. You can be
               | fairly productive in js in very little time). There are
               | packages for anything you can think of, no need to
               | reinvent the wheel most of the times. And I even like
               | it's single threaded nature. Being able to keep a process
               | waiting without needing to spin a new thread with all
               | that implies is very convenient. Edit: Original commenter
               | had already mentioned typescript.
               | 
               | So what's the deal with TS? I think on top of adding much
               | needed type safety to Javascript, TS is also one of the
               | best type systems you can ask for.
        
               | criddell wrote:
               | Would you pick it for the back end as well as the front
               | end?
        
         | stackbutterflow wrote:
         | Honest question, is specializing in a JS stack bad in terms of
         | career prospects? I've been using express + react/vuejs
         | professionally for a couple of years and I wonder if it'll be
         | in demand in the next 10-15 years.
        
           | pas wrote:
           | No, it's great. But keep up with the "ecosystem". Learn a bit
           | of frontend, backend and testing too. (Then keeping it fresh
           | won't be a problem.) And we shall see where it goes in
           | ~5-10-... years.
        
           | goodoldneon wrote:
           | You're fine -- don't listen to JS haters on Hacker News. They
           | pine for the simpler days of sites that primarily use HTML
           | and CSS, with maybe a little bit of JS sprinkled in. But
           | consumers are increasingly demanding an app-like experience
           | from the web, which requires JS frameworks.
           | 
           | As for Express, JS on the backend will be popular as long as
           | JS on the frontend is popular. JS backends have their flaws,
           | but there's a lot of value in using the same language in the
           | browser and server.
        
             | brigandish wrote:
             | > But consumers are increasingly demanding an app-like
             | experience from the web
             | 
             | This seems tendentious. Which average user was looking for
             | these changes? Can you point to a site that shifted to an
             | "app-like experience" and became successful because of it?
        
               | chenmike wrote:
               | This is a strange way to phrase the question. Sites that
               | had to shift to app-like experiences are hard to find,
               | because nowadays pretty much every web app is an app-like
               | experience from the beginning and is created with a JS
               | framework. The shift happened years ago.
               | 
               | Barring informational sites like blogs and news
               | publications, it's actually more challenging to come up
               | with new web products that are NOT app-like in nature,
               | and that do not use any kind of JS framework. Craigslist
               | may be one of the few big ones and even it is losing
               | market share to FB marketplace, which is an app-like
               | experience.
        
               | michaelt wrote:
               | Depends how you define 'app-like experience'
               | 
               | If I start with 1990s ebay, does it become app-like when
               | I add the ability to zoom images without a pageload? When
               | I add a WYSIWYG listing editor? When I let people drag
               | and drop images into their listings? When I add JS
               | infinite scrolling to search results? When I add AJAX
               | search autocomplete?
               | 
               | Or do I have to go as far as Google Docs, re-implementing
               | copy/paste functions, taking over the mouse wheel, and
               | adding my own text highlighting and zoom implementations?
        
               | Zababa wrote:
               | I think it's a "general trend" rather than a website by
               | website trend. I think that 20 years ago most of the
               | discussion online took place with things that looked like
               | websites (forums, mailing lists). These days, most of it
               | takes place in places that look like apps (Twitter,
               | Facebook).
        
               | thekyle wrote:
               | How about Google Maps and Docs
        
           | Cthulhu_ wrote:
           | Yes, but you'll find that a lot of work becomes maintenance
           | of older systems, and less new and shiny things.
           | 
           | I do think the 'craze' of new frameworks popping up left and
           | right quieted down some years ago, and things have roughly
           | settled on React, Vue, maybe Angular (which lost a lot of its
           | shine after the Angular 2 announcement fiasco), etc.
           | 
           | For a new application, I picked React + Go because I'm
           | confident that ten years down the line it will still be
           | maintainable - although, Go moreso than the front-end, which
           | doesn't feel nearly as solid and stable.
        
             | pas wrote:
             | I think Angular really delivered after they finally worked
             | out how to do emit optimized code (eg. the Ivy rendering
             | engine).
             | 
             | What is (was?) problematic with Angular is the toxic
             | leadership: https://medium.com/@jeffbcross/jeffs-letter-to-
             | the-angular-t...
             | 
             | But there are great things happening. The wider community
             | has solutions for everything. (For example here's the
             | "reactive forms are not strongly typed" issue [0] that
             | showcases both the good and the bad. The need for this
             | feature has clearly emerged in 2016. People stepped up and
             | a PR was created, but ... basically no signal from the
             | Angular team. Of course using a wrapper was an easy
             | workaround with a distinctly sour taste in the mouth. Then
             | finally something happened and an Angular team member now
             | seems to be working on it in "full steam ahead" mode.)
             | 
             | I recently had to maintain a large React + NestJS
             | application and I was seriously considering organizing a
             | terrorist cell to go back in time and ...
             | 
             | [0] https://github.com/angular/angular/issues/13721
        
             | lprd wrote:
             | Same stack as what I chose for my personal projects. This
             | is just my opinion, JavaScript on the server was a horrible
             | idea. There are far better languages to reach for when
             | writing API's and other server related stuff. I'm curious
             | how you are handling authorization/authentication (assuming
             | you are developing api's)?
        
           | askonomm wrote:
           | JS is everywhere though. You even have parasitic languages
           | that compile to JS (ClojureScript, TypeScript, etc). You got
           | Node and also Deno for back-end stuff with support for
           | multithreading. Concequently, JS is among the most popular,
           | and most used languages.
           | 
           | I do admit the ecosystem needs to find a way to have more
           | stability, because NPM outdated package tree nightmare is
           | pretty bad, but I guess that's what you get with an industry
           | that moves so fast, so in that case it's up to you as a
           | developer to not use too many dependencies.
           | 
           | Anyway this is all to say I don't think JS is going anywhere
           | for the next 10 years.
        
         | teekert wrote:
         | "Personally I want to build systems. I don't want to spend my
         | day debugging all kinds of weird problems."
         | 
         | Don't we all, I guess we all crave that snap, click, build
         | -feeling Lego gave us. But in my experience building a system
         | means debugging all kinds of weird problems. At least, when I
         | start something, I usually for a long time feel like I'm just
         | hopping from weird problem to weird problem. But, TBH, those
         | problems don't seem weird anymore as experience grows.
        
         | halfmatthalfcat wrote:
         | > I still can't understand why today people choose a
         | (Javascript) stack of build systems, ton of dependencies, and
         | all kinds of exotic tech that is the latest and most hyped.
         | 
         | So the same cannot be said for "newer" languages like Rust, Go,
         | and other's whose ecosystems and paradigms aren't completely
         | fleshed out yet?
         | 
         | This comment reeks of someone who is looking from the outside
         | in when it comes to building stuff in JS. Most comments I see
         | criticizing JS stacks are so superficial and demeaning that's
         | its obvious the commenters have little to no experience working
         | in JS.
        
           | shmel wrote:
           | Why do you compare it to Rust though? JS is as old as Java,
           | however its ecosystem seems to be much less mature (at least
           | from outside). I haven't touched Java for 15 years, but I am
           | fairly confident that people still use Apache Ant as build
           | system. Every time I try to peak into JS world, it appears
           | similar to tiktok trends in terms of how quickly people move
           | from one thing to another.
        
             | yatac42 wrote:
             | > I am fairly confident that people still use Apache Ant as
             | build system
             | 
             | As in: there are still (older) projects around that haven't
             | (yet) migrated away from Ant? Sure.
             | 
             | As in: Ant is still the go-to build system or at least
             | still commonly used? No, not at all. When I create a new
             | Java project in IntelliJ for example, I can choose between
             | Maven and Gradle as the build system. Ant isn't even
             | offered as an option.
        
             | jpgvm wrote:
             | It's even worse from the inside. The last 2 companies I
             | have worked at have had significant amounts of JS code
             | (even the majority of code) and it's inevitably became
             | unmaintainable mess through a combination of lack of solid
             | frameworks to instill structure and trying to apply it
             | outside of its domain of the browser.
             | 
             | The last 3-5 years have convinced me that JS isn't
             | appropriate outside of a browser almost ever. I am sure if
             | you think hard enough you can think of cases where it's
             | superior to some other lang but in general it's a very poor
             | choice for almost anything that isn't DOM manipulation.
             | 
             | Worst was definitely attempting to diagnose an off-heap
             | memory leak due to C extensions. Naturally the JS folk gave
             | up and dumped the problem on my lap so I proceeded to do my
             | usual "C guy" stuff and was amazed at just how bad stuff
             | like node-gyp and friends are and just how fragile
             | everything is. I found the leak and patched it and all was
             | "ok" again but just peeking inside those layers makes you
             | deeply uncomfortable with the runtime in production.
             | 
             | The rest of the problems can probably be attributed to
             | lower quality developers but point remains. Things like
             | lack of structure leading to insane architectures, pushing
             | for microservices without understanding the tradeoffs
             | because they didn't want to work on "legacy" JS code that
             | was built with last years hipster tech rather than this
             | years, etc. These problems are endemic to to culture and
             | ecosystem which IMO are inseparable from a language/tool in
             | practice despite what we want to believe in theory.
             | 
             | I don't refuse to work with JS but I definitely make my
             | concerns abundantly clear and I generally don't hold back
             | with "I told you so" when it inevitably bites people in the
             | ass.
        
             | y4mi wrote:
             | JS is as old as java, but they're both talking about the
             | build process of JS.
             | 
             | in the beginning, JS wasn't really transcoded/compiled. The
             | first time i personally found out about that was sometime
             | after 2010 with coffeescript, not sure if there were
             | preceeding examples.
             | 
             | and your claim that people move around is really false.
             | React has been the defacto standard for a pretty long time
             | at this point.
             | 
             | yes, other UI libraries/frameworks exist, but reacts
             | marketshare has been extremely dominant since it displaced
             | jquery/ember etc
        
               | jpgvm wrote:
               | I don't think anyone really has strong opinions about JS
               | on the frontend. It seems relatively fit for purpose
               | there.
               | 
               | Most contention I have seen is around attempting to apply
               | it outside the browser. That path only seems to lead to
               | misery.
        
         | hasperdi wrote:
         | These issues that you described can happen with any language,
         | ask any programmer or system engineer with long enough
         | experience they'll have stories with weird problems.
         | 
         | Regardless which language you pick, there will always be these
         | kinds of issues.
        
           | Cthulhu_ wrote:
           | > These issues that you described can happen with any
           | language
           | 
           | While you're not wrong, I do strongly feel it depends on the
           | language and architecture chosen. I'm a Go developer these
           | days, and there's a big mindset to e.g. avoid dependencies,
           | and for those dependencies to avoid including even more
           | dependencies, keeping things fairly lightweight and with a
           | low 'attack surface'. I mean I personally wouldn't mind a
           | stricter type system and native enum support and some other
           | things, but for now, I enjoy how basic it is, whilst avoiding
           | the footguns that C/C++ brings.
        
         | yen223 wrote:
         | WhatsApp was built on Erlang and FreeBSD, which feels way more
         | exotic to me
        
           | tdrdt wrote:
           | FreeBSD has been released in 1993 and has been a well known
           | OS in the hosting world.
           | 
           | Erlang has been released in 1986 and has been a well known
           | language for message systems.
           | 
           | When I think about exotic I think about some new tech that
           | sound really nice but is still very unproven.
        
             | serial_dev wrote:
             | JavaScript is from 1996, it's 25 years old, and apps built
             | with it are used by billions. With regards to the backend,
             | it has a vibrant full stack ecosystem, it's performant,
             | simple, and easy to hire talent for.
             | 
             | It's okay not to enjoy JavaScript (I'd always prefer to go
             | with Dart or Rust myself), it's okay thinking that other
             | languages (and ecosystems) are better equipped to solve
             | some problems, but calling it unproven and exotic is very
             | surprising to me.
        
           | RNCTX wrote:
           | FreeBSD was a more stable/mature platform to build on when
           | WhatsApp was designed, particularly in terms of network
           | performance.
        
             | seanw444 wrote:
             | Is the implication that FreeBSD is less mature now,
             | intended? I'm assuming not. The BSDs have grown in
             | popularity significantly in the last couple years.
        
         | johnfn wrote:
         | > Personally I want to build systems.
         | 
         | Sure. And so do I. Specifically, I don't want to _re_ build
         | systems that have already been built for me which I can use as
         | dependencies. And I don't want to spend all my time
         | troubleshooting bugs, either - I'll take a well-maintained
         | repository on GitHub with thousands of users bug-testing it for
         | me over some janky thing some guy on another team threw
         | together a few quarters ago before he quit.
        
           | choeger wrote:
           | > with thousands of users bug-testing it for me
           | 
           | Replace this with:
           | 
           | > with thousands of users just like me that once picked it as
           | a dependency and then forgot about the project
        
           | tdrdt wrote:
           | Having dependencies is not bad. But slapping a system
           | together with a ton of dependencies you don't know about is
           | bad.
           | 
           | Simple is better in my opinion.
        
             | netcan wrote:
             | I agree, but this assumes you actually know what you are
             | building. This isn't usually the case for a startup.
             | 
             | I'm sure whatsapp _wanted_ a million users, but they didn
             | 't originally know that would happen. If whatsapp later
             | turned out to be about serving fewer users with more
             | features, this becomes a story about premature
             | optimization.
        
             | rjzzleep wrote:
             | Simple isn't always better. Reusable patterns are. WhatsApp
             | followed pretty standard erlang design patterns.
             | 
             | Rails and Django to some extent force a standard pattern,
             | they're not really low dependency systems(although they are
             | compared to most node projects). But that pattern to some
             | extent makes sure that you can hand over the project to the
             | next dev and he'll be able to make sense of it.
             | 
             | Somehow node.js has become what PHP used to be.
             | 
             | Whatever happened to interoperability? How are there
             | hundreds of queuing system that use redis and rabbitmq
             | under the hood, but you can only process things in python,
             | javascript or ruby? The data structures are considered
             | private.
             | 
             | So if you want to process your code in python you have to
             | use the python message queue, if you want to process it in
             | javascript you have to find a node mq. How does that make
             | sense?
             | 
             | Want to do business intelligence on your javascript based
             | system now? Gotta write javascript code. Welcome to
             | debugging the same kind of memory leak and processing
             | issues every other queuing system has had to go through.
        
         | antihero wrote:
         | > exotic tech that is the latest and most hyped
         | 
         | See whilst obviously the JS ecosystem is quite volatile and can
         | be trend-based, I think there are benefits - the language
         | breeds innovation. Innovation doesn't always make the correct
         | choices, and some tech-conservativism is important in many
         | types of companies. But if our ancestors had just been lazy and
         | just stuck with what was known and good, we'd never even had
         | computers in the first place.
        
         | trhoad wrote:
         | A static React site, an Express backend, and a REST API is
         | hardly exotic
        
           | capableweb wrote:
           | Seems redundant to have a Express backend + REST API, why
           | can't you just build your REST API on the Express backend?
           | 
           | And also, React is complex, hardly anyone seems to know the
           | internals. The API interface might be easy to learn, but
           | "simple" is not something that you should label React with.
        
             | crate_barre wrote:
             | _why can 't you just build your REST API on the Express
             | backend_
             | 
             | Pretty sure he means to set up the REST api in Express with
             | a few simple routes.
             | 
             | JS is not complex, and pretty vanilla choice. It's what
             | people add on top of it that gives it a bad name. I'll link
             | to this guy so you can understand:
             | 
             | https://kentcdodds.com/blog/how-i-built-a-modern-website-
             | in-...
             | 
             | ^ This is the problem, not JavaScript.
        
               | Zababa wrote:
               | I'm not sure I understand why people build websites like
               | that. My theory would be that usually at some point you
               | have to go lower in the stack (learn how Linux/Node/V8
               | work or stuff like that). You can avoid going lower in
               | the stack, but the complexity has to go somewhere, and
               | that's what we see here.
        
           | BigJono wrote:
           | Yeah and if that's all people used, JS would be the best
           | language ecosystem ever.
           | 
           | Too bad I can't even remember the last time I jumped on a
           | project with less than 80 dependencies.
        
         | feketegy wrote:
         | > exotic tech that is the latest and most hyped
         | 
         | You answered your own question. Because it's "exotic" and
         | "hyped".
        
           | chinathrow wrote:
           | > exotic
           | 
           | My fast reading brain with not enough caffeine read that as
           | 'toxic'.
        
         | nodejs_rulez_1 wrote:
         | _> ...As a developer you need to support this in the future. It
         | might be nice to build today but it will be a nightmare later._
         | 
         | But the sort of people choosing a JS stack are quite quick to
         | move on. Make a mess, slap a new achievement on CV and dash.
        
           | paraknight wrote:
           | I don't agree, but I also find your username ironic. Did you
           | dash?
        
             | croes wrote:
             | He speaks from experience
        
         | netcan wrote:
         | For all sorts of reasons, good and bad.
         | 
         | One is that the future is unknown. Your goal might be to keep
         | the product simple and scale to 1bn users, but you don't really
         | know if it will play out like this.
         | 
         | Maybe you plateau at 1m users with the initial idea and pivot
         | the product somehow. Now the app does dating, social media or
         | specialised communications for highway construction teams.
         | 
         | Tradeoffs are easier to make in hindsight. Whatsapp is a good
         | demonstration of what you can gain with good trade offs. Do
         | less but well. Whatsapp did trade some things off though. Their
         | web version came late, is feature poor and a little clunky.
         | Same for a lot of features, compared to similar apps ATT. IDK
         | what exactly can be traced to the stack, but their decisions
         | around user identity are similar. Phone numbers only. One
         | device at a time. They gained simplicity, but lost options.
         | 
         | It's easy to see the benefits in hindsight. Real time is
         | harder. Play that game again and it might turn out different.
         | Most apps that set out to have 1bn never get close. At this
         | point, scalability doesn't matter and tradeoffs made for
         | maximum scalability don't seem as wise.
        
           | baktubi wrote:
           | I think the Unix philosophy should apply here. The fact that
           | WhatsApp did one thing well and it had a good business model
           | (was it $1 per year?) is the important bit.
           | 
           | When you say, "Now the app does dating..." I think the right
           | move is to scrap the project because that's a colossal
           | fuckup. Unless you have Microsoft cash to have multiple
           | colossal fuckups in a row, don't add another dating app to
           | your messaging app. Ergo, less is more.
           | 
           | WhatsApp probably started with passion and a solid vision.
           | The Zuckerberg gave them an offer they couldn't refuse.
           | Anyone will take 19 billion for a basket full of Indian
           | users.
           | 
           | Another thing WhatsApp did well is they targeted ALL phones.
           | They didn't abandon their users like the fang-bangers (sorry
           | been watching True Blood-- FAANG is an annoying acronym so
           | they are hereby fang-bangers).
        
             | 1f60c wrote:
             | I think it was 79 euro cents for a lifetime at first (this
             | thread brought back memories of my installing WhatsApp on
             | my phone while I was asleep), then it became 89 euro cents
             | per year (except for the users who had been grandfathered
             | in), and then came the Facebook acquisition and they made
             | it free for everyone.
        
             | netcan wrote:
             | Maybe.
             | 
             | I'm not saying that philosophy is bad, just that reality is
             | complicated.
             | 
             | " _Scrap the project and move on_ " works in some contexts,
             | not others. The way startups/products actually work, often,
             | is evolutionary. If your texting idea didn't work, but you
             | see a chance to pivot into something... are you really
             | going to just fire everyone and tell investors "sorry?"
             | 
             | That said, the "one thing well" philosophy really does have
             | big engineering advantages. You can't have everything. I'm
             | just raising the "retrospectives" warning.
             | 
             | In any case, the "$1 per year" was never a real business
             | model. They never even got around to actually charging
             | it... because anything that limits the usership of a
             | messaging app will sink it. It's the opposite of "support
             | everything" strategy that made them successful.
             | 
             | "Sell to Zuck" was always the plan.
        
               | baktubi wrote:
               | Yeah I agree with the idea of pivoting. I suppose to
               | clarify:
               | 
               | * Pivoting would leverage existing technology built in
               | the process of initial concept. Which to me is the
               | equivalent of scrapping the initial idea (while salvaging
               | the generally-useful IP/technology).
               | 
               | * Adding a bunch of tangential features to a product to
               | increase revenue is a colossal fuckup scenario (maybe the
               | language is a bit over dramatic).
               | 
               | For instance, Google is great at search; gmail is cool;
               | docs was innovative (albeit limited); and then...
               | https://killedbygoogle.com/
               | 
               | Unfortunately, after seeing this time and time again it's
               | tough for me to get behind mainstream tech. I loved the
               | old Microsoft/Nokia phones; and the Zune. You can tell a
               | lot or love went into the design/engineering but then
               | projects just get axed by corporate interests.
               | 
               | Meanwhile, you can by a mechanical device or appliance
               | from 1950s and it'll still work just fine.
        
               | netcan wrote:
               | Again, I don't disagree with you... just think reality
               | makes it messy.
               | 
               | Pivoting via "scrap & salvage" is pretty tough for a
               | startup. The tech behind whatsapp was evidently good, but
               | the IP behind a messaging app is probably not enough to
               | give you an edge. Users are.
               | 
               | Made up scenario: whatsapp loses the SMS replacement
               | game. They have millions of users, but not a billion.
               | Meanwhile, they find that a subset of users like to use
               | whatsapp for dating (or customer support, etc. doesn't
               | matter). They pivot to focus on those customers, and
               | evolve into something else.
               | 
               | This might be a (drama noted) colossal fuckup scenario in
               | an engineering sense. A tractor that you are now
               | converting into a ship.
               | 
               | Evolving is definitely a worse way of engineering than
               | starting with the intention of designing a ship, with
               | neatly defined tonnage, speed and size requirements.
               | Instead, it takes a miracle to implement basic ship
               | features like floating.
               | 
               | Evolving is how a lot of actual software gets invented.
               | Spreadsheets were intended for accountants. They weren't
               | meant to be used as a database, incident report
               | generators, a casual programming environment, or a HR
               | tool. It became those things by evolving.
               | 
               | It happened that way because inventing UIs is hard, and
               | evolving into them happened to work. It's still true that
               | "colossal fuckup scenarios" arise because of this
               | approach. Excel programmer spent decades making excel
               | better at things it's architecture wasn't good at. It's
               | ugly and messy, but life is sometimes ugly and messy.
               | 
               | Flexibility is valuable. Knowing the spec in advance is
               | valuable. Very valuable. They're in conflict with each
               | other to some extent
        
             | ChicagoBoy11 wrote:
             | >Another thing WhatsApp did well is they targeted ALL
             | phones. They didn't abandon their users like the fang-
             | bangers (sorry been watching True Blood-- FAANG is an
             | annoying acronym so they are hereby fang-bangers).
             | 
             | That was clearly the killer "feature." Whatsapp is
             | synonymous with communication on the developing world
             | because of this. I remember when I was introduced to it
             | fairly early on in Brazil and someone claiming that yeah,
             | anyone no matter the OS could get on it. I couldn't believe
             | it, to be honest, it felt like what iMessage was starting
             | to look like... but for everyone? I can't imagine what it
             | would've been like to support so many things, but clearly
             | it was a lot of sweat that paid off extremely handsomely.
        
         | ignoramous wrote:
         | > _I still can 't understand why today people choose a
         | (Javascript) stack of build systems, ton of dependencies, and
         | all kinds of exotic tech_
         | 
         | True, but one can do simpler things with JavaScript, too.
         | 
         | And a word on exotic tech: Well, some of it is really good in
         | helping small dev shops achieve scale of development,
         | deployment, and maintenance.
        
         | javchz wrote:
         | Well, WhatsApp had to support exotic cases like BlackBerry,
         | Symbian OS (Nokia), and Windows Phone. I will say, dealing with
         | JS stacks, seems easy in comparison to mobile fragmentation
         | from back then.
        
           | 1cvmask wrote:
           | And JAVA ME (formerly J2ME). The worst of the mobile
           | fragmentation by far.
        
           | vadfa wrote:
           | I don't see how this is related. Server-side you write and
           | expose an API. Then you have 1-3 developers for each platform
           | that write code that consumes that API. Those developers
           | don't need to know anything about the internals of the
           | server.
        
             | snovv_crash wrote:
             | But this speaks to good architecture decisions, right?
             | 
             | If they tried to make a shared framework that ran the same
             | codebase everywhere and then port it onto different
             | platforms ( _cough_ facebook) then you don 't get these
             | nice isolated issues, for example.
        
               | vadfa wrote:
               | Yes, if you don't write a native app for every platform
               | the result will be rubbish. That's not a "good
               | architecture decision", that is common sense. If they
               | don't do it that way it's to save money, because as we
               | all know Facebook is strapped for cash. (Talking about
               | Facebook and Instagram here, because as far as I know
               | WhatsApp is completely native on every platform)
        
               | snovv_crash wrote:
               | This then also means a well defined server API without
               | undocumented behaviour that individual implementations
               | depend on. I stand by my assertion that the good scaling
               | and separable builds is a sign of good architecture.
        
               | vadfa wrote:
               | Isn't a non-leaky abstraction a given too?
               | 
               | All these requirements are like saying "having a good
               | experience using this app requires an app that doesn't
               | crash all the time". Well, duh?
        
             | peterburkimsher wrote:
             | I wish that WhatsApp could make a public API. I'm very
             | thankful that Facebook is integrating WhatsApp into their
             | services, because that gives me hope that a true web
             | interface will be developed.
             | 
             | For me, it takes between 10 hours (worst case) to 5 minutes
             | (best case) to open WhatsApp.
             | 
             | My iPhone 4S is too old, so opening WhatsApp gives a
             | message "This version of WhatsApp ha..., Update WhatsApp,
             | Your update will be free of cha...". Clicking Update
             | WhatsApp opens the App Store, and clicking Update results
             | in another error message: "This application requires iOS
             | 10.0 or later."
             | 
             | So I only use WhatsApp through BlueStacks Android emulator
             | on my personal laptop. That's what leads to the 10 hour
             | worst-case response time: a full working day, including
             | commute. The 5 minute best-case is because BlueStacks takes
             | a very long time to open, and runs my laptop fans really
             | loud.
             | 
             | This is not only a pain point with WhatsApp, it's also a
             | problem with WeChat, KakaoTalk, LINE, even 9gag (for
             | posting videos).
             | 
             | Meanwhile, email continues to work just fine on my decade-
             | old phone, mbasic.facebook.com is particularly speedy,
             | iMessage still works, and Hacker News is great :)
        
               | egberts1 wrote:
               | Those are all the apps with the worst privacy policy set
               | at the Google and Apple stores.
        
               | [deleted]
        
               | agustif wrote:
               | Hey, if you do prefer that, I too have a broken phone,
               | but the new multi-device beta let's you to have whatsapp
               | in your computer with a powered-off phone. That works for
               | me.
               | 
               | https://faq.whatsapp.com/web/download-and-
               | installation/how-t...
        
           | Kelteseth wrote:
           | But this would be client side only, right? These issues would
           | also only occur in the corresponding app repository, when
           | building a separate native app for all platforms. The backend
           | for all of these apps would be the same, not the frontend.
        
           | nobleach wrote:
           | Having had to write a "mobile" app for Blackberry, I would
           | have KILLED for the option of using JS.
        
       | Pensacola wrote:
       | Only? Sound like plenty.
        
       | JoeCoo7 wrote:
       | WhatsApp has to be one of those most easily scalable projects to
       | work on. You have the freedom to divide chatrooms across
       | different databases/shards. Same goes for notifications, where
       | users can be divided since they're all isolated. The hardest part
       | is just coordinating all the moving pieces in a way that can grow
       | well, and that mostly just involves avoiding pitfalls in your
       | design that might bottleneck.
        
       | M0r13n wrote:
       | I'm honestly not that surprised about the size. In my experience,
       | efficiency and the number of engineers have not necessarily
       | correlated.
       | 
       | For example, I used to work for a company with about 500
       | employees (relatively large by German standards) and now work for
       | a competing company in the same industry with less than 150
       | employees. The core engineering team only consists of a dozen
       | engineers.
       | 
       | The output is noticeably larger. The number of new features and
       | especially the stability is significantly better. It is
       | noticeable that due to the small team, each project has one or
       | two "heroes". They are responsible for most of the commits and
       | know the system inside out.
       | 
       | These "heroes" have been working on the same or similar systems
       | for over 30 years in some cases and have experienced a lot in
       | their careers. As such, they can bring experience and advice that
       | is just out of reach for others.
       | 
       | Given the choice, I would always prefer a small team over a large
       | team.
        
         | yodsanklai wrote:
         | > Given the choice, I would always prefer a small team over a
         | large team.
         | 
         | I think that's one of the conclusion of the "mythical man-
         | month" book.
        
         | commandlinefan wrote:
         | > efficiency and the number of engineers have not necessarily
         | correlated
         | 
         | Often inversely correlated...
        
         | bluedino wrote:
         | I worked at a place that made a group of loosely-related
         | websites. Basically different spins/themes on the same
         | products. There were 4 of us on the development team. There
         | were 10 websites. Some new, some in maintenance mode.
         | 
         | I quit after a few years, and about 3-4 years later they went
         | on a hiring spree. Tripled the number of developers. Refreshed
         | some products, discontinued some, and some went into
         | maintenance mode. They still had around the same 10 websites,
         | but 12 developers.
         | 
         | The products didn't get better. The products didn't get made
         | any faster. They didn't have any less bugs. My wife actually
         | works there now, and when she tells me about the issues they
         | have I laugh a little bit.
         | 
         | It's all the same problems we had back then. Invites don't
         | work. Teams doesn't work. Uploading profile pictures has
         | issues. Entering data doesn't work. Registration is broken.
        
           | ratww wrote:
           | I had a somewhat similar experience with a large tech
           | company. Grew from <100 to 600 in a few years, but in the
           | meantime they weren't even able to update the homepage. Not
           | even the content changed, in almost three years. Everything
           | was convoluted and over-engineered. The whole thing was
           | rewritten from scratch twice, and when I left there were
           | plans for a third rewrite from scratch. COVID destroyed the
           | company: there were _zero_ customers. But the website still
           | struggled to stay online. That 's when I decided to leave.
        
         | altacc wrote:
         | Having worked in a range of companies, there is definitely an
         | inverse correlation between company size and speed, especially
         | for individual teams. At 500 people you're well into enterprise
         | level organisation, where a lot of processes are put in place
         | to minimize the amount of damage a single person can do to the
         | organisation. That also puts a speed limit on those efficient
         | and knowledgeable contributors.
         | 
         | Some companies spend a lot of time & energy trying to implement
         | operating models that try to overcome this but actually end up
         | putting in place more decision processes & oversight. Sometimes
         | the real solution is to reduce control and let the efficient
         | teams lead the way. But then what would all that management do
         | to fill their days...
        
           | mywittyname wrote:
           | I think this is more correlated to maturity than size. It's
           | just that size is highly correlated to maturity.
           | 
           | I've been in large companies where I was the admin/owner of
           | their AWS account. And I've worked at companies with 200
           | people where I have to submit tickets to get _anything_
           | created in AWS. The major difference was how mature the
           | product was.
        
             | necheffa wrote:
             | I work on a suite of products over 50 years old that have
             | been incumbent for about that long and still have a submit
             | tickets to get just about anything done.
        
             | closeparen wrote:
             | My current employer has a nice balanced approach to this:
             | everything is a code review. I may need a gatekeeper to
             | _approve_ my change, but I can always submit it.
             | 
             | It's also safer this way. Not even the proper owners should
             | routinely be on production shells or tools with live write
             | access. They also go through peer review with each other.
             | Once the infrastructure is in place for that, it makes
             | sense to open up.
        
         | jawns wrote:
         | Heroes are a red flag for me, as a manager, trying to build a
         | resilient team. Which is not to say that you can't have a range
         | of experience levels. But what happens when those heroes are
         | unavailable? Maybe they're out on vacation, or medical leave,
         | or they eventually retire or leave the organization because
         | they're tired of propping it up? Heroes tend to be a single
         | point of failure.
        
           | jbverschoor wrote:
           | You hire more "heroes". Pay what they're worth. Make them
           | happy.
           | 
           | Or you just pay an order of magnitude on hosting, performance
           | consults, bandwidth, etc. etc. etc. And STILL won't EVER get
           | to the same result, because you can't always pay (I'd say
           | almost never) your way into a stable and performant solution.
           | Look at all the others.. Nothing works as well as whatsapp.
           | 
           | Good luck hiring 500 engineers who have no idea what happens
           | throughout the whole stack.
           | 
           | I'd rather pay what you save on cloud providers and
           | snowflakes to them. (not even taking into account the result
           | of these "heroes".. the business value / dominance whatsapp
           | has because of this)
           | 
           | You can hire 1000 people who can't draw, or maybe just a
           | little bit, or you can pay an artist and get the result
           | you'll otherwise never get.
        
             | y4mi wrote:
             | > _You hire more "heroes". Pay what they're worth. Make
             | them happy._
             | 
             | i dont think its as easy as you make it sound. these
             | "heroes" generally are as productive because they dont sync
             | and already have a clear idea of what they want to
             | achieve/how to implement what they wish.
             | 
             | if you add a second person like that to the same team you
             | run a massive risk of having two competing ideas which
             | creates a lot of friction.
        
               | varelse wrote:
               | So make sure you're hiring a hero and not a primadonna?
        
               | GLGirty wrote:
               | The difference between hero or primadonna can be
               | environmental. When there are multiple experts for a
               | department/tech stack, you get heroes. If the bus number
               | is one, a hero risks converting to a primadonna.
        
               | ratww wrote:
               | I agree. I'd add that having multiple experts in
               | competition with each other (due to bad culture, bad
               | communication or even just bad personality) also has the
               | potential of turning heroes into primadonnas.
        
               | varelse wrote:
               | I find if you hire heroes with different skill sets you
               | can avoid this dilemma. Or don't hire two Supermen. Hire
               | Batman and Superman instead. But if a hero converts to a
               | primadonna then they were never a hero. On the other hand
               | if you're skimping a hero of the equipment and resources
               | they told you they needed from day one, you're the a***.
        
               | kweinber wrote:
               | Environments don't create primadonnas. It's a matter of
               | attitude and maturity. Being an expert doesn't make a
               | person insufferable.
        
               | varelse wrote:
               | In my experience, insufferable mid-level management can
               | bring out the worst in the best people, and then accuse
               | them of being primadonnas because they are absolutely
               | incapable of any self-reflection themselves.
        
               | marderfarker2 wrote:
               | At my company, "heroes" are given their own projects with
               | them given full creative control over every detail,
               | including the people they work with. I think I've read
               | somewhere on HN to never let any two person do the same
               | thing, to avoid the conflicts you mentioned.
               | 
               | Benefit of this is that these heroes can each break new
               | grounds, and their BKM shared amongst themselves making
               | the team extremely productive.
        
               | xyzzyz wrote:
               | > I think I've read somewhere on HN to never let any two
               | person do the same thing, to avoid the conflicts you
               | mentioned.
               | 
               | I've read this in Peter Thiel's "Zero to one".
        
             | tored wrote:
             | Yes, you will always pay one way or the other.
             | 
             | I work as a contractor, thus I have read a lots of code
             | from various different types of projects and the conclusion
             | is always the same, if they had spent more money on skill
             | to begin with I wouldn't be needed.
        
             | dunefox wrote:
             | > Nothing works as well as whatsapp.
             | 
             | Ehh, that's very debatable. Signal, Threema, and yes even
             | Telegram (regardless of E2E enc, etc.) work well and are
             | reliable.
        
           | civilized wrote:
           | You're right up to a point, but I've seen far worse red
           | flags. Like companies that can never acquire these heroes in
           | the first place because no one competent would work there
           | that long.
           | 
           | As somebody else mentioned, these companies end up regularly
           | throwing millions at consultant projects that always fail. In
           | other words, they pay a hefty premium for shitty temp
           | "heroes" that give them less than employee heroes would have
           | given them.
           | 
           | You see this a lot in the traditional finance sector, where
           | managers don't appreciate tech workers and relentlessly fuck
           | themselves over trying to save a dime.
        
             | jgilias wrote:
             | Yeah, somewhat counterintuitively treating tech as a cost
             | center will inevitably lead to wasted funds. But hey, that
             | just makes incumbents die faster and give way to
             | organizations treating tech as an investment. Which are the
             | ones you want to work for as a tech person anyway.
        
           | gen220 wrote:
           | If you have good trust and communication within a team, the
           | scenarios you describe are surmountable.
           | 
           | Every strength is a weakness, and every weakness is a
           | strength. IME, heroes are no more a red flag than pretending
           | that good engineers are interchangeable. It depends on the
           | context.
           | 
           | The expected outcomes of these teams are different, and
           | that's OK. If you're a very small company and don't have a
           | couple heroes, you won't build anything important. If you're
           | a very big company, the heroes that built it left years ago,
           | and you need resiliency more than new heroes (unless the
           | business is going sideways and needs saving).
        
           | Jensson wrote:
           | Managers prefer a team reliably working at 0.1x speed rather
           | than have it work at 1x speed 95% of the time. Yes, sometimes
           | people leave and you will have less velocity for a while, but
           | people pick it up and you go back to high velocity. To fix
           | that you ensure the team always works at slow velocity, that
           | way it wont get slowed down since it was already performing
           | as if you lacked those heroes from the start.
        
           | mensetmanusman wrote:
           | Use heroes to make jumps you can't make with cog machine
           | teams, then manage the prior tier platform to be resilient.
        
           | illegalmemory wrote:
           | 5 heros on vacation for 7 days, and coming back and finishing
           | the work in next 5 days is more comfortable position than 50
           | clueless engineers claiming work will be over in 4 days.
        
             | leeoniya wrote:
             | hmm, not sure how many customers would be okay with 7 days
             | of downtime, for instance.
        
               | bmeski wrote:
               | System has less of a chance of going down if you have
               | fewer noobs pushing to master all day.
        
           | bob1029 wrote:
           | > Heroes are a red flag for me, as a manager, trying to build
           | a resilient team
           | 
           | Heroes are your most powerful asset, but you have to use them
           | responsibly. The best thing you can do when handed a 10x
           | unicorn developer is to try to document 100% of the things
           | they say & do, and also make it a requirement that the hero
           | mentor others some % of the week. E.g. For 1-2 hours every
           | Friday you force them to hold a "no stupid questions"
           | session.
        
           | qaq wrote:
           | My first interaction with this management philosophy was when
           | our partner AOL replaced experienced SRE serving our project
           | with 3 far less experienced people for the same price. What
           | used to take an hour now literally would take weeks on the
           | bright side for the manager he had way more reports now.
        
             | jawns wrote:
             | I tried to clarify in my top comment that I wasn't saying
             | we should avoid having experienced people on a team, but
             | based on your response, maybe a little more clarification
             | is needed.
             | 
             | When I think about a "hero" in terms of team dynamics, it's
             | a person who is consistently relied upon to save everybody
             | else's butt, often by doing things that are flashy but not
             | particularly healthy, such as working long into the night
             | to meet an unrealistic deadline. When you have this sort of
             | Superman figure who's willing to swoop in and save the day,
             | the problem isn't so much their level of experience but the
             | unhealthy level of dependency that's placed on their
             | shoulders.
             | 
             | I'm all for having highly skilled, highly experienced
             | engineers who are productive themselves and can further
             | increase productivity by helping unblock others on their
             | team. And I agree with you that replacing them with some
             | number of juniors without the same institutional knowledge
             | can be disastrous. But if your team becomes so dependent on
             | heroics that they can't stay afloat otherwise, then when
             | that hero gets hit by a bus or just quits to take a
             | position elsewhere, you're screwed.
        
           | SkyPuncher wrote:
           | It depends on the company/stage/setup/etc.
           | 
           | Having a "hero" at a 5 person startup can mean life when
           | death was inevitable. Having that at a 500 person org likely
           | means another team is left cleaning up crap.
        
             | sangnoir wrote:
             | It's really not hard to be a "10x engineer" if you
             | disregard tech-debt and error-handling. I was part of a
             | very productive, complementary duo where my partner churned
             | out a _lot_ of code that only catered to the happy path,
             | and I 'd clean it up/"productize" it.
             | 
             | I actually enjoy that sort of work, but didn't receive as
             | many accolades as the guy "churning out features quickly",
             | even though he'd break the build and block the entire
             | company at least once a week before I paired up with him.
             | 
             | Once I saw how the sausage is made, I'm a lot more
             | sceptical of the "10x" label, either there's an invisible
             | support system, or the code base had a short half-life. Any
             | other scenario is a set-up for disaster.
        
           | M0r13n wrote:
           | This is definitely a potential risk. However, you have this
           | risk with small teams (less than 10 developers) even without
           | hero roles. The smaller the number of developers relative to
           | the complexity of the product, the higher the probability of
           | a single point of failure.
           | 
           | However, you can reduce the risk through good documentation,
           | good engineering practices and so on. As long as you are
           | aware that the team is partly made up of heroes, you can
           | prepare accordingly. So as a manager, I can try to ensure
           | that a few heroes keep productivity high while making sure
           | that the rest of the team understands the basics of the
           | system through things like code review and the like. In the
           | worst case, a teammate can then at least get up to speed
           | quickly.
        
           | AnIdiotOnTheNet wrote:
           | > But what happens when those heroes are unavailable? Maybe
           | they're out on vacation, or medical leave, or they eventually
           | retire or leave the organization because they're tired of
           | propping it up?
           | 
           | I don't know why people insist on using examples like this
           | instead of the more obvious one: What if they are hit by a
           | bus?
           | 
           | Any of the other examples have trivial solutions in the event
           | of an emergency: call them and maybe offer them some money.
           | If they are very stubborn then go to their home and beat them
           | with a wrench. But you cannot get information from a dead
           | person no matter how much money or how big a wrench you have,
           | and people die suddenly every single day. That's the worst
           | case you need to be making contingencies for.
        
           | killtimeatwork wrote:
           | Many/most teams don't have heroes at all, they only have
           | noobs :) I.e. most people stay on a team for a year or three
           | and change jobs afterwards - there's no chance of
           | accumulating knowledge. Nobody is sure how things work and
           | why they are the way they are.
           | 
           | I believe many managers are fine with this, because the team
           | is producing something (i.e. "velocity" is higher than zero),
           | but are oblivious to the fact that team could be way more
           | productive if the knowledge was actually retained in people's
           | heads and not just lousy attempts at documentation.
        
             | schnitzelstoat wrote:
             | I don't know if they are oblivious to it - but it might not
             | be possible to pay people enough to get them to stay due to
             | pay bands set by HR etc.
             | 
             | OP mentions that he is in Germany - there it might be more
             | possible as SWE doesn't pay as well as it does in the USA
             | and there are fewer companies so it might be feasible to
             | pay a lot more than the competition, whereas in the USA
             | that's unlikely to be viable outside of companies like
             | Netflix etc.
        
           | jimmont wrote:
           | It appears you have become the hero and are maintaining that
           | status.
        
           | throw1234651234 wrote:
           | I understand your point, but what managers fail to understand
           | is that someone usually "carries", i.e. is actually
           | responsible for the success of the product. I have even seen
           | it be a manager...once.
        
           | thewhitetulip wrote:
           | The worst team is the one with a single point of failure. All
           | hell breaks loose if they're unavailable. This makes me
           | wonder why managers have teams like this?
        
             | myohmy wrote:
             | Managers fail up. So they can point to their tight budget
             | in their next interview.
             | 
             | Oh, they can also blame the single point of failure for
             | being stupid and lazy. That tends to work a few times
             | before their boss catches on.
        
             | WJW wrote:
             | There is one type of team which is even worse than one with
             | a single point of failure: those teams without anyone at
             | all capable of fixing the system. Given that alternative,
             | having a single point of failure is better than having
             | nothing at all. It would of course be even better to have
             | multiple capable employees, but you cannot always have the
             | team you want with the hiring budget you have.
        
             | LeifCarrotson wrote:
             | Not all managers have a primary concern of "trying to build
             | a resilient team".
             | 
             | Sometimes, the #1 goal is building a high-performance
             | and/or high-quality product. Lack of team resilience,
             | (temporary) downtime, risk of project failure due to an
             | insufficient bus factor...they're all not good things, but
             | they can be compromised in the pursuit of that goal.
             | 
             | If you want high resilience as well as high performance and
             | also high quality, well, you should go work at NASA, and
             | hope that the Senate works out their budget issues...
        
           | vp8989 wrote:
           | Being replaceable tends to make work less satisfying. All the
           | places I've worked at that followed your advice had the most
           | churn and the least productivity/ROI on engineering $ spent.
        
             | onion2k wrote:
             | _Being replaceable tends to make work less satisfying._
             | 
             | In my experience the opposite is true. My personal goal on
             | any project is to make myself replaceable. There's nothing
             | I find more tedious than having to work on the same thing
             | for years because no one else can take over from me.
        
               | Jensson wrote:
               | > My personal goal on any project is to make myself
               | replaceable.
               | 
               | So you admit that you aren't replaceable? If the company
               | mandated that you should always be replaceable then you
               | wouldn't need that goal, it would always be fulfilled.
               | And working in a way that makes you always replaceable
               | isn't fun.
               | 
               | Edit: What we learned from your comment is that _making
               | yourself replaceable_ is fun, but _being replaceable_ isn
               | 't fun, since as you say you work to become replaceable
               | so you can start working on something new rather than to
               | stay replaceable forever.
        
               | 1123581321 wrote:
               | You're talking about something else, where you are an key
               | employee building a valuable system that doesn't need
               | you. The other person is talking about a company hiring
               | you to fit into a system that never needed you.
        
               | McWobbleston wrote:
               | I would hate to be stuck on the same product or set of
               | features, but I very much enjoy getting to a point where
               | I know all of the systems, how they relate, and roughly
               | where all the functionality lies and who's worked on
               | what. It takes a lot of time to develop that experience
               | and it's so valuable, and it seems so strange that
               | companies don't want to select for this and we're
               | dominated by a culture of job hopping every 2-3 years
               | because it's the only way to maintain appropriate pay.
               | Then companies wonder why everything is over budget and
               | never on time
        
             | jameshart wrote:
             | If you think being replaceable is unsatisfying, try being
             | indispensable. In the long run, it's way worse.
        
               | amaajemyfren wrote:
               | I think like most things there is a balance. The sweet
               | spot may be I can go hard on the project but if I feel
               | like I need some time off - there is the ability to step
               | back and have the work continue.
               | 
               | I'm sure I don't know how to achieve it though.
        
         | emerongi wrote:
         | As long as the core system is built on solid engineering, this
         | definitely makes sense.
         | 
         | From my experience in a deadline-constrained environment,
         | things end up being slapped together until they "just work".
         | Adding new features gets more and more difficult and if you're
         | still constrained by deadlines, then the solution is to hire
         | more, which is effectively a brute-force solution to meet the
         | deadlines. Output per employee gets smaller, but total output
         | should grow by some marginal amount.
        
           | allcentury wrote:
           | Absolutely. This is also when bigger and more pervasive bugs
           | enter the conversation. Decisions that kicked the can 2 years
           | ago are now a big problem and no one team can fix it.
        
       | papito wrote:
       | Because complexity kills. Something that has yet to be learned by
       | the new generation of developers. While disciples of FAANG create
       | a gazillion microservices that they spend most of their time
       | debugging in production ("observability"), smaller teams just get
       | shit done.
       | 
       | Since "monolith" has become a dirty word, it would be instructive
       | to remember that some of the major companies out there are still
       | light on distributed systems, which we all knew even back in the
       | olden days of the internet were an absolute killer of efficiency
       | and productivity. Yes, "microservices" is just another word for
       | it - believe it or not, there was life before Node and Docker.
       | 
       | Whatsapp, Dropbox, Instagram, StackOverflow - all existed or
       | still exist as one easy-to-maintain application. But go ahead,
       | "be like Google".
        
       | throwaway1777 wrote:
       | And now WhatsApp has hundreds of engineers. Take that to mean
       | whatever you want.
        
       | lumost wrote:
       | My 2cents.
       | 
       | Messaging is a weird app to focus on mau/dau. There have been
       | many messaging protocols which scale to hundreds of millions of
       | users, and there is prior art of large not particularly valuable
       | chat systems like aim, gchat etc.
       | 
       | No one is going to use a consumer messaging system which charges
       | money or shows ads. Which brings to mind substantial questions on
       | the business model others are using the messaging platform for
       | e.g. data collection.
        
         | closeparen wrote:
         | WhatsApp at this stage had a small signup fee.
        
       | johnisgood wrote:
       | I could scale to billions of users as a one-man engineer, but I
       | would need a marketing team. :D What I am trying to say is that I
       | could make a WhatsApp-equivalent relatively quickly, alone, yes,
       | but that is definitely not enough to gain billions of users.
        
         | MomoXenosaga wrote:
         | They were at the right place at the right time. But people want
         | to believe it is tech not luck...
         | 
         | There is a webshop in my country that copied Amazon in the
         | early 2000s and today they are beating Amazon.
        
       | dang wrote:
       | Discussed in the past:
       | 
       |  _Why WhatsApp Only Needs 50 Engineers for Its 900M Users_ -
       | https://news.ycombinator.com/item?id=10225096 - Sept 2015 (248
       | comments)
        
       | 12ian34 wrote:
       | The many comments alluding to the technical challenge of building
       | a chat app with 50 engineers being relatively easy are forgetting
       | that scaling said chat app to 1 billion users is absolutely not
       | easy and a function of more than just the engineering quality and
       | technical challenge. There's MUCH more to a successful product
       | than engineering...
        
         | tantalor wrote:
         | ... such as? Don't leave us hanging!
        
           | dunefox wrote:
           | I'd imagine getting the users in the first place. That has
           | nothing to do with engineering quality and scaling up,
           | though.
        
       | chatman wrote:
       | This is misleading. Likely, those 50 engineers were standing on
       | the shoulders of the giants, i.e. those open source maintainers
       | who build wonderful value for everyone out there.
        
         | ratww wrote:
         | Does this really matter? Companies with 100, 500, 1000, 5000,
         | etc employees often stand exactly in the same position and use
         | the same kinds of tools.
        
       | jimbojet wrote:
       | User count is fairly meaningless. How much was that in daily
       | active / weekly active users? How many of those were bot /
       | spammer accounts being "seasoned" to then sell to abusers?
        
       | pyuser583 wrote:
       | Scaling a single, discreet, product shouldn't take a ton of
       | engineers.
        
       | xchip wrote:
       | ...the other 450 engineers are just for spying and tracking
       | people
        
       | adrianlmm wrote:
       | For a similar project Google would use 250 enginers, 200 of those
       | would be diversity hires and the rest for the real work.
        
       | JohnJamesRambo wrote:
       | Remember this when people talk about Tether and how it can't be
       | real since it has X employees.
        
         | danrocks wrote:
         | Tether can't be real because it's a scam, not because it has X
         | employees.
        
       | dano wrote:
       | I looked at it this way. The thing most engineers, hero's or not,
       | want to work on is new projects and to build products customers
       | enjoy. The last thing they want to work on is maintenance. In my
       | last personally owned company, small as we were, the engineering
       | team and I documented how things worked as part of what we did
       | every day. We found that by having the inner workings documented
       | so that anyone in the team could effect a repair or update a
       | piece of code freed us to work on new products and services. Just
       | having the system continually documented freed our minds and kept
       | us mostly happy (everyone has a challenge from time to time) and
       | productive.
       | 
       | The documentation, not that it really matters, was done in a
       | MoinMoin wiki. It was written by engineers, for engineers. We
       | didn't stop marketing or sales or anyone else from reading it,
       | but the target audience was the engineering and operation staff
       | thus no one had to over-explain the vocabulary of our
       | architecture.
        
         | unbalancedevh wrote:
         | In my experience, documentation is the first thing to go when
         | the workload increases. There are always tasks that get
         | labelled as "important, but not urgent" that get pushed out;
         | and documentation that is already known to everyone immediately
         | involved is easy to push out beyond the product release date.
         | Of course, by then other priorities are moved up, and
         | documentation never recovers.
         | 
         | The only way I've seen documentation get the priority it needs
         | is if management mandates it as part of the release package on
         | which developers' performance evaluations are based.
        
         | arthur_sav wrote:
         | > Just having the system continually documented
         | 
         | In my experience this is the hardest part. Can't count how many
         | times my team started documenting and then slowly, for one or
         | the other reason, the documentation became outdated.
         | 
         | The cycle is: Meetings for communication -> becomes too
         | cumbersome and time consuming -> implement policy to document
         | things -> everyone start documenting (for a month or two) ->
         | docs are not up-to-date because people forget/leave etc.. -> go
         | back to meetings
        
           | albrewer wrote:
           | This is supposed to be enforced in code review or with other
           | team practices / processes.
        
           | castlecrasher2 wrote:
           | In my experience it's a culture thing. When I learned to love
           | Jira/Confluence was when the product team conducted meetings
           | directly from Jira and during task discussion almost always
           | asked us if tasks and documentation were done.
           | 
           | It could have been really annoying but the product lead was
           | extremely empathetic and that could be felt in any task
           | discussion, and his attitude made me want to do it instead of
           | making it feel like busy work like it has in other companies.
        
         | deathanatos wrote:
         | How did you get others to _read_ the docs?
         | 
         | This is ~our approach, but it seemed to stop after ~150
         | employees or so.
         | 
         | (You also seem to have the time to _write_ the docs...)
        
         | KennyFromIT wrote:
         | How did you manage to keep the documentation from going stale
         | or efforts being duplicated?
        
         | bcrosby95 wrote:
         | I must be weird. I love maintenance. The best bugs come out of
         | that. And it's hard to tell if a system was well designed
         | except in hindsight; maintenance programming is the best place
         | to be for that.
        
           | t-writescode wrote:
           | You're not alone. I may not love "maintenance"; but, I do
           | love resolving performance issues and shoring and
           | standardizing existing stuff a lot. I think I most like a
           | balance.
        
           | myohmy wrote:
           | I absolutely hate maintenance, so I love people like you.
           | That is why its great to have a small but diverse team.
        
       | TacticalCoder wrote:
       | What is amazing to me is that they did scale to 400 million
       | monthly active users with less than 50 engineers... In 2014.
       | 
       | That is: on hardware from seven years ago and older. Which is an
       | eternity in tech. Since then we've seen incredible new hardware
       | come out (EPYC 2 in 2018 for example).
       | 
       | One can wonder: up to how many MAU could a service scale today,
       | on today's hardware, with a small team? And what about tomorrow's
       | hardware...
       | 
       | This is fascinating.
        
         | hnarn wrote:
         | Seven years _can_ be an eternity in tech, but is it an eternity
         | counting from 2021? I mean, AWS launched in 2006, Netflix
         | expanded to Europe in 2012, Docker had their initial release in
         | 2013 and so on, so none of these major events are that  "new"
         | anymore. Obviously things have changed (a lot) during the past
         | seven years, but have they changed as much as they did between
         | 2007 and 2014?
         | 
         | What I'm saying is that for the general case of "scaling with
         | few engineers", it seems to me that the tools available matter
         | a lot more than the specifics of what hardware is available at
         | the time. Launching an international service "from scratch"
         | with a small team and a modest budget just wouldn't have been
         | possible 20 years ago, but not because CPUs were too slow.
        
       | seoaeu wrote:
       | It is silly to focus on number of engineers rather than on total
       | cost. Any company should keep hiring more engineers until the
       | additional salary expense exceeds the decreased operating
       | cost/increased revenue that engineer generates
        
       | up6w6 wrote:
       | This just makes more clear that we didn't try enough creating an
       | open-source instant messaging app that respects our privacy as it
       | should be. It's probably much more difficult to do the same thing
       | as WhatsApp did because of the dependency we created around it
       | but I think we just need the right initiative to do so.
        
         | baby wrote:
         | There are several hard problems linked to what people really
         | mean when they say privacy.
        
         | snarf21 wrote:
         | It is hard to get the general public to care about privacy when
         | they post pictures of every meal and share every thought and
         | emotion they have with no filter.
        
           | jodrellblank wrote:
           | It's hard to get the general public to care about inferior
           | things pushed on them by patronising people sneering at them.
           | How is a photo of some food important to privacy enough to be
           | the first thing you teach for? Seems more like punching down
           | at people you see as inferior because photographing meals is
           | "so lower class, right guys?".
        
       | bravetraveler wrote:
       | Queue endless business majors wanting to copy them
        
       | topdancing wrote:
       | You can very easily build your own WhatsApp server today by
       | simply deploying https://www.ejabberd.im/ on a box and using
       | https://conversations.im/ on an Android phone.
        
         | paxys wrote:
         | And talk to yourself on it?
        
           | topdancing wrote:
           | Not at all, I have a bunch of friends on my private server
           | and getting someone an XMPP client and account takes less
           | than a minute. No phone number or email required.
           | 
           | Sure, everyone out there isn't going to be on my server - but
           | for the close friends I care about - it's been flawless.
        
         | dopamean wrote:
         | Comments like this always remind me of that old comment on here
         | about how dropbox could easily be recreated with fsync or
         | something.
        
           | 1f60c wrote:
           | Here it is, for the uninitiated:
           | https://news.ycombinator.com/item?id=9224
        
         | southerntofu wrote:
         | Why is this being downvoted? Whatsapp is notoriously "just" a
         | highly-optimized fork of ejabberd [0]. ejabberd is written in
         | Erlang/OTP which enables quasi-horizontal scaling and amazing
         | failure recovery, along with incredible uptime.
         | 
         | What's different about Whatsapp compared to a traditional XMPP
         | experience is:
         | 
         | - it uses a single centralized server, without federating
         | 
         | - it uses phone numbers as identifiers instead of nicknames
         | 
         | - it has a tightly integrated (closed-source) client
         | 
         | So you can not only "build your own WhatsApp", but also
         | federate with others doing the same, which is pretty cool and
         | prevents a single actor from abusing the whole network.
         | 
         | [0] https://www.infoq.com/presentations/whatsapp-scalability/
        
           | AshamedCaptain wrote:
           | Doesn't anybody remember how literally Whatsapp's founder
           | wrote on one of the ejabberd mailing lists with something
           | like "I just installed the server, please help me configure
           | authentication" or the like?
           | 
           | Retrospectively that has always colored my perception of how
           | relevant technical aspects are for a successful startup (i.e.
           | not much). Marketing (and viral, user-capturing & monopolist
           | practices) are (sadly) much more important.
        
             | btown wrote:
             | If you look at
             | https://www.infoq.com/presentations/whatsapp-scalability/
             | though it gives a very different picture. Sure, the
             | _founding_ team may have tried to prototype without knowing
             | full technical details, but they knew enough to choose
             | software that could fundamentally scale towards the future,
             | and they rapidly brought on fantastically talented
             | developers with a hunger for optimization. You can 't
             | choose the right technical people without technical chops;
             | that being said, in many cases the right technical people
             | _are_ those who know how and when (and when not) to stand
             | on the shoulders of giants!
        
               | AshamedCaptain wrote:
               | It doesn't look like "they knew enough to choose software
               | that could fundamentally scale". It looks more like they
               | choose the first option that was available to them. Apple
               | also choose XMPP for Facetime as did a million other
               | companies that serve similar scales of simultaneous
               | clients.
               | 
               | They may have hired people later on (though this entire
               | article is kind of showing that no, they actually didn't
               | hire that much technical people). But by the time they
               | even hired the first Erlang engineer they had already won
               | over the European market. And for the record it was not
               | precisely due to the quality of either their Android or
               | iOS android apps -- the J2ME and Symbian clients had
               | quite a following and I would even bet were more used.
               | 
               | Original Whatsapp was a disaster and this did not prevent
               | them from gaining marketshare. You could basically login
               | as anyone just by having their phone number (which maps
               | to their JID) and defeating their XMPP obfuscation layer
               | (some XML encoding, which was easily done due to the
               | easily RE Java clients).
               | 
               | It was only later on (2011ish?) that they started getting
               | somewhat serious, which coincided with them, already
               | swimming in users (and money?), starting to get more evil
               | (with a more hardcore stance against 3rd party clients).
        
             | southerntofu wrote:
             | > how relevant technical aspects are for a successful
             | startup (i.e. not much)
             | 
             | That's why i'm very interested in human and political
             | aspects of free software. Successful FLOSS projects usually
             | have a strong community backing them, and strong
             | connections between developers and the community so that
             | the technology isn't too disconnected from practical needs
             | and UX concerns.
             | 
             | I think non-profits and cooperatives building upon free-
             | software solutions are a better alternative model. For
             | example, framasoft.org (french NGO for libre
             | culture/software) has been instrumental in developing new
             | solutions like Peertube, and in kickstarting a new wave of
             | hosting coops (chatons.org federation).
             | 
             | In the IM world specifically, i'm very interested about
             | Snikket.im project. It aims to build a complete
             | client/server distribution of XMPP software tailored for
             | specific user expectations.
        
           | toast0 wrote:
           | http://www.erlang-factory.com/sfbay2014/rick-reed is the
           | original link for that presentation, FWIW.
        
       ___________________________________________________________________
       (page generated 2021-10-25 23:02 UTC)