[HN Gopher] Everyone is still terrible at creating software at s...
___________________________________________________________________
Everyone is still terrible at creating software at scale
Author : caspii
Score : 445 points
Date : 2021-04-10 07:31 UTC (15 hours ago)
(HTM) web link (margint.blog)
(TXT) w3m dump (margint.blog)
| jonplackett wrote:
| Reading about Neuralink yesterday makes me wonder if in 50 years
| it will be possible to connect a bunch of minds together to
| achieve a task none could do on their own. Could be programming
| or some other large scale task with a lot of interdependence.
| wizzwizz4 wrote:
| That's called communication.
| jonplackett wrote:
| The whole point of the article is that communication on a
| very large scale is very hard. There's a point that adding
| more people makes things worse. What I mean is, one day,
| maybe more people will be able to collaborate by becoming one
| big brain.
| Antoninus wrote:
| This the template for a clickbait headline on HN.
| Grimm1 wrote:
| This is one of the reasons I'm in the erlang derived system of
| languages. You never have to use the distributed tools if you
| don't want to but they're really powerful and really elegant when
| you start to need them.
|
| This is only one part of the problem though because as someone
| stated below even though server systems have networking stacks
| they are not entirely designed around the concept of seamless
| computation over multiple nodes so you wind up with abstractions
| that solve the problem but do so fighting against some
| occasionally nasty gotchas that make it a lot harder than it
| needs to be.
|
| This is all to say I'm greatful for those who maintain things
| like erlang, elixir, kube, go etc. They're somewhat fighting
| against the stream so we have an easier time when we do need to
| scale.
| hrpnk wrote:
| Software crafted carefully for the beauty and structure is one
| thing. Another is software for customers where delays of shipping
| the functionality has financial impact. Will you perform a change
| with cost of 1 day or do it right and spend 5 days? What will you
| be rewarded for? Short-term thinking is right until it's wrong
| because it stalls the whole company/system.
|
| Another challenge in software comes from the added cost due to
| decisions that are tricky to reverse. Worst thing is that they
| only show at certain scale, meaning that in many cases they were
| the right decisions in the past (business context).
|
| If you only operate in a single market, you can process prices
| with a single currency, so you get away with bad modelling -
| without the currency. To scale to another market, you can just
| deploy your application twice. If the company succeeds in new
| markets, you will replicate the solution. Fast forward a few
| years and you have a migration project to add currency to your
| software and data to optimize deployment.
|
| Organizational complexity is another domain. Understanding what
| places to adjust is key. For example: how complex is it to change
| the tax rate in your software? How do you find out which
| applications to change? How do you know who needs to perform the
| change? Do you have to broadcast a question to the whole company
| and ask them for performing the change until X? How do you know
| you adjusted all places? Do you have to deploy the change
| manually or will the change be automatic based on time?
| bjornsing wrote:
| I've come to think about this more and more in terms of politics.
| The core problem is that not all people come to work with the
| intent to build high quality software in a rational and efficient
| manner. Also, what do you think film production would look like
| if you just hired a lot of "senior film developers" and let them
| hash out who should be the director and who should hold the
| microphone bar?
| mlazos wrote:
| This post seems like the author is early in their career and is
| realizing everything in the world is half broken and not perfect.
| What stood out to me is when he said "you could build what takes
| weeks in an enterprise in a weekend." I don't know about everyone
| else but I've become a much much better engineer since college
| and I still have to carefully handle every edge case for
| production and write a lot of tests before shipping something and
| I really don't think code I write in a weekend will be production
| ready at a large company. Truth is by any metric, the massive
| systems in place by Amazon are pretty amazing examples of
| scalable software. At my day job at least managers have seemed to
| figure out that they set the high level goals/milestones and then
| let the engineers figure out how we want to get to those goals.
| This system has worked pretty well (of course the deadlines can
| get tight and we accumulate technical debt) but once we hit a
| milestone we deliberately decide to take some small amount of
| time to fix technical debt, as long as we kept track of it as we
| scrambled which also requires some discipline. The way things
| distill up to management is for my VP he doesn't know what
| details you did on x day, he wants to know did you help implement
| y one of our org's high-level goals. This system works to some
| degree and I haven't thought of a better one yet.
| m00x wrote:
| Yeah, OP is definitely an early engineer, or not very
| intelligent. Enterprise software takes longer because there's
| multiple stakeholders and it's better to take a week to build
| it properly than it is to hack it and fix bugs for the next 3
| months.
|
| The decisions you take to build software has to include the
| expected scale. If you're pushing something to 10 users, you'll
| take different decisions than if you push it to 20M users. This
| is done at the planning phase, and senior engineers are aware
| of this. I'm not going to spend 2 weeks optimizing the shit out
| of a system that'll be used by 10 people.
| galaxyLogic wrote:
| > I find myself coming back to the film business,
|
| The difference I think is that film-industry has separated the
| creative parts from the production parts and the creative part is
| not really complicated at all. Write the screenplay. That can be
| innovative but it is a single author typically who will do it. No
| complications because everybody's screenplay has to work
| together. Because there is only one.
|
| In other words making movies is more like making buildings than
| making software. Buildings can be huge and very expensive but
| their design is still quite simple and based on designs the
| architects have created in the past.
| abiro wrote:
| Torvalds once said [1] that the only scalable software
| development methodology is open source, and I tend to agree for
| two reasons:
|
| 1. The project structure, tooling and documentation that lets new
| contributors jump in quickly makes software development easy to
| scale. In Coasian terms, the transaction costs around development
| are minimized.
|
| 2. It enforces hard interfaces and it clearly separates
| development from operations. Lack of discipline around these
| issues is a source of much accidental complexity.
|
| [1] I can't seem to find the quote, I read it in an interview a
| few years ago and stuck with me since.
| kasperni wrote:
| > the only scalable software development methodology is open
| source
|
| Some companies such as Microsoft, Google, Amazon would
| disagree.
| abiro wrote:
| The Linux kernel had 20k contributors since its inception,
| you'd be hard pressed to find those numbers for a single
| project at any company.
| misterS wrote:
| I guess (and I have no way to verify this, it's really just
| a guess) that the number of developers that worked on the
| Windows code base in its 35 year history is in the same
| order, if not higher.
| kasperni wrote:
| Yes, the linux kernel is a big codebase but so are a lot of
| others [1]. And you will find lots of private companies not
| mentioned that have long-standing projects with >10 mil
| LOC.
|
| If you took the 100 biggest code bases in the world. I
| would be surprised if more than 10% of them were open
| source.
|
| [1] https://www.informationisbeautiful.net/visualizations/m
| illio...
| oblio wrote:
| I know a medium sized French-American multinational.
| You've probably not even heard of it. A decade ago they
| had multiple products that had several million lines of
| code. Their entire platforms probably had 100 million. I
| can't even imagine how many contributors they had, the
| oldest project was started in the 80's and it probably
| had 1000 contributors over time, at least.
|
| And again, that's for a middle size company you haven't
| even heard of. FAANG, Microsoft, Oracle, IBM for sure
| will have stuff dwarfing that.
| abhijat wrote:
| Someone from oracle posted here a while ago, where they
| described Oracle 12.2 has 25 million lines of C
| https://news.ycombinator.com/item?id=18442941
| marcosdumay wrote:
| And yet it's a less appealing product than Postgres.
|
| An explosion on the number of lines of code is one of the
| way development teams fail.
| _dps wrote:
| Is it actually less appealing? My understanding is that
| DBAs consider Oracle very good, just very (very)
| expensive. This also lines up with my experience tuning
| queries against Oracle vs Postgres backends. The folk
| wisdom seems, to me, to be Postgres is 99% as good in the
| common cases and 90-95% as good in the difficult cases.
| marcosdumay wrote:
| Oracle DBAs consider Oracle clearly superior. SQL Server
| DBAs that know all the ways to optimize SQL Servers
| consider SQL Server clearly superior. I don't know of any
| PostgreSQL DBA, just generalists that work with Postgres,
| so I don't know about them.
|
| The truth is that Oracle is ridden with coherence bugs,
| and have a much worse performance picture out of the box
| than Postgres. But while improving the Postgres
| performance requires deep digging into the DBMS itself,
| Oracle has a lot you can optimize just on the SQL
| interface. But there are plenty of cases where Oracle
| just can not become as fast as out of the box Postgres,
| and many where it could in theory, but a bug prevents you
| from doing things right.
|
| Overall, three is no definitive ordering on the speed of
| the "good 3" DBMS. It always depends on what you are
| doing.
| kgeist wrote:
| How many of those 20k contributors worked on drivers, and
| how many - on the actual core (~150kloc)? Every driver is
| like a separate subproject, having 20k people work on
| hundreds/thousands of drivers (unrelated to each other)
| wouldn't sound as impressive.
| thrower123 wrote:
| That's the scary thing. How many real, core contributors
| does even something like the Linux kernel have? People
| who have actually written more than a couple patches that
| landed and stayed in the kernel. I'd be astounded if it's
| much more than a hundred.
|
| Most open-source projects have one to maybe five real
| contributors, outside the drive-by pull requests to fix
| some bug.
| rcxdude wrote:
| Consider the bit at the end of this blog post:
|
| https://devblogs.microsoft.com/oldnewthing/20180326-00/?p=9
| 8...
|
| Bonus chatter: In a discussion of Windows source control,
| one person argued that git works great on large projects
| and gave an example of a large git repo: "For example, the
| linux kernel repository at roughly eight years old is
| 800-900MB in size, has about 45,000 files, and is
| considered to have heavy churn, with 400,000 commits."
|
| I found that adorable. You have 45,000 files. Yeah, call me
| when your repo starts to get big. The Windows repo has over
| three million files.
|
| Four hundred thousand commits over eight years averages to
| around [130] commits per day. This is heavy churn? That's
| so cute.
|
| You know what we call a day with [130] commits?
| "Catastrophic network outage."
| jka wrote:
| To get closer to an apples-to-apples comparison, it'd be
| necessary to know whether the commit counts in each case
| include all development branches for all development
| groups. By design, git-based development can be highly
| distributed.
|
| Also, even if we normalized both cases for 'code files
| only' and/or 'kernel code only', there could still be
| architectural, code style, and development process
| differences that lead to different metrics for each
| project.
| [deleted]
| dragonwriter wrote:
| > the only scalable software development methodology is open
| source
|
| "open source" isn't a development methodology, or even a
| distinct set of methodologies.
|
| > The project structure, tooling and documentation that lets
| new contributors jump in quickly makes software development
| easy to scale.
|
| Plenty of open source projects don't have that, nor is there
| anything restricting those things to open source projects.
|
| It is true that some open source projects, because they see the
| value in new developers jumping in quickly, prioritize having
| structure, documentation, and tooling that supports that. It's
| also true that some proprietary software projects do, too,
| because the project owner sees value in that.
| abiro wrote:
| There are books written about open source development. Don't
| confuse somebody dumping source files on Github with open
| source development.
| dragonwriter wrote:
| > There are books written about open source development
|
| There are books written about different authors
| idealizations of how to do open source development. The
| fact that many people have written about different
| methodologies for approaching a particular challenge
| doesn't make the challenge a methodology.
|
| And, yes, I am aware that the very approach known as "the
| bazaar" (from the essay "The Cathedral and the Bazaar") is
| sometimes referred to erroneously as "open source
| development", which is a particularly glaring error since
| all of the examples of both "cathedral" and "bazaar"
| development were open source projects.
| watwut wrote:
| Yeah, and that does not mean those books describe how open
| source projects function in reality. But they should, how
| things are done changes.
| oblio wrote:
| There are a gazillion ways to develop open source. It's
| more like an ethos than a methodology.
| marcosdumay wrote:
| > There are a gazillion ways to develop open source.
|
| Yet every large project behaves in a similar way, with
| only a few larger variations.
| oblio wrote:
| That's why one of the seminal books about Open Source is
| the Cathedral and the Bazaar? Because "there is only one
| way to do things"? :-)
| marcosdumay wrote:
| Hum... The Bazaar on that book is a very specific way to
| do things. So specific that I don't think anybody really
| follows it.
| qznc wrote:
| Since Open Source software drops liability by definition, a
| huge complexity factor disappears.
| rakoo wrote:
| To go further I think it's not exactly open source but remote-
| first that made software development scalable.
|
| If you grossly simplify it to its innermost core, making
| development scalable means that if you have 10 times more
| people you can have 10 times more features/bugfixes/speed
| improvements/... The only way to do that is to make sure that
| an additional developer doesn't depend on other developers to
| work, and that can only happen if everything is properly
| documented, the build instructions are up-to-date, the
| processes are clear, basically anyone can start from scratch
| and get up to speed without the help of anyone else.
|
| That kind of organization traditionally doesn't happen in on-
| site companies, where newcomers are followed by senior people,
| they have to follow some introduction course to familiarize
| themselves with the processes, they have to ask many questions
| every day, they need to be synchronized with other persons
| which brings some inefficiency beacuse everyone works at their
| own pace, etc... This all disappears when everything is
| properly documentend and every contributor can work in the
| middle of the night if they wish. I think the Gitlab Handbook
| goes over this quite well and describes a framework to
| implement that kind of organization but the rules are
| retrospectively obvious for people already used to open-source
| (https://about.gitlab.com/company/culture/all-remote/guide/):
|
| - write down everything
|
| - discussion should happen asynchronously. Any synchronous
| discussion (by text or call) should be only very small points.
| Whatever the type, write down the conclusions of that
| discussions
|
| - Everything is public (to the organization), including
| decisions taken, issues, processes
| marcosdumay wrote:
| Yeah, remote-first is important, but it's not only it.
|
| Another very relevant factor is that people can just clone
| stuff, create new projects, and everything moves
| independently. So the open source development model has a
| pile of solutions for dependency management that team based
| development doesn't adopt.
|
| But the one thing I don't get is why team based development
| doesn't adopt those solution. They are not expensive. Yet,
| even when I was able to dictate into everybody's
| requirements, I wasn't able to force teams into adopting
| them. Instead they insisted on synchronizing themselves by
| much more expensive and less reliable means. My guess is that
| most developers never dug any deep into open source and have
| no idea how it's done.
| hyko wrote:
| Software is predominantly a fashion business, not a technology
| business.
|
| Just allowing the market winners to drive that fashion means we
| aren't able to progress the state of the organisational art
| scientifically.
| d--b wrote:
| Software is indeed a lot like cities.
|
| But cities ain't pretty. Dig the ground and nothing looks like
| what's on the map. To create a new building, you need to cut a
| lot of plumbing and re-route it somehow.
|
| Stuff gets old and breaks all the time, and maintenance teams are
| on the ground 24/7 to fix it. NYT subway is the mainframe of the
| city. Look at the central steam heating that's still pumping
| water into high rises.
|
| Sure, you can sell a shop, and people will be able to repurpose
| that space to suit their needs very efficiently. But isn't that
| what Docker does? Provide a cookie cutter space to run an app?
|
| But in cities, there are places where trucks can't go, where you
| can't park, where unplanned work happens, where the sewer floods.
| That's when the city 'bugs', and people need to manually
| intervene to fix it...
|
| Trying to find an ideal way of running software at scale is just
| as utopian as building the perfect city using 'super structures'.
| It's a dream of the 60s that's never going to happen.
| alkonaut wrote:
| Maybe a large US city is less of an ideal than say a medium
| sized Swiss one...
| sjm wrote:
| Ideal in what sense? NYC has more than double the GDP of the
| entire country of Switzerland and approximately the same
| population.
| golemiprague wrote:
| Cities like Tokyo might be more comparabe, and much nicer
| than NYC in many aspects, especially the public
| transportation system. For a major city NY looks pretty
| shabby and not that nice in average.
| alkonaut wrote:
| I mean in the sense of what perceived chaos or
| deterioration is "allowed" in a city or it's
| infrastructure. E.g whether a plot can be left unbuilt or
| half torn down for example, and the standards which are
| expected of public infrastructure like roads or train
| stations.
| mattmanser wrote:
| Rather meandering and over-verbose, this post never really seems
| to reach its point.
| astura wrote:
| I've legitimately never seen so many words used to say
| absolutely nothing.
| Griffinsauce wrote:
| Just like most large scale software.
| [deleted]
| choeger wrote:
| Maybe it's time to finally differentiate the tools for software
| as in functions, algorithms, classes, libraries, etc. and
| software as in e-shops, databases, sso services, streaming
| services, etc.
|
| We have good science and steady (albeit slow) progress in the
| former while it feels like the latter is more or less subject to
| stagnation. For instance, when I need to setup a DB cluster, why
| is there no tool that takes my requirements and generates
| deployment scripts, monitoring, migration tools, etc.?
| fctorial wrote:
| The latter is called industry experience.
| ChrisMarshallNY wrote:
| That's a great essay! I think he describes the issues quite well,
| and I enjoyed his metaphors (I like the city metaphor).
|
| I also -like him- have no answers for "the big picture." I ran a
| small cog on a big wheel for a long time. I feel the basic
| _quality_ (as in "few bugs") of our work was good, but I also
| feel that it took too long, and failed to flex, so it often fell
| flat with our customers. Our software was developed, using a
| hardware process, and there are vast differences between the two
| disciplines.
|
| I am fortunate, in that I don't need to play in anyone else's
| sandbox. It means that my scope/scale is quite limited, but I am
| pretty happy with what I am able to do. It's actually fairly
| ambitious for a one-man "team."
|
| Since striking out on my own, I have experimented with what I
| term "ultra-agile" techniques; with some success. Hard to codify,
| though, as they depend on a high degree of _experience_ , as well
| as the fact that I'm a bit "spectrumish."
|
| The closest to success that I have had, is to design _all_ of my
| software as an extensible infrastructure. I don 't know, exactly,
| how it will be extended, but I write "hooks" into it. I often end
| up using these "hooks," myself, in future work.
| 2OEH8eoCRo0 wrote:
| Software is more art than science. Show me the F=MA of software.
| What is the industry accepted fizz buzz?
| ailmanki wrote:
| Every system requires sooner or later another system on top.
| https://en.m.wikipedia.org/wiki/Chaos_theory
| cdogl wrote:
| The city metaphor is evocative and new to me. I like it.
|
| > Conversely, if we built our cities the way we build our
| software, you would need to enter the shop through the special
| garage, and exit through the roof to walk a wire to get to
| another custom made building from scrapped containers to do the
| checkout. And some of the windows are just painted on because
| they're an MVP.
| davidgl wrote:
| I think Stewart Brand's concept of Pace Layers fits well with
| the city metaphor of how software changes over time at
| different rates https://sketchplanations.com/pace-layers
| dagw wrote:
| That does actually sound like real cities to me. You have to
| walk 1 km to get to a point 50 m away because they never put an
| underpass in when they built the railway line. That flagship
| building stands half finished for 4 years because the people
| involved got bought out by another company that never got
| around to finishing it.
|
| And then you have that one section left over from the original
| release that all the engineers agree desperately needs to be
| refactored and upgraded, but due to cost and politics they
| never get to do it. And anyway you have a couple of power users
| who insist that the backwards and broken way that part is
| implemented is actually perfect and shout very loudly anytime
| you suggest changing it.
| mojuba wrote:
| Thought the same, but I also think we already have software-as-
| a-city if you take operating systems, cloud platforms or for
| example message queue systems on top of which you build
| components (e.g. microservices at larger scale). Message queues
| are the streets and as such they are probably not the most
| efficient but at least they are flexible enough to allow you to
| experiment with the buildings: rebuild, renovate and repurpose
| them while leaving the streets more or less intact.
|
| The problem with this industry seems to be that once something
| is solved, i.e. we already have reliable, battle-tested
| "streets", there is a big pressure to push everything further,
| build even more complex systems, faster. The pressure is rather
| natural: you will have a competitive advantage if you can push
| the limits and build something that can't be done based on the
| previous architectures, within a limited time frame.
|
| For example, building desktop apps is a solved problem. These
| are you streets, these are your building blocks. But because
| it's a solved problem there's little money to be made out of
| it. The money lies somewhere on the edges of the map (e.g.
| SaaS) where there are still no roads and no general
| urbanization plan, and it's where the businesses tend to flock
| to. Hence the chaos, uncertainty and quality problems in most
| of innovative software.
| PurpleFoxy wrote:
| Large software projects are almost as complex as cities and yet
| we have almost no one working on them. 4 developers can be
| enough to build a product used by millions. That's kind of
| incredible IMO. 4 builders/architects/engineers don't go nearly
| as far.
| WJW wrote:
| 4 architects are also enough to design a house that can be
| built millions of times. If you include the builders into the
| city projects, it would be fair to include the tens of
| thousands of people working in data centers and network
| exchanges as well.
|
| Also like a sibling comment states, 4 developers is not even
| close to a "large" software project.
| qznc wrote:
| Software allows incredible leverage by reusing stuff. I would
| never consider four developers a large software project
| though.
| lelanthran wrote:
| > Conversely, if we built our cities the way we build our
| software, you would need to enter the shop through the special
| garage, and exit through the roof to walk a wire to get to
| another custom made building from scrapped containers to do the
| checkout.
|
| I live 1.6km from my office. I drive 9.2km to the office.
| Cities aren't as straightforward as you might think.
| ailmanki wrote:
| Every system requires sooner or later another system on top.
| Entropy is increasing.
| https://en.m.wikipedia.org/wiki/Chaos_theory
| gillesjacobs wrote:
| >How is it that we've found ways to organize the work around so
| many other creative disciplines but writing software is still
| hard?
|
| I think OP heavily overestimates the organisational praxis in
| other disciplines. Nearly every creative discipline I was ever in
| was largely purely ad-hoc with very little explicitly stated
| organisational approaches to the craft. Academia, movie and music
| production, and creative writing for instance have much less
| readily-available principles than software.
|
| Software is probably the most thought about creative industry I
| can think off in modern history.
| JoeyJoJoJr wrote:
| I think the software industry could learn a lot from the
| collaborative attitudes and approaches employed in the more
| typically creative industries. Those industries you mentioned
| may not have as many formalised principles, yet collaboration
| is so imbued into the process that it naturally lends itself to
| large scale creations.
| 0xbadcafebee wrote:
| Here's why:
|
| First, consider a car. Imagine if somebody told you that a random
| company would be changing the software that runs your car, every
| day, as you drive it. Freaked out? You should be. That's what a
| modern tech company is. They just don't have the same risks, so
| they don't care so much if they fuck up, so the work isn't
| organized very well.
|
| Next consider cities. Do you want to build something? Great. I'll
| ignore all the costs (there are many).
|
| First you're gonna need a design by an accredited expert. That
| design needs a dozen permits approved by the government before
| you can even touch a trowel or hammer. Then a team of experts
| guess to town, changing grade, pouring foundation, waiting to
| cure, installing basic plumbing, and then getting an initial
| inspection. Then comes framing, inspection, plumbing, inspection,
| electrical, inspection, HVAC, insulation, drywall, fixtures,
| trim, walkways, driveways, flooring, landscaping, grading, and
| inspection.
|
| So far we have used specialist teams to build sections of the
| building and only continue when a strict inspection by
| authorities says we can continue. No customer has used the
| building yet. Also consider that this building will not scale. If
| you reach capacity, go make another building.
|
| Building software at scale is a combination of monkeying around
| with a car's internals while someone is driving it, and building
| a building. Only the risks to human life are lower, and we do not
| have teams of very specific contractors strictly following
| government approved codes and zoning laws to build one thing that
| meets one set of criteria.
|
| The people doing the job(s) in tech aren't good enough at it to
| do the things we're asking of them. And they aren't just building
| one piece and walking away, they are constantly fiddling. And
| very often, they have no master plan approved by inspectors
| according to well known and inviolable codes.
|
| On top of that, they have never sat down and figured out how to
| do all of this really efficiently amd reliably. Scrum/Kannan are
| general processes; they do not tell you specifically how to build
| a website in an extremely efficient and error-free way. But we've
| done that a million times by now, poorly. It's because we haven't
| yet codified it as an engineering discipline and stripped out the
| fat. And we haven't done that because there's no requirement to,
| because nobody's life is on the line, the way it is with cars and
| buildings.
|
| And it's hard. It's hard because we still hand-make components
| rather than buy them off the shelf. Every company I've worked at
| has re-created the same god damn thing 1000 times, by hand,
| because for some bizarre reason they thought it was a better idea
| to forge steel than to connect plumbing. Imagine if your plumber
| hand-crafted her own custom fittings for each job.
|
| Really, we do a _marvelous_ job today considering how completely
| undisciplined, unregulated, haphazard, and dangerous what we do
| is. There are many approaches we could take to simplify the
| actual process of it all, and make it efficient. But the
| complexity and difficulty will be there for a very long time. We
| can also move to pre-fab, but you 'd have to convince people that
| writing new software just to build a tech product is a bad idea.
| Good luck with that.
| praptak wrote:
| TFA mentions "Notes on the Synthesis of Form" by Christopher
| Alexander but doesn't really do the book justice: _" things get
| exponentially easier if you take problems one piece at a time"_.
|
| That's _sort of_ what the book says but it 's missing the
| critical part. The whole book is about _how_ to split the
| solution space into pieces that can be taken "one piece at a
| time", as it's obviously not the case that just any split will
| do.
|
| It goes through some real world examples of design and goes on to
| build a more general semi-formal theory of design, components and
| independence.
| dools wrote:
| I have often thought about the comparison to film, and film has
| been around for over 100 years. So in like 2080 software will be
| like film. In the year 4000 software will be like houses. In the
| year 10,000 software will be like nature.
| narrator wrote:
| I'm surprised this article didn't mention test driven development
| for building large pieces of software!
|
| I thought the test-driven development idea was impractical and
| pedantic when I first heard about it. I gave it a shot, though,
| and the thing it allowed me to do was to prevent massive systems
| from becoming impossible to modify. When a system gets so big
| that it is impossible to fit it in one's mind, most developers
| get scared to do new releases because there's probably some
| critical part of the system they forgot. They forgot all the
| requirements and QA test cases for at least one obscure part of
| the system. Even if all the QA test cases are documented, the
| process to release becomes increasingly difficult and time
| consuming.
|
| With test-driven development, you can just run all the tests, and
| if they pass, say ship it. The key is not to get too many tests,
| especially integration tests which take a long time to run and
| like to break when designers make UI changes. Usually, I start
| with the happy path integration test and then write tests as I'm
| developing for things that don't work right. About 40% of the
| stuff works the first time, and I never have to write a test for
| it. 45%, I write the test and fix it, and it passes, and I can
| forget about it. The other 15% is usually some tricky algorithm
| with many corner cases, and that's where most of the testing
| goes. I typically write one happy path integration test and then
| fill in the lower level tests as needed. When the happy path
| works, most of the system is working. That said, spending time on
| integration tests is usually a massive waste of time. It either
| works the first time or something lower down broke, and one
| should write a test for the lower level part that runs in a
| shorter amount of time than the integration test.
|
| I was able to port a large enterprise app from an Oracle backend
| to Postgres because there were tests for everything. The port
| just amounted to getting all the tests to run on the new database
| and the necessary data migration. This migration was by no means
| a trivial feat, but it was at least possible, and it ended up
| working and saving the company millions in license fees.
|
| The point being. A system with millions of lines of code is
| approachable if it has good tests. A new developer can work on
| it, and if all the tests pass, that developer didn't break
| anything. I can go back to the code I wrote years ago and still
| use it if I run the tests, and it works. I can then see how it's
| supposed to work, add features, and so forth. Without tests, this
| is very difficult because as systems become bigger, they break
| more and more if they don't have tests.
| adithyasrin wrote:
| I know this comes with experience but imagining how to scale or
| how to actually implement it is daunting. Obviously not everyone
| is going to have to need, but there's just so many tools and so
| many things to keep in mind to scale. What would you reckon is
| the best way to learn how to scale? Possibly one step at a time.
| rakoo wrote:
| Pick the most expensive servers/databases/storage and increase
| the number of users. Every 1-2 changes in orders of magnitude
| of the latter will require you to think thoroughly about how to
| best use resources and not explode your budget.
|
| Also read from the highscalability blog, there's a lot of
| experience recorded there. See for example this transcript of
| an AWS presentation about a step-by-step guide to scaling:
| http://highscalability.com/blog/2016/1/11/a-beginners-
| guide-.... Of course it's done with AWS bricks but the ideas
| are universal
| adithyasrin wrote:
| Thanks, wasn't aware of the blog. Will take a look.
| robotmay wrote:
| The way I learnt a lot myself was to run applications with
| little budget, e.g. running them on anaemic servers with few
| resources. When you have no option but to improve performance,
| because the alternative is too expensive, it gives you a good
| prod to design with it in mind.
| harimau777 wrote:
| I sometimes suspect that this is a consequence of the anti-
| engineering mindset that a lot of the software industry embraces.
| Things like YAGNI, programmers don't need to study CS, not hiring
| testers, etc.
|
| Maybe if software "engineers" were treated more like actual
| engineers then we would start seeing better results.
| thinkingemote wrote:
| Does the old adage "80% of all software projects fail" still
| hold?
| CrLf wrote:
| Define "fail". We go through our lives labeling failures as
| successes.
| redis_mlc wrote:
| If you mean large software projects, yes, if you include
| slipped deadlines.
|
| The real killers are:
|
| - no owner, or fake (insincere) owners
|
| - data cleanup
|
| - schema re-mapping (on 10,000 tables)
|
| - no usable requirements
|
| - disinterest.
|
| I've done several enterprise-scale migration projects, and have
| a 100% track record, because I was a "real owner." But when I
| look through old jiras, I always see failed versions of the
| identical project by employees who refused to step up and be an
| owner.
|
| Favorite compliment/curse, "You must have really wanted it."
|
| Most SAP projects fail, as well as most new billing projects,
| some on a colossally expensive scale - think $100 million and
| up.
| the_duke wrote:
| At scale software stops being a technology problem and becomes a
| people problem. And groups of humans don't scale.
|
| We naturally organize into hierarchies of small subgroups with
| limited interaction boundaries. Each subgroup will adopt slightly
| different methodologies, have different tooling preferences, will
| have integration and communication overhead with others, and so
| on.
|
| This can not be preventend, only mitigated. Which is perfectly
| fine. The important step is recognizing that software development
| is really not special [1], and that the limiting factor is the
| human element. Then you can build your organizational structures
| around these restrictions.
|
| [1] https://news.ycombinator.com/item?id=26713139
| p0nce wrote:
| And who manages to do that?
| oblio wrote:
| This is part of the reason microservices are becoming
| reasonably popular. They facilitate scaling people. If you
| think about it one of the biggest projects we have, the
| military, scales through assembling near-identical pieces
| structurally (but not functionally, an artillery unit has a
| totally different role to an infantry unit).
| _the_inflator wrote:
| I see it differently. In my experience with 200+ devs,
| Microservices itself is a buzzword to a certain extend and
| allows for different team to avoid talking to each other:
| they build their "own" services, which are copies which leads
| to bloat.
| heisenbit wrote:
| Microservices may help scaling but tend to come with a fixed
| amount of overhead which makes them more effort at a very low
| and low scale. They are also perfect as long as you got the
| overall architecture right and do not have to make cross-
| cutting changes because this turns a coordinating people in a
| team to coordinating people across teams problem.
| vsareto wrote:
| And predicting cross-cutting concerns is at least as
| difficult as predicting all of your business changes, which
| is at least as difficult as predicting the future.
| josephg wrote:
| Yes! Ironically, microservice architectures replicate the
| same set of advantages and challenges you have scaling
| _teams_. The abstractions (team / process boundaries)
| become much more rigid. But in exchange, within a team /
| service, change becomes easier because there's less to
| understand and navigate.
|
| How well the whole system functions depends deeply on how
| well those abstractions map the solution space. When you're
| doing something routine (eg most web development), lots of
| people have figured out reasonable ways of breaking the
| problem down already. So it works pretty well and you don't
| carry much risk that the abstractions you pick from the
| start will hurt you down the road. But when you're doing
| something more creative and exploratory - like a
| marketplace for VR, or making a new web browser or making
| something like Uber - well, then you take on abstraction
| risk. You will probably make mistakes in how you initially
| break up your teams and services. Executing well depends on
| your organizational ability to learn and refactor.
| oblio wrote:
| Well, with the ideal technical leadership, you start off
| with a monolith and clearly delineate boundaries inside it.
|
| Then as the project grows chunks just get carved out.
|
| Though real life is much messier.
| nly wrote:
| Something that only works for huge sizes isn't scalable by
| definition. Scalable means it can scale up _or_ down
| kaliszad wrote:
| Great point! Actually "scalable" in any direction is
| meaningless until you provide the scale, performance,
| reliability, cost and time requirements you have to
| consider while scaling. A single server for a typical web
| workload is just fine, if you don't need to be up more
| than ~ 99,9% of time, if you don't have more than 10s of
| thousands of users. Considering the availability, you
| could e.g. reboot into a smaller or larger VM. (I know,
| this is crazy to write on HN but a single virtual server
| is plenty reliable for most things if you are able to do
| a very fast recovery from backup to limit the extent of a
| disaster.) :-) If you don't consider "serverless", it
| really doesn't get much better than a single server
| considering cost. If you want more reliability/
| availability and more performance, you probably have to
| take other approaches. E.g. more servers, services like
| S3/ lambda, maybe a mainframe (cluster) in certain
| situations etc.
| ncallaway wrote:
| I disagree to an extent. Very few solutions are scalable
| across an entire range of a problem space.
|
| Something being scalable, to me, means that it can cover
| a _broad_ range of the problem space, but doesn't imply
| it covers 100% of it.
|
| If we imagine that the "problem space" is managing
| software projects from 1 person up through 1,000,000
| people and you told me that there was a solution that
| worked for the range of 50-1,000,000 people I would
| describe it as scalable, even if I wouldn't recommend it
| to a 5 person team.
| bob33212 wrote:
| The sub groups are going to be somewhat homogeneous internally.
| So you will end up with a group who thinks leetcode profeciency
| is important in hiring and another group who thinks QA
| experience is more important and another group who thinks
| diversity is the most important. All those groups think they
| are right and better than the other groups and they don't work
| well together.
| deckard1 wrote:
| This is all outlined by Fred Brooks in 1975. I've beat this
| dead horse before, but I suppose I'll beat it again.
|
| In the late '90s and early aughts everyone read Mythical Man-
| Month. There is even a law attributed to him, Brooks' law [1].
| It really feels like _no one_ internalized what he was actually
| saying. I suppose in a world of trends and fads, timeless
| advice is hard to recognize.
|
| > [...] the increased communication overhead will consume an
| ever-increasing quantity of the calendar time available. When n
| people have to communicate among themselves, as n increases,
| their output decreases and when it becomes negative the project
| is delayed further with every person added.
|
| Microservices, of course, exasperate this by actually
| increasing the communication required to make software.
|
| [1] https://en.wikipedia.org/wiki/The_Mythical_Man-Month
| eloff wrote:
| > Microservices, of course, exasperate this by actually
| increasing the communication required to make software.
|
| I HATE microservices. I've never seen them used
| appropriately, although I imagine it can be done. But I think
| if you increase human communication by using microservices
| you're defeating the raison d'etre of microservices. That's
| not the fault of microservices - it's the fault of the humans
| using them badly or where they shouldn't.
|
| Story time. I once contracted to a fortune 500 doing a $100M
| total rewrite with literally 100 microservices all in their
| own git repo. It was such a disaster I contemplated shorting
| their stock. I don't know how the story ended, but I feel
| confident it didn't end well.
| TheCoelacanth wrote:
| Microservices can be okay, but they are greatly overused.
| They are a solution for very large organizations with a ton
| of teams. For 100 micro-services to make any sense at all,
| you need to have at least 200-300 developers.
|
| Total rewrites, on the other hand, are never anything other
| than a complete and unmitigated disaster. The only way that
| you can possibly make a $100M rewrite successful is to do
| it Ship of Theseus style, where the rewritten code is
| incorporated into the working and actively-used production
| software long before the full rewrite is complete.
|
| Even then, the full rewrite usually won't complete, so you
| need to make sure that project is still useful even if you
| only do a partial rewrite.
| marcinzm wrote:
| >Microservices, of course, exasperate this by actually
| increasing the communication required to make software.
|
| I'd argue the exact opposite. Microservices with defined
| interfaces and APIs mean that everyone doesn't need to
| communicate with everyone. Only the people who work on a
| given microservice need to communicate amongst themselves.
| Across those groups the communication can be restricted to
| simply reading documented APIs. That in turn drastically
| lowers the amount of communication needed.
| dalbasal wrote:
| Yet... the economics of software is such that software
| companies balloon in size.
|
| Software companies/teams/applications of today are many time
| larger than in 1975. It may be more efficient to have
| software that's federated and developed by small teams but...
| Facebook's revenue is $90bn. A social network 10% FB's "size"
| will make much less than $9bn in ad revenues^.
|
| That means that product teams grow until the point Fred
| Brooks warned about play out... until the marginal team
| members' contribution is negative.
|
| ^For example, Reddit has 10%-20% facebook's users, but only
| about 2% of its revenue.
| CTmystery wrote:
| > The Team Topologies book suggests to favor teams that are end-
| to-end, that fully own a problem to be solved, supported by
| platform teams and teams that manage a very complex piece of
| technology.
|
| Does anyone have first hand experience with this working really
| well? It seems sensible but in my experience it does not work.
| The platform teams, I think, should be collaborations between a
| small number of members from each end-to-end team. The platform
| would be allowed to grow only as needed by the use cases. As soon
| as you put a platform team in charge with a mandate, what they
| build and what is needed starts diverging in the name of "Long
| term planning". Instead of fixing the struggles of today, they
| think they have the capacity and formula for fixing the problems
| of a year from now. In my experience, they do not. Would love to
| hear from others
| kasperni wrote:
| TLDR: Complex systems are complicated to build
| vishnugupta wrote:
| To add/elaborate systems get complex because the underlying
| reality is not simple, as much as we'd like it to be. No matter
| what paradigm (OO, strong/weak type etc.,) we choose or
| methodology (waterfall, agile etc) we adopt to build the
| underlying reality can't be changed or mounded to fit the
| software, it is always the other way around.
|
| Just a few days ago there was this thread about date
| calculation. One would think how complex can it be? But as one
| digs deeper they keep discovering layers and layers of special
| cases.
|
| Take Uber as another example. Push a button get a car was their
| aim. We see people commenting here that they will be able to
| build it over a weekend. But the real world of cab hailing has
| thousands of unwritten, implicit rules that are in people's
| head. All of them have to be codified and need to work
| seamlessly, and on top you add constraints of distributed
| systems and physics etc.
|
| Or take tax laws, it's a labyrinth of rules large enough that
| people have built DSLs.
|
| So, any software that's solving real world need will be
| complex. We just have to bite the bullet and deal with the
| reality.
| chrisco255 wrote:
| Fred Brooks was right:
|
| "Brooks insists that there is no one silver bullet -- "there is
| no single development, in either technology or management
| technique, which by itself promises even one order of magnitude
| [tenfold] improvement within a decade in productivity, in
| reliability, in simplicity."
|
| The argument relies on the distinction between accidental
| complexity and essential complexity, similar to the way
| Amdahl's law relies on the distinction between "strictly
| serial" and "parallelizable"."
|
| https://en.m.wikipedia.org/wiki/The_Mythical_Man-Month
| magwa101 wrote:
| One of the problems with software is that projects are definied,
| people allocated, time spent, all with a mindset that the product
| will be "done". With software, it is never "done". "Art is never
| finished it is merely abandoned" comes to mind. Building software
| is more like bees living in a beehive than the construction of a
| building. Our mental model of software is not accurate.
| ozim wrote:
| I see another person using Fred Brooks "accidental complexity" in
| a wrong way.
|
| Software development is hard because "essential complexity" is
| hard. Whatever people want to do is hard. You can make it easier
| to write software as much as you want, but you are not making
| "world" easier. That is main point of Fred Brooks essay.
|
| I see a lot of devs are making statements that now development
| for them is essential complexity. While no, software and code
| never will be essential complexity, software only helps us to
| solve essential things faster.
| billti wrote:
| Two quotes stood out to me:
|
| > What if software were built in the same way? What if the core
| parts of our business would be like streets, and all that
| newfangled stuff is something we could build on top, experiment,
| tear it down if it does not work?
|
| To me, it kind of already is. The foundation on what we build
| most software on today; TCP/IP, HTTP, Win32, POSIX APIs, the
| C-runtime library, threads & processes, etc. Much of that is
| about 50 years old now.
|
| > Somehow, the code in front of you is just the tip of the
| iceberg of a lot of mental representation of what is happening
|
| I think this is a key point. When we look at code, the comfort
| with it is relative to how well we understand that abstraction,
| and have a good model for what it is doing "under-the-hood". I
| think a lot of the burn-out and "fatigue" some people feel is
| where these abstractions are regularly reinvented (I'm looking at
| you, JavaScript frameworks), and thus you have to spend time
| learning the details of a new abstraction before you can just
| read/write a piece of code comfortably.
| hnarn wrote:
| Sometimes I feel like there are two worlds in IT: the one you
| encounter on a daily basis, and the one you read about on HN. The
| only thing consistently "terrible" about both of them is the
| often expressed lack of humility towards solutions you don't
| "like", or more often, actually, solutions you don't understand.
|
| Gottfried Leibniz suffered no small amount of flak for his
| conclusion that the reason the world looks the way it does is
| because it's already "the best of all possible worlds": you may
| think you see an improvement, but you lack divine understanding
| and therefore you don't see how it would actually make things
| worse.
|
| While this is in my opinion a silly idea, I think most people who
| work in tech could use a bit of humility influenced by this line
| of thought: if a solution exists today, and it has yet to be
| replaced, it's at least _possible_ that it 's over all the best
| solution, and the reason you're not seeing that is because of a
| lack of understanding.
|
| Of course this is not always true, but if nothing else it would
| lead to more interesting discussions than those stemming from
| someone saying "if you're not using Kubernetes you're a moron"
| and another replying "learn how to sysadmin and you'll realize
| you don't need Kubernetes in the first place".
| anorwell wrote:
| Chesterton's fence [1] captures this idea: one should not tear
| something down until they understand it.
|
| [1]
| https://en.m.wikipedia.org/wiki/Wikipedia:Chesterton's_fence
| wpietri wrote:
| We are at the very beginning of the history of computing. The
| age of print lasted 500 years, and there's every sign the age
| of computers will be longer. The notion that anything we are
| doing now has been perfected is hard to swallow. Especially
| given the way each innovation creates new opportunities for
| improvement.
|
| I'm willing to assume that this is the best of all _possible_
| worlds, that everybody did the best they could in the moment.
| But having all that history behind us gives us lessons such
| that tomorrow 's best possible world is surely better than
| today's.
|
| I agree that we should have less technical arrogance and
| jackassery in the future. But that emotional immaturity is also
| part of today's best possible world.
| gameswithgo wrote:
| A think I have been realizing a lot, is that in all aspects of
| life, a person of even average intelligence can easily imagine
| how things _ought to be_. The licensing office should be more
| organize, the software should start instantly, the government
| should be more efficient, etc etc.
|
| However we are humans, only recently evolve from something a
| bit like an ape. Hampered by emotions and constant fits of
| irrationality. Actually achieving the correct behaviors that we
| can easily imagine is _very hard_ and _very rare_.
|
| So we are correct in our criticisms but could be more calm and
| understanding about the fact that everything is terrible, and
| strive to make even very small improvements and enjoy those.
| goatinaboat wrote:
| _A think I have been realizing a lot, is that in all aspects
| of life, a person of even average intelligence can easily
| imagine how things ought to be. The licensing office should
| be more organize, the software should start instantly, the
| government should be more efficient, etc etc_
|
| I don't believe this is true, an average person can easily
| imagine a local maxima, but everyone's local maxima is
| different.
|
| For example it might be very convenient for person A if the
| licensing office were open 24 hours but probably not so
| convenient for person B who would have to cover the night
| shift every other week or person C whose taxes would need to
| pay for it. And so on.
|
| Nearly every sub-optimality from one perspective is an
| optimality from another.
| bjornsing wrote:
| True, but don't forget that the frequency and severity of
| those fits of irrationality depends _tremendously_ on how
| accepted irrationally is (and what personal consequences it
| carries). I know it sounds like a contradiction, but I think
| we need both high standards and an understanding that working
| together on intellectual tasks is hard.
| rorykoehler wrote:
| It's an interesting mental model but I would put forward that
| it should be the best possible world we are currently capable
| of creating. Due to our own biological shortcomings I don't
| think we'll ever reach the best possible outcome in anything.
| ukj wrote:
| Then why do you call it a "possible" outcome?
| rorykoehler wrote:
| It's possible but infinitely unlikely
| [deleted]
| ukj wrote:
| 1 in infinity likelihood is 0.
| rorykoehler wrote:
| I disagree. So does quantum theory
| simpleguitar wrote:
| The recent HN article about how our brains prefer additive
| solutions vs the subtractive also seems to point in the
| direction of ever increasing complexity.
|
| Maybe the field of software can really mature only when looking
| for subtractive solutions become part of the computing culture.
| jhgb wrote:
| > Gottfried Leibniz suffered no small amount of flak for his
| conclusion that the reason the world looks the way it does is
| because it's already "the best of all possible worlds": you may
| think you see an improvement, but you lack divine understanding
| and therefore you don't see how it would actually make things
| worse.
|
| Isn't this line of thought disproven by a single improvement?
| People before the improvement could have argued along the same
| lines and then be proven wrong.
| narag wrote:
| Wow, a lot of responses, this one will get buried, but here I
| go anyway.
|
| In my opinion, what makes most of the wrongs we see is that
| computing is very young and we haven't had the time to set up
| half decent solutions for anything.
|
| There is a huge demand for any solution, because a bad solution
| is better than no solution. The Kubernetes vs sysadmin is like
| the Diogenes story. There are two versions. In one of them
| someone tells Diogenes "if you praised the king, you wouldn't
| need to eat lentils" and he replied "if you ate lentils you
| wouldn't need to praise the king". In the other version it's
| just the same story only Diogenes talks first.
|
| I don't like lentils. I don't like to praise the king either.
| Both are terrible solutions, but while we have kingpraiseless
| burgers, we need to choose something that somehow works.
| runawaybottle wrote:
| This flys in the face of how progress occurs. There's a reason
| why movie theater seats are full leather reclinable with cup
| holders. It's that absurd push for imagining 'better'.
|
| Large systems in general are hard to refactor. Our impulse in
| tech is to always consider the refactor because we see how in
| other systems a refactor is impossible or takes blood (our
| governments and countries, institutions like
| education/healthcare). Those in tech have seen many systems to
| compare against (what went wrong, what went good, even on the
| most basic projects).
|
| The first thing they tell you about capitalism is that humans
| haven't encountered a better system, which is true. Make that
| argument to a programmer. They either saw a shittier system, a
| better one, a more fitting one, a proper one, a terrible one,
| etc. You pretty much can't tell us anything because we've seen
| it somewhere.
|
| Being content with systems is bad for all of us. Being content
| with your life is good for all of us.
| tracerbulletx wrote:
| It would also be nice to look at the efficiency of software
| engineering and business practices from 10 years ago, 20 years
| ago, and 30 years ago and realize just how far we've come.
| Software is running more things, bigger things, more important
| things, more complicated things, and we get better all the
| time, it's ok to live in an imperfect world and just make
| things a little better every day. Its ok to try things, find
| they're suboptimal and make changes, it's ok to retry things
| that have been done before but with a new spin.
| mightybyte wrote:
| Interesting. This kind of seems like the software version of
| the efficient-market hypothesis from financial economics. It's
| pretty much obvious that markets have inefficiencies. But at
| the same time, they're the best we've got and constructing
| something that's better in general is really hard.
| rebelos wrote:
| Good analogy. I'd say the efficient markets hypothesis would
| be better interpreted as "you cannot deterministically
| identify an inefficiency", i.e. you're always assuming some
| amount of risk with any trade. You could probably make a
| similar case for improvements more abstractly - you cannot be
| perfectly confident a priori that something will be net value
| additive over the previous configuration.
| smoe wrote:
| While I do agree on the humility and lack of understanding
| points, my perspective is more, that the current solution is
| probably not the best, but what the best solution would be is
| pretty much unknowable for any non trivial system. Instead, it
| is more likely that you are approaching a local maxima. So the
| question is, whether it makes more sense to optimize your
| current approach or seeking another maxima which implies making
| things worse for a while before they get better and you might
| get stuck halfway there.
|
| Either way might be a better choice for a given situation and
| due to lack of information and changing environments carry
| their own risks.
| copycatchiller wrote:
| yes, but there are also things done because "they have always
| been done this way." That's also a dangerous pattern imo.
| michaelcampbell wrote:
| > the one you encounter on a daily basis, and the one you read
| about on HN
|
| This is true of any "real world" subset, vs its online outlets;
| HN may just be more niche than some.
| Joeri wrote:
| Chesterton's fence as applied to software: don't replace a bad
| software solution until you understand why it is that way.
|
| Still, looking back we can see how software solutions were not
| optimal for their age, and how amazing things can be coaxed out
| of old hardware using new insights. It would be pretty
| remarkable that precisely now is when we finally achieved
| optimal software. More likely, we're still getting it wrong,
| and this is just one more step along the line.
|
| The truth can be a little of both: yes, we can do better than
| what exists now, but we're not guaranteed to unless we
| understand what came before.
| reader_mode wrote:
| I think software is relatively young and evolves faster than
| any branch I've seen simply because the compute platform
| changes so much over night. Just imagine that the PC I started
| coding on was a P1 machine with 16MB RAM. My smartwatch has 10x
| compute power. Cloud computing, IoT, GPGPU, SSD, multicore -
| major underlying constraint shifts in how we do compute.
|
| Even the best possible solution is only best in the time it was
| created, and because it's good it will get entrenched and keep
| getting used, even when the constraints it was created in no
| longer apply even remotely and it's actually a very suboptimal
| solution given new constraints.
|
| Everything from programming languages, OS kernels, file
| systems, distributed computing frameworks - soo much of it has
| ideas and solutions that are technically better but it's hard
| to overcome the inertia
| noir_lord wrote:
| The "PC" I started on was a 1Mhz z80 with 1KB of RAM - the
| first computer I owned had 48KB.
|
| I'm sat in front of a machine that is literally billions of
| times faster and many orders of magnitude more powerful in
| every other axis (storage, bandwidth).
|
| And yet, strip away the fancy IDE, the container tooling and
| it's all _still_ my brain, some text on a screen and my
| understanding of the problem.
| reader_mode wrote:
| But the problem isn't "how to do this in 48KB of ram" these
| days it's "how do I do this on a distributed compute cloud
| with high level services".
|
| Neural networks were a hypothetical approach, nowadays you
| can rent a ML pipeline for a couple of bucks and download
| prebuilt models.
|
| So taking an approach from that era (for example C or UNIX)
| has obvious legacy baggage and isn't optimal in modern
| context but is still used because of inertia.
|
| Also at this point it's less about your brain and
| understanding and more about being able to search and mash
| together. It's not just having a fancy IDE, internet and
| search fundamentally changed how you learn and solve
| problems.
| noir_lord wrote:
| Agreed on all counts.
|
| The primitives we use are at the wrong level of
| abstraction for the problems we are solving but..well I
| haven't seen anyone come up with ones better _enough_
| they displace the existing solutions.
| pdimitar wrote:
| You do have an excellent point for part of the cases.
|
| We should IMO also entertain one more possibility: namely
| pressure from the top.
|
| A lot of people who I knew as sympathetic and calm before they
| took management roles turned into something I could code in one
| minute: namely a program that asks "how much is this going to
| take?" and if your answer is above N hours/days then they say
| "no, we're not doing it". And that's not because they are
| stupid or suddenly non-sympathetic. It's because their bosses
| optimize for those metrics and measure them (and their salary,
| and their bonuses, and even their vacation days) by the same
| metrics.
|
| Perverted incentives play a big part in the current state of
| things where everyone just looks baffled at thing X and is like
| "why in the holy frak isn't this working better?".
|
| Truth is, MANY, and I mean really A LOT of people know that
| their work is sub-optimal but pressure from above prevents them
| from making it better.
|
| And people are not independent -- for most people outside IT
| and other prestige / high-skill professions, standing up to the
| boss means 3 to 9 months of unemployment and maybe no food on
| the table. Hence everybody learns to suck it up and do the bare
| minimum that keeps them employed, perpetuating the cycle.
|
| Multiply that to at least 90% of our world and it's IMO quite
| easy to understand why it is like it is.
| bradleyjg wrote:
| I've been reading these Dilbert rants for twenty five years.
| I probably made some of them on slashdot.
|
| We as an industry tried the all engineers thing and it didn't
| work.
|
| Have you considered the possibility that the boss' bosses
| have a point? That the intelligent people you once respected
| have been convinced rather than intimidated?
| tremon wrote:
| _Have you considered the possibility that the boss' bosses
| have a point?_
|
| It's the organization that shapes the individual, not the
| other way around. Changing a cog in an existing gear system
| only works if the cog is of a particular shape, and it will
| get chafed and dented if not sufficiently so. If the cog is
| too different, it will break and be replaced again.
|
| Occam's razor says that if many different people fail
| within the same organizational structure, the problem is
| not with those people.
| didibus wrote:
| > We as an industry tried the all engineers thing and it
| didn't work
|
| Example of when this was tried?
| bradleyjg wrote:
| https://medium.com/illumination/google-once-fired-all-
| manage...
| pdimitar wrote:
| I would be ashamed to call myself a thinking human if I
| never considered it.
|
| And as said in other comments, for some companies and
| businesses they are right indeed.
|
| But not in programming / IT in general. At least not where
| I worked (and where many of my former colleagues worked).
|
| It's all optimizing for short-term and never the long-term.
| And I have seen people playing the long game gaining a lot
| so I also know I am not imagining it.
| kansface wrote:
| Predictability in the short term has a definite business
| value in and of itself, regardless if it makes
| development slower - or perhaps, a value to management.
| Certainly the degree matters.
| pdimitar wrote:
| You know, I am actually VERY okay with short-term
| thinking but not when it's 100% of all decision making.
|
| Make it 75% short-term objective and give the other 25%
| to longer-term planning and/or simply playing with ideas
| and I'll be very happy to work in such a place.
|
| I am not a fanatic -- and most programmers are actually
| quite understanding (if a bit too timid sometimes)
| people.
|
| We just want the pendulum to not be on one of the sides
| 100% of the time. That's it really.
| [deleted]
| lamontcg wrote:
| There's an implicit assumption here that everyone is "doing a
| bad job" because of "perverse incentives".
|
| But I'd argue that most 'computer nerds' have pretty deeply
| seated perfectionism flaws.
|
| To take a simplified example, we tend to want perfect
| security, and like to prove that we're smarter than each
| other by understanding complicated attacks.
|
| When it comes to running a business, though, a 10 person
| startup cannot afford to act like their threat model includes
| defending from hostile nation state actors willing to do
| Stuxnet-levels of effort to hack them. If they do that much
| work on security their product will never get off the ground.
|
| The goal of most companies when it comes to security is never
| to have perfect security, but to have good enough security to
| stay out of the headlines.
|
| It is like tradeoffs that have evolved in nature. You don't
| see organisms that maximize one feature at the expense of
| everything else. That energy always needs to go into
| competing concerns.
|
| So sometimes as technical people we fetishize the perfect
| solution to some problem in the Enterprise, when actually
| doing that solution would be horrible for the Enterprise due
| to the costs.
|
| (Of course the flip side is sometimes management only cares
| about window dressing the company so they can bailout with
| some golden parachutes and that has nothing to do with
| running a healthy company -- but not every example of a
| company not being perfect is due to deeply cynical and
| perverse incentives like that, but may be entirely rational).
| pdimitar wrote:
| > _But I 'd argue that most 'computer nerds' have pretty
| deeply seated perfectionism flaws._
|
| My goal here absolutely was not to turn this sub-thread
| into a ping-pong of "but managers bad at X" and "but but
| but programmers bad at Y".
|
| I am very aware of the nuance you introduced and I've been
| guilty of being a tech perfectionist much more times that
| I'd be comfortable with sharing. But finally I learned.
|
| (Granted, many haven't and maybe never will. In this regard
| you do have a solid point.)
|
| > _So sometimes as technical people we fetishize the
| perfect solution to some problem in the Enterprise, when
| actually doing that solution would be horrible for the
| Enterprise due to the costs._
|
| Absolutely. Not denying it. It's just that what gets me is
| the _reaction_ to some of the engineers ' flaws is to go to
| the other extreme.
|
| Which is just as bad state of things. Both extremes are
| unproductive.
| itsArtur wrote:
| > A lot of people who I knew as sympathetic and calm before
| they took management roles turned into something I could code
| in one minute: namely a program that asks "how much is this
| going to take?" and if your answer is above N hours/days then
| they say "no, we're not doing it". And that's not because
| they are stupid or suddenly non-sympathetic. It's because
| their bosses optimize for those metrics and measure them (and
| their salary, and their bonuses, and even their vacation
| days) by the same metrics.
|
| I had this exact same thought recently when reflecting on my
| behavior in my new role as a "technical product owner". All
| of it was reflexive, as if I suddenly forgot all of the
| software engineering knowledge I accumulated over the years
| and became a deadline-driven cog.
|
| I don't have a solution yet; I think it comes down to that I
| don't yet speak the same language that people I report to do,
| and thus I feel like I can't defend my position well enough.
| It comes with experience, I guess!
| pdimitar wrote:
| Part of what makes you an enabler of programmers -- and not
| just a guy who screams "faster! harder!" -- is to be able
| to push back a little.
|
| There's sadly an inherent conflict of interest between the
| people doing the job on the ground (programmers) and the
| rulers (managers).
|
| Your job is to find the middle ground -- not to completely
| side with your higher-ups.
|
| That would be a good starting point for you IMO.
| Xunjin wrote:
| "Hence everybody learns to suck it up and do the bare minimum
| that keeps them employed, perpetuating the cycle."
|
| In my point of view, that's how you lose solutions to stay in
| the comfort zone, and I'm not saying the people whom actually
| "just obey orders" but those who make them.
|
| As I gain more experience, with age and some wisdom (I hope),
| I see that most top-down decisions are actually the worst way
| to decide things, people who are "at the bottom" maybe not
| understand how a entire system works however they do really
| knows (and suffer) on how their work is sub-optimal.
| yawz wrote:
| Pushing down the decision making is important for that
| reason. The "makers" know best most if not all of the time.
| The manager is there for support and for facilitating the
| learning. E.g. With decisions we make mistakes. As a
| manager it's important to ensure that the team reviews past
| decisions and learns from them to improve the decision
| making process.
| marcosdumay wrote:
| That's evidently correct, and all the empiric results point
| into correctness too. As always I recommend reading Deming.
| Or go directly to the Toyota production system. The best
| working system is one where management sets the goals and
| workers decide the means, while one trusting each other to
| do their job.
|
| https://en.wikipedia.org/wiki/Toyota_Production_System
|
| But somehow, management never accepts this reality. Even
| the Japanese auto industry is regressing into Taylorism.
| NotPavlovsDog wrote:
| There may not be that much difference between Lean
| (Toyota) and Taylorism. Don't forget that Toyota (Toyoda)
| is not that far from Fordism - "you get whatever color
| you want as long as it is black". Employees get to tweak
| the process but the fundamental model stays the same.
| They don't get to change the assembly line, nor the
| parts, nor what is assembled. And Toyota guaranteed life-
| long employment when it was implemented.
|
| I've worked the line twice - first time as a teen looking
| for any work, the second time as an undercover analyst
| hired to examine why Lean was failing at a plant, both
| doing the same type of job: electrical component
| assembly. I got physical injuries (minor) both times. The
| majority of the assembly line workers had repetitive
| stress injuries, including losing fine motor skills and
| wrist control due to use of impact wrenches and exposure
| to bad ergonomics. If a human delaminates, you can't glue
| them back.
|
| Although _rational_ (Taylorism, Management by Objectives,
| MBO, OKR, Milestones, management by goals ) vs.
| _normative_ (Lean / Toyoda, human-centric, Agile,
| humanist, culture ) division of management
| styles/rhetoric has been proposed, they should perhaps be
| seen through the prism of Creative Destriction (not
| disruption, and surprisingly, CD comes from Marx).
|
| In a vulgar approximation, imagine capital-backed
| entities served by the manager class, which are stressed
| by short deadlines and performance-tied bonuses and
| careers. They are constantly shifting basic
| approximations of solutions and perspectives, without
| challenging the system.
|
| _Give employees more autonomy, and output and managerial
| control suffer. Instill more rigid control, and risk
| losing the ability to innovate and "knowledge producing"
| (Drucker of MBO fame) talent such as programmers_.
|
| In an antagonistic system, whatever you come up with has
| its own roots of destruction on top of built-in conflict
| (Marx again). Hence the back-and-forth re-framing between
| "rational" (Taylorism) and normative "culture, baybe,
| Holocracy!" so fast that today it is a zombie that has
| both and neither.
|
| You end up with projectification, corporate "culture"
| that demands the soul _and_ meeting arbitrary
| "milestones" from employees, but the corporation itself
| can only guarantee arbitrary termination.
|
| There is a great and influential analysis "Design and
| Devotion: Surges of Rational and Normative Ideologies of
| Control in Managerial Discourse" by Barley and Kunda. I
| would suggest reading "Management fashion" by Abrahamson
| as well. Both can be found through G.scholar, including
| no-registration copies.
|
| https://www.jstor.org/stable/2393449 https://journals.aom
| .org/doi/abs/10.5465/amr.1996.9602161572
| marcosdumay wrote:
| Sorry, but if all you got from Toyotism was Lean, than
| you got somewhere already fully geared into Taylorism.
| There is much more to it, and yeah, even one of the
| tenets you will get from Deming is that short deadlines
| and performance-tied bonus and careers can't work. He
| also goes a great length about empowering employees to
| change things like part sourcing, but yeah, not the final
| product.
|
| (I don't know about worker health problems, and I imagine
| that poor as Japan was at the time, any empiric data on
| it would be useless. But, anyway, I don't think the
| management structure is the correct place to fix that one
| problem.)
|
| And I'll have to break into your Marxist narrative, but
| all that was once incredibly successful in a fully
| capitalist framework. It never became non-competitive, it
| was only replaced by order of top-down command
| structures.
| dragonwriter wrote:
| > from Deming is that short deadlines and performance-
| tied bonus and careers can't work.
|
| well, for the performance pay, Deming is more subtle: its
| not that it doesn't work, its that the performance must
| be something meaningfully under the employees control for
| performance pay based on it to work (otherwise it can't
| influence anything), _and_ it must be a holistic measure
| of company value (otherwise you get perverse incentives
| around microoptimizations), and your measure of
| performance must take into account the degree of
| uncontrolled variation in the process (otherwise you give
| false negative and positive signals due to random
| variation rather than real problems /improvements).
| NotPavlovsDog wrote:
| I would appreciate an expansion on the "break into".
|
| The approach I outline can explain and predict why the
| two Ts rotate, including at Toyota (Toyota going Taylor,
| or rather normative - rational).
|
| I prefer not to base my experience on what Deming or
| anyone else wrote, but how their writing relates to real-
| life practice (though of course as a critic of Taylorism
| as represented by MBO and other Performance Evaluation
| rational schools, he is a welcome breath of fresh air).
|
| Have you _observed_ , preferably both as a participant
| and independent, Deming idealism in a functional real-
| life system? I have only seen a shift towards
| projectification and cultural hegemony.
| marcosdumay wrote:
| What breaks the narrative is that the system is clearly
| not antagonistic. Giving the correct kind of autonomy to
| the workers increases every single item that management
| should care about, at least in a capitalist system. And
| both logic and evidence point that way.
|
| > Have you observed, preferably both as a participant and
| independent, Deming idealism in a functional real-life
| system?
|
| As a participant, never. As an independent, we are
| talking about the mentor of the movement that teached the
| Japanese management how to go from an industry that
| didn't produce anything good into the highly developed
| one that outcompeted everybody all ever the world.
| NotPavlovsDog wrote:
| > What breaks the narrative is that the system is clearly
| not antagonistic
|
| So why the shift to Taylorism at Toyota, the heart of the
| Toyota system? Why the shift to projectification from
| life-long employment at Japan? How can one explain
| chronic injury if workers get to plan and improve the
| workflow (or are wasted humans not considered waste)?
|
| And the Toyoda principles at the heart of Toyota predate
| Deming. (Ohno, Taiichi (March 1988), Just-In-Time For
| Today and Tomorrow, Productivity Press, ISBN
| 978-0-915299-20-1 )
|
| The list of paradoxes can continue indefinitely, or one
| can elect to look for the root cause.
| dasil003 wrote:
| "Top-down" has become an epithet that disgruntled engineers
| throw around, but the reality is that there are problems at
| all layers of an org, and they should be solved at the
| appropriate level.
|
| Trying to solve every problem top-down manifests as micro-
| management that beats the morale and initiative out of
| strong ICs. But on the other hand, trying to address
| everything bottoms-up leads to local maxima, no
| standardization across tech or product, and warring
| factions within the company.
|
| There is real art to this. Being able to grok the
| incentives and structures of a vertical slice through the
| org (C-level, VP, Director, Mgr, IC) as well as the tech
| (Brand, Product, Pages, Components, APIs, Middleware,
| Storage, Infra) is a rare and extremely valuable talent
| that allows solving problems that are impenetrable by any
| traditionally defined role.
| pdimitar wrote:
| Everything you said is true. And it still doesn't change
| the fact that most organizations [I've been in] utilize
| the top-down approach on 99% with the occasional 1% of
| bottom-up.
|
| That's an unbalanced state of affairs that completely
| loses very valuable feedback everywhere along the chain.
|
| Part of the job of a manager is to find this perfect
| balance on the level they are working at. And most don't.
| They just pass the pressure put on them, down below to
| their underlings.
| jakub_g wrote:
| This. It's also really difficult to build a mastery of things
| when you're constantly moving from one piece of code to
| another according to shifting priorities.
|
| And once you build mastery of $fwk/$lib/$tool, it's
| deprecated and you migrate to another one. GOTO START
| pdimitar wrote:
| Sadly yes. I really want to... stand still for a bit and
| learn things properly, you know?
| analog31 wrote:
| If the manager is a former engineer, they might just be
| following a rule of thumb, that bigger tasks are impossible
| to manage, and above a certain size threshold, carry an
| existential risk for the business.
| BurningFrog wrote:
| I think OPs point applies to organizational management as
| much as programming:
|
| > _if a solution exists today, and it has yet to be replaced,
| it 's at least possible that it's over all the best solution,
| and the reason you're not seeing that is because of a lack of
| understanding._
| cascom wrote:
| I'm not sure I'd call the incentives perverted (only
| perverted in the sense that "perfect" software is not the
| goal).
|
| Management and owners are a far from perfect group that make
| plenty of bad strategic decisions for a host of reasons, but
| they also have to deal with the real/perceived constraints of
| time, budget, market, competition, etc. that are easy to not
| appreciate when you are focused on the design/build portion
| of the business.
| zaphar wrote:
| Some of what you say about management is true. But if you
| really want to fully understand then you need to keep
| following that trail of incentives. Your bosses boss doesn't
| optimize for those metrics "just because". There are reasons
| and persuading them requires understanding those reasons. A
| company has only so much revenue. Wasting that revenue in the
| best case reduces profits. In the worst case it kills the
| company. Changing a perverse incentive requires understanding
| whether it is actually perverse or not and then proposing a
| compelling alternative that results in better results for the
| company.
| tremon wrote:
| _Changing a perverse incentive requires understanding
| whether it is actually perverse or not and then proposing a
| compelling alternative that results in better results for
| the company._
|
| While that's true, considering those incentives and
| changing them if necessary is part of my boss's job, not
| mine. My job is only to report the misaligned incentives I
| perceive to my boss. If my boss isn't able to make me
| understand his decisions, that's my cue to find a new boss.
| And I concede, those bosses are rare. I've only had two in
| twenty years of employment.
|
| It takes time to recognize the qualities you need in a boss
| to perform well. I know myself well enough in that
| department to actively look for those qualities. Even to
| the point where I prefer a less challenging job under the
| right boss than the perfect job under the wrong leadership.
| alisonatwork wrote:
| I agree that often there are good business reasons for
| killing projects, and plenty of so-called technical debt is
| actually just the result of pragmatic decision-making. Imo
| developers should get used to maintaining that stuff,
| because that's a huge part of the job in most profitable
| companies.
|
| But there is also a flip-side where I see the opposite
| happen - especially in startups and VC-funded companies.
| These companies can take massive losses while at the same
| time investing huge amounts into elaborate solutions that
| appear to primarily exist for the purpose of creating
| impressive bullet points on the resumes of careerist
| employees. In these companies, even in management, the work
| seems to be driven by everyone trying to maximize the
| impact of their own, personal "tour of duty", then making a
| profitable exit. These incentives are not the same as those
| of people working at more legacy/boring companies.
| pdimitar wrote:
| > _These companies can take massive losses while at the
| same time investing huge amounts into elaborate solutions
| that appear to primarily exist for the purpose of
| creating impressive bullet points on the resumes of
| careerist employees._
|
| Yep, that's one of the things I had in mind when I said
| "perverse incentives".
| pdimitar wrote:
| I am sure that what you say is very true for not a small
| amount of businesses.
|
| My issue is mostly with my profession -- programming.
| Sometimes we are so micromanaged and our time is so sliced
| and diced that all of this short-term thinking easily gets
| 5x more expensive on a short time frame anywhere from 3 to
| 6 months.
|
| This has been observed numerous times yet IT managers seem
| to want to never learn.
|
| It's OK to try and save money. But saving money short-term
| in order to have a huge expense down the line is just being
| irresponsible from those managers.
| hallway_monitor wrote:
| Having led a team for about a year now let me tell you
| it's not easy. You are always trying to balance many
| priorities and even with over a decade of coding
| experience, it's not always obvious which refactorings or
| tech debt stories or architectural changes are worth
| prioritizing now, which can wait till later, and which
| are engineer pipe dreams that I need to gently but firmly
| kill as quickly as possible.
|
| I have had similar managers to what was described above,
| in those types of organizations. It may be a lost cause
| but it may be an opportunity to work on your influence
| skills; being able to explain and sell your solution is
| perhaps even more important than coming up with it in the
| first place.
| pdimitar wrote:
| I wholeheartedly agree that we the engineers must work on
| our non-tech talking skills!
|
| I've been working on it for a few years now and it's
| REALLY difficult but even the small wins feel very
| rewarding and are motivating me to refine my comm skills
| further.
| tacitusarc wrote:
| This is really difficult. Perhaps the best thing is to
| track the proposed technical debt stories, and do not
| discard them when they are rejected. Instead, also keep
| track of how much pain is hit that can be associated back
| to that particular technical debt.
|
| 3 years into major project, my team finally got
| prioritized to work on and improvement that will take us
| a couple weeks, that we were not able to work on
| previously because we could not justify its value. But
| for the past 3 years the organization has lived with pain
| that this would have resolved. So now we can justify it.
| And the thing is, it's not like we didn't know 3 years
| ago when we said we wanted to make this happen but things
| are going to be terrible if we were not able to
| prioritize it but in many organizations it's not good
| enough to just say how bad things will be. People need to
| actually have felt the pain.
|
| Honestly, I think that's okay on its face. My bigger
| issue is that frequently there is no or very little trust
| around engineers who predict what the major problems will
| be an advocate for them and then it's not like when
| they're proven right management now listens to them the
| next time. I think that's the real problem.
| mavelikara wrote:
| > My bigger issue is that frequently there is no or very
| little trust around engineers
|
| This is a problem. But at any given time management hears
| programmers arguing for both sides - the "if you're not
| using Kubernetes you're a moron" and the "learn how to
| sysadmin and you'll realize you don't need Kubernetes in
| the first place" (quotes from the top of the thread). In
| many cases folks in senior management do not have any
| idea of these things, and they rely on one of these two
| opinions, resulting in the other half claiming here on HN
| that management is all bull.
|
| What the profession lacks is a code of practices - like a
| Joel Test accepted by every accredited engineer.
|
| I am not sure that is net positive for such a rapidly
| evolving space. Even now people are debating if unit
| tests are needed and other basic topics. The only
| "settled" thing about the profession seems to be that
| gotos are bad.
| spockz wrote:
| Anecdotally, we had the opportunity to get better gear
| for developers than an outdated single 24" 1080p screen
| and slow computer. Higher management invited a few
| engineers to share Ygritte ultimate setup. These varied
| widely between one large ultra widescreen and three small
| screens. The developers in the meeting started arguing
| vehemently in the meeting and in the end they couldn't
| agree and all developers were left with the same stuff.
| If you were enterprising enough you could gather two of
| those 1080p screens until someone from workplace it saw
| that and installed to desk to spec.
| mavelikara wrote:
| That is copy-book Sayre's Law:
| https://en.wikipedia.org/wiki/Sayre%27s_law
| cosmodisk wrote:
| It was a mistake to give an option to discuss this to
| start with. It's always if not this than that, x brand is
| better than y,etc. What matters: is it a
| Windows/Linux/Mac shop?
|
| No matter what brand,the base unit comes with enough RAM,
| processing power and spacious SSD.
|
| Everybody gets the same brand,as it's easier to support
| from the IT perspective.
|
| Keyboard,mouse,monitors should be an individual choice.
| Some want to ruin their vertebrae by staring at a tiny
| 13" screen and be happy about it,while some need 3
| monitors and 2 keyboards to feel the power,why not.
|
| Not sure why some companies are so tight when it comes to
| the equipment for developers,as it's always a small
| fraction of the overall cost of the position.
| psahgal wrote:
| I find that the best senior engineers don't say "if
| you're not using Kubernetes you're a moron" and "learn
| how to sysadmin and you'll realize you don't need
| Kubernetes in the first place", but instead say
| "Kubernetes is fantastic for this use case because of
| these reasons..." and "sysadmin would be easier to use
| than Kubernetes for this problem because of these
| reasons..."
|
| I'm not sure how one gets software engineers to justify
| and back up their decisions like this on a regular basis,
| but I suspect it would help management trust their
| engineers more.
| lanstin wrote:
| (Bad) Management chooses to listen to engineers that say
| agreeable things, not true things.
| mavelikara wrote:
| > I'm not sure how one gets software engineers to justify
| and back up their decisions like this on a regular basis,
| but I suspect it would help management trust their
| engineers more.
|
| More balanced senior engineers (I know anecdotally that
| there are lots of them) speaking up and challenging their
| fellow-engineers to justify their extreme positions will
| be a good start IMO.
| pdimitar wrote:
| This is definitely a very difficult spaghetti to
| untangle, agreed. But IMO a very good start would be to
| never accept such polar statements in the first place.
|
| "If you use extremist and simplistic generalizations you
| are not worth listening to in a discussion" would be the
| first policy I'd put into a team.
|
| Balanced arguments with pro/con analysis is what makes an
| engineer senior and a contributor to an argument.
| Everybody else shouldn't participate the next time
| around.
| mavelikara wrote:
| Agreed. But the balanced people are the least vocal. As
| Bertrand Russel noted: "The whole problem
| with the world is that fools and fanatics are
| always so certain of themselves, and wiser people
| so full of doubts."
|
| I am someone who moved from a programer role into
| management. My appeal to the balanced senior engineers is
| to speak up more. We don't have to swing as much as
| economists, enough to frustrate President Truman to say:
| "Give me a one-handed Economist. All my economists
| say 'on hand...', then 'but on the other..."
|
| But we can bring more nuance into discussions -
| especially calling out those among us with polar
| positions.
| cosmodisk wrote:
| As a manager, you need to find a way to pull information
| from people: with some it's a team meeting,with others
| it's over a meal during the lunch,etc. I even do this
| sometimes during the management meetings,when I see
| people clearly not quite happy about something but
| reluctant to speak,so I ask them directly what's on their
| mind and that's often enough for them to tell way more
| than you could have gotten by just expecting for them to
| speak up.
| pdimitar wrote:
| All of that is sadly true and it's a fundamental problem
| with humanity. I'd say that the only solution I am seeing
| is that the managers must proactively shut down the
| extremists early and hard.
|
| If you demonstrate that you will not submit to
| extremists, I've found that a good chunk of the more
| balanced people are then encouraged to speak up.
|
| ---
|
| TL;DR: Silence the bad and loud voices, and the wiser and
| quieter voices will start speaking.
| kclay wrote:
| In these cases wouldn't it have been to work on it slowly
| as a "side project" outside of the team normal
| priorities? I always tell my team if they can fit it in
| and not hinder the current sprint by all means go ahead
| and do it on the side.
|
| I know this goes against the normal and people want to
| attach "right now " value (hours it took reflected on
| check) but going beyond the call can also give you
| leverage when your yearly review comes up for example.
| pdimitar wrote:
| That's exactly what I am doing and aside from a few wrist
| slaps people have been okay with such an approach. I
| always tell them I did this for a few hours during the
| weekend or while I was waiting our DevOps to resolve an
| outage these particular 2 hours the last Friday. Seems to
| work really well.
| hpoe wrote:
| I'll also add I only have so much energy in the day, and
| after a couple of years of fighting against problems I
| coudn't do anything about and managers who couldn't make
| the changes, and customers that wanted it now, I burnt
| out, and instead of spending a useless zoom meeting to
| work on an improvement I use it instead to make maps for
| the D&D campaign I am running.
| [deleted]
| marcosdumay wrote:
| > Your bosses boss doesn't optimize for those metrics "just
| because".
|
| Your bosses optimize for those metrics because it's the one
| they can communicate higher up on the short time they can
| get higher ups attention.
|
| If it were because the metric was any good, there wouldn't
| be so many cases of companies bankrupted by technical and
| cultural debits pilling up.
| pdimitar wrote:
| > _Your bosses optimize for those metrics because it 's
| the one they can communicate higher up on the short time
| they can get higher ups attention._
|
| That's their problem and not mine, is it?
|
| A good manager doesn't just reflect the lightning bolt to
| their underlings. They absorb part of the shock and
| process / transform what they will transfer down below.
| If they aren't doing that then they aren't managers; they
| are uneducated opportunists in suits.
|
| > _If it were because the metric was any good, there
| wouldn 't be so many cases of companies bankrupted by
| technical and cultural debits pilling up._
|
| Really well articulated, kudos.
| bobthepanda wrote:
| It also takes time to turn a big ship, and by the time one
| realizes one needs to make the turn it may be too late.
|
| To extend this to non-tech examples, failing big companies,
| like Sears, do not disappear overnight; the decline was in
| full view for over a decade but ultimately the ship could
| not be turned. With tech the problem may be even worse;
| Uber still has not turned a net profit, but with tech
| investors like to throw money at losing businesses. Some
| turn around, but that's not guaranteed.
| legulere wrote:
| A lot of things have entirely historic reasons, that don't
| apply anymore.
| mellavora wrote:
| if you can enumerate these reasons and demonstrate that they
| don't apply any more, then you are in a good position to
| change the thing.
|
| If you don't know what these reasons are, it is hard to know
| with certainty that they no longer apply.
|
| see above reference to Chesterton's fence.
| croes wrote:
| I don't see it that way. People are creatures of habit and
| prefer to torment themselves with existing problems that
| they have learned to deal with rather than get involved in
| something new.
| iFreilicht wrote:
| People also love to create and get new and shiny things.
| Combine that with force of habit, and you get truckloads
| of engineers that tear down and replace what was working
| and only needed maintenance.
|
| Both forces exist, the important part is to balance them
| properly.
| TeMPOraL wrote:
| In my head, I've started to understand the suck in terms of
| timescales. Leaving products that are too complicated for the
| people who build it aside, even if you have the most competent
| and well-oiled team onboard, there are two things that together
| guarantee their output will suck:
|
| - Usually, the "right way" to do something is only apparent
| halfway through doing that thing, or after you finished that
| thing.
|
| - There's rarely time to redo things with benefit of hindsight,
| because the priority is always to work on business goals first.
|
| That's how things were everywhere I worked or seen, so I
| believe this is likely a general phenomenon. But the
| consequence of that is strong path dependence. It's not that
| using Kubernetes makes you a moron or a smart person. It's that
| somebody already deployed Kubernetes for a project that's too
| simple for it (or they were too inexperienced), but business
| pressure makes it impossible to spend a couple of weeks undoing
| it, so everyone is stuck with the pain. Perhaps the reason for
| the initial Kubernetes deployment was because it was the
| fastest way to fix some other issue, and the company could not
| allocate enough time for engineers to do it right.
|
| To be clear, this isn't me ranting about "stupid managers with
| their priorities". Pursuing business goals is the raison d'etre
| of a software company (it's pretty much a tautology). I'm just
| saying, it's hard to improve things without lots of slack, and
| slack is a competitive disadvantage.
|
| Perhaps another facet to this is: business lifecycle is too
| short for software lifecycle. If you were to spend time on
| redoing things properly, by the time you were done, you'd be
| out of business.
| yourabstraction wrote:
| I've seen this a lot when new people join a software project
| and immediately start telling people the "right" way to do
| everything and that all their solutions are far from optimal.
| Sometimes this can be useful if the project was a mess, but
| more often than not the new guy is failing to take time to
| understand the subtle reasons behind the team's various
| decisions. It's likely that the solutions were actually closer
| to optimal that meets the eye when you consider time
| constraints, developer resources, company politics, etc.
|
| I've been on both sides of this, and look back and cringe when
| I think about projects I've criticized before trying to
| understand them. I've also had the experience of being early at
| a startup and having later employees come in and constantly
| complain about how sub optimal everything is. The problem is
| they're missing the time crunch that was in place, and the fact
| that if subpar solutions weren't used the company would have
| never gotten to market and they would have never gotten a job
| there. Software isn't created in a vaccuum.
| jmkni wrote:
| It reminds me of that Instagram thing, you look on Instagram
| and see beautiful people living perfect lifes and feel
| completely inadequate in comparison.
|
| Then you look a little deeper and realise most of them are
| pretending, they're photoshopping their selfies, renting their
| expensive cars for a day and taking their selfies on other
| peoples property.
|
| HN can be a bit like that, you can feel completely inadequate
| compared to these super-geniuses doing everything perfectly,
| when really you're doing ok.
| josephg wrote:
| There's a bell curve of software engineering ability in the
| world. There's a few super geniuses, a bunch of smart cookies
| and a whole lot of people in the middle.
|
| HN attracts a broad range of ability, but generally upvotes
| the smartest content. So if you sample from "upvoted HN
| articles and comments", the content you read is heavily
| biased toward things written by unusually smart people. But
| there's a range.
|
| By contrast, most companies will attract and hire people like
| their current employees. So the variance at any particular
| company is much smaller than in the general pool. Everyone
| ends up working alongside people like them.
|
| So if you work at a big bank or something, you probably don't
| interact with many people like the super geniuses who write
| interesting articles that get upvoted a lot on HN. They're
| over represented amongst HN content, but most programmers are
| average and getting by just fine. And remember, hotshots are
| equally ungrounded. Lots of good engineers you meet at Google
| were hired at straight out of collage. They have no idea how
| the sausage is made anywhere else. Lots of folks here don't
| have your experience. They have no idea what it takes to do
| software at a company that sees their job as a cost centre.
| Or what it takes to get good work done in spite of political
| pressure to stagnate.
|
| Having a variety of voices gives all of us something to learn
| and something to teach.
| 908B64B197 wrote:
| > There's a bell curve of software engineering ability in
| the world. There's a few super geniuses, a bunch of smart
| cookies and a whole lot of people in the middle.
|
| We see the effects of that throughout our industry and yet
| it's never publicly acknowledged.
|
| > And remember, hotshots are equally ungrounded. Lots of
| good engineers you meet at Google were hired at straight
| out of college. They have no idea how the sausage is made
| anywhere else.
|
| One of the most eye opening experience one can do is 3x the
| compensation for a software role and see what caliber of
| engineers it attracts. I remember one company who did that
| for college hires and they were shocked. They had been
| whining for a while about the "talent shortage" and that it
| was hard finding folks in the local market. Turns out the
| folks they really wanted to hire were there all along, just
| invisible.
| dragonwriter wrote:
| > There's a bell curve of software engineering ability in
| the world.
|
| Probably not; across both the world as a whole and the
| profession specifically, a power law distribution is a lot
| more believable.
|
| Even in a particular firm, its probably closer to a power
| law, or maybe an offset bell-like curve where the median is
| above the mode and there is a much longer high tail, then a
| common bell curve.
| theli0nheart wrote:
| > _HN attracts a broad range of ability, but generally
| upvotes the smartest content. So if you sample from
| "upvoted HN articles and comments", the content you read is
| heavily biased toward things written by unusually smart
| people. But there's a range._
|
| Does HN really upvote the "smartest" content? Citation
| needed, please.
| asimjalis wrote:
| I am not certain about this either. My experience has
| been that the newer posts on the new tab are frequently
| more interesting than the posts on the leaderboard.
| bckr wrote:
| For instance, the OP is not very well written or
| informative, IMHO. Great discussion though
| efrecon wrote:
| Isn't this typical HN though: the quality is most often
| in the discussion, rather than the original link?
| bckr wrote:
| Not necessarily, although I do appreciate the overall
| quality of discussions here. Reminds me of a very good
| 00's forum (yes I am a youngling)
| michaelcampbell wrote:
| Indeed. HN is an echo chamber like most things of its
| ilk. It's just like the Emmy (etc.) awards; actors
| praising actors for acting; the rest of the world is
| largely orthogonal.
|
| The amount of pretense and smug here isn't any less about
| its domain than any other self-selected community.
| cryptica wrote:
| I'm also skeptical about that. It depends on whether or
| not the smartest content aligns with people's financial
| interests. Sometimes the smartest content goes against
| most people's financial interests so it is heavily
| downvoted.
|
| HN is a financially-oriented tribe. It's like in the
| Bitcoin community. Only the pro-Bitcoin content gets
| upvoted and anti-Bitcoin content gets downvoted. The
| smartness of the content hardly matters at all. Sometimes
| the smartest content is at the bottom of the list.
|
| Usually, I read both the top and bottom parts of HN then
| I get some of the best arguments on both sides.
| noir_lord wrote:
| The simple solution is not to look on Instagram (which is a
| bit trite).
|
| The more complex solution is to develop a worldview where you
| only compete against yourself, i.e. is what I'm doing now
| going to make me a better person in X time.
|
| It's up to each person to decide for themselves what better
| means, it could be an emotional thing "will it make me more
| empathetic?" or a financial thing "will this allow me to earn
| more money?" or something else "will it make me a better
| programmer, chess player, cyclist, *happier*?".
|
| For all the size of the world and its complexity it collapses
| down to a local optimisation problem for you as a person.
| vidarh wrote:
| It doesn't even require them to pretend (though I do realise
| many do pretend). All it requires is people to be more likely
| to post their successes, or if mistakes then at least
| mistakes on the path to success.
|
| Even if everyone were to post the mundane it wouldn't
| generally be what would get upvoted or liked in a given
| medium enough to counter the successes.
| vharuck wrote:
| This is my thinking, too. On Instagram, we're seeing the
| best photos of the people who are on their one vacation of
| the year. Even if we ourselves post a vacation photo and it
| gets shared and liked, we won't think of it as one of
| "those" beautiful photos, because we just got lucky with
| that one. We remember our embarrassingly bad photos.
|
| And on HN, comments are filled with insight backed up by
| decades of deep experience. But the Perl expert in one
| thread is unlikely to be the Russian historian in the
| other. There are few masters-of-all-trades (though they do
| exist, as if to inspire and mock us). Not to mention the
| majority of readers whose experience is rarely relevant
| enough to comment.
| finnh wrote:
| AA saying: "don't judge your insides by other people's
| outsides"
| m463 wrote:
| When you say that, I'm reminded of Feynman's book[1] describing
| the challenger accident investigation.
|
| Basically he's investigating and goes and talks to the software
| guys and quickly concludes everything is awesome, and moves
| onto investigating hardware.
|
| I was initially taken aback by this part. But he later he
| wrote:
|
| "To summarize, then, the computer software checking system is
| of highest quality. There appears to be no process of gradually
| fooling oneself while degrading standards, the process so
| characteristic of the solid rocket booster and space shuttle
| main engine safety systems."
|
| It makes sense, I mean they can fly simulations over and over
| before the shuttle is even built, let alone launched.
|
| [1] The first book was "Surely You're Joking Mr Feynman", the
| second with the challenger investigation was, "What do you care
| what other people think?"
| ignoramous wrote:
| > _... I think most people who work in tech could use a bit of
| humility influenced by this line of thought: if a solution
| exists today, and it has yet to be replaced, it 's at least
| possible that it's over all the best solution, and the reason
| you're not seeing that is because of a lack of understanding._
|
| Agree. See: https://fs.blog/2020/03/chestertons-fence/
|
| That said, we may not get far if no one makes an attempt to
| tear down those fences. Tech is like art in that respect: Lets
| practitioners experiment like no other engineering discipline.
| emodendroket wrote:
| In a bigger sense it just doesn't matter. A lot of the
| criticisms in the Unix Hater's Handbook are still trenchant
| decades later. And so what? Everyone knows how to use Unix and
| they're used to the deficiencies. Up and down the stack until
| the advantages of the new thing are so overwhelmingly obvious
| that people will jump.
| troelsSteegin wrote:
| Two rebuttals to Leibniz are "that was then, this is now", and
| "but this is special". Both are fluff until you answer the real
| question, which is "why are things the way they are"? It takes
| both humility and courage to engage with something you don't
| understand, but if you are curious, it is also deep fun.
| dalbasal wrote:
| My favourite walking dead character and comic idol is King
| Ezekiel, because "embrace the contradiction."
|
| Since we can't avoid thinking in generalizations, and they're
| usually false, the only option is to accept contingent
| thinking. Sometimes, respecting " _the way things are_ " is the
| right way. Sometimes, rejecting the way things are is the right
| way. It's often how things get better. There isn't a true "way"
| you can live by in all circumstances.
|
| There are hints though. If you're using disrespect/challenge to
| write a good essay (like this author) or to try and change
| something.... rejecting "the way things are" might be good. If
| you're just whining, probably not.
|
| Humility good. Complacency bad. These are definitions, not
| value statements. They're not necessarily contradictory, if you
| care about such things... but they can have a Hillel/Shamai^
| dynamic to them. Sometimes, an adversarial discussion like
| "Kubernetes" vs "Learn Tow to Sysadmin, You Moron" is
| enlightening. Sometimes it's no-it-all asholery. I don't think
| there are universal rules governing the distinction.
|
| In any case, I don't think this author is being "terrible." He
| does recognize that scaling is a problem that he doesn't fully
| understand and meanders through a lot of possible reasons. None
| of it is mean. The title is a little click-bait, but I don't
| think it's a foul. He's quite respectful, in fact.
|
| ^Like political left/right, but more bearded.
| pier25 wrote:
| > _There isn 't a true "way" you can live by in all
| circumstances._
|
| It's like applying principles like DRY. If you push that
| principle to the extreme, your code will be shit. DRY is a
| good principle, but not a rule to follow blindly.
| bjornsing wrote:
| > I think most people who work in tech could use a bit of
| humility influenced by this line of thought: if a solution
| exists today, and it has yet to be replaced, it's at least
| possible that it's over all the best solution, and the reason
| you're not seeing that is because of a lack of understanding.
|
| It also doesn't have to be the best, and probably isn't - the
| space of all possible bit strings large enough to encode
| Kubernetes is quite big. Being "good enough" is sufficient, and
| often the discussion really comes down to differing opinions on
| what that means. These is no uniquely and clearly defined
| objective function here. People need to stop thinking that
| theirs is the one and only.
| KajMagnus wrote:
| > _doesn't have to be the best, and probably isn't_
|
| Precisely. And, building the truly best solution, is often a
| mistake, in that it can take pointlessly long.
|
| Good enough is good enough, and others might find things to
| improve, and be correct that it'd been better in their way --
| the important thing is instead to realize that actually
| making any improvement one might have found, can be far down
| on the priority list. Maybe there'll always be other more
| important things to do, so it'll never happen.
|
| > > _" at least possible that it's over all the best
| solution"_
|
| can be rewritten as:
|
| > > _" at least possible that it's a good enough solution"_
|
| And maybe it can make sense to make improvement suggestions,
| and then, when the others say "yea but there's too much else
| to do", then, ok, maybe forget about it and be happy anyway.
| nbardy wrote:
| This is how I feel about JavaScript. I used to let the HN
| expert problem solving crowd show me all the problems and think
| of how bad it sucked. Now I'm happy it just worked most of the
| time.
| sidpatil wrote:
| > Gottfried Leibniz suffered no small amount of flak for his
| conclusion that the reason the world looks the way it does is
| because it's already "the best of all possible worlds": you may
| think you see an improvement, but you lack divine understanding
| and therefore you don't see how it would actually make things
| worse.
|
| https://en.m.wikipedia.org/wiki/Pareto_efficiency
|
| https://en.m.wikipedia.org/wiki/There_ain't_no_such_thing_as...
| de6u99er wrote:
| Doesn't matter if you use kubernetes. If the system has not
| been designed for scale-out you will ultimately lose.
|
| K8s is just a container orchestrator. And containers are just
| virtual machines.
| bckr wrote:
| That makes sense.
|
| > containers are just virtual machines
|
| I think this part is false though. Containers are, well,
| contained processes with contained storage. No simulated
| hardware like VM implies
| zo1 wrote:
| Its an unecessary complication when explaining containers
| to the uninitiated. Just tell them they are essentially VMs
| and they'll follow along with the other concepts just fine
| instead of looking at you wide eyed as they try decipher
| how "contained processes with contained storage" interacts
| with their use case. Once they get experience, they'll
| learn the details.
| betaby wrote:
| VMs are not 'simulated hardware' either honestly.
| bckr wrote:
| How would you put it, then?
| etaioinshrdlu wrote:
| To varying degrees they actually are. The hypervisor
| provides simulated devices and the kernel inside has
| device drivers to speak to them. For efficiency these
| devices are as "thin" as possible, but they are still
| simulated devices.
|
| Unless you're using something like PCIe passthrough to
| give raw hardware access to a VM.
| CuriouslyC wrote:
| People are afraid of change and uncertainty. Basically all our
| solutions are basically the best we could do without making any
| leaps into the unknown from our current position. Just because
| we couldn't have done better given our starting position, that
| doesn't mean we couldn't have done better if we'd started over
| with a new initial position.
|
| Think of it like gradient descent. The location you end up is
| the best location in the neighborhood of your starting
| position, but it's almost certainly a local minimum, and if you
| started over from a different initial position there's a good
| chance you'd find a lower minimum.
| Kinrany wrote:
| > At least, building a new project within a company should be
| easier than starting from scratch, but my hunch is that many
| companies fail that test.
|
| This observation seems both true and important.
| MattGaiser wrote:
| I have yet to find an organization that has figured out how to
| effectively transfer learned lessons to other
| projects/employees without retaining those specific people for
| long periods.
|
| Add in high levels of turnover and unless the specific person
| who learned the lesson is also on the project, there isn't a
| clear wrong way to prevent a wrong turn.
|
| I am part of a new team at a startup and everyone on the new
| team is a relatively new hire. As far as I know, we haven't
| taken any meaningful learnings. Despite the company existing
| for years, we have done everything from scratch.
| pdelgallego wrote:
| I help often customers understanding technical debt, and code
| quality. Most people focus of code metrics, but they tend to
| forget organizational metrics, and how that impact the
| software. There are several studies on this. I particularly
| like one from Microsoft [1].
|
| One of the metrics that I use address your point:
|
| Number of Ex-Engineers (NOEE): This is the total number of
| unique engineers who have touched a code and have left the
| company as of the release date of the software system
|
| Implications: This measure deals with knowledge transfer. If
| the employee(s) who worked on a piece of code leaves the
| company then there is a likelihood that the new person taking
| over might not be familiar with the design rationale, the
| reasoning behind certain bug fixes, and information about
| other stake holders in the code.
|
| A large loss of team members affects the knowledge retention
| and thus quality.
|
| [1] https://www.microsoft.com/en-us/research/wp-
| content/uploads/...
| watwut wrote:
| High levels of turnover typical suggest high dysfunction of
| some kind.
|
| Also, the way to disseminate knowledge is by talking about
| problems. As in, if you manage to create culture where people
| do that, you will occasionally hear "team x tried that and
| had peoblems, lets ask them".
| dontbeabill wrote:
| what's also interesting is the "problems" we are attempting to
| "solve" with that scale seem to be getting more and more driven
| on pure greed.
|
| does the world really need facebook, or better infrastructure,
| clean water, food, education.
|
| do we really next 2 hour prime delivery?
|
| but let's solve those with 100,000s of engineers moving fast,
| breaking things all because they get paid alot.
|
| so many developers today are just building massive tech debt to
| vest their RSU's and get out.
|
| how many are actually doing any good for the planet or their
| communities.
| Yizahi wrote:
| There is no "scale" so to say. To create MS Windows which has
| lets say 50 mloc, versus MS Calc which has maybe 1/1000 lines of
| code compared to Windows you don't "just" write x1000 times more
| code and done. Writing one Windows is many orders of magnitude
| harder than writing a thousand Calc sized apps. That's why
| "scale" don't work, it's not scale, it's difficulty spike. If
| you'll hire large enough corp, with several hundred expert
| programmers (no trainees , no juniors), and do what they think
| they want - don't interrupt them from flow, schedule almost no
| meetings, communicate strictly async via email, allow then self
| organize and self manage - I think that such a project will be an
| epic sized fail.
| de6u99er wrote:
| Scale up vs. Scale out.
|
| Those require different architectural considerations.
|
| E.g. with scale out your chache has to work suddenly in a
| distributed way.
|
| And as we all know scale-up is expensive, can quickly become very
| expensive, and more importantly it has upper limits. In most
| cases scale-out is actually the only option, even if scale-up
| would work for some while. Many times I have seen architects go
| the easy route and scale up. They either get promoted or switch
| companies until their solution either gets too expensive or can't
| scale any more.
|
| In the meanwhile other systems are interacting with this system
| and would need to make changes to adjust to a scale-out system.
|
| E.g.: A distributed cache can result in lower write performance,
| or someone else could have overwritten a certain entry because
| locking failed (e.g. SELECT FOR UPDATE).
|
| In many cases those systems join the list of legacy systems.
| bullen wrote:
| We are now seeing the limits of everything, USB2/3, Gb network
| (pushing beyond increases complexity to a ridicoulus amount),
| Gflops/W (2 is CPU peak for electronic transistors) and
| persistence (the most durable drives where made 10 years ago on
| 50nm).
|
| So now we know that one man can build, host and improve/maintain
| a product himself with the entire earths population as customers.
|
| This is a short window, before thermodynamics takes charge and
| ruin the prospects, enjoy it while it lasts!
|
| The first one to do Minecraft without hiring anyone or selling it
| wins. :P
| sokoloff wrote:
| In a few years, 2.5Gbps networking is likely to be where 1Gbps
| networking was not that long ago. What was once niche and
| expensive will be $20/port very soon, then $10, then $7...
| bullen wrote:
| See I don't think so, because of the physical limits and
| energy. I think both are peaking at the same time because
| while you might be able to invent new physics they are only
| executable in an environment of energy abundance, and peak
| energy happened 10 years ago.
|
| Try getting 1Gb/s to work without going crazy first! ;)
| sokoloff wrote:
| 802.3ab 1000BASE-T works over all of the most common
| Ethernet cables and structured wiring I've thrown at it,
| even wiring that probably fails to meet even the loose Cat5
| spec.
|
| I've not had any experiences being made crazy by it in the
| last decade. Some of the very old gear had interoperability
| issues, wouldn't auto-negotiate, etc, but that's often the
| case on any new "standard" tech. Nowadays, you can take the
| random $15 adapter, $2 cable, and $20 switch and expect
| it's all going to "just work" at gigabit speeds.
| bullen wrote:
| What router do you use?
| sokoloff wrote:
| My router is an Edgerouter-X SFP, but that is only the
| gateway from the local network to the Internet, not the
| main switch. The main switching is done by a mix of
| Ubiquiti 8-port PoE switches and a Mikrotik central
| switch.
| bullen wrote:
| And you can push 1Gb/s reliably over that?
| sokoloff wrote:
| I can push 1Gpbs everywhere on the local net. I can pull
| 265Mbps and push 12Mbps from/to the internet, limited by
| the 250/10Mbps internet plan I bought.
| bullen wrote:
| I think it's my ethernet cables... just ordered cat6
| we'll see!
| fulafel wrote:
| We have had well working 100 Gbit Ethernet for about 8 years
| now. It's been trickling down in recent years:
| https://www.servethehome.com/mellanox-
| connectx-5-vpi-100gbe-... (meanwhile the high end is at
| 200G/400G now for nics and switches respectively)
| bullen wrote:
| Sure I was talking for the home... most people can't afford
| $20.000 routers or fiber.
|
| I'm prepared to say that unless they can cram 10Gb/s into
| existing fiber (without increasing fixed- or energy- costs
| too much) we're stuck at 1Gb/s... forever!
| edgecrafter wrote:
| Infrastructure as code services is actually matching the City
| metaphor quite well. With the huge difference you can rescale and
| reshape your "city" at will
| qznc wrote:
| I agree with the post that software at scale is a big open
| question.
|
| What gives me hope is that there were successful mega projects
| though not about software. Fur example, I read about the
| Manhattan Project over Project Atlas to the Apollo Project. Those
| were all mega projects with high innovation and uncertainty like
| software development. I have no good theory what makes it
| successful though.
| sly010 wrote:
| Bad software (therefore all software) is like the recurrent
| laryngeal nerve in giraffes [0] (a freak of evolution). It is
| unlikely to be the result of a singular minds "design". It is
| more of a result of evolution through countless independent
| decisions, each of which are individually an improvement, but put
| together results in weird inefficiencies that are impossible to
| fix without starting from scratch. And when was the last time you
| were allowed to "redesign" something from scratch at work?
|
| [0] https://bioone.org/journals/acta-palaeontologica-
| polonica/vo...
| snidane wrote:
| The tooling is simply not there, so every software project keeps
| pushing the boundary of what is possible in its own unique
| fragile way.
|
| People don't want solutions to yesterday's problems. These are
| considered trivial and already solved, such as invoking a shell
| command (which just hides a lot of complexity under the hood).
| Noone will pay you for invoking existing solutions. They pay you
| to push the boundary.
|
| By tooling I mean programming languages, frameworks, libraries
| and operating systems. All of whuch have been designed for a
| single machine operation with random access memory model.
|
| This no longer holds true. In order to scale software today you
| need multi machine operations and no such OS exists for it yet.
| Only immature attempts such as Kubernetes, whose ergonomics are
| far from the simolicity that you'd expect from a unix like
| system.
|
| And random access memory model breaks down completely, because it
| is just a leaky abstraction. For full performance memory is only
| accessed linearly. Any random access structured programming
| language completely breaks down for working with GBs of data, in
| parallel and on a cluster of machines.
|
| I don't think we'll push the boundary further if we keep piling
| crap on top of existing crap. The foundations have to change and
| for that we'd better go back to the 70s when these ideas were
| being explored.
| the8472 wrote:
| > Any random access structured programming language completely
| breaks down for working with GBs of data, in parallel and on a
| cluster of machines.
|
| Why would you need a cluster of machines to work on mere
| gigabytes? You can get single-processor machines with terabytes
| of ram. Even laptops come with 64GB now.
| oblio wrote:
| Yeah, big data these days should be petabytes of data or at
| most high hundreds of terabytes coupled with very intensive
| access patterns.
| joe_the_user wrote:
| To put is cynically, everyone knows that yesterday's tools
| don't work. So it's better to try today's tools, which you
| don't yet know don't work. This allows one poorly thought out
| idea to replace another - especially when selling new stuff is
| profitable.
|
| That said, it's implausible that the main problem is scaling
| since mainframes scaled considerable data quite a while back.
| kungito wrote:
| I'd say the problem is way different and it's a people problem.
| A small percentage of people invest enougu resources to be able
| to have the big picture and write higher quality solutions.
| There is a huuge percentage of people with minimal education
| learning along the way and by learning along the way they
| reinvent things as a part of their learning process. This also
| incentovizes rising popularity of low quality but simpler
| tooling
| etothepii wrote:
| The number of developers (and developer managers) who want to
| only complete tasks where someone (else) has defined the
| inputs and the outputs is extraordinary.
|
| It is my firm belief that the vast majority of value in
| Software Development is in understanding the problem and the
| ways that Software can help. Crucially this includes knowing
| when it can't.
|
| When asked what my job was once I said, "to think and write
| code, the hard part is doing it in that order."
| shade wrote:
| This is a pet peeve of mine - I've worked with folks who
| were/are good developers but wanted to constrain their role
| to only development tasks. No effort to get exposure to
| build automation/deployment pipelines or understand how
| Docker/Kubernetes work at a high level, no desire to
| understand the bigger business domain or how our customers
| use the product. Pretty much just "give me a fully groomed
| ticket and leave me alone until I tell you it's done."
|
| On the one hand, I get it - sometimes it actually is nice
| to have a ticket that you don't have to think too much
| about the bigger context for, and can just crank out code.
|
| On the other hand, I don't think it's particularly good for
| your career security if you're limiting your own role to
| "minimally communicative code monkey" - if that's all you
| want to do, there's an awful lot of competition out there.
|
| I've made an effort the past couple of years, across
| multiple jobs now, to get some real exposure to the ops
| aspects of my roles (most recently, prototyping and then
| helping implement CI/CD pipelines for Azure Durable
| Functions) as well as making an effort to understand the
| business domain, where the gaps in our product offerings
| are, and what the competitors in that sector look like.
| It's really helpful in terms of looking for a more
| efficient way to solve a business problem, not to mention
| being able to say things like "hey, the market we're in is
| moving pretty heavily to Kubernetes, so it's really
| important that we decide how we're going to support that."
|
| I'm not saying you need to be (or can be) an expert in all
| of those things, but I think having the high level exposure
| is really important, at least when you get to a senior IC
| level. A minor bonus is it helps you develop a Spidey sense
| for when the business is pursuing strategies that don't
| seem likely to work, giving you a chance to either try to
| offer feedback/seek clarification on why, or to pull the
| plug and find a new gig if it seems bad enough.
| etothepii wrote:
| It's very frustrating and makes it easy for management to
| outsource your role. Sometimes that's the right thing,
| but given how little most "ordinary" companies leverage
| technology and how utterly most non-tech understand what
| can (and can't) be done it's usually the wrong call.
| IncRnd wrote:
| > People don't want solutions to yesterday's problems. These
| are considered trivial and already solved, such as invoking a
| shell command (which just hides a lot of complexity under the
| hood). Noone will pay you for invoking existing solutions. They
| pay you to push the boundary.
|
| People are paid to work on products which provide value to the
| business. There is always a Minimum Viable Product, MVP. Meet
| that. Exceeding that is the boundary that needs to be pushed,
| not grabbing the latest, possibly untested, tools off the
| shelf.
|
| > And random access memory model breaks down completely,
| because it is just a leaky abstraction. For full performance
| memory is only accessed linearly. Any random access structured
| programming language completely breaks down for working with
| GBs of data, in parallel and on a cluster of machines.
|
| This is why constantly re-engineering (for the purpose of
| engineering) is not the most useful method. I used to work in
| FoxPro with databases that were GBs of data. If today those GBs
| are difficult to handle, which they aren't, then there is a
| problem with how the stack was put together. GBs are trivial.
| viraptor wrote:
| > Noone will pay you for invoking existing solutions. They pay
| you to push the boundary.
|
| I wouldn't put it like that. 90% of DevOps jobs is to invoke
| the existing solutions quickly. Sure, there will be a little
| bit of pushing the boundaries here and there was needed. But
| largely I'd describe those jobs as putting the Lego blocks
| together every day - and there's lots of money in it.
| emptysongglass wrote:
| DevOps Engineer here, former Lead (moved from startup to Big
| Corp). That's literally my job day in day out. Put the Lego
| bricks together. Everybody around me tells me I'm doing a
| good job and I'm sitting here thinking, "but all I did was
| Google these things and insert some variables in some Helm
| charts?"
|
| The other part of my job is more difficult, which is
| encouraging teams to de-silo and automate wherever possible
| but so many of the employees are threatened by this idea at
| Big Corp that I'm encountering a LOT of resistance.
| mtberatwork wrote:
| I think your first point answers your second point. It's
| only human nature for people to worry about their jobs.
| quantumcd wrote:
| Same here. It's easy to forget the difference between a
| master LEGO builder and the average person just piecing
| together bricks though. Knowing what to build and when/how
| is a lot of the job too. I try and remind myself that we're
| also often there to help other people build better things,
| perhaps analogous to automatically sorting LEGO bricks so
| other people building can move faster with fewer
| compromises.
| mrweasel wrote:
| Quickly and cheaply. People who only do development tend to
| look at every problem as some code they need to write.
|
| We saved multiple customers a lot of time and money by just
| doing minor load balancer or web server configurations,
| rather than letting their developers doing a custom
| application to handle things like proxying or URL rewrites.
| Similarly we also frequently have to ask customers why they
| aren't offloading work to their databases and let it do
| sorting, selection, aggregation or search.
|
| Having just a small amount of knowledge about what your
| existing tools can actually do for you, and knowing how to
| "Lego" them together solve a large number of everyday
| problems.
| tluyben2 wrote:
| > People don't want solutions to yesterday's problems. These
| are considered trivial and already solved, such as invoking a
| shell command (which just hides a lot of complexity under the
| hood). Noone will pay you for invoking existing solutions.
|
| I am not exactly sure what you mean by this, but taken
| literally; most companies pay well for this and it's also the
| most common work programmers do all over the world. But maybe I
| misunderstand your meaning.
| oblio wrote:
| Heck, aren't Google devs complaining about "proto-to-proto"
| work ("protobuff to protobuff", not "prototype to
| prototype"), where they're basically just CRUDing together
| pre-existing services? Google pays a lot and has huge scale
| plus is considered quite innovative (I think they publish a
| lot of papers each year), yet most of their work is the
| typical enterprise middleware development job.
| goto11 wrote:
| > Noone will pay you for invoking existing solutions. They pay
| you to push the boundary.
|
| Most businesses pay developers to solve problems, not to push
| boundaries. They typically prefer developers using "boring"
| tried-and-true approaches.
|
| In my experience it is _developers_ who push for "boundary
| pushing" solutions because it is more exciting. And of course
| the vendors selling those solutions.
| hnarn wrote:
| If it was true that the only way to get "paid" is to push
| boundaries, 99% of everyone working in IT would be out of a
| job. Kurt Vonnegut once said that "the problem with the world
| is that everyone wants to build, and noone wants to do
| maintenance". I'm reminded of that quote daily working in the
| software industry -- the people "pushing boundaries" are a
| tiny minority and if they were the only ones working our
| modern world would come crashing down in an afternoon.
| anonymoushn wrote:
| You can hardly expect people to talk about all the
| maintenance they did in their performance review
| nthj wrote:
| Yes, put it in terms of churn mitigation, compliance,
| security, and data integrity and you can absolutely sell
| maintenance at your performance review
| Ekaros wrote:
| Why? Maintenance can easily be quantified. Fixed x number
| of issues, implement y number of feature request/changes.
| Limited downtime to z amount of time by doing following
| actions...
| IggleSniggle wrote:
| ...why in the world wouldn't you talk about the ways you
| made an application better, more resilient, or heck, just
| able to continue operating??
|
| No, I don't want my local government official to build a
| new damn road. I would clap and love them for fixing the
| one I drive on everyday! Even better if they do it in a
| way that's going to last for 100 years instead of 10, and
| that's the way you brag about maintenance work.
| dolmen wrote:
| Why?
|
| Work should be balanced between creating new code and
| maintaining existing code. And managers should value
| both.
| wccrawford wrote:
| Of course they _should_ , but it just doesn't seem to
| work out like that.
|
| I was basically pushed out of my previous job because I
| was the bug-fix guy and liked it.
|
| When I was the 'new stuff' guy, I got big raises every
| year. When I moved to being the bug-fix guy, and other
| people were the 'new stuff' people, suddenly I stopped
| getting good raises and barely met inflation. Even my
| coworkers were heaping praise on someone who I _knew_ was
| over-complicating things because he was doing new stuff.
|
| As I was leaving, a system that I had rewritten twice was
| up for another rewrite for more functionality. I would
| have _loved_ to do it, and I knew I could. They gave it
| to someone else. I heard, 6 months after I left, that
| they failed and gave up on the rewrite. I am absolutely
| certain I could have done it again.
|
| Getting a new job got me a 40% pay raise. It's not that I
| wasn't still worth a lot more every year. It's that they
| couldn't see it because it was all maintenance, and not
| shiny new stuff.
|
| I still prefer bugfixes, and I still rock at them. But I
| end up doing mostly new stuff and don't say anything
| because I know how that goes.
| bluesnowmonkey wrote:
| They should value both but just don't in my experience.
| Building a system can get you promoted. Performing
| maintenance to keep it running generally will not.
| Doesn't matter if the maintenance project demonstrates a
| higher level of proficiency than the original
| development. Doesn't matter that it's extending the life
| of a system providing known value, as opposed to building
| something with theoretical value.
| jokethrowaway wrote:
| You answered yourself.
|
| Maintenance is seen by business as a cost, innovation is
| seen as potentially bringing more money. The incentives
| are all wrong and that translate to engineers (and teams,
| and companies) chasing features and not fixing bugs.
|
| A rewrite is often the only way of getting enough
| resources to fix up things.
| anonymoushn wrote:
| Because the company's internal rubric for performance
| reviews awards 0 points for this kind of work.
| isodude wrote:
| It's what we do at least. More than 60% of the work is
| maintaining old code/systems.
| de_keyboard wrote:
| I think the problem is we keep using the same crappy tools
| because people are scared to be an early adopter.
|
| Meanwhile, there are things sitting on the shelf that solve
| these problems in a principled way _and_ make things simpler:
|
| - ML languages
|
| - Nix
|
| - Bazel
|
| - Erlang
|
| - Rust
|
| Some tools that are great and have large adoption:
|
| - Git
|
| - Terraform
| mavelikara wrote:
| > people are scared to be an early adopter.
|
| ML was first developed in 1973. Ocaml in 1996 and SML in
| 1997. Great tools which haven't been popular in 20-40 years
| probably have something beyond fear of early adoption
| inhibiting them.
| iFreilicht wrote:
| I think Nix will take a long while to increase adoption,
| because it is hard to learn. The language is new, the concept
| is new, the mental model is new. But you need to master all
| of those to use the tool effectively.
|
| Same goes for the other items in your list. Git had enough
| time and force behind it, and I believe the other tools will
| succeed as well. But it will take time.
| b3kart wrote:
| > ML languages
|
| What are those?
| nmca wrote:
| I suspect he means Ocaml, sml and bucklescript.
| Multicomp wrote:
| agreed. ML [1] is short for meta langauge. Because it is
| not very C-like syntax by default, it can feel exotic and
| alien to most of today's unix graybeards, 90s enterprise,
| aughts startup folks, and even the JS crowd.
|
| see also its 'newer incarnations' Scala, F#,
| Nemerle...(don't slay me)Rust(ish)
|
| [1]
| https://en.m.wikipedia.org/wiki/ML_(programming_language)
| shipp02 wrote:
| I would add E[1] language to that list.
|
| [1]http://erights.org/elang/index.html
| bpicolo wrote:
| For your typical scaling org, I think data layers are often
| the main issue. Moving from a single postgres/mysql primary
| to something that isn't that represents the biggest hurdle.
|
| Some companies are "lucky" and have either natural sharding
| keys for their primary business line, are an easily cacheable
| product, or can just scale reads on replicas. Others aren't,
| and that's where things get complicated.
| mxz3000 wrote:
| Tbh, That's why for our new projects we've completely
| ignored relational databases. They're a pain in the ass to
| manage and scale poorly.
|
| DynamoDB, on the other hand, trivially scales to thousands
| (and more!) of TPS and doesn't come with footguns. If it
| works, then it'll continue to work forever.
| de_keyboard wrote:
| How do you handle cases where strict synchronization is
| required?
|
| Banking app is a classic example.
| tpxl wrote:
| This is funny to me since modern relational databases can
| get thousands and more TPS in a single node. My dev
| machine reports 16k TPS on a table with 100M rows with
| 100 clients.
|
| > pgbench -c 100 -s 100 -T 20 -n -U postgres > number of
| transactions actually processed: 321524 > latency average
| = 6.253 ms > tps = 15991.775957
| josephg wrote:
| Yep. And 98% of software written today will never need to
| scale beyond 10k TPS at the database layer. Most software
| is small. And of the software that does need to go faster
| than that, most of the time you can get away with read
| replicas. Or there are obvious sharding keys.
|
| Even when thats not the case, it usually ends up being a
| minority of the tables and collections that need
| additional speed.
|
| If you don't believe me, look through the HN hiring
| thread sometime and notice how few product names you
| recognise.
|
| Most products will never need to scale like GMail or
| Facebook.
| cpursley wrote:
| Erlang/Elixir would like to have a word.
| rapsey wrote:
| It is very unfortunate so few are willing to give it a try
| Multicomp wrote:
| I watch Erlang / Elixir. But I really like strong static
| typing and type safety. So for now, I'm not learning
| anything beyond basic beam theory, lest I will have to
| unlearn it later.
|
| Gleam lang at least is definitely something I watch closely
| and may (may) be the BEAMs breakout moment.
| coldtea wrote:
| > _People don 't want solutions to yesterday's problems. These
| are considered trivial and already solved, such as invoking a
| shell command (which just hides a lot of complexity under the
| hood). Noone will pay you for invoking existing solutions. They
| pay you to push the boundary._
|
| You'd be surprised. Most software production re-builds and re-
| invokes existing solutions that just aren't systematized yet.
| It certainly doesn't push any boundaries...
| AlchemistCamp wrote:
| > By tooling I mean programming languages, frameworks,
| libraries and operating systems. All of whuch have been
| designed for a single machine operation with random access
| memory model.
|
| You should learn about Elixir (or anything on Erlang VM).
| eandre wrote:
| Agreed completely. Today there's such a large disconnect
| between how you _think_ about software (APIs, services,
| systems, data flows, replication, data storage, access
| patterns, and so on) and how you actually develop software,
| with plain text files on disk that don 't at all resemble our
| mental model.
|
| I've been working on a new type of framework that is about
| making how we develop software match this mental model. Where
| the higher-level primitives we think in terms of are first-
| class citizens at the language level.
|
| I think only then is it possible to reason about, and scale,
| software in a much more productive and effective way.
| qznc wrote:
| Building up the tooling is how we advanced over the last
| decades. It is a very slow process though because one tool has
| to become stable before tools on top of it can progress instead
| of adapting to changes.
|
| Large software projects do not have decades. Only few years.
|
| Distributed operating systems are nothing new though. Plan9 for
| example.
| eternalban wrote:
| > There is something peculiar about software that makes it
| different from other crafts.
|
| What distinguishes software from other _techne_ is a lack of
| _physics_. There is no 'solid ground of reality'. All other
| forms of making involve _discovering_ and then _applying_ the
| governing laws.
|
| Hardware architectures, operating system, and programming
| languages, to an extent, do furnish a _phenomenal context_ and
| where these characteristics are stable and well established (e.g.
| Memory Models), _science_ appears [1].
|
| But clearly, this ground of reality is indeed 'soft' and the
| practitioners usually re-invent not just the wheel, but the
| ground that it rolls on as well.
|
| The dilemma -- which nearly means facing a choice of a
| multiplicity of lemmas -- that confronts the designers of
| hardware, OS, and languages is precisely that tension between
| generality (and corresponding weak "world" semantics) and
| specificity (with robust but constrained semantics).
|
| [1]: https://ocw.mit.edu/courses/electrical-engineering-and-
| compu...
| flohofwoe wrote:
| My own highly simplified opinion why software development
| "doesn't scale": because software development is all research and
| zero production - with the notable exception of asset production
| in game development, which actually is quite easily scalable once
| you've figured out a workflow that works for a specific (type of)
| game. If you think about how to scale software development, don't
| think about an assembly line, but about a research institute.
| kalecserk wrote:
| I have been working with software development over the last 20
| years and agree with that point of view 100%. Software
| development is the act of learning/researching. The source code
| we create is not a "product". I really liked your metaphor of
| the research institute. Spot-on. Thanks for sharing
| konschubert wrote:
| Man, I feel like you took the words out of my mouth.
|
| 100% agree.
| namelosw wrote:
| The problem is we stop pursuing answers on this topic, thus stop
| making progress. It's basically like what Bret Victor has
| described in 'The Future of Programming'. [0]
|
| There were a lot of language zealots at the end of the last
| century, especially on evangelizing Object-Oriented Programming.
| Nowadays everybody can easily counter those arguments with 'No
| Silver Bullet' without further thinking, it's arrogance in
| disguise of humility. There are still a huge amount of accidental
| complexities to deal with in most tech stacks. Most businesses
| would die fast and leave nothing behind anyway, while the
| progression of the industry would accumulate and benefit the
| whole industry itself.
|
| Java looks slightly better for creating software at scale than C.
| C looks slightly better than FORTRAN. FORTRAN looks slightly
| better than machine code. Say there's a language that looks like
| Haskell but has tooling and ecosystems as good as Java, I believe
| it would also slightly better than Java.
|
| [0]: https://youtu.be/8pTEmbeENF4
| jiggawatts wrote:
| Conversely, writing software for a _small_ scale is absurdly
| inefficient.
|
| I consult for government departments that often have legally
| mandated requirements for making something available online. That
| could be a form submission process, some geospatial information
| presented on a map, or whatever.
|
| The problem is that when you might have 500-10K user transactions
| _annually_ , it becomes crazy expensive to write bespoke
| software, even with the most agile process and the lowest
| overhead tooling.
|
| Take cloud deployments, for instance. Sure, you can just "click
| through some wizard" pressing next-next-next-finish and be up and
| running, but the security team won't allow your infrequently-
| maintained web server on the Internet without a web application
| firewall. Setting one of those up is days of tweaking to avoid
| false positives.
|
| Need to send mail? Azure bans outbound port 25 connections, you
| have to use Sendgrid, or something like it. Time to read up on
| yet another unique and special API!
|
| Collecting fees and penalties or making payments to citizens?
| Woah there... there's a massive API surface you have to learn.
| Security on top of security is needed. Whitelisted IP addresses.
| Client certificates. That have to be rotated, _manually!_
|
| You'll forget some essential maintenance, of course, and then
| you'll have to set up triggers and alarms so you don't get burned
| the second time. Which entails mailing lists that _change
| dynamically_ because the team of contractors has a turnover rate
| faster than the typical certificate expiry time. Send too many
| alarms and all recipients will configure an Outlook rule to
| ignore them. Not enough alarms and you 'll miss issues. Just
| setting this up semi-reliably is an exercise in itself.
|
| Really basic stuff becomes difficult, when you realise that 99%
| of the alerting and monitoring features in Azure and AWS are
| designed for systems at _their scale_. It 's all about analysing
| beautifully smooth curves of graphs aggregating millions of
| points of data, where deviations are glaringly obvious. These
| tools are utterly useless when you get one real transaction per
| day, swamped by a thousand bots. The _load balancer health check_
| is 99.99% of the traffic for some of these sites!
|
| Then there's the human element:
|
| Have you tried justifying the time to set alarms for a system
| where a week-long outage might only affect a dozen customers?
|
| How about the budget to upgrade to a newer operating system for
| something that is not _technically_ broken -- merely hard to
| support now?
|
| Or have you tried doing any sort of maintenance on a system that
| was built by contractors hired for a fixed term, all of whom are
| now gone?
|
| Meanwhile departments are renamed every couple of years to suit
| the whims of the latest batch of politicians, so _everything_ has
| to rebranded. Even tiny little sites use by practically nobody.
|
| I've seen sites up for 10 years, where I estimated that they cost
| $2,000 per citizen that actually used the site! Madness.
| hacker_newz wrote:
| It is easier than ever to deploy software on a small scale. It
| sounds like your security team is failing to provide a suitable
| platform and you chose the wrong cloud provider for sending
| email.
| imwillofficial wrote:
| I mean, I think amazon does a good job. Google too, Microsoft
| gets some points too.
| pdamoc wrote:
| Alan Kay said at one point:
|
| > I have always done my work in the context of a group that is
| set up to maximize a wide spread of talents and abilities. Just
| as "science" is "a better scientist than a scientist", such a
| group is "a better programmer and systems designer than any
| individual". Really learning to program -- etc. -- is really
| learning about how to be part of a scientific/engineering/design
| team. As with a sport like basketball, there are lots of
| fundamentals that all need to do extremely well, and there are
| also "special abilities" that not every member will have, but
| which the whole team will have when it is functioning well as a
| team.
|
| I fear that very few people really spend the time needed to
| really learn the fundamentals and very few people learn a
| scientific mindset that would allow them to cooperate effectively
| with others without getting their ego in the way.
| [deleted]
| void_mint wrote:
| If you think of all of the decisions that are made between having
| no software/users/"scale", to being something like Netflix, you
| can imagine how many opportunities there are for things to become
| "terrible" (or really, for a system to be made a small percentage
| worse, which obviously accumulates over time). I would challenge
| the assertion that most sufficiently scaled systems are
| "terrible", more that a person did not like them for one reason
| or another. If a system accommodates the demands of its users and
| maintainers, I don't really think you can label it terrible.
| Scaled software isn't art, it is either good enough or it isn't.
|
| I would argue that it is becoming easier for "terrible" software
| to scale well. I would say that's a much bigger win.
| moonbug wrote:
| +1 for Design Thinking
| johbjo wrote:
| One thing that might make software fundamentally different from
| other stuff, is that before being able to make a meaningful
| contribution a person needs to invest X hours just to understand.
|
| In most other creative fields, this can be hand-waved away. No
| other engineering field has anywhere near the complexity.
|
| Therefore, all important people on a project are stakeholders of
| some sort, and anyone else is not able to make real improvements.
|
| The project will move in whatever direction satisfies the
| incentives of the important stakeholders.
| christophilus wrote:
| > No other engineering field has anywhere near the complexity.
|
| I'm a software engineer, so am not qualified to say this, but I
| highly doubt this is true.
| johbjo wrote:
| I remembered this quote from Grady Booch, at 13:00 in this
| video:
|
| https://youtu.be/adiVOdztQ34
|
| "These software artifacts we produce, are perhaps some of the
| most complex humans have ever created."
| 29athrowaway wrote:
| The problem is that there are perverse incentives for people to
| enrich themselves as quickly as possible and make problems like
| performance and scalability a liability for someone else.
|
| I can create a startup, get a massive amount of funding, sell the
| company and make millions of dollars... all without ever
| seriously caring about performance and scalability.
| dalbasal wrote:
| I enjoyed this.. fun sort of intentional-meandering style.
|
| I propose that scaling (as defined here) is a general problem,
| not just a software problem. I realize this is a hard case to
| make. Technical debt, refactoring hells and such don't plague
| film sets or factories like they do code shops. But.... Consider
| this:
|
| - Consider factories. A factory is just a huge pile of physical
| capital organised to efficiently do a specific, defined task.
| Everything is optimised to reduce marginal costs. Capital
| efficiency is strictly enforced by capital money markets.
| Marginal cost efficiency is strictly enforced by real markets.
|
| ... Either of these two constraints (marginal costs too high or
| not enough capital) usually come into play before engineers have
| had a chance to redesign, refactor, repurposed and abstract a
| factory to the point where these are a problem.
|
| - Everything that is in a movie is in a shot. Movies have a
| finite number of shots and a director can direct each one. A film
| may have a whole team replacing Toby Philpott with Jabba the
| Hutt, an army of costume designers, set designers, actor
| psychologists, etc. But, everything that gets into the movie gets
| there in a shot, and the director can work shot-by-shot and wield
| god-like control of lots of labour that way.
|
| Software is unique. Software is free from most physical
| limitations. There are no material costs. There are no capital
| costs. The only economic resource that a software enterprise
| wields are the engineers/engineering itself.
|
| When a Ford Company scales, it raise money. It uses that money to
| to build factories, buy materials, etc. These are all
| scarce/finite resources needed to start/continue making
| additional cars.
|
| Google, MSFT, FB, post-AWS amazon... When they scale up, they
| just scale up. They hire more engineers. They produce more
| software. The only "resource" being scaled up is the people
| making the software.... _Something_ has to be the limiting
| factor.
|
| In any case, the "scaling creative work" problem _does_ exist in
| factories too. The difficulty 50s era auto manufacturers faced
| competing with Toyota are sort of evidence. They struggled with
| "technical debt" in the form of car models, factory design,
| company culture and such that couldn't adapt flexibly.
|
| A lot of tesla's wins (besides marketing, fundraising and
| software) have come from their recent blank state start. Auto
| manufacturing today is highly caught up in the "Toyota Way" of
| doing things. It's been that way for decades and parts of it are
| explicit assumptions of international trade deals. It's very
| compartmentalized. Flexible within compartments. Rigid without
| compartments, whether they are outside the department or outside
| the company. It's a lot like the city metaphor this article
| mentions.
|
| The problem is... sometimes you need to design the factory and
| the car simultaneously. At times that has screwed Tesla. The
| quality control benefits of the Toyota Way are not to be taken
| lightly. At other times though, it works. The factory designer
| has been banned from designing the car and the inverse for 40
| years. I think their version of technical debt is the reason.
| williesleg wrote:
| I find the people of india in particular write shit code. Then
| they move on before they're discovered.
| Gravyness wrote:
| The world is terrible at creating software because the developers
| are the last to be paid.
| marsdepinski wrote:
| I'm not.
| turbinerneiter wrote:
| Very often, even tough I only ever write very short programs (100
| to 1000 lines) in Python, C++ (and learning Rust), I look at my
| finished program and feel like there is a million things to
| improve and a billion other ways to do it. Each variable could
| have a better name, each abstraction could be abstracted more,
| the whole thing could be rewritten in a different style or
| language.
|
| Whereas if I design something in CAD, I send it to the printer,
| try it, improve it once, print again and never ever again think
| of it as long as it does it's job.
| macando wrote:
| When you start drawing diagrams of non-trivial software, for
| instance Blender (an open source tool for building 3D models
| and animations), you start to understand how complex and
| complicated software is.
|
| No other machine built by humans is that complex.
| cosmodisk wrote:
| Except this:
| https://en.m.wikipedia.org/wiki/Large_Hadron_Collider
| Ebisu77 wrote:
| Ah yes, the famous Large Hadron Collider, the most complex
| piece of human engineering that doesn't have any software.
| cosmodisk wrote:
| I think most of it is just software
| qznc wrote:
| I'd say manufacturing plants are easily as complex as
| Blender.
| macando wrote:
| How about car software?
|
| I used Blender as an example because it's open source.
| qznc wrote:
| I'm working on driver assistance stuff. In terms of raw
| functionality the software is not that complex. The
| complexity comes from other aspects: Safety-critical,
| hardware-software-codesign, commercial, resource
| constrained. Then, of course, organizational disfunction
| creates accidental complexity as it does everywhere. All
| of that together, and a software developer barely
| achieves a thousand lines of code per year.
| spongeb00b wrote:
| Regarding comments about manufacturing plants and the LCH,
| sure both of these examples aren't software directly (but
| they also wouldn't be possible without software), but they
| are also always under constant fixes, feature upgrades and
| optimisations.
| rusk wrote:
| I think under market conditions you would be continuously
| tweaking the model, and manufacturing processes, based on
| feedback eg wrt robustness, comfort etc and also taking into
| account changing usage parameters/expectations, changes in
| material supply costs, regulatory requirements etc
|
| Of course these concerns matter more if you're making a lot of
| these "widgets" but even in the case of a one off you're going
| to have servicing costs and the occasional
| redesign/replacement/upgrade ...
|
| It's not that far out of line with software - widely used
| products go through tight highly iterative development cycles
| whereas one off solutions tend to be just "good enough" with
| bug fixing and the occasional feature request.
| [deleted]
| konschubert wrote:
| Software development is research work.
|
| If you take processes from people who build machines and apply
| them to software, it will fail. Software means _inventing_
| machines.
| coldtea wrote:
| > _Everyone is still terrible at creating software at scale_
|
| Terrible is a relative term. Terrible compared to what?
|
| Who said/showed (much less proved) that there's a better way and
| we just don't follow it to achieve it some optimum?
|
| There's also a semantic confusion here. Compared to e.g. the car
| industry we're infinitely better at "creating software at scale".
| We can create a billion copies of a software and distribute it
| our the world, with marginal cost close to zero.
|
| But the author doesn't mean "creating software at scale" like
| when they say "car production at scale". They mean production of
| "large software".
|
| Well, let's see the car industry make a car with the scope, flux
| requirements, shifting environments, etc that large scale
| software has...
| adreamingsoul wrote:
| IMO, technical leadership isn't allowed to build at scale or the
| leadership doesn't exist.
|
| The race to the bottom incentivises profit over principle. Boards
| don't care if something is built to scale if the competitor gets
| to market with a cheaper solution. Scaling is a day two problem.
|
| I'm bearish on my future employment as someone who questions the
| motivations for profit driven development.
|
| I naively thought the market would reward the best product,
| instead I see the cheapest product being rewarded. What happened
| to innovation and doing what's best for society?
| yoz-y wrote:
| You answered the question yourself. Cost is by far the most
| important factor, so if you focus on quality you will not be
| able to sell your product.
| qznc wrote:
| As long as our resources are limited we always need to
| consider the tradeoffs. This is not specific to software.
| benrbray wrote:
| I think it's fairly common for even established companies to
| prioritize short-term costs over long-term costs, even when
| the long-term costs of NOT investing in good tooling are
| astronomical. So absolute cost isn't really the deciding
| factor.
| noir_lord wrote:
| > reward the best product, instead I see the cheapest product
| being rewarded.
|
| That's because you are looking at best from one angle and the
| market from another.
|
| To the "market" the best may well be the _cheapest_ thing that
| mostly does it 's job/most of the time.
|
| When I was young I used to think users cared about bugs and
| they do but they only care when the bugs are sufficient in
| number and degree to cross a threshold where they outweigh the
| utility of the software, That sort of 1 in 100 times it loses
| an hours work, well if the 99 times it saves me an hours work,
| _shrug_.
|
| What we (as techies and I'm generalising wildly here) want to
| build is heirloom quality carpentry, what we get to actually
| build is franken-furniture from ikea packs that we hope has
| some some instructions and won't fall apart next week.
| ipaddr wrote:
| One bug in a shopping cart preventing checkout ruins the
| entire product. Bugs come in a different shapes/sizes.
| noir_lord wrote:
| > bugs are sufficient in number and degree
|
| In this case that would be a high degree bug.
| fctorial wrote:
| This isn't a software specific problem. 90% of work in every
| business is inefficient.
| axelfreeman wrote:
| A lot of guessing and hunching and try to find one answer to a
| couple of problems where the question isn't clear. So I don't
| know what point this article wants to make.
| Dfgh210410 wrote:
| We have actually degenerated back to CLI. "You don't need UI"
| lol.
| systematical wrote:
| I really hate titles like this.
| [deleted]
| [deleted]
| macando wrote:
| Yesterday I watched a video about how Netflix scaled its API with
| GraphQL Federation.[0] The video contains some neat
| visualizations that helps you see how complex data accees
| problems at scale can get. And this is just the services level
| they talk about.
|
| No mentioning of the underlying infra with all its complexities
| needed to acheive the goals of flexibility, reliability, speed
| and cost cutting.
|
| You don't have to be of Netflix size - when you start getting
| tens of thousands of users, complexity hits you real fast.
|
| [0] https://youtu.be/QrEOvHdH2Cg
| markb139 wrote:
| There is at least one massively scaled system that relies heavily
| on software, that doesn't often get a mention because it just
| works. When was the last time you picked up a land-line phone and
| it didn't work because of a fault on the system?
| Applejinx wrote:
| I thought somebody might mention it, but since nobody did: a best
| boy grip is the chief assistant to the person in charge of
| building rigs and frameworks for things like cameras and lights.
|
| Best boy basically means 'chief assistant', second in command
| kind of thing.
|
| Grips build the structures which cameras and lights are hung on.
| They don't touch the electrics, gaffers and electricians do that:
| grips make the towers, stands, tracks: the physical stuff on
| which you put things.
| lostdog wrote:
| All these specialized jobs--best not, gaffer, makeup--really
| highlight what's different between software and movie
| production. On a movie set, nearly everyone is doing exactly
| the same thing they did on the last movie set. In many software
| jobs you're never repeating your last project.
| 0x4d464d48 wrote:
| "Ironically, in my experience, just bouncing off ideas of one
| another is not the way this works well. What you can end up with
| is people throwing their ideas into the ring just for others to
| find faults in it. In the worst case, this can turn into a
| competition who knows most and who is smartest."
|
| This right here is exactly why I learned to say nothing in
| brainstorming sessions and try to dodge then outright if I can.
|
| My experience is these are typically for someone higher up in the
| pecking order needing their own ideas validated.
|
| My usual work around is if something needs to be done, quietly
| work on a POC on your own and solicit feedback or even bring in
| stakeholders.
| Ericson2314 wrote:
| Computing _is_ a lot like American cities, in the that the
| infrastructure (yak shaves) that would make everything easier are
| almost invariably under-invested.
|
| One of my favorite examples would
| https://en.wikipedia.org/wiki/Capsicum_(Unix). It's incremental
| which is good. But people wrote the kernel part and declared
| mission accomplished like Bush in 2003, ludicrous! One has to do
| that and _then_ overhaul the userland _and_ either get those
| patches upstreamed. And these days "userland" isn't a collection
| of init, shitty scripts and Gtk/Qt, but a bunch of libraries,
| especially programming language standard libraries.
|
| This would dramatically change the security _and_ ergonomics
| landscape, because so much global state programmers
|
| "Containers" was always the wrong metaphor because software is
| about _composition_ , but containers are inert and only their
| mass and volume composes (very crude). Better to think about
| plumbing or rail.
|
| Another perhaps more opinionated example is getting everyone to
| use Nix (or fine, something like it.) Whether with container
| style virtual global state, or nicer capsicum, we need to make it
| trivial to install _and develop_ the entire commons. All "builds
| on my machine but you can use it" just leads to a lack of
| integration so no one can help the original author smooth over
| the gaps. It will also allow machines to be demystified, allowing
| people to toy with all the software on the system, which will
| help reduce the programmer alienation which allows so much
| accidental complexity to occur in the first place.
|
| But yeah, almost nobody is doing these things at the scale they
| deserve, and even the megacorps drown in their own technical debt
| like sunbelt cities that are just metastasized suburbs. Eveyrone
| is in the "well I'll eat shit and shut up as long as my
| competitor is too" mind site. It's disgusting.
___________________________________________________________________
(page generated 2021-04-10 23:01 UTC)