[HN Gopher] What is negative engineering?
___________________________________________________________________
What is negative engineering?
Author : dom_fr
Score : 57 points
Date : 2022-12-11 10:53 UTC (1 days ago)
(HTM) web link (future.com)
(TXT) w3m dump (future.com)
| extasia wrote:
| Isn't what the article describes the concept of "defensive
| programming"?
| dejj wrote:
| "Defensive programming" seems more apt to presentation. I also
| consider it part of mitigation in FMEA.
|
| https://en.wikipedia.org/wiki/Failure_mode_and_effects_analy...
| jasode wrote:
| Fyi ... other related terminology ... _" chaos monkeys"_
| engineering from Netflix, designing for _" fault tolerance"_ ,
| etc. See list from :
| https://en.wikipedia.org/wiki/Chaos_engineering#See_also
|
| Another list is _" cross-cutting concerns"_ which can include
| error-detection & correction, process monitoring, etc:
| https://en.wikipedia.org/wiki/Cross-cutting_concern#Examples
|
| And _" site reliability engineering"_ popularized by Google:
| https://en.wikipedia.org/wiki/Site_reliability_engineering
| juancn wrote:
| That's just sound engineering.
|
| Besides the usual quality considerations (correct, available,
| performant, etc.), well engineered software systems should be:
|
| - observable: you need to be able to tell the state of the system
| to a sufficient degree (metrics, status reports, etc.)
|
| - operable: it should be possible to influence the behavior of
| the system even in case of failure (feature flags, tools for
| retrying or pausing jobs, throttle processing speed, etc.)
|
| Failure modes should be considered as early as you can, and
| provide sufficient tooling to deal with them.
|
| Most of the effort on large scale or mission critical systems is
| dealing with failure modes, rather than the happy path and making
| sure it's extremely difficult to make it fail catastrophically
| without recourse.
|
| Failure should be considered business as usual and dealt with
| accordingly, hopefully automatically but you need to leave
| tooling in place for those situations you didn't consider.
| MPSimmons wrote:
| >Negative engineering is the time-consuming and sometimes
| frustrating work that engineers undertake to ensure the success
| of their primary objectives
|
| You don't need another name for this. This is part of the
| engineering process.
| angarg12 wrote:
| Glad to see I'm not the only one baffled by this article. If
| you've ever developed and maintained software in production,
| you had to deal with failures. Is just part of the regular
| software development lifecycle.
| croo wrote:
| I usuall call this "meta-programming" or "meta-work". The work
| you need to do in order to make way for actual work.
| P5fRxh5kUvp2th wrote:
| How can you prove your blogging chops if you don't name stuff?
|
| You see this a lot in academia as well, they absolutely love to
| name things, and think the teaching and learning of those names
| is important in some manner.
| drewcoo wrote:
| > What is Negative Engineering?
|
| "Negative engineering" is a term coined and championed by
| Jeremiah Lowin, apparently. Most search engine hits on the term
| are his or kindly refer to him.
|
| He uses it to mean coding for the negative test cases.
| bestcoder69 wrote:
| For some context in case people don't know, this is Future.com,
| the a16z PR site that they're shutting down soon for lack of
| interest.
| cratermoon wrote:
| For the naysayers, yes, this should be part of the engineering
| process. But there are thousands of teams of programmers not
| doing this. Arguably, they aren't doing engineering, they are
| coding, or something else. In which case you can correctly say
| "this _is_ be part of the engineering process ", and point out
| that teams (some of which I've consulted with) who ignore this
| aren't doing engineering.
|
| And yet, the number of systems embedded in our daily lives that
| are built without "the engineering process" is stunning. They are
| not typically the essential systems, but just everyday things we
| depend on and for which the loss would be much more than
| inconvenient, but short of catastrophic.
|
| The author's points may be a rehash of old news, or even trivial,
| for a substantial portion of HN readers, but for many programmers
| and their managers, this could make them part of the "lucky
| 10,000"[1].
|
| 1 https://xkcd.com/1053/
| [deleted]
| kshay wrote:
| > Imagine how frustrated you'd be if every now and then, an
| application simply refused to add items to your cart, navigate to
| a certain page, or charge your credit card
|
| Sort of interesting that this is framed as a wouldn't-it-be-
| awful-if hypothetical; I feel like on the contrary, and for some
| value of "every now and then," this is most people's daily
| experience of interacting with most commerce websites.
|
| > The truth is, these minor refusals happen surprisingly often,
| but users never know because of systems dedicated to intercepting
| those errors and running the erroneous code again.
|
| ...Or users do know, and just try again? Fault tolerance has been
| outsourced to the customer.
| airbreather wrote:
| The dangers of implicit state
|
| https://medium.com/@DavidKPiano/the-facetime-bug-and-the-dan...
| aatd86 wrote:
| And yet many programmers do not like reasoning about errors
| because it "hInDErS tHe hApPy pAtH"... :)
|
| That's funny to me.
|
| Actually, I don't even know why call it _negative_ engineering.
| That should be engineering, period.
| zmgsabst wrote:
| A quip I picked up from a Boeing engineer:
|
| "Engineering is designing failure."
|
| Obviously that has darker tones in aerospace, but I think it
| applies to even CRUD apps:
|
| - my app will choke and fall over if TPS exceeds 1k/sec
|
| - my app will drop data when we exceed 2TiB in the DB total
|
| - etc.
|
| You're "engineering" when you design and control those modes in
| your system.
| jonathaneunice wrote:
| "Negative engineering" as described here is also especially
| challenging to test. Requires extraordinary backflips to
| construct reasonable approximations to the edge conditions,
| corner cases, external factors, and unreasonable contingencies
| against which you're trying to defend.
|
| The same "Happy Path is just 20% of code" imbalance extends
| through test rig, tribal knowledge, and effort required. In some
| cases, 20% seems fantastically generous. IME preventing failures
| that only occur at heavy load or during other errors may be 95%+
| of your effort and represent 99%+ of your "value add."
| [deleted]
| sublinear wrote:
| > They _insure_ the outcomes that positive engineering tools are
| used to achieve.
|
| Ugh such a tacky abuse of words. Context around that sentence
| implies it should be "ensure". No, this is just yet another ad
| promoting gimmicky work for consultants to sell.
|
| I understand HN is definitely the place to post this stuff, but I
| was expecting something more interesting.
| OJFord wrote:
| I think it's emphasised exactly to avoid it being misread as
| 'ensure', I don't read it as an intended pun. Doesn't really
| make sense to 'ensure the outcomes', you 'ensure a good
| outcome' or something.
| PeterisP wrote:
| The context around that sentence does quite explicitly intends
| to insure the outcomes because you can't really ensure it. I
| mean, the paragraph literally starts with "Insurance as code".
| helmholtz wrote:
| You insure _against_ something. You ensure something. GP is
| right.
| giantg2 wrote:
| You also drink ensure, if you're old enough.
| strken wrote:
| If you're talking about the grammar, then the phrase "My
| car is insured" is fine. You aren't required to specify
| what it's insured against; that can be left implicit.
|
| If you're talking about the language, insurance is wrong:
| retries _do_ ensure positive outcomes. However, assurance
| is wrong as well: unlike retries, observability tools don
| 't ensure positive outcomes, they just mitigate the
| negative ones.
|
| It's just bad writing, using a tortured metaphor to explain
| a definition that doesn't make any sense.
| giantg2 wrote:
| I'm a negative engineer.
| Oxidation wrote:
| See, treating this as an add-on is why software has a reputation
| as "not real engineering".
|
| Engineering is all about turning science into _products_. Anyone
| with a chemical lab and enough post-grad grunt can _make_ a
| compound. That 's just the start. Getting it to customers, at
| scale, with a saleable price and not selling poison,
| contamination, or thin air and not turning your plant into a
| crater or superfund site _is_ the task. It just happens to
| involve a bit of chemistry along the way.
|
| You can't just complain from the smoking ruins of your startup
| "but the happy path we tweeted about when we did it in a flask
| worked fine, how could we know it would be hard to make 50 tons
| of FOOF a day?"
|
| It's not always glamorous, it's not always fun, it's not even
| always innovative, it's almost certainly never easy, but it is
| what you're actually selling, and it's why _you 're_ the one
| selling it and not someone else.
|
| Now, can you cut corners and make it work, say, 99% of the time
| and that's good enough to ship it and sit back[1]? Sure you can.
| But if someone comes along at 99.9% and the same price, or even a
| higher price and your customers will pay for that extra 9, you're
| in trouble.
|
| [1]: or ship 99% code thinking it's 100% because you didn't check
| shanebellone wrote:
| I found this example particularly engaging.
|
| I would not be comfortable shipping code that has undefined
| behavior, outcome, or consequence. It takes a small amount of
| effort to build functions/methods with defined but encompassing
| outputs.
|
| Although complexity probably scales parabolically in certain
| use cases.
| m_mueller wrote:
| > when we did it in a flask worked fine
|
| kinda funny how Flask has a toy server that's probably behind a
| lot of startup MVPs....
| l0b0 wrote:
| Excellent analogy, and one of the reasons just putting a bunch
| of smart people together isn't good business model. A large
| fraction of them might be capable of building algorithmically
| outstanding software, but that skill set is completely
| orthogonal to building a _product,_ testing it, integrating it
| with third party software and /or hardware, maintaining it, and
| ultimately migrating away from it to the next generation of
| software.
| runnerup wrote:
| > Anyone with a chemical lab...not turning your plant into a
| crater or superfund site...how could we know it would be hard
| to make 50 tons of FOOF a day?
|
| For additional context to the mental imagery, I thought it
| would be appropriate to link FOOF:
| https://en.wikipedia.org/wiki/Dioxygen_difluoride
| wantoncl wrote:
| More imagery:
|
| https://www.science.org/content/blog-post/things-i-won-t-
| wor...
| cratermoon wrote:
| "Satan's Kimchi" is the name of my new KPop/Black Metal
| crossover band
| stdbrouw wrote:
| I don't understand this kind of inferiority complex. In
| software engineering we can often afford to wing it in a way
| that other kinds of engineering can't because they are strongly
| locked into what they've built already and because human life
| is at play. You can't really redo the foundation of a pre-
| existing building without a truly heroic amount of work, and
| you can't reboot a person if their pacemaker fails. When
| presented with those same conditions, software engineering
| tends to be just as rigorous as any other kind of engineering,
| e.g. NASA, medical software, and so on. I'd argue _that_ is
| what defines engineering: you optimize for the things that
| matter and don 't get carried away trying to pursue the most
| elegant or robust system at all costs - that would be science
| or art. Your competitor who is not cutting corners won't beat
| you with their 99.9% product, they will just never get off the
| ground. Meanwhile, you're at 99.5% and climbing.
| [deleted]
| cratermoon wrote:
| > In software engineering we can often afford to wing it in a
| way that other kinds of engineering can't
|
| Ariane flight V88 haunts your nightmares.
| Swizec wrote:
| Galloping Gertie haunts civil engineer's nightmares.
|
| Hyatt Regency walkways too.
|
| Apollo 1.
|
| Plenty of examples, actually. All engineering fields have
| their share of catastrophic failures. For better or worse,
| that's how we learn.
| MrJohz wrote:
| Different problems also have different scales. The road
| that they've repaved outside my house has had far less
| scrutiny than the new bridge they've built down the road.
| And NASA have very different coding guidelines to
| Shopify.
| jrockway wrote:
| After reading https://www.hillelwayne.com/post/are-we-really-
| engineers/ I am convinced that Software Engineering is the
| same as any other engineering field. Sure, it looks like like
| and death is at play in the other fields and all we do is
| debate what color fonts should be on a web page, but their
| projects seem a lot like that too. In every engineering
| field, projects usually go well enough to get another
| project, which is the best you can really hope for in the
| real world.
|
| Our whole "I can't do work today, I have to apply these
| security fixes 83 levels deep" also sounds a lot like:
| https://practical.engineering/blog/2022/8/2/you-spend-
| more-o...
|
| All in all, yeah, I don't understand the inferiority complex.
| Engineering always looks amazing to people that aren't in the
| thick of it, but we hide a lot of the flaws, and fix the
| other ones through iteration.
| toss1 wrote:
| >> In software engineering we can often afford to wing it in
| a way that other kinds of engineering can't ...
|
| Just because you _can_ doesn 't mean that you _should_. Of
| course, with software that is truly noncritical, handling
| failure modes can be optimized away in favor of shipping-the-
| new-version sooner. That doesn 't mean that it is a good
| thing.
|
| The fact that this is so prevalent in the software industry
| is why the general public sees most software and indeed
| technology ("software is eating the world...") as a plague --
| time and time again, NO effort is made to ensure that things
| fail-safe, or even that they fail-gracefully.
|
| So the user experience is that tech is good enough that we
| sort of have to use it (it's better than paper), but so bad
| that it is endlessly frustrating, whether it's endless
| glitches in client or web apps, data that gets mysteriously
| lost or corrupted, Bluetooth still screwing up connections,
| UI 'designs' that hide key functions behind a 'clean'
| interface that is nonetheless inscrutable for the occasional
| user... the entire industry has created a world no one wants
| to live in, one little decision at a time.
|
| (And that doesn't even begin to account for software that
| kills people by bad decisions or oversights.)
|
| The fact that an article such as this one is even necessary
| -- that it is NOT simply assumed that an absolutely key part
| of the design process of every piece of software is to
| consider all failure modes and how to handle them in a way
| that is safe and graceful -- casts a real shadow over the
| software industry, and shows why it is still separate from
| real engineering (despite software often being more complex).
| [deleted]
| [deleted]
| rapsey wrote:
| > See, treating this as an add-on is why software has a
| reputation as "not real engineering".
|
| Software is way more of a craft than engineering.
| hardware2win wrote:
| >The creative application of scientific principles to design
| or develop structures, machines, apparatus, or manufacturing
| processes, or works utilizing them singly or in combination;
| or to construct or operate the same with full cognizance of
| their design; or to forecast their behavior under specific
| operating conditions; all as respects an intended function,
| economics of operation and safety to life and property
|
| Fits software eng. just fine
| rapsey wrote:
| > The creative application of scientific principles to
| design or develop
|
| Not even close. But we sure love to claim the principles
| were scientific. A programmer becomes good at his craft
| with time and experience. Not by studying in school and
| learning the principles there. A classic difference between
| a craft and engineering.
|
| How many large construction projects fail due to incorrect
| engineering compared to large software projects?
|
| > or to construct or operate the same with full cognizance
| of their design; or to forecast their behavior under
| specific operating conditions;
|
| Almost never.
| elvis10ten wrote:
| > Almost never
|
| Isn't this the key point? That *most* "software
| engineering" is not engineering. I can build a bridge and
| not be an engineer; similarly I can build software and
| not be an engineer. The properties of the bridge/software
| is what makes the difference isn't it?
|
| > How many large construction projects fail due to
| incorrect engineering compared to large software
| projects?
|
| Yes, poor engineering is rampant. But I will argue that
| malleability also plays a large role in the delta. In
| traditional engineering you build a car (and you are
| "done"). In software you build a car and get asked to
| make it an aeroplane without going back to the drawing
| board.
| rapsey wrote:
| Not comparable at all. You can pay to have a bridge
| constructed and the project will be defined by an
| architect, built and inspected following a rigorous and
| standardized set of steps.
|
| You can also pay to have software built. The specs will
| not be even remotely as precisely defined and the
| building of the product will not even remotely follow a
| standardized set of steps. It will come down to a team
| and individual programmers implementing features as they
| see fit.
|
| Software that was running space shuttles was famously
| engineered. The costs in terms of time and money was
| astronomical.
|
| > Yes, poor engineering is rampant. But I will argue that
| malleability also plays a large role in the delta. In
| traditional engineering you build a car (and you are
| "done"). In software you build a car and get asked to
| make it an aeroplane without going back to the drawing
| board.
|
| Which is why it is not engineering. It is a craft much
| closer to writing a novel than it is to engineering as it
| looks in other disciplines.
| elvis10ten wrote:
| > Which is why it is not engineering.
|
| I think using that as a basis for this conclusion is
| wrong. Software changes a lot because it's possible and
| "CHEAP" to do so compared to engineering in the meat
| world. This is a business pressure/decision. If companies
| can easily and cheaply modify bridges arbitrarily, I
| believe they will; and this wouldn't have any bearing on
| whether it's engineering.
|
| > Software that was running space shuttles was famously
| engineered.
|
| This is exactly my point: Most swans are white, doesn't
| mean all swans are white.
| jcranmer wrote:
| Construction is frequently slapdash. A friend's wife, who
| works as an engineer, commented that it often seemed like
| the construction workers were trying to screw everything
| up on purpose. As-built is frequently different from the
| original plans, and that's before you take into account
| all of the change requests that happen in construction,
| such as the change that caused the Kansas City Hyatt
| disaster (oops, a single change request doubled the load
| on a critical beam, causing it to exceed its design
| strength and fail in the middle of a major party and
| killing a hundred people).
| rapsey wrote:
| Construction has a standardized set of steps, which when
| followed will reliably produce a desired outcome.
| Programming has no such thing because it it can not, it
| is a fundamentally different discipline that is unlike
| any other that we call engineering. Negligence is not
| relevant.
| drpyser22 wrote:
| Programming certainly can follow standardized design and
| implementation methodologies. I don't think there is
| anything fundamental about software engineering that
| prevents that. It doesn't usually happen because there's
| not enough agreement on all levels of what is the best
| methodology. Software engineering is young and still sees
| significant evolution. Principles of software design and
| engineering are largely agreed upon and applied, but not
| in a single, concrete and consistent standard because
| experts still disagree too much on the specifics of
| applying those principles, and on other controversial
| principles.
|
| And of course there is often no regulations that would
| require and guarantee the application of such a standard,
| and no standardized QA by customers.
| jcranmer wrote:
| Can you give an example of what you think one of these
| "standardized set of steps" in construction is?
|
| My suspicion is that, if you start trying to get
| specific, you can actually find standardized steps at the
| same level of abstraction in programming. But since
| construction superficially looks like "stamp out multiple
| copies of the same thing" whereas programming is
| superficially "everything is its own special sauce,"
| people try comparing them at different levels of
| abstraction.
| hardware2win wrote:
| >Not even close. But we sure love to claim the principles
| were scientific.
|
| Software engineer applies computer science theory in real
| world where he she is constrained by real world factors
|
| If computer science is not enough "scientific" the idk
|
| >A programmer becomes good at his craft with time and
| experience. Not by studying in school and learning the
| principles there. A classic difference between a craft
| and engineering.
|
| All of them are required to become good software
| engineer.
|
| Solid theory, proficiency in hands and exp.
| Dudeman112 wrote:
| >Software engineer applies computer science theory in
| real world where he she is constrained by real world
| factors
|
| A good 90% of people with that title are flying blind as
| if they lived the wild west
|
| A good 90% of the rest almost never do that (regardless
| if because they don't know how, or the company won't let
| them)
|
| The rest will make the news of they fail, so they put an
| amount of effort and time which reflect that
|
| We skipped the "mechanic" equivalent in the software
| industry and call anyone that can pass fizzbuzz, and who
| does work like fizzbuzz, an engineer
|
| Software engineers do a lot of tinkering, and almost no
| engineering. Which is fine by me, actual engineering is a
| PITA and tinkering is where the fun is at
| hardware2win wrote:
| >We skipped the "mechanic" equivalent in the software
| industry and call anyone that can pass fizzbuzz, and who
| does work like fizzbuzz, an engineer
|
| If I have engineer in job_title and engineering degree in
| informatics, then I m an engineer by your
| "standards/definitions"?
| mdip wrote:
| As I was reading this article, that itch in the back of my head
| kept being triggered. I didn't encounter anything obvious, but I
| felt like something was missing. Then I remembered a project I
| inherited at a past job.
|
| tl;dr Good idea, but it can be easy to end up with _very_ complex
| code flow if best practices /language-specific patterns aren't
| followed.
|
| I'm not sure if my experience is an example of Negative
| Engineering success or failure (a bit of both, perhaps). I had
| inherited a tool that monitored a conferencing environment
| looking for specific conferences that also were not started and
| had non-moderators waiting to be admitted[0]. It was a code base
| written by someone who was not familiar with the language it was
| written in, resulting in code that was very non-ideal.
|
| I had joked that the fact that the system worked was "an
| accident". It was a perfect example of "failing it's way to
| success". Every time a targeted conference was discovered, an
| exception was thrown. When the exception was thrown, the catch
| block would retry the request slightly differently. It would then
| add an invisible (which started the conference), monitor for the
| conference state to indicate it was started and remove the
| invisible user from the conference. The add/remove had to be done
| because the "invisible users" could not join two conferences, so
| we had a pool that would grow/shrink as needed. Nearly every step
| of the way, of course, there were many circumstances that were
| appropriate to retry, to catch and recover and (ultimately) to
| crash[1].
|
| Now, my approach to this kind of a problem is "if you can check
| that a request will fail and fix it before making it, do that",
| so I have the "invisible user pool" ensure it adds users before
| the pool is empty (and if it is empty, creates/returns a fresh
| user instead of throwing). Nearly every command that could throw
| had an invariant that could be confirmed before the call yet the
| developer opted to let things crash and handle the problems as
| they surfaced.
|
| You can imagine the kind of Rube Goldberg device each part of
| this solution was. Part of that was the language and the
| developer's inexperience with it. Abusing Try/Catch, as this
| developer did, made the code feel like it was just a bunch of
| GOTO statements. The _fixed_ solution had nearly as many Try
| /Catch blocks -- it's a distributed network application, after
| all, nearly every call had something external that could go awry.
| The difference was that there was an obvious "happy path" and the
| "catch" logic had been limited to either "sleep/retry", "abandon"
| or "fix; returning to the happy path".
|
| On the balance, my experience would suggest that Negative
| Engineering, when applied _exclusively_ ... works[2]. You just
| might not be able to explain _why_ it works very easily if it 's
| designed carelessly and (at least in some languages) the patterns
| you're required to use lend themselves to code that can be
| challenging to follow.
|
| [0] Not a terribly difficult thing to do but not something that
| the API provided a way to solve so it involved listening for
| conferences with the API and reading data from the reverse-
| engineered database the service used... not a difficult task.
|
| [1] The service had its own supervisor which handled recovery so
| for application failures (database unreachable/conferencing
| environment down), the right thing to do was exit.
|
| [2] It logged critical errors left and right, but I wasn't put on
| the project to fix bugs, I was put on it to add a feature ...
| something that could no longer realistically be done due to the
| tech debt/complexity the solution had reached being designed the
| way it was.
| PhasmaFelis wrote:
| > Imagine how frustrated you'd be if every now and then, an
| application simply refused to add items to your cart, navigate to
| a certain page, or charge your credit card. The truth is, these
| minor refusals happen surprisingly often, but users never know
| because of systems dedicated to intercepting those errors and
| running the erroneous code again.
|
| I want to live in the world this guy is from.
___________________________________________________________________
(page generated 2022-12-12 23:01 UTC)