[HN Gopher] Rethinking Software Testing: Perspectives from the W...
___________________________________________________________________
Rethinking Software Testing: Perspectives from the World of
Hardware (2019)
Author : Jtsummers
Score : 34 points
Date : 2021-07-10 17:37 UTC (5 hours ago)
(HTM) web link (software.rajivprab.com)
(TXT) w3m dump (software.rajivprab.com)
| baybal2 wrote:
| It needs to be said that software formal verification is much,
| much harder than hardware, and it will put much harder limits on
| they way you design a programs.
|
| I suspect it would not be rare for a software project written
| this way to have more work hours spent on verification, than the
| development itself.
|
| The use of shared code would be even more pervasive to the point
| whole library will likely be replaced rather than doing a single
| LOC change.
|
| Hardware verification is very much like a pyramid. Verified cells
| are made of verified, and characterised to death semiconductor
| devices. Verified gates are made of verified cells. Verified
| logic components are made of verified logic gate designs.
| Verified libraries are built upon verified logic components.
| Verified blocks built upon verified libraries. And verified
| macroblocks built upon verified blocks.
| MasterIdiot wrote:
| I remember a talk where someone who did both software and
| hardware development for Intel talked about "Hardware development
| cycle fro software developers" or something like that at some
| conference, and the cost of simulating CPUs or fabricating them
| for a round of testing was absolutly mindblowing. Also - the fact
| the a 1 in a billion bug will happen multiple times a second on a
| modern CPU.
|
| P.S Did anyone else happen to run into this talk? I tried to find
| on youtube and could'nt find anything related.
| triska wrote:
| This article contains many good points.
|
| However, in my opinion, it is still overly lax: The aim should
| always be _complete coverage_.
|
| The article hints at this with:
|
| _"Test every combination of events? Test every output for every
| combination of events? Test every possible corner case? You would
| need an absolutely enormous number of tests in order to achieve
| that!"_
|
| However, it does not really follow through on it, and even
| continues with a dedicated section on "Why Random Testing Works".
| At least _symbolic_ execution with coarse approximations should
| be used to cover all cases subsumed by more abstract domains.
|
| The reason for this is that the number of possible cases is
| usually so large, and the areas where problems lie often so
| small, that random tests are unlikely to discover all mistakes.
| The article mentions precisely this situation: "The bug only
| manifested itself during a small set of overlapping corner
| cases." QuickCheck and fuzzing are popular and do find mistakes
| in practice. However, how many mistakes are _not_ found by these
| tools, if such random approaches already discover mistakes?
|
| In my opinion, an ideal test case only terminates if a mistake is
| found. If not, it will simply keep on searching, potentially for
| months, years, ... You need to be able to start the search for
| mistakes and keep it running for as long as you want.
|
| Logic programming languages like Prolog excel at writing test
| cases, because they provide search as an _implicit_ language
| feature. It is easy to write a query that asks: _" Are there
| cases where such-and-such holds?"_, and the system will search
| for them _exhaustively_. The ability to concisely write test
| cases that cover vast areas of the search space is a major
| attraction of logic programming languages.
| throwamon wrote:
| > In my opinion, an ideal test case only terminates if a
| mistake is found. If not, it will simply keep on searching,
| potentially for months, years, ... You need to be able to start
| the search for mistakes and keep it running for as long as you
| want.
|
| That's a good point, but the fundamental problem is that any
| sufficiently complex space takes way more than a practical
| amount of time to be completely traversed (easily reaching or
| surpassing age-of-the-universe magnitudes), so isn't that where
| fuzzy testing comes in? If you use something like Prolog, isn't
| it going to perform the search in an ordered manner,
| drastically reducing the chances that a hard-to-spot mistake is
| found? And at least in the general case, don't you have to
| restart the search from scratch every time logic changes?
| satisfice wrote:
| Yet another article on testing written by someone who defines
| testing in whatever way conveniently ignores the icky part:
| people.
|
| Every time you want to say "manual" testing, ask yourself why you
| never say "manual programming." Testing is a human, social
| process, just as programming is. Output checking can be
| automated, but so what? It's not the essence of testing any more
| than running a compiler is the essence of programming.
| throwaway984393 wrote:
| Stakes are 100% the reason most tech is crap. You can't get away
| with shipping buggy hardware, people would ask for their money
| back. You can't get away with shipping a medical device that
| hurts people, you'd get sued. You can't get away with building a
| car that randomly turns on the brakes while you're going 70MPH,
| nobody would buy your cars (and you'd get sued).
|
| With consumer software, there are no stakes. The business knows
| it can make money with buggy software. The developers know the
| business doesn't care. In fact, most people in the business don't
| seem to care if it works at all. The only time they care is if
| there is a legal consequence, or if they will get fired. They are
| not always stupid, but they are _always_ lazy.
|
| And that is never going to change. There are parts of this world
| that are perfectly acceptable as just being shitty, and at least
| 95% of software in the world fits into that. Things don't get
| better without a motivation.
| ChrisMarshallNY wrote:
| _> And that is never going to change._
|
| Not necessarily, but the cure could be worse than the disease.
|
| The problem is that the stakes are getting higher and higher
| for _all_ software. Every app developer is trying to turn their
| program into a PID collector, and they are storing the PID in
| leaky buckets.
|
| I mean, why would I get an account setup screen to play a
| solitaire game? Sheesh.
|
| But politicians and bureaucracies have a nasty habit of
| imposing fairly ridiculous and/or out-of-date remedies.
|
| They could make it a requirement to fax your PID to play
| solitaire.
| mrh0057 wrote:
| People do care but there is a lack of choice. Then you have
| another issue with enterprise software is the people who are
| using the software are generally not the ones buying it. It
| comes down to does it check the right boxes.
|
| Business do care it is just much more difficult than people
| realize to make software simple and easy to use. Then you have
| the fact most software projects fail and way over budget so
| they think there is no other way. Therefore they end up
| settling since at least they got something that sort of works
| which is better than nothing.
| Jtsummers wrote:
| Several postings previously, but no discussion on any of them.
| marsven_422 wrote:
| But but according to HN design for testability is a total waste
| and relying on interfaces is useless C# bloat!!
| ibushong wrote:
| I was doing ASIC design and verification for 5 years before I
| switched to software and I think it led me to write much higher
| quality code and tests. I was used to making 100% sure everything
| was correct, because fixing bugs in silicon is extremely
| difficult and expensive. You are also forced to spend more time
| defining architecture and interfaces ahead of time, which can be
| very useful/important for certain software projects.
|
| Integration tests usually seem to find the most important bugs,
| but on hardware they take hours or days to run. With that
| limitation removed in software development, they're almost too
| good to be true.
|
| I'd also be interested in seeing how formal verification could be
| applied to software development.
| exdsq wrote:
| Formal verification is used in software development! Check out
| Coq, Lean, Agda, Lean, TLA+, or any links when you google those
| tools :)
| fouric wrote:
| Lean: so good, we mentioned it twice.
|
| I've heard that ACL2 is also a thing that exists.
|
| To me, one of the most intimidating parts of getting into
| formal verification is deciding which system to use - as you
| mentioned, there are several. What are the strengths and
| weaknesses of each?
| namibj wrote:
| They did do formal verification for a microkernel:
| https://sel4.systems
|
| It's afaik the most complex piece of software with full formal
| verification to date.
| bradleyjg wrote:
| There's some software projects that look a lot like hardware
| and would benefit from crossover learnings. There's also lots
| of software projects where the hard part is not building out
| the code but instead figuring out what to build. A lot of the
| differences in common practices are due to that, not because
| software developers are foolish or lazy.
|
| To give concrete examples:
|
| Intel knows what it needs to accomplish in order to win back
| the top spot. There are tons of challenges to get there, but
| the end goal is clear. That's an entirely different kind of
| problem from Twitter needing to figure how to monitize.
| megous wrote:
| Reference models are fun, but for anything more than a simple web
| app, if you try to make them sufficiently different in
| implementation, you end up re-implementing not only your business
| logic, but also a RDBMS functionality in your testsuite.
|
| Also verifying state of the app against the reference model after
| each testing step that modifies the state of the app also gets
| quickly out of hand. Even more so if you try to do it via public
| interfaces only.
|
| And there's no guarantee your model is right.
| dboreham wrote:
| Also transitioned from hardware design to software. Same
| cognitive dissonance at much of what I see surrounding testing
| software.
| thequantizer wrote:
| Having a background in electrical and software engineering I
| found this post very fun to read and was nodding along with some
| of the conclusions
| ChrisMarshallNY wrote:
| I spent 30 years at hardware companies, writing software.
|
| I'm fairly familiar with this conundrum.
|
| The single biggest issue that hardware folks seem to have with
| software, is refusal to accept that software, in order to be
| valuable, _has to run by different rules from hardware_.
|
| They. Just. Can't. Accept. That.
|
| If it works for hardware, it _must_ work for software. Attempts
| to explain that the agility inherent in software is a _huge_
| advantage, are met with stony glares, and accusations that
| "programmers are lazy slobs that hate quality."
|
| So software developers are sidelined, treated like crap, not
| supported in efforts to achieve quality processes (it _must_ be
| the same as hardware, or it 's "lazy slob" stuff), insulted,
| tripped up, and basically ridiculed. It totally sucks.
|
| Software is _different_ from hardware; even firmware is
| different, these days. That simple assertion needs to be the
| baseline for any true reconciliation.
|
| But I agree that modern software is of appallingly bad quality,
| and there's a dearth of lessons learned/mentoring in Quality
| practices.
|
| Most practices (like TDD) are useful, but still fairly nascent,
| and tend to be "one-trick ponies." They will work nicely for
| _some_ projects, but can be disasters for others.
|
| Quality is a continuum; not a point. In hardware, we can get away
| with treating it as a point, but we hobble software, when we try
| to do it the hardware way.
|
| Since leaving my last job, I have been working with ideas I
| developed during my career, that were never supported by the
| companies I worked for. These ideas are working fairly well, but
| they won't work for everyone.
| detaro wrote:
| > _The single biggest issue that hardware folks seem to have
| with software, is refusal to accept that software, in order to
| be valuable, has to run by different rules from hardware._
|
| Except for the moment where the hardware contains a mistake,
| then it's on the software to be flexible and contort to
| _somehow_ workaround the hardware issue, because software is
| magic and can adapt to everything.
|
| That said, I see plenty companies that are not as bad, actively
| work to change or at least accept that they don't "get"
| software entirely and that they get better results if they
| leave it to those who do.
|
| And also agreed in the other direction: Lots of resources about
| building software today describe things that work for a SaaS
| product you can deploy 5x a day. You can't do that if you ship
| hardware, especially hardware that can't phone home daily for a
| software fix.
___________________________________________________________________
(page generated 2021-07-10 23:00 UTC)