[HN Gopher] Effective Property-Based Testing
___________________________________________________________________
Effective Property-Based Testing
Author : mullr
Score : 60 points
Date : 2021-02-03 18:39 UTC (4 hours ago)
(HTM) web link (blog.auxon.io)
(TXT) w3m dump (blog.auxon.io)
| spion wrote:
| For JS and TypeScript, the best property testing library I've
| encountered so far is fast-check https://github.com/dubzzz/fast-
| check
| mekatronix wrote:
| These are great tips. Thanks.
| spockz wrote:
| I would like to make a shout out to quickcheck (Haskell,
| https://hackage.haskell.org/package/QuickCheck) as it was my
| first introduction to property based testing and has since let me
| to use property based testing everywhere in the last 10 years.
| Then there is scalacheck (https://www.scalacheck.org/). Both let
| you write your own generators and are quite good at shrinking
| down to minimal use cases.
|
| The suggestion elsewhere in this thread to decrease the number of
| iterations during normal testing and crank it up during nightlies
| is also good.
|
| The only thing I'm still missing from the libraries is a
| convenient mechanism of remembering previously failing generated
| inputs and use them as a static list of testcases next to the at
| runtime generated ones like a regression test of sorts.
|
| Edit: typos
| im_down_w_otp wrote:
| Indeed. It's an unfortunate general omission in the ecosystem.
| It's one of the things that our product does (in its
| appropriation from different inspirations like MC, PBT, SBFL,
| etc., but we move things up to assessing systems rather than
| programs) where contradictions and counter-examples are curated
| specifically for making it easier to reuse them. They're
| provided back to the user as executable "properties" in our DSL
| to apply to past, present, future data.
|
| _disclaimer: Auxon co-founder_
|
| Also, big (h/t) to Quickcheck from me as well. Getting into it
| via Erlang many, many years ago was among the more impactful
| and transformative developments in my approach to thinking
| about improving software quality.
| spockz wrote:
| In the article you (auxon) wrote to avoid using type based
| generators and unbounded collections and instead write your
| own generators. Usually I find it more convenient to write
| filters on the generated types then create my own generators.
| As the post mentions, this can lead to many discarded inputs.
| Have you ever considered how we could use the predicates in
| the filters to create Specialized generators that only
| generate inputs that will be accepted by the filter? You
| probably need some meta programming or reification of the
| predicates for that to work.
| mristin wrote:
| If you use Python and want to infer test strategies from
| contracts, you might want to check out this library of mine: [1].
|
| There are also plugins for IDEs (Pycharm, VS Code and vim), which
| can be quite helpful during the development.
|
| [1]: http://github.com/mristin/icontract-hypothesis
| Smaug123 wrote:
| Brilliant article.
|
| If I were to add just one thing to the list: metatest. Write a
| test that asserts that your generated test cases are
| "sufficiently comprehensive", for whatever value of
| "sufficiently" you need. In an impure language, this is as easy
| as having the generator contain a mutable counter for "number of
| test cases meeting X condition" for whatever conditions you're
| interested in. For example, say your property is "A iff B". You
| might want to fail the test if fewer than 10% of the generated
| cases actually had A or B hold. (And then, of course, make sure
| your generators are such that - say - A and B hold 50% of the
| time; you want an astronomically small chance of random metatest
| failure.)
|
| (I did a brief intro to this in the Metatesting section of a talk
| I did two years ago:
| https://github.com/Smaug123/talks/blob/master/DogeConf2019/D... .
| On rereading it now, I see there's a typo on the "bounded even
| integers" slide, where the final `someInts` should read
| `evenIntegers`.)
| lowercase1 wrote:
| This is a good overview of property based testing but the
| mentions of Cucumber threw me off.
|
| In my job Cucumber seems to add little more than just commenting
| and sequencing functions, tasks that are better suited to your
| programming language of choice, while adding overhead and
| complexity.
|
| What am I missing?
| leeuw01 wrote:
| Auxon also looks like an interesting company.
| [deleted]
| delightful wrote:
| If so, company should post here:
|
| Ask HN: Who is hiring? (February 2021)
|
| https://news.ycombinator.com/item?id=25989764
| rck wrote:
| This is a great set of ideas for using property-based testing.
| I've found it useful to think of code in terms of invariants and
| contracts, and property-based testing lets me express those very
| directly in code. No other testing method comes close.
| the_gastropod wrote:
| Property testing is awesome, but it does significantly slow down
| test suites. Are there any standard practices for striking a
| balance of getting the added confidence/safety of using property
| tests without introducing large delays into your CI pipeline?
| hinkley wrote:
| My understanding, shallow as it is, is that property testing
| goes great with pure functional code, because without side
| effects you can run tests in parallel, taking advantage of all
| of these cores and build servers we have.
|
| If your tests are coupled, you're already in a bad way whether
| you know it or not. Dumping property testing on top of that
| without addressing the underlying cause sounds like a recipe
| for misery.
|
| It's probably a great stick and carrot if you're pushing a tech
| debt reduction agenda though.
| Jtsummers wrote:
| Item 5 in TFA is one that I found related to performance most.
| Using filters to toss out bad input means you're potentially
| generating thousands of inputs for dozens of tests. Aggregate
| this across all your tests and it's an incredible waste of
| time. Constructing proper inputs _directly_ can get you closer
| to the minimum time required to execute the tests and help
| achieve that desired performance threshold. (NB: Not used
| professionally, only got one office to even entertain them but
| they wouldn 't take it and run with it even after I showed them
| a dozen errors I'd found in an hour of using them. Consequently
| my experience is still limited.)
| mullr wrote:
| We like to run the tests as part of CI with a relatively small
| number of iterations, and then turn the knob way up in a
| nightly or weekly scheduled test job.
| Smaug123 wrote:
| Yup. We've got some stuff set up so that in the IDE your
| tests finish in ~100ms; in precommit testing pipelines you
| get several seconds; in nightly tests you get two hours.
| k__ wrote:
| You could fan-out tests to multiple FaaS instances.
|
| Cloudflare Workers, for example, have no cold-starts. But they
| are JS/WASM only
| delightful wrote:
| Does anyone have any thoughts on the graphic showing the spectrum
| of testing options listed in the article (or additional resources
| that cover at a high-level this topic of understanding range of
| testing options)?
|
| link to the graphic for ease of reference:
|
| https://blog.auxon.io/images/posts/effective-property-based-...
___________________________________________________________________
(page generated 2021-02-03 23:01 UTC)