[HN Gopher] The sad state of property-based testing libraries
___________________________________________________________________
The sad state of property-based testing libraries
Author : nequo
Score : 97 points
Date : 2024-07-04 15:15 UTC (7 hours ago)
(HTM) web link (stevana.github.io)
(TXT) w3m dump (stevana.github.io)
| Smaug123 wrote:
| I had a quick look through the linked "Testing Telecoms Software
| with Quviq QuickCheck" paper, but couldn't immediately see
| anything to answer my question: "why is this stateful stuff not
| better rolled by hand?". The OP gestures at this with the key-
| value pair model of a key-value store, but why would one not just
| write a state machine; why does it require a framework? I
| literally did this last week at work to test a filesystem
| interaction; it boiled down to:
|
| type Instruction = | Read of stuff | Write of stuff | Seek of
| stuff | ...
|
| And then the property is "given this list of instructions, blah".
| The `StateModel` formalism requires doing basically the same
| work! I really don't see that `StateModel` is pulling its weight:
| it's adding a bunch more framework code to understand, and the
| benefit is that it's getting rid of what is in my experience a
| very small amount of test code.
| two_handfuls wrote:
| My understanding is that the parallel QuickCheck will check
| that in your multithreaded program, all possible interleavings
| result in a final state that can also be reached by a
| sequential invocation of the commands.
|
| That would be the benefit.
| Smaug123 wrote:
| Is it _common_ to work in a runtime that lets the test
| harness control the order in which threads race to access
| mutable state? I wouldn 't be surprised if Haskell could do
| it, but I think I'd have to write a custom interpreter of
| .NET or JVM bytecode, for example.
| neonsunset wrote:
| I believe Java has exactly that, but I couldn't recall the
| project name.
|
| In the case of .NET, there is
| https://github.com/microsoft/coyote which works by
| rewriting IL to inject hooks that control concurrent
| execution state.
|
| It would have been much more expensive to have a custom
| interpreter specifically for this task (CoreCLR never
| interprets IL).
|
| This approach somewhat reminds me of precise release-mode
| debugging and tracing framework for C++ a friend of mine
| was talking about, which relies on either manually adding
| the hooks to source files or doing so automatically with a
| tool.
| Smaug123 wrote:
| Ah, now you mention it I did try Coyote once; it crashed
| out instantly on F#, which is why I forgot about it.
| AlexErrant wrote:
| I'm just here to commiserate with F# being the red-headed
| stepchild of dotnet. Everytime people say "dotnet" they
| _really_ mean "C#".
| neonsunset wrote:
| Given F# compiler outputs legal IL that is expected to
| execute in a particular way, and CoreCLR doesn't fail on
| importing it, and then executing it, it has probably more
| to do with either Coyote or other tooling that interacts
| with this setup.
|
| That is, if the issue persists. There's one that mentions
| F# submitted in 2020, but nothing else since:
| https://github.com/microsoft/coyote/issues/39
|
| Edit: Whoops, meant to reference this one:
| https://stackoverflow.com/questions/61694051/how-to-use-
| micr...
|
| I don't understand why there is less tolerance for having
| to make accommodations or take extra steps when working
| with F# than, let's say, when using Kotlin or Scala. Both
| follow a similar pattern where they _can_ use pretty much
| every Java library by virtue of targeting JVM but can 't
| use a huge amount of Java-only tooling that does meta-
| programming or instrumentation beyond what is provided by
| JVM bytecode specification, and yet it's not seen as such
| a huge ordeal from my surface impressions.
| Smaug123 wrote:
| _Does_ that issue mention F#? I may be blind but I see no
| evidence that it does.
|
| In general the F# way is to write completely different
| stuff, right, either hiding away the underlying C# or
| completely replacing it. Witness the existence of the
| SAFE stack, Giraffe to hide the egregious ASP.NET, the
| totally different approaches with statically resolved
| type parameters in F# vs enormous hierarchies of
| interface types in C#, Myriad AST generation vs Roslyn
| stamping out strings, etc. I have no opinion on the
| languages hosted on the JVM, but there is a really
| nontrivial impedence mismatch between idiomatic C# and
| F#!
| drewcoo wrote:
| I would say that stateful stuff is better covered by model-
| based testing. Feel free to mix test styles - it's your code!
| NickM wrote:
| There are some tests where you're right, but often the tricky
| part is shrinking failing cases. If you only want to generate
| sequences of state transitions that are considered "valid" then
| you typically have to have some sort of model state that
| dictates which test steps are valid for a given state, and you
| need to make sure that you don't remove test steps during
| shrinking in a way that triggers a spurious failure by
| violating the preconditions that you followed to generate each
| step originally.
|
| If any operation is valid in any state and you just want a
| totally random sequence of arbitrary operations, then yeah a
| stateful proptest framework may be overkill. But if you need to
| maintain a model state and specify preconditions for different
| operations, having a dedicated framework saves a lot of work.
|
| I wrote a blog post about this stuff last year, if you're
| interested in some more in-depth examples:
| https://readyset.io/blog/stateful-property-testing-in-rust
|
| Also, as others have mentioned, parallel state machine testing
| is another really cool benefit you can get from having a
| dedicated framework, but it's not the only benefit.
| mjaniczek wrote:
| The author focuses on the state machine and parallel aspect of
| PBT, but there are other aspects that might have larger effect.
|
| One of these is coverage guided PBT, check eg. Dan Luu's article:
| https://danluu.com/testing/
|
| Another one, which I'm biased towards, is making shrinking
| automatic while keeping all the invariants you created while
| generating the values. I have a talk on this but the TL;DR is
| that
|
| - derived QuickCheck-like shrinker functions that work on values
| (shrink : a -> [a]) have issues with the constraints, making
| people turn off shrinking instead of dealing with the issues
|
| - rose tree "integrated shrinking" (eg. Hedgehog) follows the
| constraints of the generators, but has issues with monadic bind
| (using results of generators to dispatch to another generator)
|
| - the only approach that seems to magically "just work" is
| Hypothesis' "internal shrinking", which uses a layer of
| indirection to shrink lists of random choices instead of the
| values themselves. The disadvantage is that the generators are
| now parsers from the lists of bytes that can fail (introducing
| some inefficiency) and user can make a crazy generator that the
| internal shrinker will not be able to shrink perfectly.
| Nevertheless, it's the best DX of the three approaches, and given
| it's a small miracle people are testing at all, feels to me like
| the approach most worth building on as a testing library author.
| AlexErrant wrote:
| > - rose tree "integrated shrinking" (eg. Hedgehog) follows the
| constraints of the generators, but has issues with monadic
| bind.
|
| We're at the limits of my amateur knowledge, but I believe this
| is a fundamental limitation of monadic bind/generators.
| Instead, you should prefer applicative generators for optimal
| shrinking. https://github.com/hedgehogqa/haskell-
| hedgehog/issues/473#is...
|
| In other words, applicative generators do not use "results of
| generators to dispatch to another generator", but instead
| shrinking is optimal due to the "parallel" nature of
| applicatives (I'm using "parallel" in the monadic sense, and
| not the sense of article's "threading" sense). Since
| applicatives are "parallel", they can shrink the generators
| independently. (Whereas monadic generators are in "series" and
| therefore shrinking one necessarily changes the behavior of the
| subsequent generator, as you noted.)
|
| Please feel free to link your talk if it's public!
| ctxc wrote:
| I now realize how far I am from amateur knowledge, I don't
| understand any of this.
|
| But always interesting to discover an avenue I haven't
| explored before :)
| AlexErrant wrote:
| Haha yeah I kinda went off the deep end with applicatives.
| Here's a short primer on applicative vs monadic shrinking
| behavior using F# syntax
| https://github.com/hedgehogqa/fsharp-
| hedgehog/issues/419#iss...
|
| You can think of `let!` as `let + await`, and `let! x ...
| and! y` as `await Parallel([x, y])`.
|
| Please feel free to ask any questions if it's still
| confusing!
| Smaug123 wrote:
| Not _quite_ accurate with the Parallel example, if I
| understand you correctly. Don Syme is explicit that
| applicative `async` should not implicitly start work in
| the thread pool (https://github.com/dotnet/fsharp/issues/
| 10301#issuecomment-7...).
| AlexErrant wrote:
| My usage of "parallel/await" is entirely metaphorical; I
| was kinda going for a Javascript-esque syntax with "await
| Parallel" - I'm assuming most people aren't familiar with
| F#'s `and!`. It doesn't make sense for applicative
| generators to (necessarily) use the thread pool.
| nine_k wrote:
| To have an idea of monads being sequential, think about a
| JS Promise:
| promise.then(...).then(...).then(...)...
|
| Promises are (almost) monadic, and the chain is sequential.
| You can't make it parallel, and that's the point: monadic
| binding represents sequential computation, aka "the
| semicolon operator".
|
| To have an idea about applicatives being parallel, think
| about a list of functions, and a list of values; each
| function would be applied to a corresponding value,
| resulting in a list of results: results =
| [] for ix in range(functions): f =
| functions[ix] x = values[ix]
| results.append(f(x))
|
| It's pretty obvious that you can do each _f(x)_ in parallel
| and in any order, instead of the sequential code above.
|
| (Why would one care about this all? Because many daily
| things are functors, applicatives, and monads, e.g. a list
| is usually all three.)
| mananaysiempre wrote:
| Sequence points in C and CPS/ANF compilation in Lisp are
| kind of related to this. Both monads and applicatives
| enable you to define f(g(), h()), where f, g, and h are
| stateful computations of some sort, but monads force you to
| specify which of g and h is invoked first [so it's more
| like x = g(), y = h(), f(x, y)] while with applicatives the
| implementor of the stateful model can decide what happens
| in such cases.
|
| [Disclaimer: I don't know how QuickCheck actually works, so
| I'd appreciate a sanity check (hah) of the following from
| somebody with actual knowledge on the matter.]
|
| GP's point, if I understood it correctly, is as follows. If
| you're doing randomized testing, then g and h could perhaps
| be defined as { for (i=0; i<n; i++) { int x = rand(); if
| (!fork()) return x; } } (and yes, in monad-land the moral
| equivalent of fork() is admissible as what I vaguely called
| a "stateful computation"). You see how strict ordering of
| side effects enforced by monads essentially forces you into
| a depth-first search of the state space (although I guess
| you could do iterative deepening?), while applicatives can
| allow for different exploration strategies (ones more
| interesting than C's official behaviour of "that's UB, you
| fool").
| sunshowers wrote:
| I write stateful property tests with Rust's proptest quite
| regularly, I just tend to handcode them which is quite
| straightforward. See https://github.com/sunshowers-code/buf-
| list/blob/main/src/cu... for a nontrivial example which found 6
| bugs.
|
| For parallel testing I guess it can be useful at times, but often
| it's easier to just run a bunch of tests in parallel instead.
| gamegoblin wrote:
| I do a lot of manual proptesting in Rust that all look
| something like: let mut rng =
| rand::thread_rng(); for action_count in 1..4 {
| for _ in 0..10_000 { let seed =
| rng.gen::<u64>(); eprintln!("let seed =
| {seed};"); let mut rng =
| ChaChaRng::seed_from_u64(seed);
|
| i.e. top level true randomness, then a bunch of nested loops
| (only 2 here, but some tests have more) to go from low-
| complexity cases to high-complexity
|
| then generate a seed to seed a deterministic PRNG, and print it
| out so if the test fails, I just copy and paste the error seed
| to replay the error case
|
| I have found doing this manual proptesting to be faster, more
| flexible, and generally less fuss than using any frameworks or
| libraries
|
| That said, for really robust concurrency testing, I cannot
| recommend enough the AWS Shuttle library
| (https://github.com/awslabs/shuttle) which can find insanely
| complicated race conditions. I wrote a little tutorial on it
| here: https://grantslatton.com/shuttle
|
| We used it at AWS to verify the custom filesystem we wrote to
| power AWS S3.
| nextaccountic wrote:
| How do you do shrinking? IMO that's the property testing
| killer feature
| gamegoblin wrote:
| That's why my outer loop goes from low-complexity cases to
| high-complexity cases, it has basically the same effect as
| shrinking (without actually having to do any work)
| josephg wrote:
| I do this sort of thing too. I don't have an automated way
| to shrink my input - but that's usually fine in practice.
|
| For example, say I'm testing a data structure. I'll have an
| outer loop that picks a seed and an inner loop that does
| ~100 mutations of a data structure instance, testing
| assertions each time. If there's a failure, I'll try a
| bunch of seeds to look for one that fails the fastest. (In
| the fewest possible inner loop iterations).
|
| It's not a perfect system, but for most bugs I can usually
| get a test case that only needs 5-10 steps before a crash
| occurs. And if the reduction step finds a different bug?
| That's no problem at all. I fix what I found and go back to
| trying more seeds.
| sshine wrote:
| Amazing survey.
|
| Big fan of property testing.
|
| A particularly big fan of Hedgehog.
|
| I tried writing a Hedgehog-inspired library in Rust and realized
| how complex the underlying domain is.
|
| For example, the RNG is "splittable" [1][2] which means you can
| take a deterministic seed and split it in two, e.g. for parallel
| generators to work independently but still deterministically. The
| effort that has gone into this feature level is a little numbing.
| I have an awe similar to that of the "fast inverse square root"
| hack when I see code like this: -- | A predefined
| gamma value's needed for initializing the "root" instances of
| -- 'Seed'. That is, instances not produced by splitting an
| already existing -- instance. -- -- We
| choose: the odd integer closest to @2^64/ph@, where @ph = (1 +
| [?]5)/2@ is -- the golden ratio. --
| goldenGamma :: Word64 goldenGamma =
| 0x9e3779b97f4a7c15
|
| when realizing that those numbers don't come easily. [3]
|
| [1]:
| https://hackage.haskell.org/package/hedgehog-1.4/docs/src/He...
|
| [2]: https://gee.cs.oswego.edu/dl/papers/oopsla14.pdf
|
| [3]: https://github.com/hedgehogqa/haskell-hedgehog/issues/191
|
| Wonder why most property-testing libaries don't have features
| like this?
|
| The libraries require training to use. And they're not that easy
| to write.
|
| > _the current state-of-the-art when it comes to property-based
| testing is stateful testing via a state machine model and reusing
| the same sequential state machine model combined with
| linearisability to achieve parallel testing_
|
| Okay, okay. I admit I've never performed property-based stateful
| testing, nor in parallel. So that may be the coolest feature out
| there, because it addresses one of the hardest problems in
| testing.
|
| But I think that yet other things have happened with modern
| property-testing libraries (e.g. Hypothesis, PropEr, Hedgehog,
| Validity):
|
| Shrinking for free [4], generators for free [5], defining the
| probability distribution of your sub-generators in a composable
| way.
|
| Maybe those features are not as significant, but they're equally
| missing from almost all property-test libaries.
|
| [4]: Gens N' Roses: Appetite for Reduction * Jacob Stanley * YOW!
| 2017 https://www.youtube.com/watch?v=LfD0DHqpeVQ
|
| [5]: https://tech.fpcomplete.com/blog/quickcheck-hedgehog-
| validit...
| 01HNNWZ0MV43FF wrote:
| Huh. I wonder why they don't use one of those random-access
| PRNGs like PRNS, which is basically a hash of a counter. Maybe
| not good enough in the speed-quality space?
| pydry wrote:
| Lack of parallelization really doesnt make me feel that sad. I'm
| constrained by many things while coding but CPU horsepower isnt
| one of them.
| mrkeen wrote:
| What's stopping your test suite from running instantly?
| pydry wrote:
| Typically a desire to have tests that are somewhat realistic.
| mrkeen wrote:
| What does that mean?
| meindnoch wrote:
| You realize that by "parallelism" the author is referring to
| the detection of concurrency issues, and not the parallel
| running of tests, right?
| k__ wrote:
| Typescript/JavaScript libraries are in the list, but JS it's
| single threaded, so I don't know if there is really something
| missing here.
| nequo wrote:
| Concurrency is not the same as parallelism. I've understood
| TFA to mean concurrency when it mentions parallelism. This
| would explain the presence of JS libraries which could test
| for concurrency problems in asynchronous code.
| epolanski wrote:
| JS is single threaded, but the process lives in an event
| loop.
|
| You can easily write a JS function that can have a race
| condition by launching two asynchronous processes (such as
| writing to local storage in a browser or the file system
| both operating on the same data).
|
| I had this happen just today while scraping different
| websites writing to the same files and overwriting each
| other.
| ncruces wrote:
| With the advent of coverage based fuzzing, and how well supported
| it is in Go, what am I missing from not using one of the property
| based testing libraries?
|
| https://www.tedinski.com/2018/12/11/fuzzing-and-property-tes...
|
| Like, with the below fuzz test, and the corresponding invariant
| checks, isn't this all but equivalent to property tests?
|
| https://github.com/ncruces/aa/blob/505cbbf94973042cc7af4d6be...
|
| https://github.com/ncruces/aa/blob/505cbbf94973042cc7af4d6be...
| mrkeen wrote:
| I'm not sure how Go's fuzz tests differ from what you linked in
| your article, but you article said proper fuzzers need to run
| for days or weeks, and that PBT should approximately always
| chosen over fuzz testing.
|
| But I'd take one step back, and ask a more meta question about
| testing: does a successful test mean successful code, and vice-
| versa? Is there anything in Go's contract that specifies that
| the same inputs to the same code will yield the same output?
| bananapub wrote:
| fuzzing is clearly not a replacement for tests?
| josephg wrote:
| It kind of is. I often use randomised tests (fuzzing) for
| data structures and algorithm implementations. Throw a lot of
| asserts in and see if an hour of random input can find any
| edge cases I've missed. Usually the first time I run
| something like this it finds problems instantly. It's very
| humbling.
|
| I find when I do this I don't need to write by hand anywhere
| near as many tests to get my software working well. I also
| usually turn any failures found by the fuzzer into standalone
| unit tests, to make any regressions easier to find later.
| ncruces wrote:
| I agree, it can be. My example above matches yours (fuzzing
| a data structure).
|
| I coded some tests that ensure the data structure is
| useful, many of them test examples from papers describing
| the data structure, but that don't necessarily cover all
| the corner cases.
|
| Then I fuzzed it. I used Go's fuzzer, which is geared
| towards parsers and stuff. It can generate a stream of
| bytes and use that for fuzzing. The data structure is a
| set/map. So I interpret the stream of bytes as commands to
| add/remove random elements from the set/map. After I add an
| element, contains needs to return true; after I remove one,
| contains need to return false; if I put in a mapping,
| finding it must return what I just put there, etc. And at
| every step all the data structure invariants (that ensure
| logarithmic search, etc) need to hold.
|
| That was stupidly effective at finding a few bugs, all
| within seconds, all with sequences of less than a dozen
| operations. Then it stops. And you get 100% coverage.
|
| I'm assuming that, apart from ergonomics, where I kinda
| build my own state machine transitions out of a stream of
| bytes, the tooling actually seems more effective than
| property testing libraries.
|
| Still curious to understand what I'm missing out.
| AlexErrant wrote:
| You're asserting properties, so IMO this meets the definition
| of PBTs ("every node of level greater than one has two
| children").
|
| However, depending on the lib, you can get some nice quality of
| life improvements. One "nice to have" is shrinking. See the
| "Shrinking" section here
| https://tech.fpcomplete.com/blog/quickcheck-hedgehog-validit...
|
| Having combinators to compose generators is also great.
|
| Libs may also have a known set of "bad" values that cause
| exceptional behavior.
| gavinhoward wrote:
| AFL++, a fuzzer, has a tool to minimize (shrink) test cases.
| zarathustreal wrote:
| "Properties" in the Property-based Testing sense refers to
| mathematical properties such as equality, associativity,
| commutivity, etc
| ncruces wrote:
| The Go fuzzer, when it finds a failure, will also walk back
| and try to shrink inputs (and still trigger the same
| failure).
|
| Not sure how effective it is compared to other options, but
| I'm not totally missing out there.
|
| It also builds a corpus of interesting inputs over time
| (those that cause new branches to be taken, since that's its
| goal: improve coverage).
|
| I'm less sure about combinators.
| kccqzy wrote:
| You can totally combine coverage based fuzzing with property
| based tests. When I was at Google, I really enjoyed their
| internal tooling for combining both. You simply write a
| property based test as usual, but when it comes to execution,
| the testing framework compiles your test in a special way to
| get coverage and then adjust the random input to hit increased
| coverage. Of course they run the test across a cluster of
| machines completely automatically.
|
| Traditional property based testing is implemented simply as a
| library, so they don't necessarily have coverage information to
| guide their random input generation.
| hyperpape wrote:
| The distinction between property based testing and fuzzing is
| basically just a rough cluster of vibes. It describes a real
| difference, but the borders are pretty vague and precisely
| deciding which things are fuzzing and which are PBT isn't
| really that critical.
|
| - Quick running tests, detailed assertions --> PBT
|
| - Longer tests, just looking for a crash --> fuzzing.
|
| - In between, who knows?
|
| https://hypothesis.works/articles/what-is-property-based-tes...
| ruuda wrote:
| The major omission in this article is fuzzing. Not only is it
| practical and in wide (and growing use), it's also far more
| advanced than QuickCheck's approach of generating random inputs,
| because fuzzing can be _coverage-driven_. Property-based testing
| came out of academia and fuzzing came out of security research,
| initially they were not connected. But with the advent of in-
| process fuzzing (through libFuzzer), which encourages writing
| small fuzz tests rather than testing entire programs; and
| structure-aware fuzzing, which enables testing more than just
| functions that take a bytestring as input, in my view the two
| techniques have converged. It's just that the two separate
| communities haven't fully realized this yet.
|
| One pitfall with non-coverage-driven randomized testing like
| QuickCheck, is that how good your tests are depends a lot on the
| generator. It may be very rarely generating interesting inputs
| because you biased the generator in the wrong way, and depending
| on how you do the generation, you need to be careful to ensure
| the generator halts. With coverage-driven fuzzing all of these
| problems go away; you don't have to be smart to choose
| distributions so that interesting cases are more common, coverage
| instrumentation will automatically discover new paths in your
| program and drill down on them.
|
| But isn't fuzzing about feeding a large program or function
| random bytestrings as inputs, whereas property-based testing is
| about testing properties about data structures? It is true that
| fuzzers operate on bytestrings, but there is no rule that says we
| can't use that bytestring to generate a data structure (in a
| sense, replacing the role of the random seed). And indeed this is
| what the Arbitrary crate [1] in Rust does, it gives tools and
| even derive macros to automatically generate values of your data
| types in the same way that QuickCheck can. The fuzzing community
| calls this Structure-Aware Fuzzing and there is a chapter about
| it in the Rust fuzzing book [2]. There are also tools like
| libprotobuf-mutator [3] that substitute fuzzers' naive mutation
| strategies, but even with naive strategies fuzzers can usually
| get to 100% coverage with appropriate measures (e.g. recomputing
| checksums after mutation, if the data structure contains
| checksums).
|
| I am using this extensively in my own projects. For example, RCL
| (a configuration language that is a superset of json) contains
| multiple fuzzers that test various properties [4], such as
| idempotency of the formatter. In the beginning it used the raw
| source files as inputs but I also added a more advanced generator
| that wastes less cycles on inputs that get rejected by the
| parser. The fuzzer has caught serveral bugs, and most of them
| would have had no hope of being found with naive randomized
| testing, because they required a cascade of unlikely events.
|
| Structure-aware fuzzing is not limited to generating data
| structures either, you can use it to generate reified commands to
| test a stateful API, as you describe in the _Stateful property-
| based testing_ section. The Rust fuzzing book has an example of
| this [5], and I use this approach to fuzz a tree implementation
| in Noblit [6].
|
| [1]: https://docs.rs/arbitrary/latest/arbitrary/ [2]:
| https://rust-fuzz.github.io/book/cargo-fuzz/structure-aware-...
| [3]: https://github.com/google/libprotobuf-mutator [4]:
| https://docs.ruuda.nl/rcl/testing/#fuzz-tests [5]: https://rust-
| fuzz.github.io/book/cargo-fuzz/structure-aware-... [6]:
| https://github.com/ruuda/noblit/blob/a0fd1342c4aa6e05f2b1c4e...
| sunshowers wrote:
| I spent some time looking at the arbitrary crate in Rust and
| was left unsatisfied at the shrinking story, which I think is
| 90% of the value of PBT.
| matklad wrote:
| Do you have a model problem which is tricky to shrink?
|
| I implemented a stupid simple shrinker for arbitrary, and I'd
| love to know a specific example where it fails to shrink in a
| good way:
|
| https://github.com/matklad/arbtest/blob/0191f93846e9f7e38254.
| ..
|
| I know at lest two interesting approaches for making that way
| smarter, but I don't yet have a problem where my dumb
| approach isn't sufficient.
| choeger wrote:
| The obvious downside is the number of examples computed. A simple
| unit test can easily take 100 times as long when property-tested.
|
| Besides, it would be really cool to have property-based
| integration or hybrid unit/integration tests. Or even property-
| based E2E tests. Unfortunately, the setup of the example will
| almost always take too long for a relevant set of runs.
|
| For instance: If you have a basic data model (say in sqlalchemy)
| and want to write property-based tests (say in hypothesis), you
| can relatively quickly derive strategies for the models (but
| beware of recursion and primary keys). But writing that model
| instance into the DB for running an example just takes too long
| for swift testing.
| PeterisP wrote:
| The simple answer to a question posed in the article "On the
| other hand one could ask why there isn't a requirement that
| published research should be reproducible using open source tools
| (or at least tools that are freely available to the public and
| other researchers)?" is that the obvious immediate outcome of
| such a requirement is that papers failing that requirement - like
| the Quviq QuickCheck papers, which seem to have been useful to
| the author and others - simply would not get published, and the
| community would lose out on that gift of information.
| spencerchubb wrote:
| I think it would be good to have some publishers require
| reproducibility, and some publishers that don't. Every
| requirement is exclusionary, and there are always edge cases
| where a paper can be useful even if it doesn't fulfill a
| requirement.
| josephg wrote:
| That's already the case. Eurosys includes an Artifact
| Evaluation for submitted papers - which typically includes
| code and data to allow reviewers to reproduce the work
| described in the paper. It's optional, but encouraged.
|
| This page lists the artifact evaluation criteria for a
| handful of conferences:
|
| https://sysartifacts.github.io/
| ashton314 wrote:
| As an intermediate point in the Require Reproducibility - No
| Requirement spectrum, strong encouragement to have
| reproducible artifacts is attractive as well. I just got a
| paper accepted to ECOOP (European Conference on OO
| Programming; has lost it's OO-focus and is now a general PL
| conference) and it's trying something relatively new:
| artifact evaluations are considered as part of the submission
| process. Our paper had a reproducible _and_ reusable
| artifact, and I think that helped our case with the
| reviewers.
| mardifoufs wrote:
| Couldn't it be possible to make the source available for the
| reviewers only? With whatever is needed to make the code run
| too.
|
| Maybe that's already being done though!
| rdtsc wrote:
| I wonder if it's missing the original QuviQ Erlang QuickCheck in
| the list? The full product is proprietary, but there is a
| freeware version QuickCheck Mini available as well:
| http://www.quviq.com/downloads/
| lmm wrote:
| I've tried to use property based testing but always found it
| falls between two stools. If I understand a property well enough
| to write a strict test of it, I can generally push it into the
| type system and make it true by construction. And if I just want
| a "smoke test" then a single arbitrary input is easier.
___________________________________________________________________
(page generated 2024-07-04 23:00 UTC)