[HN Gopher] Fuzz testing: the best thing to happen to our applic...
       ___________________________________________________________________
        
       Fuzz testing: the best thing to happen to our application tests
        
       Author : bluestreak
       Score  : 29 points
       Date   : 2023-08-17 16:15 UTC (6 hours ago)
        
 (HTM) web link (questdb.io)
 (TXT) w3m dump (questdb.io)
        
       | yosefk wrote:
       | The relative rarity of input (pseudo-)randomization in SW testing
       | is near inexplicable to me, except by the very low cost of all
       | but the most commonly reproducing bugs paid by the SW vendor.
        
         | mqus wrote:
         | In the regular testsuite (think CI) you want to have
         | predictable results. Doing them again and again on the same
         | code should give the same results so you can properly see with
         | which code change things got wrong. Maybe it's simpler to
         | explain it the other way around, for every new path your
         | fuzzer(or other randomized test) tests, it also doesn't test a
         | path it tested in a previous run and you probably want to add
         | the failing paths it found to your regular test suite.
         | 
         | Don't get me wrong, we should have more randomization, but it's
         | not good everywhere, which might explain why we don't have as
         | much of it.
        
       | jerrinot wrote:
       | Hello, I'm Jaromir, one of the core engineers at QuestDB team. I
       | just noticed this blog is trending! Andrei - the author - lives
       | in Bulgaria and he is probably already sleeping. Happy to answer
       | any question the blog left unanswered.
        
         | yosefk wrote:
         | What tests did you have before adding fuzzing?
        
           | jerrinot wrote:
           | it's the usual spectrum of tests:
           | 
           | 1. correctness: from small units tests to relatively complex
           | integrations tests. they typically populate a test database
           | and query it via various interfaces, such as REST or the
           | Postgres protocol. we use Azure Pipelines to execute them -
           | testing in MacoOS, Linux (both Intel and ARM) and Windows.
           | 
           | 2. performance: we tend to use the TSBS project for most of
           | our performance testing and profiling. fun fact: we actually
           | had to patch it as the vanilla TSBS was a bottleneck in some
           | tests. Sadly, the PR with the improvements is still not
           | merged: https://github.com/timescale/tsbs/pull/186
           | 
           | edit: I thought I would link some of the more interesting
           | tests: Since QuestDB supports the Postgres wire protocol we
           | have to gracefully handle even various half-broken Postgres
           | clients. But how do you write a test with mimicking a client
           | generating invalid requests? No sane client will generate
           | broken requests. So we use Wireshark to record network
           | communication between the broken client and our server and
           | then use this recorded communication in tests. Example: https
           | ://github.com/questdb/questdb/blob/3995c31210c70664d4b3...
        
       ___________________________________________________________________
       (page generated 2023-08-17 23:00 UTC)