[HN Gopher] Functional Tests as a Tree of Continuations (2010)
       ___________________________________________________________________
        
       Functional Tests as a Tree of Continuations (2010)
        
       Author : amenghra
       Score  : 23 points
       Date   : 2025-03-13 20:46 UTC (2 hours ago)
        
 (HTM) web link (www.evanmiller.org)
 (TXT) w3m dump (www.evanmiller.org)
        
       | turtleyacht wrote:
       | End-to-end (e2e) tests are slow and flaky. They don't have to be,
       | but effort to fix breakage starts consuming most available time.
       | 
       | One idea is to separate scraping from verification. The latter
       | would run very fast and be reliable: it only tests against stored
       | state.
       | 
       | Then scraping is just procedural, clicking things, waiting for
       | page loads, and reading page elements into a database.
       | 
       | Some consequences are needing integrity checks to ensure data has
       | been read (first name field selector was updated but not
       | populated), self-healing selectors (AI, et al), and certifying
       | test results against known versions (fixing the scraper amid UI
       | redesign).
       | 
       | A lot of effort is saved by using screenshot diffing of, say,
       | React components, especially edge cases. It also (hopefully)
       | shifts-left test responsibility to the devs.
       | 
       | Ideally, we only have _some_ e2e tests, mostly happy paths, that
       | also act as integration tests.
       | 
       | We could combine these ideas with "stacked databases" from the
       | article and save on duplication.
       | 
       | Finally, the real trick is knowing, in the face of changes, which
       | tests _don 't_ have to run, making the whole run take less time.
        
       | zem wrote:
       | on one hand I suspect too much code has explicit and implicit
       | global state for this technique to be useful; on the other hand
       | using this from the beginning might prevent introducing that sort
       | of global state in the first place.
        
         | James_K wrote:
         | You would lose the performance optimisations of a copy, but you
         | can still express tests in this tree fashion with global state.
         | You just need to recurs the tree from the top down and run the
         | entire path up to each node. A simple example:
         | test_fork<A, B, C>(a: A, f: (A) -> B, gs: List<(B) -> C>) ->
         | List<C> {         fs.map(g -> g(f(a)))       }
         | 
         | As you can see, the first step of the test is re-run from
         | scratch every other branch of it.
        
       | djha-skin wrote:
       | I wouldn't call 5-level-nested code "surprisingly clean", and
       | continuations are cursed. I wouldn't want to have to debug tests
       | that relied on continuations unnecessarily.
        
         | James_K wrote:
         | The code doesn't have to be nested, it can be factored into
         | methods. Additionally, these aren't technically continuations
         | in the sense of the "continuation passing style" popular in web
         | programming, so don't confer the same drawbacks. Tests are
         | essentially organised as a pipeline of functions which gets
         | run, it's just that the pipeline has forks in it where the data
         | flows along both forks.
        
       | simonw wrote:
       | "One of the most essential practices for maintaining the long-
       | term quality of computer code is to write automated tests that
       | ensure the program continues to act as expected, even when other
       | people (including your future self) muck with it."
       | 
       | That's such a great condensation of why automated tests are
       | worthwhile.
       | 
       | "To write your own testing framework based on continuation trees,
       | all you need is a stack of databases (or rather, a database that
       | supports rolling back to an arbitrary revision)."
       | 
       | PostgreSQL and SQLite and MySQL all support SAVEPOINT these days,
       | which is a way to have a transaction nested inside a transaction.
       | I could imagine building a testing system on top of this which
       | could support the tree pattern described by Evan here (as long as
       | your tests don't themselves need to test transaction-related
       | behavior).
       | 
       | Since ChatGPT Code Interpreter works with o3-mini now I had that
       | knock up a very quick proof of concept using Python and SQLite
       | SAVEPOINT, which appears to work:
       | https://chatgpt.com/share/67d3621a-14ac-8006-b762-e2f006e27e...
        
       ___________________________________________________________________
       (page generated 2025-03-13 23:00 UTC)