[HN Gopher] Test Smarter, Not Harder: Focus on Outcomes, Not Out...
___________________________________________________________________
Test Smarter, Not Harder: Focus on Outcomes, Not Outputs
Author : todsacerdoti
Score : 34 points
Date : 2023-12-31 19:16 UTC (3 hours ago)
(HTM) web link (markus.oberlehner.net)
(TXT) w3m dump (markus.oberlehner.net)
| gehen88 wrote:
| I was hoping for some interesting insight to be shared, but no
| it's just a shallow opinion piece on why E2E is better than unit
| tests for web interfaces, without really explaining why.
|
| The author misses that component tests don't have to be like unit
| tests at all, they are more like integration tests if you test
| entire pages (which are components too), use actual user
| interactions and mock only the I/O (mostly network) layer. It's
| like E2E without spinning up a backend.
| zer00eyz wrote:
| As someone advocates for and runs E2E tests, you're going to be
| hard pressed to find any one writing about it.
|
| The main objection heard will be: "It takes x hours and n
| servers to set up an E2E environment."
|
| Infrastructure is hard, and with cloud functionally unlimited,
| so people write a mock and move on with their life.
|
| To do E2E well your culture needs to support it. You need to be
| able to spin up, run and summarize an E2E test in less than 20
| minutes. Set your test up to END on the hour (not start, end)
| SO that if there is an outage no one can make the excuse that
| the "meeting already started". If the test fails every one who
| has changes in that build drops what they are doing and treats
| it like a production outage. The technological magpies,
| collecting shiny things and adding them to the stack has to
| stop, because you dont want to add "new tech" to build time.
| You may need to run some of your own infra to get this done.
| Some servers in a closet some where will do wonders for your
| bills.
|
| Because you're always building your environment from scratch,
| in the event of a real disaster you know what it takes to put a
| large chunk of your environment back together. Your like going
| to have a pretty lean stack, one that scales diagonally (at
| this point go look at AWS spot pricing and how much you can
| save if you can scale that way).
|
| That isn't to say you should not unit test. Things that are
| clearly stand alone, like well defined validation, things that
| are ugly (that 20 condition switch statement that powers some
| esoteric edge case of business "logic") should get unit tests.
| Places where you can leverage fuzzing easily should have unit
| tests! Anything that has a dependency tree or needs a mock,
| should be skipped and addressed with an end to end test.
|
| What will you get in the end? A production environment with
| every bit of unneeded complexity stripped from it (and probably
| a lower cloud bill). An aversion to adding anything that
| doesn't have VALUE to your stack. A ruthless eye for what
| qualifies as "bloat" (hint, a LOT of it is bloat). The removal
| of code for codes sake (mocks) and layers of duplication (an
| E2E create user test can exercise a lot of code, and replace a
| lot of unit tests at various levels). The whole team having a
| much better understanding of how your infrastructure is built
| and deployed.
| somewhereoutth wrote:
| I've said it before, but I'll say it again:
|
| If the system is used by humans, then it needs to be tested by
| humans. Ideally but not necessarily _not_ the customers.
|
| If it is used by just machines then machine testing is fine.
|
| (Per the article, unit tests are clown coding really - nobody
| cares)
| kvdveer wrote:
| Does this logic extend to /parts of software/? Most lines of
| code are not UIs, so they are not used by humans, meaning they
| should be tested by machines?
|
| Do we consider webbrowsers to be humans here?
| vegetablepotpie wrote:
| I shutter thinking my boss _could_ read that article because unit
| testing is already viewed as _extra work_ that is unnecessary.
|
| It's apparent that the author has never worked on a legacy
| product.
|
| End-to-end testing works great until your project grows in
| complexity and something doesn't work right. Then you play the
| game of looking for _where_ and _why_ while your boss is
| frustratingly asking you how long it's going to take while
| telling all the other bosses that you're incompetent. Worst case,
| you break something, and you don't realize it, or your coworkers
| break something you wrote and they don't realize it.
|
| Unit tests _can_ protect you from those situations.
| drpossum wrote:
| Consider finding a new boss
| ysavir wrote:
| E2E is not a replacement for unit tests. E2E tests are more
| difficult to write, as they can require a lot of setup to get all
| the exact scenario needed for each piece of functionality we need
| to test, they tend to be flaky, they can be very slow, etc.
|
| The purpose of unit testing is to isolate the various scenarios
| by making sure that each unit/component/whatever correctly
| responds to its various inputs. In this way, we only need to set
| up the relevant scenario for that unit. If we tested each of
| these piece of functionality in an E2E test, then each test would
| also need to set up the user session, settings, etc. It would be
| a nightmare to author and maintain, compared to the simple,
| straight forward unit tests.
|
| In contrast, E2E tests are good at making sure that the whole
| system works well together, and that each unit is getting inputs
| and providing outputs, and that the page doesn't break in the
| process. Both are needed to maintain confidence, and each solves
| a different problem.
| cmckn wrote:
| Agreed. An e2e test suite can continue passing with significant
| bug rot throughout the system, as they're often given generous
| timeouts, retries for flakes, etc. So once you realize
| something is really, truly broken; it's going to be a nightmare
| to track down and properly fix if the e2e suite is your only
| data point.
| hinkley wrote:
| I have seen projects where people push back on making major
| functional changes and the only rational explanation is that
| they don't want to have to rewrite a bunch of brittle E2E
| tests.
|
| That really took the value of E2E down several notches in my
| estimation.
| TuringNYC wrote:
| >> I have seen projects where people push back on making
| major functional changes and the only rational explanation is
| that they don't want to have to rewrite a bunch of brittle
| E2E tests.
|
| Same for any testing. it should be part of the cost and Cost
| Benefit Analysis.
| zer00eyz wrote:
| > they tend to be flaky
|
| If your E2E test is flaky, then how is your production
| environment reliable?
|
| > they can require a lot of setup
|
| Setup vs mock.
|
| > they can be very slow, etc.
|
| We have a good handle on scaling, slow is now a product of the
| testing time being smeared across your org like peanut butter.
|
| > The purpose of unit testing...
|
| Does E2E replace unit testing, no. Things that are already
| units or esoteric (business "logic", you know it's in your code
| base). But if your writing mocks your better off un fucking
| your environment that E2E testing is viable, and make it a
| first class citizen. Everything is a crud app of some sort:
| web, api, mobile, smoke signal... Dropping data in these front
| ends, can be validated out of the back end, and in the app
| itself.
|
| To put a fine point on it: Unit tests make me feel good about
| checking in. End to end tests make me confident to deploy!
| TuringNYC wrote:
| >> If your E2E test is flaky, then how is your production
| environment reliable?
|
| IME, E2E tests are flaky despite the underlying system not
| being flaky because E2E has to do a series of things, often
| including waiting/polling, and in real life people will wait
| longer or compensate but in tests you want to do it all
| perfectly and fast.
| colinmorelli wrote:
| I think in this scenario, different people will have a
| different perspective on whether asking people to "wait
| longer" constitutes flakiness or not.
|
| From an end-user's perspective, there's very little
| difference between "the task didn't work" and "the task
| worked, but took longer than your patience to wait." Both
| appear to be broken.
| ysavir wrote:
| The end user isn't exposed to it.
|
| The flakiness comes from needing an automated browser
| interaction acting against a page with changing input. If
| your page is static, sure, the E2E test is probably
| straightforward and quick. But the moment any dynamic
| functionality is introduced to the page, the test's
| interactions need to account for the page's current
| context (or wait for an intended context). And that has
| to be coordinated amongst the actual test code, the
| testing browser, the driver, etc. If anyone one of them
| hiccups, you may end with a false negative, and those
| happen more often than we'd like.
| wouldbecouldbe wrote:
| Yeah you have never wrote end to end tests or run them only
| in utopia.
|
| Whether you use playwright or cypress getting the correct
| wait events figured out is a huge pain.
|
| Only if you run pure server side applications that's simple.
| gkbrk wrote:
| It's almost like server-side applications are more
| reliable.
| zer00eyz wrote:
| >> Only if you run pure server side applications that's
| simple.
|
| End to End testing is less about "making the front end do
| all the work" and more about being free of mocks and
| synthetics.
|
| Just because the front end exercises my server, does not
| mean server devs get a pass on testing API down. The tests
| I write should inform the data your sending and receiving,
| they should function as an example of inputs and outputs of
| what to expect in the system. Those timing issues are
| either a real front end problem, or a server problem (and
| my own tests should have noted that I did not meet the
| SLA).
| corethree wrote:
| No but if choosing one or the other E2E is the far better
| choice. Ideally you have both.
| epicureanideal wrote:
| I was hoping this had to do with school/university tests!
| drewcoo wrote:
| > it's about building better software faster
|
| Great. Go over in that corner and argue with the other testers
| and some managers about defining "better" while the rest of us
| get some work done.
|
| > strong emphasis on unit tests, and while they have their place,
| let's not forget about a category of tests that I consider much
| more valuable when it comes to testing complex web application
| user interfaces: End-to-End
|
| First, testing UI doesn't mean the tests have to be End-to-
| End(E2E).
|
| Second, I have the opposite experience, where teams seem to want
| test devs to write slow, brittle E2E tests that require
| maintaining fragile environments that often need to cross-
| coordinate among many repos deploying to it (plus any manual
| changes that always seem to happen).
|
| I see very little emphasis on unit tests. And I see almost none
| on contract tests, the fast way to break E2E tests into grok-able
| pieces that can run on every push (just like unit tests).
|
| > let's shift our focus from output to outcome
|
| I don't really see that a focus on E2E tests outcomes. I just see
| that as slow, brittle, high-maintenance, and something requiring
| an environment (thus not running the tests on every push - even
| slower feedback).
|
| Maybe try to focus on fast, meaningful feedback instead?
| Animats wrote:
| This is an ad for some book on Vue.js. The article is just
| generic verbiage to get traffic for the ad.
| cbdumas wrote:
| My first real software job I worked primarily in Python and
| diligently added and maintained unit tests for every code path.
| This effort really paid off as my team and I could be so much
| more certain that changes and refactors wouldn't break things in
| prod.
|
| A few years into that job, gradual typing came to Python and I
| realized that a _substantial_ fraction of the unit tests we were
| writing were in fact just making up for a lack of static analysis
| in Python. Since then I 've come to believe that with decent
| typing and static analysis, unit tests are most useful during
| development of new code and only marginally useful after that.
| Integration tests are much harder to set up initially but are
| much more valuable.
| ericyd wrote:
| E2E tests are useful, but I definitely do not agree that they
| should be the primary target when writing tests. Setting up the
| correct data conditions for an E2E test to accurately exercise
| the code is often challenging or sometimes impossible,
| particularly since E2E tests often execute against a hosted
| environment with shared resources. Unit tests probably often have
| room for improvement, but in my experience they are extremely
| useful for checking code during development before I've produced
| a final product that could be tested E2E.
| Animats wrote:
| I was just looking at the unit tests I have in a game-related
| Rust program. "cargo test" runs these. So what did I test?
|
| - A complicated index of objects in a virtual world. The test
| creates some dummy objects, moves them around, and runs a
| consistency check on the data structures after each change. This
| has caught bugs in the past.
|
| - A lock-by-string primitive, used when I don't want two threads
| working on the same URL.
|
| - Serializers and deserializers, usually where some data comes in
| from the outside. Quoted strings in the test cases deserialize,
| reserialize, and check.
|
| - There are geometry generators which generate spheres, boxes,
| cylinders, etc. Those all have unit tests.
|
| - Some math functions, such as angle between two quaternions, and
| some computations around transformation matrices and the viewing
| frustrum that tell what's visible on screen.
|
| - Some transformations on meshes.
|
| - Some simple parsers.
|
| Mostly things with algorithmic complexity, data format
| dependency, or non-trivial internal consistency. Hard parts that
| the Rust compiler won't catch, and which are hard to debug in a
| larger context.
|
| With those tests in place, and in 100% safe Rust, not much
| mysterious happens that's internal to the program.
|
| External relationships with the game server are another matter.
| That's handled with lots of internal checks and logging.
| eckesicle wrote:
| Of course context matters but _generally_, end to end tests
| should be mostly avoided:
|
| - the execution time of end to end tests are orders of magnitude
| larger than unit tests. In a medium ish application your end to
| end tests will take several seconds to spin up, for example by
| spinning up fresh databases (even with rollback in gets slow) and
| other detachable services. A unit test is usually a malloc away
| and thousands can be run in parallel on your local box in a
| second. End to end tests easily take hours of compute time time
| to run in the real world. Good luck running the suite on your
| laptop. Nothing kills productivity quicker than a test suite that
| runs slowly
|
| - unit tests are also independent of other code so it's easy to
| get some confidence that your code does the right thing by only
| testing the touched files in your diff before running the entire
| suite on CI.
|
| - unit tests encourage good code quality since it's easier to
| test injectable code. E2E instead incentivises copy pasted test
| suites that are thousands of lines of code long.
|
| - in the general case end to end tests are asynchronous and
| require orders of magnitude more effort to run. Asynchronous
| tests are flaky. Flaky builds slow down your velocity.
|
| - E2E tests rot really fast. One change leads to thousands of
| broken E2E tests.
|
| - E2E is not a replacement for monitoring your application. The
| class of bugs you will catch in your E2E suite is a strict subset
| of the class of bugs you will catch with monitoring and metrics.
| Build the latter instead and roll out your code incrementally to
| your users.
|
| - E2E tests are complicated to set up and maintain in ops. Unit
| tests not so much. Somewhat unexpectedly your development
| velocity scales pretty much linearly with stack complexity.
|
| In summary (and again context matters and there aren't any
| absolutes, but ...):
|
| - Don't do automated E2E tests.
|
| - Do unit tests
|
| - Do metrics and monitoring
|
| You have a limited budget, spend it where you get a good bang for
| your buck.
| tired_and_awake wrote:
| Surprised by the responses here.
|
| Unit tests and smoke/E2E are like peanut butter and jelly,
| they're better together. And yeah they're expensive to build but
| it's almost always cheaper than deploying crap and hurting your
| customers.
|
| Why all the drama?
| drpossum wrote:
| I assume they're different from me and write bug-free code that
| never needs to be refactored the first time.
| Sleaker wrote:
| E2E also has the potential of being a cross-team nightmare when
| multiple teams own individual components of a process rather than
| a single team owning the entire tech stack. So focusing on
| outcomes rather than output doesn't necessarily work because the
| teams doing the work are only given responsibility for an output.
| This is mostly an organizational problem but it does trickle down
| to the team level. And yah, it's not 'agile' which I feel like
| this post is just reiterating one of those pillars.
|
| I think this post glosses over the reality that tech workers
| primarily want to make things work correct, and they generally
| would write good tests (whichever kind that means for their
| problem) if they had the time and resources. But the final words
| seem to indicate doing a check on if the test is valuable... But
| that's not really where the problem or difficulty lies. It's
| often an organizational/team barrier (at least in the places I've
| worked) not a tech problem. And the commentary is often 'well
| this test is as good as we can do without engaging with all of
| these known pains'
___________________________________________________________________
(page generated 2023-12-31 23:02 UTC)