[HN Gopher] SMURF: Beyond the Test Pyramid
___________________________________________________________________
SMURF: Beyond the Test Pyramid
Author : BerislavLopac
Score : 64 points
Date : 2024-10-19 20:22 UTC (1 days ago)
(HTM) web link (testing.googleblog.com)
(TXT) w3m dump (testing.googleblog.com)
| pydry wrote:
| They should at least admit they made a mistake with the "test
| pyramid".
| codeflo wrote:
| Not the least of which is that people who take stuff like that
| as gospel instead of as heuristics tend to endlessly argue what
| is and isn't a "unit".
| petesergeant wrote:
| > who take stuff like that as gospel
|
| In 2011 I wrote a blog-post[0] about the dangers of taking
| TDD too literally, and until I disabled comments on it ~5
| years later people were still angrily shaking their fists at
| each other online. 139 angry comments for a very short post.
|
| 0: https://writemoretests.com/2011/09/test-driven-
| development-g...
| pydry wrote:
| It's not even good as a heuristic. It's simply a bad idea
| which only got off the ground because it came from the Google
| testing blog.
| candiddevmike wrote:
| Conjoined triangles of testing
| petesergeant wrote:
| I feel like "testing trophy" has been in vogue for a while now,
| and definitely feels more right as someone who's made a career
| of unfucking test suites, but there's almost no area of
| software engineering as involved in navel-gazing as testing.
| grahamj wrote:
| This is all pretty obvious
| sverhagen wrote:
| And yet, people focus endlessly on unit tests, pyramid in hand,
| saying things like: they're the best kind of test, or at least
| better than integration tests. It needs some maturity to
| articulate the nuances. And while I've tried, I think SMURF may
| be a good aid in that. While I've moved away from the religious
| focus on unit tests, long ago, I appreciate learning about
| SMURF today.
| grahamj wrote:
| That's fair. It's easy to get philosophical about such things
| so something you can point to that's more based on metrics
| can help a discussion be more objective.
|
| otoh countering opinion with fact doesn't always work well -
| it might just turn into quibbling over where on each axis
| different test types' strengths lie ;)
| sverhagen wrote:
| I think that a lot of folks have never heard anything else
| but the testing pyramid, repeated over and over. I find
| them often very open to other ideas, in my case I've
| previously heard about the "testing trophy", and found
| willing audiences.
| usbsea wrote:
| This is obvious, as anoter commenter said, but this is
| nonetheless useful.
|
| You can use it to show graduates. Why have them waste time
| relearning the same mistakes. You probably need a longer blog
| post with examples.
|
| It is useful as a check list, so you can pause when working
| earlier in the lifecycle to consider these things.
|
| I think there is power in explaining out the obvious. Sometimes
| experienced people miss it!
|
| The diagram can be condensed by saying SMUR + F = 1. IN other
| words you can slide towards Fidelity, or towards "Nice
| Testibility" which covers the SMUR properties.
|
| However it is more complex!
|
| Let's say you have a unit test for a parser within your code. For
| a parser a unit test might have pretty much the same fidelity as
| an intergation test (running the parse from a unit test, rather
| than say doing a compilation from something like Replit online).
| But the unit test has all the other properties be the same in
| this instance.
|
| Another point is you are not testing anything if you have zero
| e2e tests. You get a lot (a 99-1 not 80-20) by having some e2e
| tests, then soon the other type of tests almost always make
| sense. In addition e2e tests if well written and considers can
| also be run in production as synthetics.
| candiddevmike wrote:
| What's useful here? There's nothing actionable, no way to
| quantify if you're doing "SMURF" correctly. All the article
| describes is semi-obvious desirable qualities of a test suite.
| viraptor wrote:
| You're not "doing SMURF". It's not an approach or a system.
| It's just a specific vocabulary to talk about testing
| approaches better. They almost spell it out: "The SMURF
| mnemonic is an easy way to remember the tradeoffs to consider
| when balancing your test suite".
|
| It's up to your team (and really always has been) to decide
| what works best for that project. You get to talk about
| tradeoffs and what's worth doing.
| stoperaticless wrote:
| > What's useful here?
|
| It is up to the reader to figure out this one.
| imiric wrote:
| This is interesting, but I see a few issues with it:
|
| - Maintainability is difficult to quantify, and often subjective.
| It's also easy to fall into a trap of overoptimizing or DRYing
| test code in the pursuit of improving maintainability, and
| actually end up doing the opposite. Striking a balance is
| important in this case, which takes many years of experience to
| get a feel for.
|
| - I interpret the chart to mean that unit tests have high
| maintainability, i.e. it's a good thing, when that is often not
| the case. If anything, unit tests are the most brittle and
| susceptible to low-level changes. This is good since they're your
| first safety net, but it also means that you spend a lot of time
| changing them. Considering you should have many unit tests, a lot
| of maintenance work is spent on them.
|
| I see the reverse for E2E tests as well. They're easier to
| maintain, since typically the high-level interfaces don't change
| as often, and you have fewer of them.
|
| But most importantly, I don't see how these definitions help me
| write better tests, or choose what to focus on. We all know that
| using fewer resources is better, but that will depend on what
| you're testing. Nobody likes flaky tests, but telling me that
| unit tests are more reliable than integration tests won't help me
| write better code.
|
| What I would like to see instead are concrete suggestions on how
| to improve each of these categories, regardless of the test type.
| For example, not relying on time or sleeping in tests is always
| good to minimize flakiness. Similarly for relying on system
| resources like the disk or network; that should be done almost
| exclusively by E2E and integration tests, and avoided (mocked) in
| unit tests. There should also be more discussion about what it
| takes to make code testable to begin with. TDD helps with this,
| but you don't need to practice it to the letter if you keep some
| design principles in mind while you're writing code that will
| make it easier to test later.
|
| I've seen many attempts at displacing the traditional test
| pyramid over the years, but so far it's been the most effective
| guiding tool in all projects I've worked on. The struggle that
| most projects experience with tests stems primarily from not
| following its basic principles.
| stoperaticless wrote:
| > I don't see how these definitions help me write better tests
|
| > What I would like to see instead ...
|
| If you hire me, I can address those needs.
| joshuamorton wrote:
| > If anything, unit tests are the most brittle and susceptible
| to low-level changes.
|
| I don't find this to be the case if the unit tests are precise
| (which they should be).
|
| That is, if you are writing non-flaky unit tests which do all
| the "right" unit-testy things (using fakes/dependency injecting
| well and so isolating and testing _only_ the unit under test),
| you should end up with a set of tests that
|
| - Fails _only_ when you change the file /component the test
| relates to
|
| - Isn't flaky (can be run ~10000 times without failing)
|
| - Is quick (you can do the 10000 run loop above approximately
| interactively, in a few minutes, by running in parallel
| saturating a beefy workstation)
|
| This compares to integration/e2e tests which inherently break
| due to other systems and unrelated assumptions changing
| (sometimes legitimate, sometimes not), and can have rates of
| flakyness of 1-10% due to the inherent nature of "real" systems
| failing to start occasionally and the inherently longer test-
| debug cycle that makes fixing issues more painful (root causing
| bug that causes a test to fail 1% of the time is much easier
| when the test takes .3 CPU-seconds than when it takes 30 or 300
| CPU-seconds).
|
| Very few tests I see are _actually_ unit tests in the above
| sense, many people only write integration tests because the
| code under test is structured in inherently un- or difficult-
| to test ways.
| imiric wrote:
| That's true. I probably used brittle in the wrong sense
| there.
|
| What I mean is that after any code change that isn't a strict
| refactoring you will inevitably have to touch one or more
| unit tests. If you're adding new functionality, you need to
| test different code paths; if you change existing
| functionality, you need to update the related test(s); if
| you're fixing a bug, you need to add a unit test that
| reproduces it, and so on. All this means is that unit tests
| take the most effort to maintain, so I'm not sure why the
| article claims they have "high" maintainability, or that
| that's a good thing. In contrast, higher-level tests usually
| require less maintenance, assuming they're stable and not
| flaky, since you're not touching them as often.
|
| > Very few tests I see are actually unit tests in the above
| sense, many people only write integration tests because the
| code under test is structured in inherently un- or difficult-
| to test ways.
|
| Very true. I think the recent popularization of alternative
| guidelines like The Testing Trophy is precisely because
| developers are too lazy to properly structure their code to
| make pure unit testing possible, and see the work of
| maintaining unit tests as too much of a chore, so they make
| up arguments that there's no value in them. This couldn't be
| farther from the truth, and is IMO an irresponsible way to
| approach software development.
| js8 wrote:
| > There should also be more discussion about what it takes to
| make code testable to begin with.
|
| IME, testable pretty much just means referentially transparent.
| imiric wrote:
| Sure. But how do you achieve that in practice? All functions
| can't be pure, and at some point you need to handle messy I/O
| and deal with nondeterminism. How you structure your code in
| the best way to do that, while also ensuring you can write
| useful tests? None of this is trivial.
| js8 wrote:
| After I wrote my comment, I realized I was actually
| somewhat wrong (I have been in this debate, on the anti-TDD
| side, for quite some time now, but it was kind of hard to
| figure out why for me, but I think I finally got it).
| Testability is more than just the referential transparency,
| RT it is a necessary but not sufficient condition for
| testability.
|
| > But how do you achieve that in practice?
|
| The way to achieve referential transparency is, in my mind,
| functional programming. Specifically, use monads to model
| side effects. You can model any computer system as a
| composition of lambda terms exchanging data (specific
| subset of terms) monadically, so in theory, this can be
| achieved. So you can imagine any program as a tree of
| functions, each builds a more complex function as a
| composition of smaller, simpler functions, until the whole
| program is put together in the main() function.
|
| However, I need to add two other conditions that for a
| program to be testable: Each function to which you
| decompose your program has to (2) be reasonably short (has
| to have limited number of compositions) and (3) has to have
| a clear specification, based on which it can be determined,
| whether the function in itself is correct. The condition
| (2) is strictly speaking not required, but because we are
| humans with limited ability to understand, we want it to
| help us create (3).
|
| Now I believe that the more you have RT and (3), your
| program is more testable. This is because that testing is
| pretty much just partial type-checking by sampling - you
| create some sample values of the type you expect, and you
| verify that the program produces expected values. The
| advantage of sampling is that you don't have to formally
| specify your types, so you don't need complete formal
| specification. The conditions RT and (3) are pretty much
| necessary if you want to properly type your programs (for
| example, we could specify every function using a type in
| dependent type theory).
|
| So testable (is a spectrum) really means "close to type-
| checkable" (which is a binary). I however need to address a
| misconception (which I had), the types that we assign to
| functions (i.e. the specification) do not come from the
| program itself, but rather from the domain knowledge, which
| we expect to impart into the program. Literally, types are
| (or can be) the specification.
|
| And by the way, the condition (2) determines how small are
| the units of the program you can be testing.
|
| Now after the above setup, let me get to the main point,
| which I will call testability tradeoff: The conditions (2)
| and (3) are mutually exclusive for some programs, i.e.
| there is a tradeoff between (2) making units small and (3)
| giving them a good (easy to interpret) specification.
|
| Let me give some extreme examples of testability tradeoffs
| for different programs to illustrate this concept. Library
| of functions has usually only little testability tradeoff,
| because most functions are independent of each other, so
| each of them (on the API level) can satisfy both (2) and
| (3). On the other end of spectrum you have things like a
| trained neural network or program that implements a tax
| code - even if you can decompose those programs into little
| pieces to satisfy condition (2), it is not possible to then
| assign these pieces a meaning sensible enough to construct
| useful tests per condition (3). Such programs are simply
| not understandable in detail (or better to say, we don't
| know how to understand them).
|
| The hidden assumption of TDD folks (and proponents of
| massive unit testing, in the test pyramid) is that we can
| always convert program as much to have (2) and (3), i.e. in
| their view, the testability tradeoff can be always made as
| low as needed. But I don't think this is true in practice,
| and I have given examples above - we have complex useful
| programs that cannot be decomposed to little pieces, where
| each of the little pieces can be meaningfully specified in
| the business domain. Such programs, I claim, cannot be
| effectively unit tested, and can only be e2e or integration
| tested. (However, they can be type-checked against the full
| specification.)
|
| So because (as stated above) testability of a program is
| pretty much your ability to meaningfully assign (expected)
| types and typecheck, now I think that TDD proponents, when
| they talk about testability, want to have as much
| specification as possible. Which is kind of funny, because
| they started as a kind of opposition to that idea. Ah well,
| paradoxes of life..
|
| Anyway, I know my response is a bit messy, but hopefully I
| explained the main idea enough so it will make more sense
| on rereading.
| sverhagen wrote:
| This model ("mnemonic") feels like a good tool to reason about
| your testing strategy. I ran across the "testing trophy" in the
| past, which really changed my thinking already, having been
| indoctrinated with the testing pyramid for such a long time
| before that. I wanted to share my favorite links about the
| testing trophy, for those interested:
|
| https://tanzu.vmware.com/content/videos/tanzu-tv-springoneto...
|
| https://kentcdodds.com/blog/the-testing-trophy-and-testing-c...
| jbjbjbjb wrote:
| Not to be pedantic, but practically speaking it looks like there
| are two dimensions: fidelity and then the rest (the SMUR).
| eiathom wrote:
| It always amazes me how speed and testing are placed in the same
| bracket. I want solid verification, and a strong pattern to
| repeat verification no matter what. This then allows for fast
| implementation. So then something involving integrating a number
| of components as possible reliably makes the most sense
| (verification wise): I want to verify value early. It is eyebrow
| raising this pyrmaid nonsense has hung around.
| stoperaticless wrote:
| > I want solid verification, and a strong pattern to repeat
| verification no matter what.
|
| Everybody has a time budget for test execution time. I doubt
| that you will wait one year for test suite to finish.
|
| It is feasible to test all possible inputs for "int x+ int y",
| but it is not feasible to test all inputs for usual GUI
| application.
|
| Trade offs have to be recognised, some balance must be struck.
|
| Speed is one of the factors that has to be considered.
| shepherdjerred wrote:
| You don't have to consider every possibility but you can
| randomly search the tree of every possible state.
| stoperaticless wrote:
| One needs expected result defined for each of those. I
| doubt that is feasible.
| shepherdjerred wrote:
| You'd define invariants that must be met. This has been
| done before.
|
| https://en.wikipedia.org/wiki/Search-
| based_software_engineer...
|
| e.g. testing implementations of Paxos:
| https://github.com/emichael/dslabs
|
| At Facebook:
| https://engineering.fb.com/2018/05/02/developer-
| tools/sapien...
|
| As an example, your search graph might define your nodes
| as possible application states and the edges as actions
| to transition to a new state.
|
| Some edges might be "create a post" and "change post
| privacy settings". You might then define an invariant
| saying "after a user creates a post the new post is in
| their follower's timeline if privacy settings allow".
|
| Your search might eventually come across some set of
| actions that lead to this invariant being violated which
| is a discovered bug, e.g. if a user A creates a post ->
| user A changes settings to private -> user B can still
| see the post.
|
| Since the search space is infinite, the effectiveness is
| heavily influenced by how much time you can spend
| searching the tree & how fast you can evaluate states.
| This is a kind of exploratory test & wouldn't replace
| unit/integration tests, but it does let you catch certain
| kinds of bugs that would be impossible to find otherwise.
| the_af wrote:
| When the test suite is too slow, it becomes unwieldy, it gets
| in the way (instead of being part of a valuable feedback loop)
| and everyone starts looking for shortcuts to avoid running it.
| wpietri wrote:
| For sure. And I'd add that the specificity of unit tests is
| hugely valuable. If I do some sort of refactoring and found
| I've broken an end-to-end test, it may tell me very little
| about where the problem is. But the pattern of unit test
| breakage is very informative. Often just seeing the list of
| broken tests is enough to tell me what the issue is without
| even having to look at an error message.
| nemetroid wrote:
| End-to-end tests verify high-level expectations based on the
| specification of the system. These high-level expectations
| generally stay stable over time (at least compared to the
| implementation details verified by lower-level tests), and
| therefore end-to-end tests should have the best maintainability
| score.
| wpietri wrote:
| The expectations can be pretty stable, but because they cover
| so much of the system, they tend to be more fragile. End to end
| tests are also often flakier because they're dealing with the
| system at levels meant for human interaction, like by driving
| actual browsers. Because they encompass so much, they're also
| the slowest. When you have a problem with an end-to-end test,
| they can take way more time to debug because of that.
|
| So I'd agree with them; E2E tests are the hardest to maintain.
| bastawhiz wrote:
| > end-to-end tests should have the best maintainability score.
|
| End to end tests encompass far more total details than
| implementation or unit tests. If you're testing a website,
| moving a button breaks a test. Making the button have a
| confirmation breaks the test. The database being slower breaks
| the tests. The number of items in a paginated list changing
| breaks the tests. You're testing not just the behavior and
| output of interfaces, you're testing how they're composed. The
| marketing team putting a banner in the wrong place breaks the
| tests. The product team putting a new user tour popover on the
| wrong button breaks the tests. The support team enabling the
| knowledge base integration in your on-page support widget
| breaks the tests.
|
| Moreover, the cost of fixing the tests is also often higher,
| because end to end tests are necessarily slower and more
| complicated. Tests often become flaky because of a larger
| number of dependencies on external systems. It's often not
| clear _why_ a test is failing, because the test can 't possibly
| explain why its assertion is no longer true ("The button isn't
| there" vs "The button now has a slightly different label").
| nemetroid wrote:
| Not all systems are web sites. Those issues are not inherent
| to the concept of end-to-end testing.
| bastawhiz wrote:
| It still illustrates how E2E testing encompasses far more
| details than other forms of testing, which necessarily
| makes them more brittle
| nemetroid wrote:
| Sure, they often are less reliable, which is the R of the
| smurf and not the M.
| shepherdjerred wrote:
| I have found testcontainers to be an excellent way to write
| integration/end-to-end tests as easily as unit tests.
|
| It takes care of the chore of setting up test environments,
| though it won't solve all of your problems.
|
| I took this approach when testing an application at my last
| workplace. It made writing tests significantly easier, and, IMO,
| fun.
|
| https://testcontainers.com/
| DanHulton wrote:
| I _really_ have to dispute the idea that unit tests score the
| maximum on maintainability. The fact that they are _so_ tightly
| tied to lower-level code makes your code _miserable_ to maintain.
| Anyone who's ever had to work on a system that had copious unit
| tests deep within will know the pain of not just changing code to
| fix a bug, but having to change a half-dozen tests because your
| function interfaces have now changed and a healthy selection of
| your tests refuse to run anymore.
|
| The "test diamond" has been what I've been working with for a
| long while now, and I find I greatly prefer it. A few E2E tests
| to ensure critical system functionality works, a whole whack of
| integration tests at the boundaries of your services/modules
| (which should have well-defined interfaces that are unlikely to
| change frequently when making fixes), and a handful of unit tests
| for things that are Very Important or just difficult or really
| slow to test at the integration level.
|
| This helps keep your test suite size from running away on you
| (unit tests may be fast, but if you work somewhere that has a
| fetish for them, it can still take forever to run a few
| thousand), ensures you have good coverage, and helps reinforce
| good practices around planning and documentation of your
| system/module interfaces and boundaries.
| clhodapp wrote:
| That entirely gels with my experience as well.
|
| The testing-type divide feels similar to the schism around
| ORMs, where one camp (mine) find that ORMs end up costing far
| more than the value they bring, while the other claim they've
| never had such issues and they would never give up the
| productivity of their favorite ORM.
|
| Both sides appear to be describing their experiences
| accurately, even though it feels like one side or the other
| should have to be definitively right.
| geodel wrote:
| I feel exactly same for unit tests and ORMs. I'd like to what
| kind business applications these two items really delivered.
| In my experience they both worked in most trivial sense.
| Either projects were mostly POCs and no one really cared
| about production issues. Or they were so overstaffed that
| team can continue to support to provide daily work to devs.
| lojack wrote:
| > Anyone who's ever had to work on a system that had copious
| unit tests deep within will know the pain of not just changing
| code to fix a bug, but having to change a half-dozen tests
| because your function interfaces have now changed and a healthy
| selection of your tests refuse to run anymore.
|
| In my experience this problem tends to be caused by heavily
| mocking things out more so than the unit tests themselves.
| Mocking things out can be a useful tool with its own set of
| downsides, but should not be treated as a requirement for unit
| tests. Tight coupling in your codebase can also cause this, but
| in that case I would say the unit tests are highlighting a
| problem and not themselves a problem.
|
| Perhaps you're talking about some other aspect of unit tests?
| If that's the case then I'd love to hear more.
| MrJohz wrote:
| I'd also add to this that people often end up with very
| different ideas of what a unit test is, which confuses things
| further. I've seen people who write separate tests for each
| function in their codebase, with the idea that each function
| is technically a unit that needs to be tested, and that's a
| sure-fire way to run into tightly-coupled tests.
|
| In my experience, the better approach is to step back and
| find the longer-living units that are going to remain
| consistent across the whole codebase. For example, I might
| have written a `File` class that itself uses a few different
| classes, methods, and functions in its implementation - a
| `Stats` class for the mtime, ctime, etc values; a
| `FileBuilder` class for choosing options when opening the
| file, etc. If all of that implementation is only used in the
| `File` class, then I can write my tests only at the `File`
| level and treat the rest kind of like implementation details.
|
| It may be that it's difficult to test these implementation
| details just from the `File` level - to me that's usually a
| sign that my abstraction isn't working very well and I need
| to fix it. Maybe the difficult-to-test part should actually
| be a dependency of the class that gets injected in, or maybe
| I've chosen the wrong abstraction level and I need to
| rearchitect things to expose the difficult-to-test part more
| cleanly. But the goal here isn't to create an architecture so
| that the tests are possible, the goal is to create an
| architecture that's well-modularised, and these systems are
| usually easier to test as well.
|
| There's an argument that this isn't a unit test any more -
| it's an integration test, because it's testing that the
| different parts of the `File` class's implementation work
| together properly. My gut feeling is that the distinction
| between unit and integration is useless, and trying to decide
| whether this is one or the other is a pointless endeavour. I
| am testing a unit either way. Whether that unit calls other
| units internally should be an implementation detail to my
| tests. Hell, it's an implementation detail whether or not the
| unit connects to a real database or uses a real filesystem or
| whatever - as long as I can test the entirety of the unit in
| a self-contained way, I've got something that I can treat
| like a unit test.
| chipdart wrote:
| > I _really_ have to dispute the idea that unit tests score the
| maximum on maintainability. The fact that they are _so_ tightly
| tied to lower-level code makes your code _miserable_ to
| maintain.
|
| I disagree to the point I argue the exact opposite. Unit tests
| are undoubtedly the ones that score the highest on
| maintainability. They are unit tests after all, meaning they
| are self-contained and cover the behavior of individual units
| that are tested in complete isolation.
|
| If you add a component, you add tests. If you remove a
| component, you delete it's tests. If you fix a bug in a
| component, you both add one of more tests that reproduce the
| bug and assert the expected output and use all existing tests
| to verify your fix doesn't introduce a regression. Easy.
|
| Above all, unit tests serve as your project's documentation of
| expected behavior and intent. I can't count the times where I
| spotted the root cause of a bug in a legacy project just by
| checking related unit tests.
|
| > (...) a whole whack of integration tests at the boundaries of
| your services/modules (which should have well-defined
| interfaces that are unlikely to change frequently when making
| fixes), and a handful of unit tests for things that are Very
| Important or just difficult or really slow to test at the
| integration level.
|
| If it works for you then it's perfectly ok. To me, the need for
| "a whole whack of integration tests" only arises if you failed
| to put together a decent coverage of unit tests. You specify
| interfaces at the unit test level, and it's at the unit test
| level where you verify those invariants. If you somehow decided
| to dump that responsibility on integration tests then you're
| just replacing many fast-running targeted tests that pinpoint
| failures with many slow-rumming broad-scope tests that you need
| to analyze to figure out the root cause. On top of that, you
| are not adding checks for these invariants in the tests you
| must change if you mush change the interface. Those who pretend
| this is extra maintainability needs from unit tests are
| completely missing the point and creating their own problems.
| s6af7ygt wrote:
| I think the whole problem is just terminology. For example
| take your comment. You start talking about unit tests and
| units, but then suddenly we're talking about components. Are
| they synonymous to units? Are they a higher level, or a lower
| level concept?
|
| People have such varying ideas about what "unit" means. For
| some it's a function, for others it's a class, for others yet
| it's a package or module. So talking about "unit" and "unit
| test" without specifying your own definition of "unit" is
| pointless, because there will only be misunderstandings.
| thiht wrote:
| At one of my previous jobs we did very few unit tests
| (basically only pure functions) and tons of
| behavior/integration tests (ie. run the service with a real
| database, real queues, etc. but mock the HTTP dependencies,
| call its API and check we get the correct result and side
| effects) and it was the most stable and easy to work with test
| suite I've ever seen. It was extremely reliable too.
|
| Not mocking the database and other pipes is the single best
| improvement everyone can make on their test suites.
|
| We also had a test suite that followed the same principle but
| started all the services together to reduce the mocked surface,
| it executed every hour and was both incredibly useful and
| reliable too.
|
| The test pyramid is false wisdom.
| Vadim_samokhin wrote:
| I think test pyramid is a great idea, in theory. In practice,
| having lots of unit tests with mocked dependencies doesn't make
| me sure that everything works as it should. Thus, I use real
| database in my unit tests. There, I test serialization errors,
| authentication problems, network issues, etc. All the real
| problems which can occur in real life. Leaving these scenarios
| for integration tests layer will turn a test pyramid to a test
| diamond.
|
| And what was the rationale behind mocking a database in the first
| place, speed? Disable synchronous wal writes, or run your
| postgres instance in ram. Your test suite execution speed will
| skyrocket.
| satisfice wrote:
| This entire heuristic is not even about testing. The people who
| created it aren't interested in testing-- they want an excuse to
| release their shitty products.
|
| They believe that experiencing a product is just an afterthought.
| They are like chefs who never taste the food they cook.
|
| Testing is a process of investigation and learning. What this
| post covers is mechanical fact checking.
___________________________________________________________________
(page generated 2024-10-20 23:02 UTC)