[HN Gopher] Multiple assertions are fine in a unit test
___________________________________________________________________
Multiple assertions are fine in a unit test
Author : nsoonhui
Score : 297 points
Date : 2022-11-05 08:56 UTC (14 hours ago)
(HTM) web link (stackoverflow.blog)
(TXT) w3m dump (stackoverflow.blog)
| 8n4vidtmkvmk wrote:
| i need to send this to all my coworkers. i wrote some tests kind
| of similar to this, except i had a bunch of setup code then
| asserted something was true, changed 1 thing and then verified it
| became false so that it was super clear that it was the 1 thing
| that causes it to become false.
|
| they were like no, copy paste the entire setup and change the
| field to false in the new setup.
|
| im like how are you supposed to tell which of the dozen
| conditions triggered it now? you have to diff the 2 tests in your
| head and hope they don't get out of sync? ridiculous
| peheje wrote:
| Is DeleteReservationActuallyDeletes a good test name?
| latch wrote:
| I haven't heard of the single-assertion thing in at least 10
| years, probably 15. In the early 2000s, when I was starting out
| and doing .NET, it used to be something you'd hear in the
| community as a very general guideline, more like "there's
| something to be said about very focused tests, and too many
| assertions might be a smell." At the time, I got the impression
| that the practice had come over from Java and converted from a
| rule to a guideline (hardly the only bad practice that the .NET
| community adopted from Java, but thankfully they largely did move
| the needle forward in most cases).
|
| (I wrote Foundations of Programming for any 2000s .NET developer
| out there!)
|
| To hear this is still a fight people are having...It really makes
| me appreciate the value of having deep experience in multiple
| languages/communities/frameworks. Some people are really stuck in
| the same year of their 10 (or 20, or 30) years of experience.
| ed25519FUUU wrote:
| Like you I was surprised to hear this is a thing or is even
| controversial. Admittedly I've only been programming for about
| 10 years, but I haven't heard (or seen) this come up even one
| time. Every test I've ever seen has usually had multiple
| mutations and assertions, all of them testing the same premise.
| asabla wrote:
| > I wrote Foundations of Programming for any 2000s .NET
| developer out there!
|
| Holy moly! Think I still have your book somewhere. So thank you
| for that.
|
| In my last +10 years of .net development I haven't heard
| anything about single-assertion.
|
| > Some people are really stuck in the same year of their 10 (or
| 20, or 30) years of experience.
|
| I think this has manifested even more with the transition into
| .net core and now .net 5 and beyond. There are so many things
| changing all the time (not that I complain), which can make it
| difficult to pick up what's the current mantra for the language
| and framework.
| batiste wrote:
| It has been a long time I have met somebody stupid enough to push
| this kind of meaningless rules forward.
|
| Test setup is sometimes very complicated AND expensive. Enforcing
| only 1 assertion per test is moronic beyond description.
| SeriousM wrote:
| I keep it simple by using the AAA pattern:
|
| Arrange: whatever you need for the setup. Act: a single line of
| code that is under test. Assert: whatever you want to assert,
| multiple statements allowed and usually required.
|
| We put the 3A in code as comments as boundaries and that works
| more than perfect for the whole team.
|
| Oh and it's readable!
| eddsh1994 wrote:
| These aren't unit tests, they are integration tests (API calls to
| an external system). Integration tests have had multiple
| assertions forever (look at how Cypress works where each test
| step is an assertion), the idea of single assertions are for unit
| tests.
|
| The issue with multiple assertions for unit tests is that they
| hide errors by failing early extending the time to fix (fail at
| Assert 1, fix, re-run, fail at Assert 2, fix...). If you want
| multiple assertions you should have a single assertion that can
| report multiple issues by returning an array of error strings or
| something. The speed increase of multiple assertions vs multiple
| tests is usually tiny, but if you do have a long unit test then
| that's basically the only reason I can think of to use multiple
| assertions.
| hamandcheese wrote:
| The speed at which I can fix bugs is dramatically slower than
| the speed at which I can re-run a unit test.
|
| Even when I do have multiple failing tests, I focus on and fix
| them one test at a time.
| gorjusborg wrote:
| The trouble is that the type of developer that blindly follows
| this sort of shaming/advice is exactly the type that blindly
| followed someone's rule of thumb to use one assertion per test.
|
| There are no hard and fast rules.
|
| I agree with keeping the number of assertions low, but it isn't
| the number that matters. Keeping the number of assertions low
| helps prevent the 'testItWorks()' syndrome.
|
| Oh, it broke, I guess 'it does not work' time to read a 2000 line
| test written 5 years ago.
| 0xmarcin wrote:
| I think it depends on the test. If you are checking that e.g.
| context.currentUser() returns current user name, email, id and
| roles I would probably wrote just 2 tests. One for the user
| attributes and one for checking roles.
|
| jUnit provides a helpful assertAll(...) method that allows us to
| check multiple assertions without stopping and the first failed
| one.
|
| I my tests I often use "thick" asserts like
| assertEqualsWithDiff(dtoA, dtoB), that compare 2 objects as a
| whole and prints property names with values that do not match.
| Not everyone likes this approach, but for me it is the best
| balance between time spend on the test and the benefit that I get
| from it.
| SideburnsOfDoom wrote:
| This is correct - multiple asserts are OK, but there are still
| good guidelines:
|
| A good unit test has the phases Arrange, Act, Assert, end of unit
| test.
|
| You can use multiple assert statements in the "assert" phase, to
| check the specifics of the single logical outcome of the test.
|
| In fact, once I see the same group of asserts used 3 or more
| times, I usually extract a helper method, e.g.
| "AssertCacheIsPopulated" or
| "AssertHttpResponseIsSuccessContainingOrder" these might have
| method bodies that contain multiple assert statements, but the
| question of is this a "single assert" or not, is a matter of
| perspective and not all that important.
|
| The thing to look out for is - does the test both assert that
| e.g. the response is an order, and that the cache is populated?
| Those should likely be separate tests as they are logically
| distinct outcomes.
|
| The test ends after the asserts - You do not follow up the
| asserts with a second action. That should be a different test.
| [deleted]
| BlargMcLarg wrote:
| Out of everything coming out of testing and the pain of testing
| old code, this seems like such a trivial thing to discuss about.
| Then again, automated testing seems like a breeding ground for
| inane discussions wasting more time than picking a less-than-
| ideal solution and moving on.
| dwgebler wrote:
| It's not that there's anything inherently wrong with multiple
| assertions in a unit test, it's that it's often a smell you're
| testing multiple behaviours in a single test - which is the thing
| you should avoid.
| mrhands556 wrote:
| > You may be trying to simulate a 'session' where a client
| performs many steps in order to achieve a goal. As Gerard
| Meszaros writes regarding the test smell, this is appropriate for
| manual tests, but rarely for automated tests.
|
| Integration tests are typically easier to write / maintain and
| thus are more valuable than small unit tests. Don't know why the
| entire premise argues against that.
| msaltz wrote:
| I think two important requirements for good unit tests are that
| 1) If you look at a particular test, you can easily tell exactly
| what it's testing and what the expected behavior is, and 2) You
| are able to easily look at the tests in aggregate and determine
| whether they're covering all the behavior that you want to test.
|
| To that end, I think a better guideline than "only have one
| assertion per test" would be "only test one behavior per test".
| So if you're writing a test for appending an element to a vector,
| it's probably fine to assert that the size increased by one AND
| assert that the last element in the vector is now the element
| that you inserted.
|
| The thing I see people do that's more problematic is to basically
| pile up assertions in a single test, so that the inputs and
| outputs for the behavior become unclear, and you have to keep
| track of the intermediate state of the object being tested in
| your head (assuming they're testing an object). For instance,
| they might use the same vector, which starts out empty, test that
| it's empty; then add an element, then test that its size is one;
| then remove the element, test that the size is 0 again; then
| resize it, etc. I think that's the kind of testing that the "one
| assertion per test" rule was designed to target.
|
| With a vector it's easy enough to track what's going on, but it's
| much harder to see what the discrete behaviors being tested are.
| With a more complex object, tracking the internal state as the
| tests go along can be way more difficult. It's a lot better IMO
| to have a bunch of different tests with clear names for what
| they're testing that properly set up the state in a way that's
| explicit. It's then easier to satisfy the above two requirements
| I listed.
|
| I want to be able to look at a test and know exactly what I don't
| mind if a little bit of code is repeated - you can make functions
| if you need to help with the test set up and tear down.
| jpollock wrote:
| I work to "assert one thing", and the assertions are added to
| help with debugging. I sometimes even assert starting conditions
| if they can at all change.
|
| Without a "assert one thing", what tends to happen is there will
| be cut-and-paste between tests, and the tests overlap in what
| they assert. This means that completely unrelated tests will have
| an assertion failure when the code goes wrong.
|
| When you do a refactor, or change some behavior, you have to
| change _all_ of the tests. Not just the one or two that have the
| thing you're changing as their focus.
|
| Think of tests that over-assert like screenshot or other diff-
| based tests, they are brittle.
| turtleyacht wrote:
| Use mutation testing as well, if it's available for your
| language. This evaluates the quality of your test suite by
| automatically switching up your app logic and running the same
| tests.
|
| Nothing like removing one line, breezing through code review, and
| bringing down production. "But all tests passed!"
| dicroce wrote:
| also, only testing public interfaces is perfectly fine and may
| actually be preferable as it leaves you free to refactor
| internals freely without breaking tests.
|
| tbh unit testing is a balancing act between reaping code quality
| benefits and bogging yourself down with too much testing
| updating.
| hgomersall wrote:
| This is an interesting point IMO. We tend to focus at the API
| level far more, and implicitly test inner functionality (that
| is, the inner functionality must be correct for the outer tests
| to pass). Sometimes testing inner functionality explicitly is
| required when the outer tests are not complete, or when
| behaviour is defined by the inner code. We also as far as
| possible use defensive techniques and extensive type
| constraints (which is a joy in Rust).
|
| I'm constantly thinking where we need to put tests though, and
| in still not fully convinced I get it right. My rule of thumb
| is that each test should map to a specification point, and that
| spec is a necessary documentation line for the test.
| ascendantlogic wrote:
| I've always operated on the principal that each test should be
| testing one logical concept. That usually translates into one
| concrete assertion in the test code itself, but if you need to
| assert multiple concrete conditions to test the logical concept
| it's not a bad thing. At the end of the day tests are just there
| to make you more confident in code changes and they are a tax you
| have to pay for that comfort. However you arrive at "I feel
| comfortable making changes", go with it.
| oytis wrote:
| Who said multiple assertions are wrong in the first place? First
| time I hear about it really
| devit wrote:
| The problem seems to be that assert is implemented as a regular
| function.
|
| It must be implemented as a macro so that the line number and
| assertion expressions are printed, allowing to easily identify
| the failed assertion.
|
| If a language doesn't support such macros and has no ad-hoc
| mechanism for this case, it should not be used, or if it must the
| assert function must take a string parameter identifying the
| assertion.
| jmillikin wrote:
| Some languages allow a stack trace to be obtained in normal
| code, which enables position reporting without macros. Python
| and Go are good examples.
|
| If you know the file and line of the assertion, plus the values
| that are being checked, there's not as much need for a
| stringified version of the expression.
| TeMPOraL wrote:
| > _If you know the file and line of the assertion, plus the
| values that are being checked, there 's not as much need for
| a stringified version of the expression._
|
| It does save time. With the actual condition reproduced, half
| the time I don't even need to check the source of the failed
| test to know what went bad and where to fix it. Consider the
| difference between: FAILED: Expected 1, got 4
| In /src/foo/ApiTest.cpp:123
|
| vs. FAILED: response.status evaluated to 4
| expected: 1 In /src/foo/ApiTest.cpp:123
|
| vs. FAILED: response.status evaluated to
| Response::invalidArg (4) expected:
| Response::noData (1) In /src/foo/ApiTest.cpp:123
|
| This is also why I insist on adding custom matchers and
| printers in Google Test for C++. Without it, 90% of the time
| a failed assertion/expectation just prints "binary objects
| differ" and spews a couple lines of hexadecimal digits.
| Adding a custom printer or matcher takes little work, but
| makes all such failures print meaningful information instead,
| allowing one to just eyeball the problem from test output
| (useful particularly with CI logs).
| jmillikin wrote:
| Ah, this might be a difference of idiom. I would expect
| that assertion to print output like this:
| FAILED: Expected RESPONSE_NO_DATA, got RESPONSE_INVALID_ARG
| In /src/foo/ApiTest.cpp:123
|
| For non-enumerated integers, the output would be something
| like: FAILED: Expected ResponseCode(200),
| got ResponseCode(404) In /src/foo/ApiTest.cpp:123
|
| If values are formatted with some amount of self-
| description, then printing the name of the variable being
| evaluated is not usually informative.
| quickthrower2 wrote:
| Multiple assertions make sense in the case where otherwise you'd
| have to copy most of test A again to make test B
| ImPleadThe5th wrote:
| There is this team of fresher-to-the-field devs at my company who
| insist this is law.
|
| To the point where if they are testing object equality, they
| check each field in its own test.
|
| I have no idea where they got the notion from.
| fnordpiglet wrote:
| Stop writing unit tests and write comprehensive integration tests
| instead. Problem solved.
| cultofmetatron wrote:
| I didn't know you were only supposed to put one assertion per
| unit test. some of mine have 5-10 assertions. why?. because I
| don't have time to write tests all day but I recognize there are
| tests. As long as it catches the bugs before it goes to prod and
| can be modified easily, who cares?
| BlargMcLarg wrote:
| Unfortunately and from experience, far more seem to care about
| the _how_ than the _what_ when it comes to testing. This
| comment section alone is pretty telling: people spending their
| Saturday to write dogmatic rules.
| alkonaut wrote:
| An assert message says what went wrong, and on which code line.
| How on earth does it help to make just one? The arrange part
| might take seconds for a nontrivial test and that would need to
| be duplicated both in code and execution time to make two
| asserts.
|
| If you painstakingly craft a scenario where you create a
| rectangle of a specific expected size why wouldn't it be
| acceptable to assert both the width and height of the the
| rectangle after you have created it?
|
| assert_equal(20, w, ...
|
| assert_equal(10, h, ...
|
| A dogmatic rule would just lead to an objectively worse test
| where you assert an expression containing both width and height
| in a single assert?
|
| assert_true(w == 20 && h == 10,...)
|
| So I can only assume the rule also prohibits any compound/Boolean
| expressions in the asserts then? Otherwise you can just combine
| any number of asserts into one (including mutating state within
| the expression itself to emulate multiple asserts with mutation
| between)!
| flavius29663 wrote:
| That can be considered one logical assertion though. You're
| asserting the size of the rectangle. You can even extract an
| assert helper function AssertRectangleSize(20,10)
| alkonaut wrote:
| Exactly. But if I assert N properties of an object is that
| then one assert logically for any N? At what point does it
| stop?
|
| Applying any rule dogmatically is often bad, and this is no
| exception. The is that we don't like lacking rules.
| Especially it goes to hell when people start adding code
| analysis to enforce it, and then developers start writing
| poor code that passes the analysis.
|
| One assert imo isn't even a good starting point that might
| need occasional exceptions.
| ljm wrote:
| I've seen people take a dogmatic approach to this in Ruby
| without really applying any critical thought, because one
| assertion per test means your test is 'clean'.
|
| The part that is glossed over is that the test suite takes
| several hours to run on your machine, so you delegate it to a
| CI pipeline and then fork out for parallel execution (pun
| intended) and complex layers of caching so your suite takes 15
| minutes rather than 2 and a half hours. It's monumentally
| wasteful and the tests aren't any easier to follow because of
| it.
|
| The suite doesn't have to be that slow, but it's inevitable
| when every single assertion requires application state to be
| rebuilt from scratch, even when no state is expected to change
| between assertions, especially when you're just doing
| assertions like 'assert http status is 201' and 'assert
| response body is someJson'.
| hamandcheese wrote:
| > I've seen people take a dogmatic approach to this in Ruby
|
| I came up in ruby, heard this, and quickly decided it was
| stupid.
| OJFord wrote:
| > I've seen people take a dogmatic approach to this in Ruby
| without really applying any critical thought, because one
| assertion per test means your test is 'clean'.
|
| I can't speak for Ruby, but what I would call 'clean' and
| happily dogmatise is that assertions should come at the end,
| after setup and exercise.
|
| I don't care how many there are, but they come last. I really
| hate tests that look like: setup()
| exercise(but_not_for_the_last_time) assert
| state.currently = blah state.do_other()
| something(state) assert state.now = blegh
|
| And so on. It stinks of an integration test forced into a
| unit testing framework.
|
| I like them to look like: foo =
| FooFactory(prop="whatever") result = do(foo)
| assert result == "bar"
|
| I.e. some setup, something clearly under test, and then the
| assertion(s) checking the result.
| mollerhoj wrote:
| Yes, you got us rubyists there. :-( Its the unfortunate
| result of trying to avoid premature optimization and strive
| for clarity instead. Something thats _usually_ sound advice.
|
| Enginnering decisions have tradeoffs. When the testsuite
| becomes too slow, it might be time to reconsider those
| tradeoffs.
|
| Usually though, I find that to road to fast tests is to
| reduce/remove slow things (almost always some form of IO) not
| to combine 10 small tests into one big.
| ljm wrote:
| I think it's a sound strategy more often than not, it's
| just that RSpec's DSL can make those trade-offs unclear,
| especially if you use Rubocop and follow its default RSpec
| rules.
|
| It just so happens that your tests become IO bound because
| those small tests in aggregate hammer your DB and the
| network purely to set up state. So if you only do it once
| by being more deliberate with your tests, you're in a
| better place.
| TeMPOraL wrote:
| > _So I can only assume the rule also prohibits any compound
| /Boolean expressions in the asserts then? Otherwise you can
| just combine any number of asserts into one_
|
| That's what's bound to happen under that rule. People just
| start writing their complex tests in helper functions, and then
| write
| assert_true(myTotallySimpleAndFocusedTestOn(result))
| jeremyjh wrote:
| Apparently this is yet another "best practice" preached by
| people using crappy tools.
| ImPleadThe5th wrote:
| The way it's been explained to me is that because one assert
| failing blocks the other assertions from running you don't get
| a "full" picture of what went wrong.
|
| So instead of:
|
| - error W doesn't equal 20
|
| Fix that
|
| Run test again
|
| - error H doesn't equal 10
|
| Fix that
|
| Run test again
|
| It's - Error Width doesn't equal 20
|
| - Error Height doesn't equal 10
|
| Fix both
|
| Run test
|
| I think the time savings are negligible though. And it makes
| testing even more tedious, as if people needed any additional
| reasons to avoid writing tests.
| ChrisMarshallNY wrote:
| I always have multiple assertions in my unit tests. I test around
| areas of functionality, as opposed to individual functions. It's
| a bit arbitrary, but there you have it...
|
| I also use Apple's XCTest, which does a _lot_ more than simple
| assertions.
|
| If an assertion is thrown, I seldom take the assertion's word for
| it. I debug trace, and figure out what happened. The assertion is
| just a flag, to tell me where to look.
| willjp wrote:
| I think part of the motivation for one assertion per test comes
| from the fact that you stop getting information from a test as
| soon as one assertion fails.
|
| I think it was a guidance, like the SRP where you should be
| testing one thing in each test case. I also think a growing
| number of assertions might be a sign your unit under test is
| wearing too many responsibilities.
|
| Maybe it's better to say "few assertions, all related to testing
| one thing"
| cratermoon wrote:
| The Go idiom is to use table-driven tests[1]. It's still an
| evolving practice, so you'll see different variants, but the
| essence is that you have a slice of your inputs and expected
| outputs, iterate through the slice, and run the assert(s) on each
| element. var flagtests = []struct { in
| string out string }{ {"%a", "[%a]"},
| {"%-a", "[%-a]"}, {"%+a", "[%+a]"}, //
| additional cases elided {"%-1.2abc", "[%-1.2a]bc"},
| } func TestFlagParser(t *testing.T) { var
| flagprinter flagPrinter for _, tt := range flagtests {
| t.Run(tt.in, func(t *testing.T) { s := Sprintf(tt.in,
| &flagprinter) if s != tt.out {
| t.Errorf("got %q, want %q", s, tt.out) } })
| } }
|
| Sometimes there will be an additional field in the test cases to
| give it a name or description, in which case the assertion will
| look something like: t.Fatalf("%s: expected:
| %v, got: %v", tt.name, tt.out, got)
|
| Another evolving practice is to use a map instead of a slice,
| with the map key being the name or description of the test case.
| This is nice because in Go, order is not specified in iterating
| over a map, so each time the test runs the cases will run in a
| different order, which can reveal any order-dependency in the
| tests.
|
| 1 https://dave.cheney.net/2019/05/07/prefer-table-driven-tests
| skatanski wrote:
| This always rubbed me the wrong way. I think a better way would
| be to ensure the assertions to be readable, as simple as can be.
| The worst case of this being broken was the assertions were done
| in an helper method used in a base class of a test, and
| navigating to it required multiple hops, it took time to build
| the context of the test as well. The only downside of multiple
| assertions is that when the first one fails we won't know if the
| subsequent ones are passing.
| nharada wrote:
| Totally agreed, I even think duplication is totally fine in
| tests if it brings more readability. You should be able to read
| the test to verify its correctness (after all you don't test
| the test code), and multiple helper functions and levels of
| abstraction hinder this goal.
| smhoff256 wrote:
| The reason for one assert pet test is that it forces you to test
| one thing at a time. It sets your mind in the mode of single
| responsibility.
| marcosdumay wrote:
| And what is the gain from testing a single thing at a time?
|
| On a specific case, what is the gain from not testing the pre
| and post-conditions of your important test?
| Jolter wrote:
| I've seen code guidelines where this article's asserts would be
| unacceptable. If I saw this error in a test output, I'd be pretty
| miffed that it doesn't tell me what the expected and actual
| values are. Actual status code: NotFound.
| Expected: True Actual: False
| code_runner wrote:
| I have a bad habit of writing a mondo-super-all-encompassing
| test, and I have to break it up when I realize it's a house of
| cards.
|
| I think the better rule is "just test one thing, don't use a lot
| of overlapping assertions between tests".
|
| Number of assertions, as noted by this post and this comment
| section, is not a smell at all.
| TeMPOraL wrote:
| > _The excellent book xUnit Test Patterns describes a test smell
| named Assertion Roulette. It describes situations where it may be
| difficult to determine exactly which assertion caused a test
| failure._
|
| How is that even possible in the first place?
|
| The entire job of an assertion is to wave a flag saying "here!
| condition failed!". In programming languages and test frameworks
| I worked with, this typically includes providing _at minimum_ the
| expression put in the assertion, _verbatim_ , and precise
| coordinates of the assertion - i.e. name of the source file +
| line number.
|
| I've never seen a case where it would be hard to tell _which
| assertion failed_. On the contrary, the most common problem I see
| is knowing which assertion failed, but not how the code got
| there, because someone helpfully stuffed it into a helper
| function that gets called by other helper functions in the test
| suite, and the testing framework doesn 't report the call stack.
| But it's not that big of a deal anyway; the main problem I have
| with it is that I can't gleam the exact source of failure from CI
| logs, and have to run the thing myself.
| charrondev wrote:
| The majority of the testing I've written have been jest tests
| and PHPUnit tests, and note PHPUnit is my favourite. It's easy
| to built up custom assertions, and all of the in built
| assertions have the ability to provide an additional failure
| message during a failure.
|
| Assertions throw an exception and the test runner catches them
| along with any exceptions thrown by the code in test, marks the
| test as a failure, and reports the given error message and a
| full stack trace.
| 411111111111111 wrote:
| it('fails', async () => { expect(await
| somePromise()).toBe(undefined) expect(await
| someOtherPromise()).toBeDefined() })
|
| No chance figuring out where it failed, it's likely just gonna
| run into a test suite timeout with no line reference or
| anything.
| torginus wrote:
| I remember writing a small .NET test library for that exact
| problem - You could pass in a lambda with a complex condition,
| and it evaluated every piece of the expression separately and
| pretty printed what part of the condition failed.
|
| So essentially you could write
| Assert(()=>width>0 && x + width < screenWidth)
|
| And you would get: Assertion failed:
| x is 1500 width is 600 screenWidth is 1920
|
| It used _Expression <T>_ to do the magic. Amazing debug
| messages. No moralizing required.
|
| This was a huge boon for us as it was a legacy codebase and we
| ran tens of thousands of automated tests and it was really
| difficult to figure out why they failed.
| jmount wrote:
| Nice framework. Also once you allow more than one assertion,
| there is no need for top-level && in assertions (making them
| simpler tests).
| torginus wrote:
| While there is no strict _need_ , sometimes assertions
| logically belong together.
| erik_seaberg wrote:
| You definitely want x || y, x ^^ y, and x implies y.
| superjan wrote:
| Is this shared somewhere?
| mananaysiempre wrote:
| FWIW, in Python, this is pytest's entire thing (although it
| ends up grovelling into bytecode to achieve it).
| opminion wrote:
| https://stackoverflow.com/a/700325 to write out the
| internal structure of a LambdaExpression.
|
| If you just want the assignments then it's simpler:
|
| Add to the Evaluate method a test for MemberExpression and
| then:
|
| The variable name is:
|
| ((MemberExpression)expr).Member.Name
|
| The value is:
|
| Expression.Lambda(expr).Compile().DynamicInvoke()
| throwaway6977 wrote:
| Not quite the same, but available on nuget-
| 'fluentAssertions' gives you something akin to this. I've
| had decent success with having our juniors use it vs less
| verbose assertion libraries. I don't know about evaluating
| individual expressions in a line separately, but it does
| give you clean syntax and similar error messages that are
| very readable-
|
| "GetProductPage().GetProductPrice().Should().Be().LessThan(
| ...)"
| torginus wrote:
| Nah, it was some corporate project. But if you are
| interested, I could rewrite it. It would be a fun weekend
| project.
| mattmanser wrote:
| It's also worth figuring out for yourself though! They're
| suprisingly easy to use (look up Expression<>).
|
| It's incredibly useful once you know how, and encourages
| you to stop using reflection, passing magic strings as
| arguments, or having to use nameof().
| nerdponx wrote:
| This is what the Python test framework Pytest does, among
| many other similar useful and magical things. I believe that
| the Python developer ecosystem as a whole would be
| substantially less productive without it.
| paphillips wrote:
| Related to this, for anyone not fully up to date on recent C#
| features there is also the _CallerArgumentExpression_ [1],
| [2] feature introduced in C# 10. While it is not a pretty
| printer for an expression, it does allow the full expression
| passed from the call site as an argument value to be captured
| and used within the method. This can be useful for custom
| assert extensions.
|
| For example: public void CheckIsTrue(bool
| value, [CallerArgumentExpression("value")] string? expression
| = null) { if (!value) {
| Debug.WriteLine($"Failed: '{expression}'"); }
| }
|
| So if you call like this: _CheckIsTrue(foo != bar && baz ==
| true)_, when the value is false it prints "Failed: 'foo !=
| bar && baz == true'".
|
| [1] https://learn.microsoft.com/en-us/dotnet/csharp/language-
| ref... [2] https://learn.microsoft.com/en-
| us/dotnet/csharp/language-ref...
| sasmithjr wrote:
| I love using Unquote[0] in F# for similar reasons; it uses
| F#'s code quotations. Assuming the variables have been
| defined with the values you state, the assertion is written
| as: test <@ width > 0 && x + width <
| screenWidth @>
|
| And part of the output is: width > 0 && x +
| width < screenWidth 500 > 0 && x + width < screenWidth
| true && x + width < screenWidth true && 1500 + 500 <
| 1920 true && 2000 < 1920 true && false
| false
|
| [0]: https://github.com/SwensenSoftware/unquote
| beezlewax wrote:
| Is there something like this available for javascript?
| dalmo3 wrote:
| It might not be the general solution you're looking for,
| but idiomatic Jest does the job for me.
| expect(width).toBeGreaterThan(0);
| expect(x+width).toBeLessThan(screenWidth);
|
| When an assertion fails it will tell you: "Expected: X;
| Received: Y"
| tobyhinloopen wrote:
| Not really, no.
|
| I like to use Jest's toMatchObject to combine multiple
| assertions in a single assertion. If the assertion fails,
| the full object on both sides is shown in logs. You can
| easily debug tests that way.
|
| The only way to make it even possible is to do some eval
| magic or to use a pre-processor like babel or a typescript
| compiler plugin.
|
| But if you find something, Lemme know.
| fiddlerwoaroof wrote:
| Well, Function.toString() should print the code of a
| lambda in JavaScript. So I think you could do it without
| a pre-processor: use Babel as a library to parse the body
| of the function; run each sub-expression separately and
| display the results.
| xeromal wrote:
| I'm not aware of any related to testing, but perhaps you
| could use something like this in tandem with some tests to
| pull it off?
|
| https://github.com/gvergnaud/ts-pattern
| eurasiantiger wrote:
| You could parse the failing expression using babel and get
| an AST back.
| xhxivuvy wrote:
| Phrodo_00 wrote:
| I think the advantage of having an assertion per test is that
| it makes sure that all of your assertions are executed. In a
| lot of test frameworks (that use exceptions for assertions for
| example) the first assertion fail will stop the test.
|
| That doesn't mean you have to duplicate code, you can deal with
| it in other ways. In Junit I like to use @TestFctory [1] where
| I'll write most of the test in the Factory and then each
| assertion will be a Test the factory creates, and since they're
| lambdas they have access to the TestFactoy closure.
|
| [1] https://junit.org/junit5/docs/current/user-guide/#writing-
| te...
| edgyquant wrote:
| This is a feature. I want it to fail fast so I can see the
| first issue not crawl the stack for error logs.
| un_montagnard wrote:
| I personally prefer seeing everything that I broke, not
| just the first one
| Izkata wrote:
| I vaguely remember a test framework I saw a decade+ ago that
| had both "assert*" to fail immediately and something else
| ("expect*" maybe?) to check and continue.
| spiffytech wrote:
| I've seen several test frameworks that don't abort a test case
| after the first failed assertion.
|
| When you get many tests each emitting multiple failures because
| one basic thing broke, the output gets hard to sort through.
| It's easier when the failures are all eager.
| codeflo wrote:
| Which ones? I've used at least a dozen at this point, across
| C++, C#, JavaScript, Rust -- and all of them throw (the
| equivalent of) exceptions on assertion failures.
| k__ wrote:
| My experience with testing framework was that they do all
| tests and then mark the ones that failed.
| carnitine wrote:
| That has nothing to do with not knowing which assertion
| in a given test has failed.
| jmillikin wrote:
| GoogleTest (C++) and Go's built-in testing framework both
| support non-fatal assertions.
|
| They're used for code like this:
| assert_eq(list.len(), 1) expect_eq(list[0].username,
| "jdoe") expect_eq(list[0].uid, 1000)
|
| The idea being that if multiple properties are incorrect,
| then all of them will be printed out to the test log.
| masklinn wrote:
| That seems like it'd easily get confusing, when the
| assertions are dependent. Which is often the case e.g. if
| the list is empty, testing the properties of the first
| item make no sense.
| jmillikin wrote:
| That's why the first check is a hard assertion (returning
| on error), and the others are soft (continuing on error).
|
| If the list is empty, then the log will contain one error
| about the length being zero. If the list has one item but
| it has the wrong properties, the log will contain two
| errors.
| masklinn wrote:
| > That's why the first check is a hard assertion
| (returning on error), and the others are soft (continuing
| on error).
|
| See that's so completely unclear I utterly missed that
| there were two _different_ calls there. Doesn 't exactly
| help that the functions are the exact same length, and
| significantly overlap in naming.
| TeMPOraL wrote:
| That's just a matter of familiarity, though. And if you
| make a mistake, you'll discover it the first time the
| test fails - either you'll see too little output, or
| you'll see the test throw an exception or crash.
| HereBeBeasties wrote:
| I bet if it were in monospace and you were reading code
| in context rather than on HN you'd have noticed.
|
| That said, NUnit has much better syntax for this, where
| you put parallel/multiple assertions like this in an
| explicit block together:
| https://docs.nunit.org/articles/nunit/writing-
| tests/assertio...
|
| Much less boilerplate duplication in the actual test
| framework, too.
| masklinn wrote:
| > I bet if it were in monospace
|
| It is.
|
| > and you were reading code in context rather than on HN
| you'd have noticed.
|
| (X) doubt
| hedora wrote:
| > _I 've never seen a case where it would be hard to tell which
| assertion failed._
|
| There are a set of unit testing frameworks that do everything
| they can to hide test output (junit), or vomit multiple screens
| of binary control code emoji soup to stdout (ginkgo), or just
| hide the actual stdout behind an authwall in a uuid named s3
| object (code build).
|
| Sadly, the people with the strongest opinions about using a
| "proper" unit test framework with lots of third party tooling
| integrations flock to such systems, then stack them.
|
| I once saw a dozen-person team's productivity drop to zero for
| a quarter because junit broke backwards compatibility.
|
| Instead of porting ~ 100,000 legacy (spaghetti) tests, I
| suggested forking + recompiling the old version for the new
| jdk. This was apparently heresey.
| turblety wrote:
| Yup, it's why I built `just-tap` [1] which trys to minimise
| as much magic that a lot of these frameworks try to "help"
| you with.
|
| 1. https://github.com/markwylde/just-tap
| autarch wrote:
| TAP is better than some things, but it has some serious
| issues that I wrote about on my blog a while back -
| https://blog.urth.org/2017/01/21/tap-is-great-except-when-
| it...
|
| Basically it's nearly impossible to fully parse it
| correctly.
| hedora wrote:
| Is test2 a flag for TAP?
|
| If you have to pick one or the other, then you're
| breaking the common flow (human debugging code before
| pushing) so that management can have better reports.
|
| The right solution would be to add a environment variable
| or CLI parameter that told tap to produce machine
| readable output, preferably with a separate tool that
| could convert the machine readable junk to whatever TAP
| currently writes to stdout/stderr.
| autarch wrote:
| Test2 is a Perl distribution that replaces a bunch of
| older test stuff. See https://metacpan.org/pod/Test2
|
| But unlike TAP, it's fairly Perl-specific as opposed to
| just being an output format. I imagine you could adapt
| the ideas in it to Node but it'd be more complex than
| simply implement TAP in JS.
|
| And yes, I think the idea of having different output
| formats makes sense. With Test2, the test _harness_
| produces TAP from the underlying machine-readable format,
| rather than having the test code itself directly product
| TAP. The harness is a separate program that executes the
| tests.
| sitkack wrote:
| What is this madness?
|
| Nothing should have to be parsed. Write test results to
| sqlite, done. You can generate reports directly off those
| test databases using anything of your choice.
| your-program test-re.sqlite output.html
| hedora wrote:
| Yeah, but sqlite doesn't scale, and SQL isn't a
| functional language:
|
| https://web.archive.org/web/20110114031716/https://browse
| rto...
| hedora wrote:
| Here are a few mistakes I've seen in other frameworks:
|
| - Make it possible to disable timeouts. Otherwise, people
| will need a different runner for integration, long running
| (e.g., find slow leaks), and benchmark tests. At that
| point, your runner is automatically just tech debt.
|
| - It is probably possible to nest before and afters, and to
| have more than one nesting per process, either from
| multiple suites, or due to class inheritance, etc. Now, you
| have a tree of hooks. Document whether it is walked in
| breadth first or depth first order, then never change the
| decision (or disallow having trees of hooks, either by
| detecting them at runtime, or by picking a hook
| registration mechanism that makes them inexpressible).
| jasonhansel wrote:
| I'd add one more: clearly document what determines the
| order in which tests are run.
|
| On the one hand, running tests in any order should
| produce the same result, and would in any decent test
| suite.
|
| On the other hand, if the order is random or
| nondeterministic, it's really annoying when 2% of PRs
| randomly fail CI, not because of any change in the code,
| but because CI happened to run unrelated tests in an
| unexpected order.
| zmj wrote:
| I'll disagree with this. Every time I've seen that, the
| interference between tests was also possible between
| requests in production. I'd rather my test framework give
| me a 2% chance of noticing the bug than 0%.
| regularfry wrote:
| What's annoying is not being able to reproduce the 2%
| cases so you can't fix it even when you've noticed them.
| Sensible test tools give you the random seed they used to
| order the tests so you can reproduce the sequence.
| thomaslangston wrote:
| Test order should be random, so that the ability to run
| them in parallel and distribute them across multiple
| hosts is not lost by missing enforcement of test
| isolation.
| schubart wrote:
| > On the one hand, running tests in any order should
| produce the same result, and would in any decent test
| suite.
|
| Therefore the tool _should_ run the tests in random
| order, to flush out the non-decent tests. IMHO.
| Jare wrote:
| If you do this, then the tool should be running the tests
| all the time, not just on new commits.
| greggman3 wrote:
| We recently switched our system to a heartbeat system
| instead of a timeout system. The testing framework
| expects to see messages (printf, console.log, etc...)
| often. So a test testing a bunch of combinations might
| take 45 seconds to run but for each combination it's
| printing "PASS: Combination 1, 2, 3" every few ms.
|
| This way the framework can kill the test if it doesn't
| see one of these messages in a short amount of time.
|
| This fixed our timeout issues. We had tests that took too
| long, specially in debug builds and we'd end up having to
| set too large a timeout. Now though, we can keep the
| timeout for the heartbeat really short and our timeout
| issues have mostly gone away.
| ch4s3 wrote:
| Junit is especially bad about this. I often wonder how many
| of these maxims are from people using substandard Java tools
| and confusing their workarounds with deeper insights.
| sitkack wrote:
| You should write an episode of Seinfeld.
|
| I was a TL on a project and I had two "eng" on the project
| that would make test with a single method and then 120 lines
| of Tasmanian Devil test cases. One of those people liked to
| write 600 line cron jobs to do critical business functions.
|
| This scarred me.
| ckastner wrote:
| > _One of those people liked to write 600 line cron jobs to
| do critical business functions._
|
| I was a long-time maintainer of Debian's cron, a fork of
| Vixie cron (all cron implementations I'm aware of are forks
| of Vixie cron, or its successor, ISC cron).
|
| There are a ton of reasons why I wouldn't do this, the
| primary one being is that cron really just executes jobs,
| period. It doesn't serialize them, it doesn't check for
| load, logging is really rudimentary, etc.
|
| A few years ago somebody noticed that the cron daemon could
| be DoS'ed by a user submitting a huge crontab. I
| implemented a 1000-line limit to crontabs thinking "nobody
| would ever have 1000-line crontabs". I was wrong, quickly
| received bug reports.
|
| I then increased it to 10K lines, but as far as I recall,
| users were hitting even that limit. Crazy.
| kragen wrote:
| Is Dillon cron a fork of Vixie cron?
| jlarocco wrote:
| > There are a set of unit testing frameworks that do
| everything they can to hide test output (junit), or vomit
| multiple screens of binary control code emoji soup to stdout
| (ginkgo), or just hide the actual stdout behind an authwall
| in a uuid named s3 object (code build).
|
| The test runner in VS2019 does this, too and it's incredibly
| frustrating. I get to see debug output about DLLs loading and
| unloading (almost never useful), but not the test's stdout
| and stderr (always useful). Brilliant. At least their command
| line tool does it right.
| torginus wrote:
| xUnit is terrible. It has a horrible culture, the author seems
| to have a god complex. It is overly complex and opinionated in
| the worst way possible.
|
| Many times I searched for 'how do I do something with xUnit'
| and found a github issue with people struggling with the same
| thing, and the author flat out refusing to incorporate the
| feature as it was against his principles.
|
| Other times I found that I needed to do was override some core
| xUnit class so it would do the thing I wanted it to do - sounds
| complex, all right lets see the docs. Oh there are none, 'just
| read the source' according to the author.
|
| Another thing that bit us in the ass is they refused to support
| .NET Standard, a common subset of .NET Framework and Core,
| making migration hell.
| HereBeBeasties wrote:
| NUnit isn't much better. You'd think that they would have
| good test coverage and therefore high confidence to make
| changes, especially to fix actual bugs, but I gave up trying
| to get patches in because the core devs seem so afraid of
| breaking anything, even when it's an obviously-isolated
| private, hidden bug fix or performance improvement.
|
| "We can't land this tiny fix because we have a release
| planned within three months" sort of thing.
| zdragnar wrote:
| How ironic, a unit test framework is so fragile and poorly
| covered by tests they are not able to determine if a change
| would cause any regressions.
| torginus wrote:
| Tbh, one of the differences between xUnit and nUnit, is the
| way generated test cases work, like specifying test cases
| in an xml file. nUnit has the _TestCaseSource_ attribute
| for this, while xUnit has the _Fact_ attribute.
|
| One of the key differences, is there are no test cases
| generated, nUnit tests just wont run, while xUnit will
| throw.
|
| Since it's completely legal and sensible for a certain kind
| of test to have no entries in an xml, we needed to hack
| around this quirk. When I (and countless others) have
| mentioned this on the xunit github, the author berated us
| that how dare we request this.
|
| So nUnit might be buggy, but xUnit is fundamentally
| unfixable.
| codenesium wrote:
| I'm curious what things you're trying to do that requires you
| to overload xunit classes? We use xunit for everything and
| haven't found any gaps so far.
| shagie wrote:
| With Junit, you could use assertAll
| https://junit.org/junit5/docs/5.0.0-M2/api/org/junit/jupiter...
|
| Aside from that, you could do things like int
| failures = 0; failures += someAssertion ? 0 : 1;
| failures += anotherAssertion ? 0 : 1; failures +=
| yetAnotherAssertion ? 0 : 1; assertZero(failures);
|
| With the appropriate logging of each assertion in there.
|
| Consider the situation of "I've got an object and I want to
| make sure it comes out in JSON correctly"
|
| The "one assertion" way of doing it is to assertEqual the
| entire json blob to some predefined string. The test fails, you
| know it broke, but you don't know where.
|
| The multiple assertions approach would tell you where in there
| it broke and the test fails.
|
| The point is much more one of "test one _thing_ in a test " but
| testing one thing can have multiple assertions or components to
| it.
|
| You don't need to have testFirstNameCorrect() and
| testLastNameCorrect() and so on. You can do testJSONCorrect()
| and test one _thing_ that has multiple _parts_ to verify its
| correctness. This becomes easier when you 've got the
| frameworks that support it such as the assertAll("message", ()
| -> assertSomething, () -> assertSomethingElse(), ...)
|
| https://stackoverflow.com/q/40796756
| catlifeonmars wrote:
| For the JSON example, some testing libraries (jest) will
| output a unified diff on assertion failure.
| Macha wrote:
| I've noticed the opposite in a Java codebase I work in. Tests
| where the test is assertEquals(toJson(someObject),
| giantJsonBlobFromADifferentFile). Of course the test runner has
| no idea about formatting strings that happen to be json, so I
| end up having to copy these out into an editor, formatting them
| and eyeballing the difference, or for even larger ones having
| to save them out to files and diff them. And of course most of
| the fields in the mock aren't relevant to the class under test,
| so I'd trade them out for 5-6 targeted asserts for the relevant
| fields happily.
|
| The problem is, since it's a legacy codebase, there's many
| fields which are only tested incidentally by this behaviour, by
| tests that actually aren't intending to test that
| functionality.
| TeMPOraL wrote:
| I had similar case recently, in C++. I ended up spending a
| few hours writing a simple JSON differ - a bit of code that
| would parse two strings into a DOM object graph using a
| rapidjson, and then walk down them simultaneously -
| basically, I implemented operator== which, instead of
| terminating early, recorded every mismatch.
|
| Then, I packaged it into a Google Test matcher, and from now
| on, the problem you describe is gone. I write:
| EXPECT_THAT(someObject,
| IsEqAsJSON(someBlobFromADifferentFile));
|
| and if it fails, I get output like this:
| Expected someObject to be structurally equivalent to
| someBlobFromADifferentFile; it is not; - #/object/key
| - missing in expected, found in actual - #/object/key2
| - expected string, actual is integer -
| #/object/key3/array1 - array lengths differ; expected: 3,
| actual: 42 - #/object/key4/array1/0/key3 - expected
| "foo" [string], actual "bar" [string]
|
| Etc.
|
| It was a rather simple exercise, and the payoff is immense. I
| think it's really important for programmers to learn to help
| themselves. If there's something that annoys you repeatedly,
| you owe it to yourself and others to fix it.
| tsss wrote:
| > I think it's really important for programmers to learn to
| help themselves. If there's something that annoys you
| repeatedly, you owe it to yourself and others to fix it.
|
| It's a cultural problem. _I_ can do that, but my colleagues
| will just continue to write minimum effort tests against
| huge json files or database dumps where you have no idea
| why something failed and why there are a bunch of
| assertions against undocumented magic numbers in the first
| place. It's like you're fighting against a hurricane with a
| leaf blower. A single person can only do so much. I end up
| looking bad in the daily standup because I take longer to
| work on my tickets but the code quality doesn't even
| improve in a measurable way.
| david_allison wrote:
| This can be improved, it'd be worth Googling for a better
| solution than what you have.
|
| https://github.com/skyscreamer/JSONassert seems decent.
|
| but it can be done from scratch in a few hours (I'd recommend
| this if you have 'standardized' fields which you may want to
| ignore):
|
| Move to a matcher library for assertions (Hamcrest is
| decent), and abstract `toJSON` into the a matcher, rather on
| the input.
|
| This would change the assertion from:
|
| `assertEquals(toJson(someObject),
| giantJsonBlobFromADifferentFile)`
|
| to:
|
| `assertThat(someObject,
| jsonEqual(giantJsonBlobFromADifferentFile))`
|
| The difference here is subtle: it allows `jsonEqual` to
| control the formatting of the test failure output, so on a
| failure you can:
|
| * convert both of the strings back to JSON
|
| * perform a diff, and provide the diff in the test output.
|
| Decent blog post on the topic: https://veskoiliev.com/use-
| custom-hamcrest-matchers-to-level...
| Macha wrote:
| So the same team does actually use JSONAssert in some
| places, but it can still provide unhelpful comparisons.
|
| I'd actually rather people just assert on the fields they
| need, but it's a larger team than I can push that on.
| 0x445442 wrote:
| That test seems to be testing whether or not the library used
| to deserialize json works. I don't think that's valid unless
| the code base you are working on is Gson or Jackson or the
| like.
|
| Assuming that's not the case and you're interested in the
| state of two object graphs then you just compare those, not
| the json string they deserialize to.
| seadan83 wrote:
| As an aside, FWIW there are libraries to do JSON and XML
| comparisons in assertions. Output is good and you can control
| whether ordering is important.
| sschueller wrote:
| In phpunit you can send dataset through a test function . If
| you don't label them you will have a jolly time finding out
| which one of the sets caused the failure.
| kazinator wrote:
| Think about the personality of someone who is so dissatisfied
| with the lack of verbosity in his test suites, that he needs a
| side project of writing a book about unit testing. Of course
| they will advocate testing one assertion per function, and make
| up nonsense to justify their recommendation.
|
| Secretly, they would have the reader write 32 functions to
| separately test every bit of a uint32 calculation, only
| refraining from that advice due to the nagging suspicion that
| it might be loudly ridiculed.
| magic_hamster wrote:
| > How is that even possible in the first place? The entire job
| of an assertion is to wave a flag saying "here! condition
| failed!".
|
| I envy you for never having seen tests atrocious enough where
| this is not only possible, but the common case.
|
| Depending on language, framework and obviously usage,
| assertions might not be as informative as providing the basic
| functionality of failing the test - and that's it.
|
| Now imagine this barebones use of assertions in tests which are
| entirely too long, not isolating the test cases properly, or
| even completely irrelevant to what's (supposedly) being tested!
|
| If that's not enough, imagine this nightmare failing not after
| it has been written, but, let's say 18 months later, while
| being part of a massive test suite running for a while. All you
| have is a the name of the test that failed, you look into it to
| find a 630 lines long test "case" with 22 nondescript
| assertions along the way. You _might_ know which line failed
| the test, but not always. And of course debugging the test
| function line by line doesn 't work because the test depends on
| intricate timing for some reason. The person who wrote this
| might not be around and now this is your dragon to slay.
|
| I think I should stop here before triggering myself any
| further. Therapy is expensive.
| ramraj07 wrote:
| So because some idiot somewhere wrote a 100 assertion unit
| test we should ban anyone from writing even 2 assertions in
| one test?
| robertlagrant wrote:
| Devious commenter was describing a (normal) scenario where
| a unit test is not precise. No need to follow up with an
| aggressive "so what you're saying is".
| magic_hamster wrote:
| Not at all. It makes sense in some tests. I addressed the
| part asking how it's even possible to not know what
| happened.
|
| As for multiple asserts, that is really meaningless. The
| test case should test one thing. If it requires several
| asserts that's okay. But having a very long test function
| with a lot of assertions, is strongly indicating that
| you're testing more than one thing, and when the test fails
| it will be harder to know what actually happened.
| Joeboy wrote:
| I guess you might be talking about a different language /
| environment than I'm used to, but even in the 100
| assertion test case you get useful tracebacks in python.
| Testing lots of things at the same time means strictly
| speaking you're writing an integration test rather than a
| unit test, but still I don't see how it's a bad _test_.
| It 's easy and stops buggy PRs going into production.
|
| The test failures I see that are _actually_ hard to debug
| are ones where the failures are difficult to reproduce
| due to random input, tests running in parallel and
| sharing the same filesystem etc. I don 't think I've ever
| not known what assert was failing (although I guess in
| theory you could make that happen by catching
| AssertionError).
| RHSeeger wrote:
| > Testing lots of things at the same time means strictly
| speaking you're writing an integration test rather than a
| unit test
|
| There's nothing wrong with integration tests, but they're
| not unit tests. It's fine to have both, but the
| requirements for a good unit test and those for a good
| integration test diverge. The title of this post, at
| least, was specific to unit tests.
| bluGill wrote:
| A unit tests tests one unit. And integration tests covers
| more than one unit. I think everyone agrees with that,
| but nobody has defined unit.
|
| The longer I program the more I am convinced that the
| larger your unit the better. The unit tests is a
| statement that you will never refactor across this line,
| and that eliminates a lot of flexibility that I want.
|
| It turns out that debugging failed integration tests is
| easy,the bug is in the last thing you changed. Sure the
| test covers hundreds of lines, but you only changed one.
| __alexs wrote:
| I recently went to the effort of trying to work out where
| the term unit test came from in some desperate effort to
| find what a unit was meant to be.
|
| After much googling and buying or ancient text books I
| hit a dead end. At this point I think "unit" is just
| noise that confuses people into making distinctions that
| don't exist.
| Izkata wrote:
| IIRC the "unit" in "unit test" was meant to mean
| "semantic unit" ("the access module", for example, should
| be distinct with a well-defined interface that all the
| tests go through), but very quickly turned into
| "syntactic units" ("a single function", for example,
| where the "well-defined interface" ends up just being
| function arguments/return value) because most people
| didn't understand what the original proponents meant.
| switchbak wrote:
| As I recall the TDD mailing list has some background on
| the use of the word "unit", it goes WAY back, I believe
| it goes back to the mainframe/ punch card era.
| Regardless, I think it roughly translates to C's notion
| of the unit of compilation.
|
| Which is obviously not what people really mean these
| days, but the phrase stuck. The early Xp'ers even found
| it an issue back then.
|
| For a while people tried to push the term "micro tests",
| but that didn't really take off.
|
| I agree with Gerard Mezaros and Martin Fowler and
| typically follow their (very mainstream) definitions on
| this stuff. Integration and functional testing have their
| own ambiguities too, it's definitely a frustrating
| situation to not have solidly defined foundational terms.
| JackFr wrote:
| > It turns out that debugging failed integration tests is
| easy,the bug is in the last thing you changed. Sure the
| test covers hundreds of lines, but you only changed one.
|
| That's not true.
|
| A correct change might expose an existing bug which
| hadn't been tested or expose flaky behavior which existed
| but hadn't been exercised. In both cases the solution is
| not to revert the correct change, but to fix the buggy
| behavior.
| sdenton4 wrote:
| Michael Pollan was right: Write tests, not too many,
| mostly integration.
| teddyh wrote:
| Contrary viewpoint: _Integrated Tests Are A Scam_ (J.B.
| Rainsberger):
|
| https://www.youtube.com/watch?v=fhFa4tkFUFw
| sdenton4 wrote:
| Watched a bit of this... It's typical test-driven
| zealotry; the main criticism of integration tests seems
| to be that they don't force your hand in system design in
| the way that unit tests do? Which seems very silly, but
| then, I'm not a person who goes to conferences about
| testing philosophy.
| switchbak wrote:
| Did you miss his follow-up? "Integration tests are a scam
| is a scam". For real. I like J.B., but I think he muddies
| the water too much and overall understanding suffers.
| RHSeeger wrote:
| > The unit tests is a statement that you will never
| refactor across this line, and that eliminates a lot of
| flexibility that I want.
|
| I certainly don't see it as that. I see it as "this is
| the smallest thing I _can_ test usefully". Mind you,
| those do tend to correlate, but they're not the same
| thing.
| dmitriid wrote:
| > this is the smallest thing I _can_ test usefully
|
| Then you're testing useless things.
|
| Usefulness is when different parts of a program work
| together as a coherent whole. Testing DB access layer and
| service layer separately (as units are often defined) has
| no meaning (but is often enforced).
|
| Queue in memes about "unit tests with 100% code coverage,
| no integration tests" https://mobile.twitter.com/thepract
| icaldev/status/6876720861...
| RHSeeger wrote:
| >> this is the smallest thing I _can_ test usefully
|
| > Then you're testing useless things.
|
| We'll have to agree to disagree then.
|
| > Testing DB access layer and service layer separately
| (as units are often defined)
|
| Not at all. For me, a unit is a small part of a layer;
| one method. Testing the various parts in one system/layer
| is another type of test. Testing that different systems
| work together is yet another.
|
| I tend to think in terms of the following
|
| - Unit test = my code works
|
| - Functional test = my design works
|
| - Integration test = my code is using your 3rd party
| stuff correctly (databases, etc)
|
| - Factory Acceptance Test = my system works
|
| - Site Acceptance Test = your code sucks, this totally
| isn't what I asked for!?!
|
| The "my code works" part is the smallest piece possible.
| Think "the sorting function" of a library that can return
| it's results sorted in a specific order.
| dmitriid wrote:
| And the only actual useful tests are functional
| (depending on how you write them) and above.
|
| If those fail, it means that neither your design nor your
| code works.
|
| The absolute vast majority of unit tests are meaningless
| because you just repeat them again in the higher level
| tests.
| jcelerier wrote:
| Why ? In c++ if I have a test case with
| REQUIRES(v1::foo(0) == v2::foo(0));
| REQUIRES(v1::foo(1) == v2::foo(1));
|
| And the second assert fails the error message will tell
| me exactly that, the line, and the value of both function
| calls if they are printable. What more do you want to
| know "what actually happened"?
| jjav wrote:
| > when the test fails it will be harder to know what
| actually happened
|
| This should not ever be possible in any semi-sane test
| environment.
|
| One could in theory write a single test function with
| thousands of asserts for all kinds of conditions and it
| still should be 100% obvious which one failed when
| something fails. Not that I'd suggest going to that
| extreme either, but it illustrates that it'll work fine.
| [deleted]
| simplotek wrote:
| > So because some idiot somewhere wrote a 100 assertion
| unit test we should ban anyone from writing even 2
| assertions in one test?
|
| You're falling prey to slippery slope fallacy, which at
| best is specious reasoning.
|
| The rationale is easy to understand. Running 100 assertions
| in a single test renders tests unusable. Running 10
| assertions suffers from the same problem. Test sets are
| user-friendly if they dump a single specific error message
| for a single specific failed assertion, thus allowing
| developers to quickly pinpoint root causes by simply
| glancing through the test logs.
|
| Arguing whether two or three or five assertions should be
| banned misses the whole point and completely ignores the
| root cause that led to this guideline.
| BlargMcLarg wrote:
| >Test sets are user-friendly if they dump a single
| specific error message for a single specific failed
| assertion, thus allowing developers to quickly pinpoint
| root causes by simply glancing through the test logs.
|
| As if this actually happens in practice, regardless of
| multiple or single asserts. Anything that isn't non-
| trivial will at most tell you _what_ doesn 't work, but
| it won't tell you _why_ it doesn 't work. Maybe allowing
| an educated guess when multiple tests fail to function.
|
| You want test sets to be user friendly? Start at taking
| down all this dogmatism and listening to the people as to
| why they dislike writing tests. We're pushing
| 'guidelines' (really more like rules) while individuals
| think to themselves 'F this, Jake's going to complain
| about something trivial again, and we know these tests do
| jack-all because our code is a mess and doing anything
| beyond this simple algorithm is a hell in a handbasket".
|
| These discussions are beyond useless when all people do
| is talk while doing zero to actually tackle the issues of
| the majority not willing to write tests. "Laziness" is a
| cop-out.
| mjul wrote:
| Just use you best judgement and you will be fine.
|
| Patterns are context specific advice about a solution and
| its trade-offs, not hard rules for every situation.
|
| Notice how pattern books will often state the relevant
| Context, the Problem itself, the Forces influencing it, and
| a Solution.
|
| (This formulaic approach is also why pattern books tend to
| be a dry read).
| mrits wrote:
| 22 assertions in a test is a lot better than 22 separate
| tests that fail for the same reason.
| Jabbles wrote:
| With 22 separate tests you have the possibility of knowing
| that only a subset of them fail. Knowing which fail and
| which pass may help you debug.
|
| In Go, in general, tests fail and continue, rather than
| causing the test to stop early, so you can tell which of
| those 22 checks failed. Other languages may have the option
| to do something similar.
|
| https://pkg.go.dev/testing#T.Fail
| bluGill wrote:
| They are the same. I don't care, as I'll fix them one at a
| time, and if the fix happens to fix more than one great.
| mrits wrote:
| Fixing one at a time like that is a good way to get into
| an endless cycle. Zoom out a bit and make a plan before
| you start coding.
| bluGill wrote:
| That should be understood wothout saying. Even with a
| plan I'm going to fix one at a time.
| ethbr0 wrote:
| > _... where this is not only possible, but the common case._
|
| Couldn't not read that in Peter Sellers' voice
| https://m.youtube.com/watch?v=2yfXgu37iyI&t=2m36s
|
| > _... to find a 630 lines long test "case" with 22
| nondescript assertions along the way._
|
| This is where tech team managers are abrogating their
| responsibility and job.
|
| It's the job of the organization to set _policy_ standards to
| outlaw things like this.
|
| It's the job of the developer to cut as many corners of those
| policies as possible to ship code ASAP.
|
| And it's the job of a tech team manager to set up a detailed
| but efficient process (code review sign offs!) that paper
| over the gap between the two in a sane way.
|
| ... none of which helps immediately with a legacy codebase
| that's @$&@'d, though.
| RHSeeger wrote:
| > It's the job of the developer to cut as many corners of
| those policies as possible to ship code ASAP.
|
| I can't tell if this is supposed to be humor, or if you
| actually believe it. It's certainly not my job as a
| developer to ship worse code so that I can release it ASAP.
| Rather, it's my job to push back against ASAP where it
| conflicts with writing better code.
| FartyMcFarter wrote:
| > You might know which line failed the test, but not always.
|
| If that's the case, the test framework itself is severely
| flawed and needs fixing even more than the tests do.
|
| There's no excuse to have an assert function that doesn't
| print out the location of the failure.
| magic_hamster wrote:
| Even if the framework is fine, you can see something like
| an elaborate if-else tree, or even a try-catch block, and
| after it's all done, there's a condition check with
| `fail()`. So the point of failure could be manually
| detached from the actual point of failure.
|
| Granted, this is not the way to do things. But it happens
| anyway.
| mynameisvlad wrote:
| I mean in your example it's someone choosing not to use
| asserts. Which is a problem, don't get me wrong, but it's
| not the problem being talked about here.
|
| The comment thread is about "Assertion Roulette" --
| having so many assertions you don't know which went off.
| Which really seems like a test framework issue more than
| a test issue.
| Kuinox wrote:
| Tech debt can often be fixed by more tech debt instead of
| fixing the root problem.
| marcosdumay wrote:
| > Depending on language, framework and obviously usage,
| assertions might not be as informative as providing the basic
| functionality of failing the test - and that's it.
|
| Well, avoiding ecosystems where people act dumb is a sure way
| to improve one's life. For a start, you won't need to do
| stupid things in reaction of your tools.
|
| Yes, it's not always possible. But the practices you create
| for surviving it are part of the dumb ecosystem survival kit,
| not part of any best practices BOK.
| danuker wrote:
| I use the following pattern for testing regexes:
| expected_positive = [ 'abc', 'def',
| ...] for text in expected_positive:
| self.assertTrue(matcher(text), f"Failed: {text}")
|
| Before I added the assertion error message, `f"Failed:
| {text}"`, it was quite difficult to tell WHICH example failed.
| [deleted]
| profunctor wrote:
| If you're using pytest you just paramaterize the tests and it
| tells you the exact failing case. Seems to be a basic feature
| I would be surprised to know doesn't exist across almost all
| commonly used frameworks.
| danuker wrote:
| We are using unittest.
|
| But thanks for bringing it up, it seems it also has
| "subtest" support, which might be easier than interpolating
| the error message in some cases:
|
| https://docs.python.org/3/library/unittest.html#distinguish
| i...
| Dunedan wrote:
| There is also `parameterized` [1] if you want to do it by
| using decorators.
|
| [1]: https://pypi.org/project/parameterized/
| masklinn wrote:
| > We are using unittest.
|
| Unless you need the library style in order to drive it,
| switch to pytest. Seriously.
|
| - assert rewriting is stellar, so much more comfortable
| than having to find the right assert* method, and tell
| people they're using the wrong one in reviews
|
| - runner is a lot more practical and flexible: nodeids,
| marks, -k, --lf, --sw, ...
|
| - extensions further add flexibility e.g. timeouts,
| maxfail, xdist (though it has a few drawbacks)
|
| - no need for classes (you can have them, mind, but if
| functions are sufficient, you can use functions)
|
| - fixtures > setup/teardown
|
| - parameterized tests
| jjgreen wrote:
| You mean "subTest", I switched to pytest just to get away
| from the camel (and it's in the standard library!)
| ericvsmith wrote:
| Came here to mention unittest.subtest. It's for exactly
| this case.
| lancebeet wrote:
| That looks like a case where I would use a parameterized test
| rather than a for loop inside the test.
| dd82 wrote:
| this has downsides if you're comparing attributes with a
| method result and checking whether said attrs match what
| you expect. Either you run each test N times for N attr
| comparisons, accepting the cost of setup/teardown, or do a
| loop and fire off an assert error with text on which
| comparison failed.
|
| Since you already have the object right there, why not do
| the latter approach?
| lancebeet wrote:
| If the setup/teardown is expensive I would do it in
| reusable fixtures. The reason I wouldn't choose the
| latter approach is that it would usually be less
| convenient in the long run. You'd need to replace your
| asserts with expects to avoid it throwing on the first
| error (if this isn't what you want), you'll often need to
| manually add data to the assertion (as GP did) that you
| would otherwise get for free, and you'll need to look at
| the assertion error rather than the test case to know
| what actually failed. This can be quite inconvenient if
| you e.g. export your test results in a CI/CD pipeline.
| dd82 wrote:
| normally in a CI/CD pipeline, you'll see what asserts
| failed in the log output. Github Actions with pytest
| shows the context of the failed asserts in the log
| output. TBH, thought this was standard behavior, do you
| have experience with a CI pipeline that differs?
|
| All the other points you make as negatives are all
| positives for me. Biggest thing is, if you're making this
| change that alters things so drastically, is that really
| a good approach.
|
| Also, fixtures aren't magic. If you can't scope the
| fixture to module or session, that means by default it
| runs in function scope, which would be the same thing as
| having expensive setup/teardown. And untangling fixtures
| can be a bgger PITA than untangling unexpected circular
| imports
| olex wrote:
| This seems like a test design issue to me. Best practice is
| to avoid for-each loops with assertions within tests - using
| parametrized tests and feeding the looped values as input is
| almost always a better option. Figuring out which one failed
| and why is one advantage it gives you in comparison. Another
| one is that all inputs will always be tested - your example
| stops on the first one that fails, and does not evaluate the
| others after that.
| danuker wrote:
| > your example stops on the first one that fails, and does
| not evaluate the others after that.
|
| I don't think this is a big problem; trying to focus on
| multiple examples at once is difficult.
|
| It might be a problem if tests are slow and you are forced
| to work on all of them at once.
|
| But in that case I'd try to make the tests faster (getting
| rid of network requests, disk/DB access by faking them away
| or hoisting to the caller).
| olex wrote:
| Hm. I think my main issue there is not the speed, but
| rather seeing the whole picture at once. You mentioned
| you use this pattern to test regular expressions; say you
| modify the regexp in question with some new feature
| requirement, and now the very first of a dozen test
| inputs fails. You fix it, but then each one of the
| following keeps failing, and you can only find an elegant
| solution that works for all of them after seeing all the
| failures, having ran the test and modified the code a
| dozen times. Wouldn't it be nicer to see all fails right
| away and be able to find a solution to all of them,
| instead of fixing the inputs one-by-one?
| danuker wrote:
| In my experience, from doing some TDD Katas[0] and timing
| myself, I found coding slower and more difficult when
| focusing on multiple examples at once.
|
| I usually even comment out all the failing tests but the
| first one, after translating a bunch of specifications
| into tests, so I see the "green" when an example starts
| working.
|
| Maybe it would be easier to grok multiple regex examples
| than algorithmic ones, but at least for myself, I am
| skeptical, and I prefer taking them one at a time.
|
| [0] - https://kata-log.rocks/tdd
| [deleted]
| johtso wrote:
| Again pytest makes things so much nicer in this regard.
| Having to comment things out sucks.
|
| With pytest you can use the -x flag to stop after the
| first test failure.
|
| Even better you can use that in combination with -lf to
| only run the last failed test.
| masklinn wrote:
| > With pytest you can use the -x flag to stop after the
| first test failure.
|
| > Even better you can use that in combination with -lf to
| only run the last failed test.
|
| Fwiw `--sw` is much better for that specific use-case.
|
| `--lf` is more useful to run the entire test suite, then
| re-run just the failed tests (of the entire suite). IIRC
| it can have some odd interactions with `-x` or
| `--maxfail`, because the strange things happen to the
| cached "selected set".
|
| Though it may also be because I use xdist a fair bit, and
| the interaction of xdist with early interruptions (x,
| maxfail, ...) seems less than perfect.
| tsimionescu wrote:
| In my own experience, this has often been a good way of
| going in circles, where I end up undoing and redoing
| changes as fixing one thing breaks another, until I take
| a step back to find the proper algorithm by considering
| multiple inputs.
|
| Of course, ymmv depending on how good your initial
| intuition is, and how tricky the problem is.
| dd82 wrote:
| not really. one thing this is useful for is extracting out
| various attributes in an object when you really don't want
| to compare the entire thing. Or comparing dict attributes,
| and figuring which one is the incorrect one.
|
| for example, expected_results = {...}
| actual_obj = some_intance.method_call(...) for
| key, val in expected_results.items(): assert
| getattr(actual_obj, key) == val, f"Mismatch for {key}
| attribute"
|
| You could shift this off to a parametrized test, but that
| means you're making N more calls to the method being
| tested, which can have its own issues with cost of test
| setup and teardown. With this method, you see which key
| breaks, and re-run after fixing.
| olex wrote:
| Ok, in this case a parametrized test is not the best
| approach, I agree. But I would still want to avoid the
| for-each and "failing fast". One approach would be to
| gather the required attributes in an array or a struct of
| some sort, and then do a single assert comparison with an
| expected value, showing all the differences at once.
| However, this requires the assertion framework to be able
| to make such a comparison and return a nicely readable
| error message, ideally with a diff.
| dd82 wrote:
| Right, and not many actually do. with python and pytest,
| you could leverage difflib, but that's an additional
| thing that adds unnecessary complexity. My approach is
| simple enough, good enough, and doesn't require
| additional fudging around with the basics of the
| language's test libs.
|
| also,
|
| >your example stops on the first one that fails, and does
| not evaluate the others after that.
|
| I would argue this is desirable behavior. there are soft
| checks, ie, https://pypi.org/project/pytest-check/, that
| basically replace assertions as raised exceptions and do
| your approach. But I do want my tests to raise errors at
| the point of failure when a change occurs. If there's
| alot of changes occurring, that raises larger questions
| of "why" and "is the way we're executing this change a
| good one"?
| z9znz wrote:
| Testing is hard when the code it tests is OOP, has mutations
| everywhere, and has big functions that do too many things.
|
| It's practically impossible to thoroughly test such code with one
| assertion per test; it would mean having dozens of tests just for
| one object method. Correspondingly, the fixtures/factories/setup
| for tests would balloon in number and complexity as well to be
| able to setup the exact circumstance being tested.
|
| But the example in TFA is, imo, bad because it is testing two
| entirely different (and unrelated) layers at once. It is testing
| that a business logic delete of a thing works correctly, and that
| a communication level response is correct. Those could be two
| separate tests, separating the concerns, and resulting in simpler
| code to reason about and less maintenance effort in the future.
|
| We want to know if the DeleteAsync(address) behaves correctly.
| Actually we want to know if DeleteReservation() works,
| irrespective of the async requirement. Testing whether
| AnythingAsync() works is something that is already done at the
| library or framework level, and we probably don't need to prove
| that it works again.
|
| Write a test for DeletReservation() which tests if a valid
| reservation gets deleted. Write another related test to ensure
| that a non-existent or invalid reservation does not get deleted,
| but rather returns some appropriate error value. That's two,
| probably quite simple tests.
|
| Now somewhere higher up, write the REST API tests. ApiDelete()...
| a few tests to establish that if an API delete is called, and the
| business logic function it calls internally returns a successful
| result, then does the ApiDelete() return an appropriate response?
| Likewise if the business logic fails, does the delete respond to
| the API caller correctly?
| drewcoo wrote:
| > Testing is hard when the code it tests is OOP, has mutations
| everywhere, and has big functions that do too many things.
|
| Yes, that means the software being tested is not very testable.
| And should probably be refactored.
|
| https://en.wikipedia.org/wiki/Testability
| commandlinefan wrote:
| > when the code it tests is OOP
|
| In my experience when code isn't OOP, that means all static
| functions with static (I.e. global) data which isn't hard to
| test, it's actually impossible because you can't mock out the
| static data.
| z9znz wrote:
| I didn't downvote you, but I have a hard time either
| understanding your meaning or imagining the scenario you
| describe. Can you give an example?
|
| OOP functions are usually harder to test because they expect
| complete objects as arguments, and that tends to require a
| lot more mocking or fixtures/factories to setup for the test.
|
| FP functions typically operate on less complex and more open
| data structures. You just construct the minimum thing
| necessary to satisfy the function, and the test is
| comparatively simple. None of this has anything to do with
| global data. Using global data from within any functions is
| generally a bad idea and has nothing to do with FP or OOP.
| lifeisstillgood wrote:
| Things I find awkward with unit test (and it might just be me) is
| Inwant to write a test like
|
| def testLotsOfWaysTofail(): d = {"Handle Null": (None, foobar),
| "Don't allow under 13 to do it": (11, foobar), "or old age
| pensioner": (77,wobble)} for ... generate a unit test dynamically
| here
|
| I have built metaclasses, I have tried many different options. I
| am sure there is a near solution.
|
| But I never seem to have it right
| oweiler wrote:
| What you are looking for is called property based testing
| _dain_ wrote:
| isn't this just a parametrized test? or do you want to generate
| the test cases automatically like in property-based testing?
| lifeisstillgood wrote:
| Yes parameterised testing. Just all the packages i have tried
| seemed to make it awkward to impossible - hence metaclasses
| [deleted]
| ckp95 wrote:
| Does this do what you want?
|
| https://github.com/ckp95/pytest-parametrize-cases
| @parametrize_cases( Case("handle null",
| age=None, x="foobar"), Case("don't allow under
| 13s", age=11, x="foobar"), Case("or old age
| pension", age=77, x="wobble"), ... # as many as
| you want ) def
| test_lots_of_ways_to_fail(age, x): with
| pytest.raises(ValueError):
| function_under_test(age, x)
| lifeisstillgood wrote:
| Just wanted to pop in and say ckp95 actually mailed me
| this reply in case I missed it. The extra effort kinda
| restored my faith in humanity - giving a shit about
| strangers problems matters these days. Nice one.
|
| cf https://youtu.be/wUyI3ssmj_U
| inglor_cz wrote:
| It is interesting how far purism can go. I wouldn't have thought
| that some people obsess about having just one assertion in a
| test. This seems to be a case of being mentally enslaved by your
| own categories.
| oxff wrote:
| I mean, obviously. Obviously I'm not going to assert every header
| in their own unit test for a single request path in most cases.
| Assert what makes sense as a logical whole / unit.
| bazoom42 wrote:
| Unit tests are one of the many great ideas in our industry which
| have been undermined by people treating it is a set of rituals
| rather than a tool.
| seadan83 wrote:
| I wonder how much of this is the journeyman problem (aka, the
| expert beginner)
|
| I believe writing test code is its own skill. Hence, like a
| coder learning SRP and dogmatically applying it, so does a
| person that is forced to write unit tests without deep
| understanding. (And of course, bad abstractions are worse than
| code duplication)
|
| I think it's very possible to have a developer with 10 yes
| experience but effectively only 2 years experience building
| automated test suites. (Particularly if they came from a time
| before automated testing, or if the testing and automated tests
| were someone else's job)
| projektfu wrote:
| This is such a bad example because the level of testing is
| somewhat between unit and acceptance/functional testing. I can't
| tell at a glance if "api" is something that will hit a database
| or not.
|
| The first part where he says you can check in the passing test of
| code that does nothing makes me twitch, but mainly because I see
| functional testing as the goal for a bit of functionality and
| unit tests as a way of verifying the parts of achieving that
| goal. Unit tests should verify the code without requiring
| integration and functional tests should confirm that the units
| integrate properly. I wouldn't recommend checking in a test that
| claims to verify that a deleted item no longer exists when it
| doesn't actually verify that.
|
| Deciding on the granularity of actual unit tests is probably
| something that is best decided through trial and error. I think
| when you break down the "rules", like one assertion per test, you
| need to understand the goals. In unit testing an API I might have
| lots of tiny tests that confirm things like input validation,
| status codes, debugging information, permissions, etc. I don't
| want a test that's supposed to check the input validation code to
| fall because it's also checking the logged-in state of the user
| and their permission to reach that point in the code.
|
| In unit tests, maybe you want to test the validation of the item
| key. You can have a "testItemKey" test that checks that the
| validation confirms that the key is not null, not an empty
| string, not longer than expected, valid base64, etc. Or you could
| break those into individual tests in a test case. It's all about
| the balance of ergonomics and the informativeness and robustness
| of the test suite.
|
| In functional testing, however, you can certainly pepper the
| tests with lots of assertions along the way to confirm that the
| test is progressing and you know at what point it broke. In that
| case, the user being unable to log in would mean that testing
| deleting an item would not be worthwhile.
| jameshart wrote:
| The issue here appears to be imperative testing, with assertion
| failures as an early exit.
|
| If all your assertions are always evaluated, you don't have this
| issue?
| kazinator wrote:
| At work, I put in a change which allows multiple assertions in a
| C testing framework. There is no exception handling or anything.
|
| A macro like EXPECT_ASSERT(whatever(NULL));
|
| will succeed if whatever(NULL) asserts (e.g. that its argument
| isn't null). If whatever neglects to assert, then EXPECT_ASSERT
| will itself assert.
|
| Under the hood it works with setjmp and longjmp. The assert
| handler is temporarily overridden to a function which performs a
| longjmp which changes some hidden local state to record that the
| assertion went off.
|
| This will not work with APIs that leave things in a bad state,
| because there is no unwinding. However, the bulk of the assertion
| being tested are ones that validate inputs before changing any
| state.
|
| It's quite convenient to cover half a dozen of these in one
| function, as a block of six one-liners.
|
| Previously, assertions had to be written as individual tests,
| because the assert handler was overriden to go to a function
| which exits the process successfully. The old style tests are
| then written to set up this handler, and also indicate failure if
| the bottom of the function is reached.
| dvh wrote:
| My problem with assertions is that they can be turned off. I've
| seen things that could never happen happen in production on daily
| basis.
| TrianguloY wrote:
| The one assertion per test doesn't mean you need to use only one
| assertion call but rather that you only need to do one assertion
| block. Checking everything after a response is considered 1
| assertion, no matter how many assert calls you need.
|
| The issue is when you use multiple assertions for multiple logic
| statements: do > assert > do > assert... In that example imagine
| that you were also checking that the reservation was successful.
| That would be considered bad, you should create a different test
| that checks for that (testCreate + testDelete) and just have the
| precondition that the delete test has a valid thing to delete
| (usually added to the database on the setup).
| superjan wrote:
| I think the issue is that you'll always have one of those
| teammates who see this as an excuse to test the entire happy
| flow and all its effects in a single test case. I think what
| you want is reasonable, but how do you agree when it is no
| longer reasonable?
| TrianguloY wrote:
| If you logic depends on that happy path, make a test for it.
| But as I explained in another comment that test should not
| justify the lack of individual feature tests, which should
| not only test the happy path but other corner cases too.
|
| On my company we developers usually create white-box
| unitary/feature tests (we know how it was implemented, so we
| check components knowing that). But then we have an
| independent QA team that creates and run black-box flow tests
| (they don't know how it was implemented, only what it should
| do and interact)
| superjan wrote:
| Sounds like a fine approach, and I wasn't criticizing.
| Mostly I was pondering out loud why people come up with
| blanket statements what good tests should look like.
| christophilus wrote:
| Sometimes, it takes a lot less code to test a particular code
| by doing it in a long test with multiple assertions.
|
| Migrate up -> assert ok -> rollback 1 -> assert ok -> rollback
| 2 -> assert ok
|
| I don't see much benefit to breaking it up, and you're testing
| state changes between each transition, so the entire test is
| useful and simpler, shorter, and clearer than the alternative.
| twic wrote:
| Sometimes it's also more meaningful to the reader this way.
|
| Imagine an object which sees numbers, tries to pair up
| identical numbers, and reports the set of unpaired numbers. A
| good test would be:
|
| Create object
|
| Assert it has no unpaired numbers
|
| Show it 1, 2, and 3
|
| Assert it has 1, 2, and 3 as the unpaired numbers
|
| Show it 1 and 3
|
| Assert it has 2 as the unpaired number
|
| This test directly illustrates how the state changes over
| time. You could split it into three tests, but then someone
| reading the tests would have to read all three and infer what
| is going on. I consider that strictly worse.
| seadan83 wrote:
| This can be a path where things do go bad. Let's say thus
| test pattern is a success and then is replicated for many
| tests. Now, the schema or migration changes. A small change
| there now breaks the entire test suite. At this point the
| number of failing tests only indicates how many hours you
| will be fixing assertions.
|
| Another failure mode is when test scaffolding builds up.
| Imagine that migrate up part becoming multiple schemas, or
| services. It then fails, now finding exactly where to fix the
| test scaffolding becomes a multi-hit exercise.
|
| I'm not saying the example is bad, but it can put you on a
| path where if you constantly build on top of it, it can bad
| (eg, developers that don't care for tests nor test code
| quality, or just want to go home, and they just add a few
| assertions, add some scaffolding, copy-paste it all and
| mutate some assertions for a different table & rinse-wash-
| repeat across 4 people, 40 hours a week for 3 years...)
| christophilus wrote:
| Sorry the example was vague. It was a test for the
| migration library itself-- something I wrote recently when
| playing around with building a 0-dependency web framework
| for Bun.
|
| I wouldn't actually write tests for migrations themselves.
| TrianguloY wrote:
| That test is asserting a flow, not features. How can you be
| sure that the second rollback fails because that rollback
| code is wrong and not because the previous two functions made
| some unexpected changes? Or rather, how can you be sure that
| the second rollback is ok if you dont know the state it was
| run from? Maybe the migration set an unexpected flag that
| made the rollbacks pass that test without working properly.
|
| This is also the reason why tests should be run in arbitrary
| order, to avoid unexpected interactions due to order.
|
| Flow tests can be useful in some situations, but they should
| never replace individual feature tests.
| zabil wrote:
| I like the one assertion per unit test case rule. It makes me
| design cleaner code and write readable test cases. I also use
| object equality vs asserting on individual fields.
| someweirdperson wrote:
| A maximum of one assert per test seems even worse than the church
| of "a maximum of one test for every instruction that needs
| coverage".
|
| Both seem to be driven by misunderstanding through simplification
| of some otherwise meaningful ideas.
| 0x445442 wrote:
| A test should have as many assertions as needed to test an
| interface. If you find you're needing a lot of assertions to do
| that then I suspect either the interface under test is too large
| or you're testing multiple stack frames. In my experience, it's
| usually the latter; I call it accidental testing. Those delegate
| calls should have their own tests.
| wccrawford wrote:
| First off, I _do_ put more than 1 assertion in a test. But it
| definitely leads to situations where you have to investigate why
| a test failed, instead of it just being obvious. Like the
| article, I test 1 thing per test, but sometimes that means
| multiple assertions about the outcome of a test.
|
| IMO there's no point in checking that you got a response in 1
| test, and then checking the content/result of that response in
| another test. The useful portion of that test is the response
| bit.
| _lambdalambda wrote:
| Even with multiple assertions the test failure reason should be
| quite clear as most testing frameworks allow to specify a
| message which is then output in the testing summary.
|
| E.g. `assertEqual(actual_return_code, 200, "bad status code")`
| should lead to output like `FAILED:
| test_when_delete_user_then_ok (bad status code, expected 200
| got 404)`
| TeMPOraL wrote:
| Even if you don't specify the message, _at the very minimum_
| , the output should look like: FAILED:
| test_when_delete_user_then_ok Assertion failed:
| `actual_return_code' expected `200', got `400'.
|
| Note it mentions the _actual expression_ put in the assert.
| Which makes it almost always uniquely identifiable within the
| test.
|
| That's the _bare minimum_ I 'd expect of a testing framework
| - if it can't do that, then what's the point of having it?
| It's probably better to just write your own executable and
| throw exceptions in conditionals.
|
| What I _expect_ from a testing framework is _at least_ this:
| FAILED: test_when_delete_user_then_ok Assertion
| failed: `actual_return_code' expected `200', got `400'.
| In file: '/src/blah/bleh/blop/RestApiTests.cs:212'.
|
| I.e. to also identify the file and the line containing the
| failing assertion.
|
| If your testing framework doesn't do that, then again, what's
| even the point of using it? Throwing an exception or calling
| language's built-in assert() on a conditional will likely
| provide at least the file+line.
| iLoveOncall wrote:
| > IMO there's no point in checking that you got a response in 1
| test, and then checking the content/result of that response in
| another test. The useful portion of that test is the response
| bit.
|
| If I understood this part correctly, you are making the
| dangerous assumption that your tests will run in a particular
| order.
| wccrawford wrote:
| No, I definitely am not making that assumption. With a bad
| response, but a good response code, 1 test would fail and the
| other would succeed, no matter the order. I just don't think
| that that valid response code is a useful test on its own.
| It's much better with both assertions in the same test,
| unless you have some reason to think that response code
| failure would signify something special on its own.
| daviddever23box wrote:
| ...which is why it may be worthwhile to chain some assertions
| together in a single test.
| tetha wrote:
| IMO, the opposite also has to be considered. I've briefly
| worked with some code bases that absolutely did 1 assert per
| test. Essentially you'd have a helper method like
| "doCreateFooWithoutBarAttribute", and 3-4 tests around that -
| "check that response code is 400", "check that error message
| exists", and so on. Changes easily caused 4-5 tests to fail all
| at once, for example because the POST now returned a 404, but
| the 404 response also doesn't contain the error message and so
| on.
|
| This also wasted time, because you always had to look at the
| tests, and eventually realized that they all failed from the
| same root cause. And sure, you can use test dependencies if
| your framework has that and do all manner of things... or you
| just put the asserts in the same test with a good message.
| commandlinefan wrote:
| > have to investigate why a test failed
|
| Still better than investigating why the whole system failed,
| though.
| philliphaydon wrote:
| Maybe it's different in other languages but in JS and .NET the
| failed assertion fails and you investigate the failed
| assertion. You wouldn't ever have a situation that isn't
| obvious.
|
| If an assertion says "expected count to be 5 but got 4" you
| wouldn't be looking at the not null check assertion getting
| confused why it's not null...
| indymike wrote:
| I've seen this pattern often with teams that are out of their
| normal wheelhouse (i.e. Python shop doing Go or JS shop doing
| Java) and think they cannot extend a bad test library. The other
| place you see this a lot is where developer hours are being
| billed. I can sympathize with devs who are shallow on a test
| library, but the billing hours one is a dark pattern for
| optimizing cost plus billing.
| andrewallbright wrote:
| I've spent some time testing and thinking about testing. I think
| the universal guidance that should be followed is this...
|
| Do whatever you need/want to that makes your software successful.
| The rules are meant to be broken if it means you shipping.
| choeger wrote:
| What? People really would criticize that code because it has two
| assertions? How are they ever testing _any_ state changes?
|
| And to the author: Your bubble is significantly different from
| mine. Pretty much every competent developer I've worked with
| would laugh at you for the idea that the second test case would
| not be perfectly fine. (But that first iteration would never pass
| code review either because it does nothing and thus is a waste of
| effort.)
| mollerhoj wrote:
| To answer your question: We zealots test for the fact that
| something changes to some degree. E.g with rubys rspec library:
|
| expect { foo.call() }.to change { bar.value }.by(2)
|
| That is, regardless of the absolute value of bar.value, I
| expect foo.call() to increment it by 2.
|
| The point of the 1 assertion per test guideline is to end up
| with tests that are more focused. Giving that you did not seem
| to think of the above technique, I'd say that this guideline
| might just have helped you discover a way to write better specs
| ;-)
|
| Guidelines (that is, not rules) are of course allowed to be
| broken if you have a good reason to do so. But not knowing
| about common idioms is not a good reason.
|
| You might argue that the above code is just sugar for 2
| assertions, but thats beside the point: The test is more
| focused, there -appears- to be only one assertion, and thats
| what matters.
| choeger wrote:
| I think you forgot at least one valid assertion and implied
| another one:
|
| foo.call() might have a return value.
|
| Also, the whole story invocation shouldn't throw an
| exception, if your language has them. This assertion is often
| implied (and that's fine), but it's still there.
|
| Finally the test case is a little bit stupid, because very
| seldom code doesn't have any input that changes the
| behavior/result. So your assertion would usually involve that
| input.
|
| If you follow that though consequently, you end up with
| property-based tests very soon. But property-based tests
| should have as many assertions as possible for a single point
| of data. Say you test addition. When writing property-based
| tests you would end up with three specifications: one for one
| number, testing the identity element and the relationship to
| increments. Another one for two numbers, testing
| commutativity and inversion via subtraction, and one for
| three numbers, testing associativity. In every case it would
| be very weird to not have all n-ary assertions for the
| addition operation in the same spot.
| mollerhoj wrote:
| When you say I 'forgot' an assertion, are you implying that
| test should include all possible assertions on the code?
| That would perhaps cover more surface, but my goal (read
| zealot ideology) here is to have the tests help document
| the code:
|
| test "pressing d key makes mario move 2 pixels right" {
|
| expect { keyboard.d() }.to change { mario.x }.by(2)
|
| }
|
| I _could_ test the value of the d() function, but I dont
| because I don 't care what it returns.
|
| Didnt understand the "whole story invocation" and exception
| part, am I missing some context?
|
| Sure property-based testing can be invaluable in many
| situations. Only downside is if the tests become so complex
| to reason about that bugs become more likely in the tests
| than the implemenation.
|
| I've sometimes made tests with a manual list of inputs and
| a list of expected outputs for each. I'd still call that 1
| assertion tests (just run multiple times), so my definition
| of 1 assertion might too broad..
| AlphaSite wrote:
| I think a much better rule of thumb is: "A lot of small unit
| tests are better than a few big ones". Same thing, but
| clearer intent and less rigid.
| krona wrote:
| You tested a postcondition. What about preconditions and
| invariants, do you have separate tests for those assertions
| too, or just not bother?
| mollerhoj wrote:
| Please correct me if I'm wrong, but would a precondtion not
| just be the postcondition of the setup?
|
| Invariants would either have to be publically available and
| thus easily testable with similar methods, or, one would
| have to use assertions in the implemention.
|
| I try to avoid the latter, as it mixes implemations and
| 'test/invariants'. Granted, there are situations (usually
| in code that implements something very 'algorithm'-ish)
| where inline assertions are so useful that it would be
| silly to avoid them. (But implementing algos from scratch
| is rare in commercial code)
| philliphaydon wrote:
| That's because the example test only requires 1 assertion.
|
| Any rule that says there should be only 1 assertion ever is
| stupid.
| mollerhoj wrote:
| OP asked how any state change would be tested with a single
| 'assertion' and I provided an answer. Absolute rules are
| stupid, but our codebase has just short of 10k tests, and
| very few have more than one assertion.
|
| The only reason I can really see to have more than one
| assertion would be to avoid having to run the
| setup/teardown multiple times. However, its usually a
| desirable goal to write code that require little
| setup/teardown to test anyways because that comes with
| other benefits. Again, it might not be practical or even
| possible, but that goes of almost all programming "rules"..
| scruple wrote:
| Interesting. Our (Rails) codebase is around 25,000 tests
| and less than half have a single assertion. Personally,
| there's some calculus in my head when I'm writing a test
| that determines if/when the scenario I'm testing needs
| multiple assertions.
| mollerhoj wrote:
| rspec or minitest? ;-) Could rspecs 'expect change' idiom
| be the difference?
|
| I find that reducing assertions per spec where I can a
| good guideline. E.g. combining expect(foo['a']).to eq(1)
| and expect(foo['b']).to eq(2) into expect(foo).to
| include('a' => 1, 'b' => 2) yields better error messages.
| mannykannot wrote:
| The amount of setup and teardown necessary to test
| something is a property of the system under test. It is
| not susceptible to one's opinion as to how things should
| be.
| mollerhoj wrote:
| There are usually different ways to design a system. Its
| often the case that designing the system such that it is
| easy to test (with little setup/teardown) has other
| benefits too. E.g. It often indicates low coupling and a
| more simple design.
|
| That being said, there can of course me other tradeoffs
| e.g. performance and even cases where simple test setups
| are downright impossible.
| philliphaydon wrote:
| If there's 2 tests which are identical but assert 2
| different things. It should be a single test with 2
| assertions.
|
| Can always refactor to 2 tests if the test setup changes
| and the assertions begin to differ or become too complex.
| dd82 wrote:
| one assert per test seems... as you said, indicative of
| zealotry. if you already have your object there, why not
| test for the changes you expect?
|
| So you have one test that indicates that a log error is
| outut. then another that tests that the property X in the
| return from the error is what you expect. then another
| test to determine that propery Y in return is what you
| expect?
|
| that to me is wasteful, unclear, bloated. About the only
| useful result I can see that is it allows bragging about
| how many tests a project has.
| jeremyjh wrote:
| If two tests call the same method with the same setup and
| arguments just to assert two different outcomes I would
| suggest _that_ is the code smell.
| mannykannot wrote:
| Furthermore, if you have a one-assertion rule, some
| bright spark will realize he can write a single assertion
| that checks for the conjunction of all the individual
| postconditions.
|
| That's one way to get _dogma-driven_ assertion roulette,
| as you will not know which particular error occurred.
| seadan83 wrote:
| If all assertions are at the end of the test, then yes.
| Sometimes this can be made nice with custom matchers, eg:
|
| assertThat(fooReturningOptional(), isPresentAndIs(4))
|
| Or
|
| assertThat(shape, hasAreaEqualTo(10))
|
| Or
|
| AssertThat(polygonList, hasNoIntersections())
|
| Custom matchers can go off the deep end really easily.
| One of those cases of learn the principle, then learn
| when it does not apply
| drewcoo wrote:
| > But that first iteration would never pass code review either
| because it does nothing and thus is a waste of effort.)
|
| That first iteration would not be subject to code review. The
| author is using TDD.
|
| https://en.wikipedia.org/wiki/Test-driven_development
| pydry wrote:
| There's a lot of not very competent people in the industry who
| cling tightly to dogma.
|
| Testing (especially unit) is an area of tech weirdly with a lot
| of dogmatism. I think Uncle Bob is the source of some of it.
| musingsole wrote:
| I'm convinced if you read Uncle Bob carefully and follow all
| his suggestions... you'll have completely incapacitated
| whatever organization you infiltrated.
| klysm wrote:
| Then you need to hire consultants to come fix it!
| ziml77 wrote:
| It took me a long time to feel okay with all the times I broke
| the single assertion "rule" after reading Clean Code. In fact I
| only recently stopped feeling bad at all when I went to
| reimplement a bunch of abseil's flat hash map tests in C#. All of
| the tests assert multiple things. If a project as big as that can
| have multiple asserts on a basic data structure then I can too
| throwaway0asd wrote:
| My learning from test automation:
|
| * The primary goal of test automation is to prevent regression. A
| secondary goal can be performance tuning your product.
|
| * Tests are tech debt, so don't waste time with any kind of
| testing that doesn't immediately save you time in the near term.
|
| * Don't waste your energy testing code unless you have an
| extremely good reason. Test the product and let me the product
| prove the quality of your code. Code is better tested with
| various forms of static analysis.
|
| * The speed with which an entire test campaign executes
| determines, more than all other factors combined, when and who
| executes the tests. If the test campaign takes hours nobody will
| touch it. Too painful. If it takes 10-30 minutes only your QA
| will touch it. When it takes less than 30 seconds to execute
| against a major percentage of all your business cases everybody
| will execute it several times a day.
| commandlinefan wrote:
| * tests are a handy way to run one single function in isolation
| without having to spin up an instance and navigate the UI, too.
| jeremyjh wrote:
| Tests are not tech debt. You could have bad, brittle tests that
| you could consider debt but just having tests isn't debt. Debt
| implies there is something you could do about it in the future
| to pay it down, which isn't the case for a good test suite.
| ronnier wrote:
| It's debt. When you can't add new features quickly because
| you have nightmarish tests to fix and you spend more time on
| the tests than the product, I'd say it's debt. Especially
| with the insane mocking setups.
| jlarocco wrote:
| I think that's projection.
|
| Your tests being a nightmare doesn't imply my tests are a
| nightmare.
| marcosdumay wrote:
| Tests _can_ carry tech debit, just like any code. They
| certainly are not identified by it.
|
| Tests are one of the ways you have to ensure your code is
| correct. Consequently, they are business-oriented code that
| exist to support your program usage, and subject to its
| requirements. How much assurance you need is completely
| defined by those requirements. (But how you achieve that
| assurance isn't, and tests are only one of the possible
| tools for that.)
| hbrn wrote:
| They are still debt since they don't directly contribute
| to product value: you can delete all your tests and your
| software will keep functioning.
|
| It doesn't mean it's a debt worth taking, though IME most
| companies are either taking way too much or way to
| little. Not treating tests as debt typically leads to
| over-testing, and it is _way_ worse than under-testing.
|
| Also, what you're talking about (business-oriented
| requirements) is more akin to higher level tests
| (integration/e2e), not unit tests.
| Rexxar wrote:
| You can also delete the source code after compiling it
| and your software will keep functioning. Does it mean
| that code don't directly contribute to product value ?
| jeremyjh wrote:
| If you can simply delete it that's not debt, it's just
| cost. My bank doesn't let me delete my loan obligation
| and still live in my house.
| jeremyjh wrote:
| Yes, those are the bad tests I was referring to. NOT having
| tests greatly increases the debt burden of your production
| code because you cannot refactor with any confidence and so
| you simply won't.
| hbrn wrote:
| > Yes, those are the bad tests I was referring to
|
| This is the No True Scotsman issue with testing. When it
| fails, you just disregard the failure as "bad tests". But
| any company that has anything that resembles a testing
| culture will have a good amount of those "bad tests". And
| this amount is _way_ higher than people are willing to
| admit.
|
| > you cannot refactor with any confidence
|
| Anecdotally, I've had way more cases where I wouldn't
| refactor because too many "bad tests" were breaking, not
| because I lacked confidence due to lack of tests.
|
| There are many things beyond tests that allow you
| refactor with confidence: simple interfaces, clear
| dependency hierarchy, modular design, etc. They are _way_
| more important than tests.
|
| Tests are often a last resort when all of the above is a
| disaster. When you're at a place where you _need_ tests
| to keep your software stable you are probably already
| fucked, you 're just not willing to recognize it.
|
| You shouldn't have zero tests, but tests should be
| treated as debt. The fewer tests you _need_ to keep your
| software stable, the better your architecture is. Huge
| number of tests in a codebase is typically a signal of
| shitty architecture that crumbles without those crutches.
| kmac_ wrote:
| Multiple asserts are fine, multiple aspects in a single test are
| not. For example like a transaction test that checks if account
| balance is ok and if an user gets a notification. It boils down
| to a single responsibility, so when there are multiple aspects
| then it means that the class/interface should be refactored.
| rhdunn wrote:
| I view it more as "only test one operation per unit test". If
| that needs multiple asserts (status code, response content,
| response mime type, etc.) to verify, that is fine.
|
| IIUC, the guideline is so that when a test fails you know what
| the issue is. Therefore, if you are testing more than one
| condition (missing parameter, invalid value, negative number,
| etc.) it is harder to tell which of those conditions is failing,
| whereas if you have each condition as a separate test it is clear
| which is causing the failure.
|
| Separate tests also means that the other tests run, so you don't
| have any hidden failures. You'll also get hidden failures if
| using multiple assertions for the condition, so will need to re-
| run the tests multiple times to pick up and fix all the failures.
| If you are happy with that (e.g. your build-test cycle is fast)
| then having multiple assertions if fine.
|
| Ultimately, structure your tests in a way that best conveys what
| is being tested and the conditions it is being tested under (e.g.
| partition class, failure condition, or logic/input variant).
| ronnier wrote:
| I've come to the conclusion that none of this matters for most
| parts of a system. I worked in the most horrendous code and
| systems you can imagine but it turned into a multi billion
| dollar company. Then everyone starts talking about code quality
| and rewrites etc and new features stall as beautiful systems
| are written and high test coverage met and competing companies
| surpass us and take market share with new and better features.
| We've gotten religious over code and tests in the software
| industry abd should probably shift back some.
| dxdm wrote:
| I've worked on shitty code with shitty tests that ran the
| core of the business. Even while doing that, it was horrible
| to work with, held important features back and drove talented
| people away, leaving everything to stagnate in a "this works
| enough" state. When the wind changed, it was hard to turn the
| ship around, important people got nervous, and things got
| into a bad spiral.
|
| None of this is the failure of code and tests alone; but both
| can be indicative of the structural health and resilience of
| the wider situation.
| lodovic wrote:
| Wait - people are doing real http calls in unit tests, over a
| network, and complaining about multiple asserts in the test code?
| [deleted]
| batiste wrote:
| I have met people that thinks HTTP calls are great!
|
| - "But it tests more things!"
|
| Well ok, but those are integration tests, not unit tests... It
| is unacceptable that a unit tests can fail because of external
| system...
| flqn wrote:
| It's not unusual to spin up a local dev server in the same
| environment (or on the same machine, ie at localhost) as the
| tests. There's an argument to say these aren't "unit" tests but
| your definition of "unit" may vary.
| warrenmiller wrote:
| They can be mocked surely?
| drewcoo wrote:
| It's not nice to mock people "doing real http calls in unit
| tests," even if they deserve it.
| warrenmiller wrote:
| Oh dear
| nsoonhui wrote:
| For mocking, the parent comment means "swapping out the
| external calls with dummy calls", not "laughing at the
| developer"
| ThunderSizzle wrote:
| Woops, did you see the joke fly right past you?
| BurningFrog wrote:
| You failed a humor unit test.
| kevincox wrote:
| I'm 98% sure that they understood that and it was a joke.
| mollerhoj wrote:
| Hopefully the responses are stored locally and replayed on
| subsequent runs.
| tinyspacewizard wrote:
| Still a bit flaky. In an OOP language mocking is appropriate
| or in an FP language defunctionalization.
| mollerhoj wrote:
| Heh yeah but it can be used to write tests that check your
| assumptions about a 3 party api. Granted, It'll only fail
| once the test are rerun without the cache, but it can still
| be a valuable technique. It can be valuable to have a
| testsuite that a) helps check assumptions when implementing
| the connection and b) helps locate what part of it later
| starts to behave unexpectedly.
| tester756 wrote:
| define "real"
|
| yes, we perform "somewhat real tests" - tests start the app,
| fake db, and call HTTP APIs
|
| it's really decent
| chopin wrote:
| Where I work milestone and release builds cannot make http
| calls other than to our maven repo. It's meant to further
| reproducible builds but your tests can't do this as well. I
| fire up an in-memory database to make my tests self containing.
| jmillikin wrote:
| Is it weird that not only have I never heard of the "rule" this
| post argues against, but I can't even conceive of a code
| structure where it would make sense?
|
| How would a test suite with one assertion per test work? Do you
| have all the test logic in a shared fixture and then dozens of
| single-assertion tests? And does that rule completely rule out
| the common testing pattern of a "golden checkpoint"?
|
| I tried googling for that rule and just came up with page after
| page of people arguing against it. Who is _for_ it?
| littlecranky67 wrote:
| > Is it weird that not only have I never heard of the "rule"
| this post argues against
|
| This "rule" is known mostly because it is featured in the
| "Clean Code" book by Robert C. Martin (Uncle Bob). You should
| have heard of it ;)
| TeMPOraL wrote:
| Perhaps the author is better off not having heard of it then,
| and by implication, not having read "Clean Code" in the first
| place. The book is full of anti-patterns.
| [deleted]
| Kwpolska wrote:
| The book is so full of bad advice I'm not surprised this
| "rule" comes from there as well.
| teddyh wrote:
| As MikeDelta reports1, the book _doesn't_ actually say
| that.
|
| I've come to learn to completely disregard any non-specific
| criticism of that book (and its author). There is
| apparently a large group of people who hate everything he
| does and also, seemingly, him personally. Everywhere he (or
| any of his books) is mentioned, the haters come out, with
| their vague "it's all bad" and the old standard "I don't
| know where to begin". Serious criticism can be found (if
| you look for it), and the author himself welcomes it, but
| the enormous hate parade is scary to see.
|
| 1. https://news.ycombinator.com/item?id=33480517
| jmillikin wrote:
| Heard of it, but never read it.
|
| Looking at the Amazon listing and a third-party summary[0] it
| seems to be the sort of wool-brained code astrology that was
| popular twenty years ago when people were trying to push
| "extreme programming" and TDD.
|
| [0] https://gist.github.com/wojteklu/73c6914cc446146b8b533c09
| 88c...
| willsmith72 wrote:
| Wait what's wrong with XP and TDD? Where I'm from those are
| absolutely considered best practice for most situations
| matthews2 wrote:
| Are there any "rules" in Clean Code that don't need to be
| disregarded and burned? https://qntm.org/clean
| pydry wrote:
| There's plenty of sensible advice in there it's just that
| he argues for the sensible stuff and the idiotic stuff with
| equal levels of conviction and if you are junior you aren't
| going to be able to distinguish them.
|
| It would be easier if it were all terrible advice.
| stephencanon wrote:
| Fortunately you readily find the good advice in lots of
| other places, so you can simply ignore that book.
| teddyh wrote:
| Where in that book is the rule stated? I ask because I have
| heard the author explicitly state that multiple assertions
| are fine (using essentially the same explanation as
| TrianguloY did in this comment:
| https://news.ycombinator.com/item?id=33480120).
| MikeDelta wrote:
| Chapter 9 talks about unit tests and there is a paragraph
| called 'Single Assert per Test', where he says it is a good
| concept but that he is not afraid to put more asserts in
| his tests.
|
| That paragraph is followed by 'Single Concept per Test'
| where he starts with: 'Perhaps a better rule is that we
| want to test a single concept per test.'
|
| So, technically he doesn't say it.
| ThunderSizzle wrote:
| That maps better with the lectures I've seen of him on
| YouTube, and I concur with it.
|
| When I first wrote tests years ago, I would try to test
| everything in one test function. I think juniors have a
| tendency to do that in functions overall - it's par to
| see 30-100+ line functions that might be doing just a
| little too much on their own, and test functions are no
| different.
| daviddever23box wrote:
| It feels as if folks are splitting hairs where a haircut
| (or at least a half-hearted attempt at grooming) is
| required. Use good judgment, and do not blindly follow
| "rules" without understanding their effects.
| [deleted]
| mollerhoj wrote:
| I believe theres a rubocop that checks for it:
|
| https://docs.rubocop.org/rubocop-minitest/cops_minitest.html...
| raverbashing wrote:
| Nothing like machine-enforced nitpicking of the worse kind
|
| But even this rule/plugin has a default value of 3 which is a
| more sane value.
| nesarkvechnep wrote:
| Are people so entrenched in dogmas that we need "Stop doing X"
| type articles every few months?
| quonn wrote:
| Unfortunately, yes. People with good judgement are difficult to
| find.
| zzzeek wrote:
| "A foolish consistency is the hobgoblin of little minds"
|
| - whoever
|
| I definitely write tests with multiple assertions, the rule I try
| to follow is that the test is testing a single cause/effect. that
| is, a single set of inputs, run the inputs, then assert as many
| things as you want to ensure the end state is what's expected.
| there is no problem working this way.
| [deleted]
| [deleted]
___________________________________________________________________
(page generated 2022-11-05 23:01 UTC)