https://stackoverflow.blog/2022/11/03/multiple-assertions-per-test-are-fine/

*
 
Essays, opinions, and advice on the act of computer programming from
Stack Overflow.
Search for: [                    ] [Search]
    
Latest Newsletter Podcast Company
[110222-Stack-Overflow-Multiple-assertions-per-test-are-fine-1200x630]
code-for-a-living November 3, 2022

Stop requiring only one assertion per unit test: Multiple assertions
are fine

One test case, not one test assertion.
Avatar for
Mark Seeman

Assertion Roulette doesn't mean that multiple assertions are bad.

When I coach teams or individual developers in test-driven
development (TDD) or unit testing, I frequently encounter a
particular notion: Multiple assertions are bad. A test must have only
one assertion.

That idea is rarely helpful.

Let's examine a realistic code example and subsequently try to
understand the origins of the notion.

Outside-in TDD

Consider a REST API that enables you to make and cancel restaurant
reservations. First, an HTTP POST request makes a reservation:

POST /restaurants/1/reservations?sig=epi301tdlc57d0HwLCz[...] HTTP/1.1
Content-Type: application/json
{
  "at": "2023-09-22 18:47",
  "name": "Teri Bell",
  "email": "terrible@example.org",
  "quantity": 1
}

HTTP/1.1 201 Created
Content-Type: application/json; charset=utf-8
Location: /restaurants/1/reservations/971167d4c79441b78fe70cc702[...]
{
  "id": "971167d4c79441b78fe70cc702d3e1f6",
  "at": "2023-09-22T18:47:00.0000000",
  "email": "terrible@example.org",
  "name": "Teri Bell",
  "quantity": 1
}

Notice that in proper REST fashion, the response returns the location
of the created reservation in the Location header.

If you change your mind, you can cancel the reservation with a DELETE
request:

DELETE /restaurants/1/reservations/971167d4c79441b78fe70cc702[...] HTTP/1.1

HTTP/1.1 200 OK

Imagine that this is the desired interaction. Using outside-in TDD
you write the following test:

[Theory]
[InlineData(884, 18, 47, "c@example.net", "Nick Klimenko", 2)]
[InlineData(902, 18, 50, "emot@example.gov", "Emma Otting", 5)]
public async Task DeleteReservation(
    int days, int hours, int minutes,
    string email, string name, int quantity)
{
    using var api = new LegacyApi();
    var at = DateTime.Today.AddDays(days).At(hours, minutes)
        .ToIso8601DateTimeString();
    var dto = Create.ReservationDto(at, email, name, quantity);
    var postResp = await api.PostReservation(dto);
    Uri address = FindReservationAddress(postResp);

    var deleteResp = await api.CreateClient().DeleteAsync(address);

    Assert.True(
        deleteResp.IsSuccessStatusCode,
        $"Actual status code: {deleteResp.StatusCode}.");
}

This example is in C# using xUnit.net because we need some language
and framework to show realistic code. The point of the article,
however, applies across languages and frameworks. The code examples
in this article are based on the sample code base that accompanies my
book Code That Fits in Your Head.

In order to pass this test, you can implement the server-side code
like this:

[HttpDelete("restaurants/{restaurantId}/reservations/{id}")]
public void Delete(int restaurantId, string id)
{
}

While clearly a no-op, this implementation passes all tests. The
newly-written test asserts that the HTTP response returns a status
code in the 200 (success) range. This is part of the API's REST
protocol, so this response is important. You want to keep this
assertion around as a regression test. If the API ever begins to
return a status code in the 400 or 500 range, it would be a breaking
change.

So far, so good. TDD is an incremental process. One test doesn't
drive a full feature.

Since all tests are passing, you can commit the changes to source
control and proceed to the next iteration.

Strengthening the postconditions

You should be able to check that the resource is truly gone by making
a GET request:

GET /restaurants/1/reservations/971167d4c79441b78fe70cc702[...] HTTP/1.1

HTTP/1.1 404 Not Found

This, however, is not the behavior of the current implementation of
Delete, which does nothing. It seems that you're going to need
another test.

Or do you?

One option is to copy the existing test and change the assertion
phase to perform the above GET request to check that the response
status is 404:

[Theory]
[InlineData(884, 18, 47, "c@example.net", "Nick Klimenko", 2)]
[InlineData(902, 18, 50, "emot@example.gov", "Emma Otting", 5)]
public async Task DeleteReservationActuallyDeletes(
    int days, int hours, int minutes,
    string email, string name, int quantity)
{
    using var api = new LegacyApi();
    var at = DateTime.Today.AddDays(days).At(hours, minutes)
        .ToIso8601DateTimeString();
    var dto = Create.ReservationDto(at, email, name, quantity);
    var postResp = await api.PostReservation(dto);
    Uri address = FindReservationAddress(postResp);

    var deleteResp = await api.CreateClient().DeleteAsync(address);

    var getResp = await api.CreateClient().GetAsync(address);
    Assert.Equal(HttpStatusCode.NotFound, getResp.StatusCode);
}

This does, indeed, prompt you to properly implement the server-side
Delete method.

Is this, however, a good idea? Is the test code easy to maintain?

Test code is code too, and you have to maintain it. Copy and paste is
problematic in test code for the same reasons that it can be a
problem in production code. If you later have to change something,
you have to identify all the places that you have to edit. It's easy
to miss one, which can lead to bugs. This is true for test code as
well.

One action, more assertions

Instead of copy-and-pasting the first test, why not instead
strengthen the postconditions of the first test case?

Just add the new assertion after the first assertion:

[Theory]
[InlineData(884, 18, 47, "c@example.net", "Nick Klimenko", 2)]
[InlineData(902, 18, 50, "emot@example.gov", "Emma Otting", 5)]
public async Task DeleteReservation(
    int days, int hours, int minutes,
    string email, string name, int quantity)
{
    using var api = new LegacyApi();
    var at = DateTime.Today.AddDays(days).At(hours, minutes)
        .ToIso8601DateTimeString();
    var dto = Create.ReservationDto(at, email, name, quantity);
    var postResp = await api.PostReservation(dto);
    Uri address = FindReservationAddress(postResp);

    var deleteResp = await api.CreateClient().DeleteAsync(address);

    Assert.True(
        deleteResp.IsSuccessStatusCode,
        $"Actual status code: {deleteResp.StatusCode}.");
    var getResp = await api.CreateClient().GetAsync(address);
    Assert.Equal(HttpStatusCode.NotFound, getResp.StatusCode);
}

This means that you only have a single test method to maintain
instead of two duplicated methods that are almost identical.

But, some of the people I've coached might say, this test has two
assertions!

Indeed. So what? It's one single test case: Cancelling a reservation.

While cancelling a reservation is a single action, we care about
multiple outcomes:

  * The status code after a successful DELETE request should be in
    the 200 range.
  * The reservation resource should be gone.

Developing the system further, we might add more behaviors that we
care about. Perhaps the system should also send an email about the
cancellation. We should assert that as well. It's still the same test
case, though: Successfully cancelling a reservation.

There's nothing wrong with multiple assertions in a single test. The
above example illustrates the benefits. A single test case can have
multiple outcomes that should all be verified.

Origins of the single assertion notion

Where does the only one assertion per test notion come from? I don't
know, but I can guess.

The excellent book xUnit Test Patterns describes a test smell named
Assertion Roulette. It describes situations where it may be difficult
to determine exactly which assertion caused a test failure.

It looks to me as though the only one assertion per test 'rule' stems
from a misreading of the Assertion Roulette description. (I may even
have contributed to that myself. I don't remember that I have, but to
be honest I've produced so much content about unit testing over the
decades that I don't want to assume myself free of guilt.)

xUnit Test Patterns describes two causes of Assertion Roulette:

  * Eager Test: A single test verifies too much functionality.
  * Missing Assertion Message

You have an Eager Test when you're trying to exercise more than one
test case. You may be trying to simulate a 'session' where a client
performs many steps in order to achieve a goal. As Gerard Meszaros
writes regarding the test smell, this is appropriate for manual
tests, but rarely for automated tests. It's not the number of
assertions that cause problems, but that the test does too much.

The other cause occurs when the assertions are sufficiently similar
that you can't tell which one failed, and they have no assertion
messages.

That's not the case with the above example. If the Assert.True
assertion fails, the assertion message will tell you:

Actual status code: NotFound.
Expected: True
Actual:   False

Likewise, if the Assert.Equal assertion fails, that too will be
clear:

Assert.Equal() Failure
Expected: NotFound
Actual:   OK

There's no ambiguity.

One assertion per test

Now that you understand that multiple assertions per test are fine,
you may be inclined to have a ball adding assertions like there's no
tomorrow.

Usually, however, there's a germ of truth in a persistent notion like
the one test, one assertion 'rule'. Use good judgement.

If you consider what an automated test is, it's basically a
predicate. It's a statement that we expect a particular outcome. We
then compare the actual outcome to the expected outcome to see if
they are equal. Thus, in essence, the ideal assertion is this:

Assert.Equal(expected, actual);

I can't always attain that ideal, but whenever I can, I feel deep
satisfaction. Sometimes, expected and actual are primitive values
like integers or strings, but they might also be complex values that
represent the subset of program state that the test cares about. As
long as the objects have structural equality, such an assertion is
meaningful.

At other times I can't quite find a way to express the verification
step as succinctly as that. If I have to add another assertion or
two, I'll do that.

Conclusion

There's this notion that you're only allowed to write one assertion
per unit test. It probably originates from real concerns about
badly-factored test code, but over the years the nuanced test smell
Assertion Roulette has become garbled into a simpler, but less
helpful 'rule'.

That 'rule' often gets in the way of maintainable test code.
Programmers following the 'rule' resort to gratuitous copying and
pasting instead of adding another assertion to an existing test.

If adding a relevant assertion to an existing test is the best way
forward, don't let a misunderstood 'rule' stop you.

Tags: testing, unit tests
Podcast logo The Stack Overflow Podcast is a weekly conversation
about working in software development, learning to code, and the art
and culture of computer programming.

Related

[170622-Stack-Overflow-Upping-our-unit-testing-game-1200x630] 
code-for-a-living July 4, 2022

How Stack Overflow is leveling up its unit testing game

We neglected unit tests for a long time because our code base made
them difficult. But now we're putting in the work to change that.
Avatar for Wouter de Kort
Wouter de Kort
The Overflow Newsletter Banner newsletter July 15, 2022

The Overflow #134: Avoiding the difficulty bomb 

Perl in 2022, the legality of Googling the illegal, and fuzz tests
Avatar for Ryan Donovan
Avatar for Cassidy Williams
Ryan Donovan and Cassidy Williams
[081222-Stack-Overflow-How-to-interrogate-unfamiliar-code-1200x630] 
code-for-a-living August 15, 2022

How to interrogate unfamiliar code

Readable code is great, but not all code will be immediately
readable. That's when you get your interrogation tools.
Avatar for
Isaac Lyman
[101122-Stack-Overflow-How-observability-driven-development-creates-elite-performers-1200x630]
code-for-a-living October 12, 2022

How observability-driven development creates elite performers

Most organizations struggle to change their culture or find a formula
for success in difficult-to-mature processes. They don't always
understand their own systems. 
Avatar for
Colin Fallwell, Field CTO at SumoLogic

6 Comments

[245] David Bakin says:
3 Nov 22 at 1:13

IIRC (but could very well be wrong) initially in test frameworks
there were only assertions that threw exceptions. Plus it may also
have been the case that assertions didn't take descriptive comments
that were part of any resulting fail message in the log.

The first meant that in any given test run you would get limited
information: only the first failed assertion would run, nothing after
that. AND IT WAS THE HABIT of programmers to do a lot of setup in
individual unit tests (fixture setup notwithstanding) - e.g., first
read a record, _assert_ if the read failed, then call some method and
assert if _that_ failed, where the _second_ assert was the one you
were really interested in.

Then too, without messages and with multiple assertions that pass
followed with one that failed you had to pay close attention to the
log and the _line number_ listed there for _where_ the failure
happened. Especially if you had multiple assertions that "looked the
same", e.g., multiple tests `Assert(result != null)` after a bunch of
calls to getters.

So "no multiple assertions in tests" was something designed to force
you to use test _names_ - the only way of annotating what a
particular test was testing that showed up in the log - to be the
_specific pointer_ to what it was you were testing that failed.

Newer ("modern") frameworks allow both assertions that fail the test
but let it keep running (frequently spelled "Expect") and
per-assertion descriptions. Using both of these (plus using proper
structuring techniques - e.g., either setup methods in fixtures OR
just as good but not often done for some reason by people writing
unit tests: just abstracting setup into a method) alleviates the
problems noticed by users of those earlier frameworks.

IIRC.

Reply
[245] David Bakin says:
3 Nov 22 at 1:18

(BTW, w.r.t. to your last example of distinguishing assertions in the
log where you show two _very similar_ messages and then claim, thus
"there's no ambiguity" - why not just add a proper message as an
argument to those assertions which explains what your test for
`NotFound` or whatever _is actually testing_, semantically, in terms
the user (i.e., developer trying to troubleshoot some regression in
the build) will actually understand?

_That's_ where you gain readability/understandability for your tests,
that's where the tests become time-saving supplements to the dev when
troubleshooting (instead of frustrating time sinks), and that's where
a _very little extra_ time spent writing the test pays off.

IMO.)

Reply
[245] David Bakin says:
3 Nov 22 at 1:28

(Wish there was an _edit_ ability for these comments, oh well.)

Better example for the multiple assertions in a test: Call a method
passing in an object, on return from the method call _several_
methods and/or getters of the parameter object to ensure (from the
outside) it has the correct state. Ignoring whether this is the right
way to write O-O code, much less the right way to write the test
itself, "one assertion per test" means that _each_ of those tests of
state of that same object needs its own test. And, like I said above,
I think the reason is that early frameworks threw an exception on
every assert - no alternative to that - so if you wrote a test in
this way each test run would only show _one_ reason the state was
wrong (and the same one each time too, until you fixed it and got the
next one)

Reply
[01d] bartoszkp says:
3 Nov 22 at 7:52

I think this article is confusing. The problem with this is rule is
that it intends to say that each test should verify a single business
requirement, which can be thought of like a single business-level
assertion. It does mean that physically, depending on your testing
framework you will sometimes have more than one assert statements.
The rule about single conceptual assertion or single business-level
assertion IMHO should stand. This article seems to again conflate
assert statements with asserting business requirements and throws out
the baby with the bathwater by suggesting universally that always
multiple assertions are fine (with a vague caveat about "using
judgement").

The problem of extensive duplication is usually solved by having
parametrized tests, not by multiplying unrelated assertions.

Reply
[07b] Richard Wise says:
4 Nov 22 at 12:10

Interesting blog post and I like a lot of the messages and links that
you share, but I disagree with using GET returning 404 being your
source of truth that the entry has been deleted. In my opinion, the
unit under test here is the DeleteReservation function, not the
combination of `GetReservation` and `DeleteReservation`. To quote
from the article - "It's one single test case: Cancelling a
reservation", but clearly it is actually `Trying to get a reservation
after cancelling should fail`. To me, what you have is a `session`
(i.e. multiple steps), which is still a very valid test, but I
disagree that this is a unit test.

Instead, following black-box API unit testing, I would have one test
for `DeleteReservation` that validates that the delete call returns
the correct response and then have another test for `GetReservation
after deleted` that would setup the data and then call
`GetReservation`. The problem with this approach (and black-box unit
testing in general) is that tests cannot be fully independent because
if `DeleteReservation` is broken, the setup for `GetReservation after
deleted` will fail

Alternatively, following white-box API unit testing, I would have one
test for `DeleteReservation` that validates that the delete call
removes from the DB and returns the correct response. This is a good
example of where multiple assertions per unit test are valid, since
it is not practically possible to combine validation of DB deletion
call with HTTP response. I would recommend using something like
https://kotest.io/docs/assertions/soft-assertions.html allows
combining multiple assertions into a single assertion. Note that
semantically we have a single assertion (DeleteReservation should do
what it is expected to do), it is just a compound assertion -
`DeleteReservation` should `remove the reservation from the DB and
return the correct response`.

Just some thoughts!

Reply
[123] Robert Harvey says:
4 Nov 22 at 10:40

I've been waiting for someone to say this for awhile.

Reply

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked 
*

          [                                             ]
          [                                             ]
          [                                             ]
          [                                             ]
          [                                             ]
          [                                             ]
          [                                             ]
Comment * [                                             ]

Name * [                              ]

Email * [                              ]

Website [                              ]

[ ] Save my name, email, and website in this browser for the next
time I comment.

[Post Comment] 

 [                                             ] 
 [                                             ] 
 [                                             ] 
 [                                             ] 
 [                                             ] 
 [                                             ] 
 [                                             ] 
D[                                             ] 

This site uses Akismet to reduce spam. Learn how your comment data is
processed.

(c) 2022 All Rights Reserved.
Proudly powered by WordPress

Stack Overflow

About Press Work Here Contact Us Questions

Products

Teams Advertising Collectives Talent

Policies

Legal Privacy Policy Terms of Service Cookie Settings Cookie Policy

Channels

Blog Podcast Newsletter Twitter LinkedIn Instagram
*