[HN Gopher] Stop mocking your system
___________________________________________________________________
Stop mocking your system
Author : entanglement
Score : 75 points
Date : 2021-07-04 12:36 UTC (10 hours ago)
(HTM) web link (blog.bitgloss.ro)
(TXT) w3m dump (blog.bitgloss.ro)
| TeeMassive wrote:
| > And guess what... you have to keep the mocks in sync with the
| real things
|
| It is a valid problem but it's also possible to write mocks that
| are small and simple enough that this becomes trivial.
|
| In my experience when testing a module necessitates a complex
| mesh of mocks and an intricate knowledge of inbound dependencies
| than this means that the problem are not the tests nor mocks but
| rather a problem tight coupling. The tests are simply telling us
| that in an unexpected way.
|
| My rule of thumb is that if I can't mock it with a "every thing
| at default except what I want to test" than there's something
| wrong with the code, not the mocks.
|
| > What you almost always want, when mocking, is really just
| different input data for your module
|
| What you almost always want, when mocking, is really just
| different input data for your module, data normally provided by a
| long stream of collaborators. Just build your program in such a
| way that you can send it this data, regardless of the runtime
| (unit test framework, test env or production env). Yes, it is
| possible and highly desirable and don't be in denial right now.
|
| Mocks _are_ data in a nice, encapsulated and understandable form.
| I 'd rather have a simple mock than "data provided by a long
| stream of collaborators". I can't recall how many hours I've lost
| by trying to reverse engineer huge pile of data and configuration
| files because the "tech lead" wanted a "life insurance" to make
| sure we broke nothing.
| jnaddef wrote:
| The author seems to believe people either mock everything or
| don't mock anything. Obviously using mocks for all your tests is
| a very bad idea, but that's not how things are done generally.
|
| Unit tests allow you to validate a unit's behavior very quickly.
| If your unit test takes more than 1 second to run it is probably
| a bad unit test (some would argue 1/100 second max so your whole
| unit test suite can complete in a few seconds). In unit tests you
| use mocks not only to keep the test hermetic, but also to keep
| the execution time as low as possible.
|
| Then you should have integration & e2e tests where you want to
| mock as little as possible, because you want a behavior as close
| as production as possible. For those you care less about how long
| they take. That's because you usually don't run those tests at
| the same stage as unit tests (development vs release
| qualification).
|
| The author does not make the distinction between different types
| of testing, the resulting article is of pretty poor quality imho.
| billytetrud wrote:
| I've seen a lot of tests where people just mock everything by
| default without thinking. Smart programmers at a good company.
| It's an issue that does deserve more recognition. Abuse of
| mocks is bad for tests.
| jnaddef wrote:
| I know which company you are talking about :). I agree that
| abuse of mocks is bad for tests 100%. But when I clicked the
| link I was hoping to read an article giving a nuanced
| description of mocks, with some analysis on when to use and
| when to avoid mocks. Instead the article is just an opinion
| piece that just says "Stop using mocks" as if that was
| actually an option.
| pydry wrote:
| >The author seems to believe people either mock everything or
| don't mock anything.
|
| The author is saying that people frequently mock things that it
| would be more economic to just run because you've _got_ the
| real thing right there. Building a model for it is an expensive
| waste that probably won 't even match reality anyway and will
| demand constant maintenance to sync up with reality.
|
| If you're overtly concerned with the speed of your test suite
| or how fast individual tests run then you're probably the kind
| of person he's talking about. Overmocking tends to creep in
| with a speed fetish.
| jnaddef wrote:
| When I am developing a feature, I want to know very fast
| whether or not my code's logic is correct. It is not rare
| during the development cycle to run the same test dozens of
| times because I made a silly mistake (or a few), and
| obviously if the test takes 30 minutes to complete it
| completely wastes my day of work.
|
| Having a set of very fast running tests is absolutely
| necessary in my opinion.
|
| Once I have validated that the piece of code I wrote is doing
| what I intended, then I want to run other tests that do not
| use mocks/fakes, e2e tests that can possibly take a whole day
| to complete and will allow me to see if the whole system
| still works fine with my new feature plugged in. But this
| comes AFTER fast unit tests, and definitely cannot REPLACE
| those.
| wccrawford wrote:
| This sounds exactly right to me. You write mocks for the
| things that could take too much time to run frequently with
| the real code. (And I'm assuming you'd also write it for
| things that you don't want to make actual changes
| somewhere, such as a third-party API that you don't
| control.)
|
| But if it could be run locally, quickly, you wouldn't
| bother mocking it.
|
| If that's all correct, I think you and I would do the same
| things. All the people screaming "no mocks!" and "mock
| everything!" are scary, IMO.
| pydry wrote:
| Yeah, he's talking to you for sure.
| jnaddef wrote:
| Yep, and he did a very bad job at it (and so do you) if
| the goal was to change my mind. Do you maybe have
| arguments?
| alecbz wrote:
| I've certainly seen people who mock almost everything to test
| units at the smallest scale possible because they think that's
| what they're supposed to.
|
| E.g., I once saw someone test a factory method like:
| def make_thing(a, b, c): return thing(a, b, c)
|
| with a unit test where they mocked `thing`, and ensured that
| calling `make_thing(a, b, c)` ended up calling `thing(a, b,
| c)`.
|
| They write just a shit ton of tests like this for every single
| method and function, and end up writing ~0 tests that actually
| check for any meaningful correctness.
| AtlasBarfed wrote:
| harkens back to the early obsession with "100% code coverage"
| and java robots were coding tests on bean getters/accessors.
|
| 100% code coverage was a bad breadth-first metric when unit
| tests should be depth based on many variant inputs. Also,
| "100% code coverage" ignores the principle that80% of
| execution is in 20% of the code/loops, so that stuff should
| get more attention than worrying about every single line
| being unit tested.
|
| Well, unless you were in some fantastical organization of
| unicorn programmers that had an infinite testing budget and
| schedule...
| mikl wrote:
| Stop telling me what to do.
|
| (Using imperative tense when presenting your ideas to strangers
| is a really douchey thing to do)
| lilyball wrote:
| Why stop there? Why not delete your test methods too and just
| test in production?
|
| Mocks are just test code, same as your test functions. And
| they're necessary fur unit tests. If the thing you're testing
| talks to another component without a mock, it's now an
| integration test instead of a unit test.
|
| Unit tests test the API surface of a component. They're useful
| for ensuring a component adhere to its documented API contract.
| It's also useful for testing how it reacts to edge cases.
| Integration tests are useful for ensuring components work
| together, and testing that data flows through the system
| properly. But integration tests make it harder to test various
| edge cases.
|
| For example, if my component talks to yours and issues several
| requests to your component with callbacks, it may be that 99% of
| the time your component calls the callbacks in the same order. So
| I can't really validate in an integration test that my component
| works properly if the callbacks are involved in a different
| order. By mocking your component in a unit test, I can test any
| order I want.
|
| Unit tests also tend to be more deterministic. By mocking, I can
| eliminate sources of non-determinism from outside my component.
| As a trivial example, my component might need random numbers, so
| I might mock the random number generator to return a specific
| chosen sequence that I know will exercise different code paths.
|
| Really the lesson that should be taught here instead of "don't
| use mocks" is "unit tests aren't sufficient, write integration
| tests too".
| BirdieNZ wrote:
| > Why not delete your test methods too and just test in
| production?
|
| This, but unironically.
|
| Depending on your use case, any local tests, whether mocked or
| using a swarm of local containers attempting to represent
| production may be a far stretch from production reality. Put
| everything behind feature flags and test your contracts, then
| release to production regularly and test against live data and
| live services.
| aranchelk wrote:
| Ghost of Christmas Future:
|
| I did some contracting work for a company with a 1.5 hour test
| suite that ran on every deploy, the section with mocked tests
| only took a couple of minutes -- the rest were end-to-end (no
| mocks).
|
| The worst parts of those non-mocked tests:
|
| * They would interfere with each other, and could not be run in
| parallel.
|
| * They were subject to real-world variability and were not
| entirely deterministic.
|
| Management wouldn't budget fixing or replacing the tests. Bugs
| still regularly found their way into production; arguments to
| pare down the tests were vigorously rejected.
|
| My takeaways: If possible, use a language with a strong type
| system to avoid writing as many tests as possible. Move as much
| application logic as you can into pure code (so it can be tested
| in isolation). Observe the test pyramid.
| midrus wrote:
| Static typing has nothing to do with testing, or bugs. Static
| typing is about tooling, autocompletion and being able to
| follow the code.
|
| Using static typing as a way to reduce the number of tests is
| just terrible advice in my opinion.
|
| Also, the fact that tests interfere with each other, etc makes
| me think of just bad tests, not with a problem with the concept
| of e2e tests itself.
| gregmac wrote:
| > QA says: "it doesn't work". Dev says: "it must be working, all
| the tests pass".
|
| I've never experienced this, and I'm kind of doubtful this is a
| true reaction. I guess maybe it could happen in a team that is
| totally dysfunctional, where there's zero trust between QA and
| developers .. but in that case mocks are _not_ the problem.
|
| The realistic reaction is "huh, guess we are missing some test
| cases".
| fiddlerwoaroof wrote:
| Yeah, when I've run into this sort of bug, I always try to
| figure out how to make the unit tests reproduce the bug as a
| failing test case before attempting to fix the bug.
| mahidhar wrote:
| I've generally had a little bit more success with mocking when
| I'm hiding that dependency behind my own interface. So for
| example in Java, instead of trying to mock the AWS provided
| class, I write my own class (like a facade or repository pattern)
| which has a very simple interface of a success case and maybe a
| couple of relevant failure cases. It's calling the AWS library
| within it. But my mocks are at the level of my facade class which
| I find easier. The drawback is I'm not sure if there's a good
| general strategy to test the implementation of that facade. Most
| of the times the implementation is simple enough that I can do
| some simple integration tests for the most relevant cases, but
| there's always a risk that I am missing out some weird edge cases
| and I don't know how to properly deal with that.
| fiddlerwoaroof wrote:
| I think the nice thing about this is that the facades usually
| don't change: you might add methods, but, once the code is
| written, it only changes if you're doing something major like
| switching databases. Code that is written and then never
| changed tends to be less buggy than code that changes, so this
| sort of pattern tends to reduce bugs at integration points.
| catern wrote:
| As the blog post says, sometimes people mock because they're
| _incapable_ of running the real system. That 's just another
| reason why it's important to be able to run your system...
| http://catern.com/run.html
| dontbeabill wrote:
| so glad someone else has said this. truth
| rubyist5eva wrote:
| Write tests. Not too many. Mostly integration.
|
| (ps. Mock only when necessary)
|
| This has served me quite well.
| dontbeabill wrote:
| we had a entire system fail because it was based on how the TDD
| test author (who left the company) felt the system should work.
|
| basically, TDD sank us, because the code worked to make the test
| pass, versus the code working to support the business use cases.
|
| sure, this wasn't TDD problem, but was caused by a TDD zealot who
| didn't bother to really do proper integration testing and work
| with stakeholders.
| sdeframond wrote:
| Recently, I thought about a process I call "Test Coverage Driven
| Testing". It is similar to TDD, but more adapted to when we write
| tests after the code (you know you do too, at least occasionally,
| don't lie).
|
| It goes roughly like this:
|
| - write one integration test for the "happy path".
|
| - using some test coverage report, identify untested cases.
|
| - write unit test for those.
|
| I find I helps me find a good balance between time invested
| writing tests and benefits reaped from those tests.
|
| Do you have a similar process?
| teddyh wrote:
| > _- using some test coverage report, identify untested cases._
|
| From what I understand, that is not reliable; a line of code
| can be "covered" - i.e. executed - but still not be tested
| under all circumstances. If you have pre-existing code you need
| to write tests for, what you need is probably a tool for
| mutation testing.
| samatman wrote:
| I would make an exception for SQLite. Here's how you do it:
|
| - Write your schema so it's compatible with SQLite and your
| production DB, such as PostgreSQL.
|
| - When you're running tests, use SQLite instead of Postgres to
| run those tests.
|
| - Use SQLite in production. Go ahead, do it. You don't need
| Postgres, SQLite is fine. I promise. Trust me
| rad_gruchalski wrote:
| I agree with the premise of this post. Instead of mocking, design
| your components in such a way that they are easy to spin up in
| tests. Or use containerized versions of services.
|
| I started disliking the idea of mocks a few years ago. I was
| writing a system based on the Play framework. Play framework used
| to (still does?) come with a dedicated integration testing
| environment. The problem with it was that the setup of the
| testing world slightly differ from the real world. I was bitten a
| few timer by the real world setup process while the tests were
| executing flawlessly. In essence, there was no simple way to test
| the real world construction before deployment to production.
|
| Since then, the only integration tests I accept as real tests are
| the ones that test the production code path. Database? In
| containers. Kafka? In containers. etcd? In containers. There are
| exceptions, though. Proprietary APIs like SQS, or Kinesis, or
| Firestore are the difficult ones. I usually replace them with
| hand written memory bound implementations with an abstracted API
| adhering to some general top level constraints (put / get /
| consume / publish). This does not prevent errors rooted in the
| wrong understanding of the design principles of the dependencies,
| for example, consistency guarantees or delivery ordering, but
| those are usually possible to cover with additional tests further
| down the line.
| yepthatsreality wrote:
| I personally use mocks or stubs only in unit tests. Everything
| else should be live test or recorded network responses but
| always run through live code. Unless the service really depends
| on it, I do not reccommend setting up local swarm of services.
| This breaks down if you shift to distributed micro services and
| has little benefit for a single service app (ex crud + database
| app, no reason to test the database in the app tests).
| u801e wrote:
| I try to adopt the approach of using a functional core with an
| imperative shell (even if the latter is handled in a OO fashion).
|
| This way, I can unit test the functional core without the need to
| use mocks or stubs, and rely on integration tests to test the
| imperative shell.
| hendershot wrote:
| what the author is talking about in a short rant is classical vs
| mockist TDD.
|
| for better content on the subject:
|
| https://martinfowler.com/articles/mocksArentStubs.html
|
| https://martinfowler.com/articles/mocksArentStubs.html#Class...
|
| https://www.thoughtworks.com/insights/blog/mockists-are-dead...
|
| "The classical TDD style is to use real objects if possible and a
| double if it's awkward to use the real thing."
|
| I've been working in a classical TDD style for the past 8 years,
| after at least that many years of Mockist TDD. A code base built
| in classical TDD style is much easier to maintain and change, but
| it does require more test setup which can easily be pulled into
| re-usable test data scenarios etc. We'll use fakes for services
| that are external(S3, DynamoDB, third party APIs, etc), we'll use
| real DB code pointing to h2 instead of say postgres other than
| that there's no test doubles. I would not go back to using mocks
| by choice.
| aliasEli wrote:
| One of the most obvious problems with mocking is that the team
| that develops some code usually also develops the mocks that are
| used for testing it. So precisely the same misunderstandings will
| be present in the code as in the mocks. In other words you are
| not really testing anything.
|
| From my experience most errors are at boundaries between code
| from different teams. Mocking does not help here.
|
| My favorite form of tests are what I usually call subsystem
| tests. Try to test as much code as is feasible with each single
| test.
|
| Usually there are parts of your system that can be expensive to
| setup or use. For example, creating and filling a real database
| can be slow. In this case you could use an in memory database.
| Creating it and filling it with some representative data can be
| very fast. This database could be used by multiple teams, and is
| vastly superior to mocking.
|
| A similar approach can be used with other expensive parts like
| remote procedure calls [0], or input from browsers.
|
| This approach works when you design your system so that it can
| easily switch between using the real (expensive) resources and
| the ones that are only used for testing. But that is not very
| difficult.
|
| [0] Both REST and SOAP are RPC
| BeetleB wrote:
| Two words:
|
| False dichotomy.
| jeppester wrote:
| Another fanatic blog post about how something is always correct
| and another thing is always wrong.
|
| Blog posts like this will lure you into thinking that there is a
| single right approach. Don't fall for it.
|
| Do what makes sense. If it doesn't work try something else the
| next time. Becoming good is about growing your ability to make
| the right calls, not blindly following a methology.
| JulianMorrison wrote:
| I feel this can be summarised as "integration tests > unit
| tests". And yes.
|
| However, for tests you want to run locally and quickly, the
| question should be, can you get the speed advantage of unit tests
| with the largest possible amount of integration? Can you only
| mock things that are "outside the system" (such as DB, networked
| services, and file system)?
|
| IMO the biggest risk of integration testing is non-cleanup. I
| feel that is one of the major positive use cases for mocking. A
| mock DB will not retain garbage from previous tests, or previous
| runs of the same test.
| 0xbkt wrote:
| Containers do a good job on this. One should check out the
| Testcontainers[1] project that makes a good use of containers
| in language level.
|
| [1] https://github.com/testcontainers/testcontainers-go
| JulianMorrison wrote:
| They can do a good job, but you need to be very careful. "Oh,
| it's faster if you start the container once and then run all
| the tests in it." Yeah but now your tests are all peeing in
| each other's pool.
|
| The upside of mocks is that the values returned are literally
| hard coded.
| fatbird wrote:
| If the system doesn't work but all the tests pass, then you're
| missing one of two things: code coverage, or articulated
| requirements. Either code that you're not testing is crucially
| responsible for the system outcome, in which case it should be
| tested; or the demands of a unit of code with full coverage and
| passing tests don't include something that a consuming unit needs
| it to do.
|
| Neither of these conditions is an argument against unit testing
| or mocking. Both indicate a fairly basic gap in how you're
| testing that should be fixed directly. If integration tests would
| catch this gap, you're still missing the basic thing you need
| while incurring the cost and risks of integration tests.
| denton-scratch wrote:
| This resonates with me.
|
| I'm not at all an expert in testing; but I instinctively take the
| attitude that every line of test code is code that itself has to
| be tested, even though it's never going into production. I don't
| entirely trust that instinct, but I'm suspicious of test rigs and
| mocks. But I don't know how to test a "module" without them. Que
| faire.
| alecbz wrote:
| True "unit" tests are:
|
| * faster to run
|
| * give you less confidence in the correctness of the system (per
| time spent writing them)
|
| * when they fail, give you more information about where the
| failure is
|
| The more integration-y/e2e-y a test is, the more it strays from
| this: slower to run, more confidence that the system is correct,
| less info about where the failures are.
|
| I think people have learned to undervalue the properties of
| integration tests and overvalue the properties of unit tests. Is
| it nice to know exactly what's broken based on the test failure?
| Sure. Is it _as_ nice as having confidence that the whole system
| is working? Probably not, in a lot of cases.
|
| (I don't think there's general rules about which kinds of tests
| are easier to write. Sometimes setting up a real version of a
| component for test is harder, other times setting up a mock
| version for the test is harder.)
| rualca wrote:
| > I think people have learned to undervalue the properties of
| integration tests and overvalue the properties of unit tests.
| Is it nice to know exactly what's broken based on the test
| failure? Sure. Is it _as_ nice as having confidence that the
| whole system is working? Probably not, in a lot of cases.
|
| This comment alone demonstrates a collosal misunderstanding
| regarding unit tests.
|
| Unit tests are not used to verify if the system is working.
| They were never used for that. At all. Unit tests only specify
| invariants which are a necessary but insufficient to verify
| that the system works as expected. These invariants are used to
| verify that specific realizations of your input and state will
| lead your implementation to produce specific results, no matter
| which code you change. If you expect function foo to return
| true if you pass it a positive integer, you want to know if a
| code change suddenly leads it to return false or throw an
| exception. That's what unit tests are for.
|
| If unit tests worked anything remotely similar to the way you
| misrepresented them, we would not need integration or end-to-
| end tests at all, would we?
| curun1r wrote:
| The other point about unit tests is that they allow you to test
| far more permutations. At a micro level, you can test more of
| the possible types of inputs to your function. When you move to
| integration tests, the permutations multiply to the point that
| it would require billions of tests to test the same range of
| inputs.
|
| This is why both are so necessary. Integration testing tests
| whether the whole thing is coherent when put together. But it's
| terrible for testing edge cases and error scenarios. That's
| where unit testing comes in. It gives the developer a chance to
| codify every edge case they've considered to automate the
| process of catching regressions.
| arkh wrote:
| Don't forget a very important thing: unit test are hindering
| any refactoring. People will resist it because it means they
| have to rewrite their tests. And if you don't have enough e2e
| tests, you have nothing to check your refactoring efforts have
| not broken anything.
| GrumpyNl wrote:
| I have noticed that 80% of the time the unit test fails, the
| unit test has to be rewritten.
| rileymat2 wrote:
| Do they hinder refactoring or rewriting?
|
| When I am doing legitimate refactoring in simple steps by the
| book, unit tests are helpful and speed it up. When I am
| rewriting, I see your issue.
|
| For example, a sprout method or sprout class refactor do not
| change the tests.
| hamandcheese wrote:
| If a test is in my way, I delete it.
| amw-zero wrote:
| Then what is the point of it being there in the first
| place?
| rualca wrote:
| > Don't forget a very important thing: unit test are
| hindering any refactoring. People will resist it because it
| means they have to rewrite their tests.
|
| Well, unit tests verify a contract. If a developer wants to
| change code but is incapable of respecting a contract (i.e.,
| preserving old invariants or adding new ones that make sense)
| then he should not be messing with that code at all,
| shouldn't he?
|
| In that sense alone, it's quite clear that unit tests pay off
| and add a lot of.value, namely in avoiding needless
| regressions from Rambo developers who want to change code
| without caring if it even works.
| spaetzleesser wrote:
| Very true. A lot of unit tests, especially the ones with
| mocks tend to test a lot of internal behavior. So when you
| refactor your system, you often have to rewrite unit tests
| although the system actually performs correctly.
| ternaryoperator wrote:
| Regardless of what this article says, mocks are really useful
| when your project is partially written and you want to test the
| pieces you have in hand. Mocking then is absolutely the right
| approach and might entail mocking many things (which will
| eventually be replaced by working code.)
| chris_j wrote:
| I used to use mocks an awful lot more than I do nowadays. I
| learned to do that style of testing from the book Growing Object
| Oriented Software, Guided by Tests (the "GOOS" book, which is
| still well worth a read, even if you don't subscribe to that
| style of test driven development). I'm still of the view that
| mocks are extremely useful if a) you're working in a highly
| object oriented style (where systems are composed of objects that
| communicate with their collaborators by method call) and b)
| you're unit testing relatively small units of code at a time.
| Mocks are very useful if it's important to you that object X must
| call object Y, do a computation and then call object Z with the
| result.
|
| There are two key insights that helped me to eliminate the need
| for so many mocks: a) modern languages that support functional
| programming allow you to separate the concerns of computation and
| interaction with collaborators, and it's an awful lot easier to
| test drive a pure function that it is to test drive a graph of
| collaborating objects; b) modern hardware is sufficiently fast
| that it's much more feasible to spin up a whole service (or many
| services) in order to run tests on them, and you don't need to
| write fine grained unit tests merely in order to make the tests
| run quickly enough.
| lanecwagner wrote:
| I recently wrote an eerily similar piece. My focus is more on
| unit test but the idea is the same: https://qvault.io/clean-
| code/writing-good-unit-tests-dont-mo...
| gravypod wrote:
| > But dude, I don't want to use a real database (or AWS endpoint
| or rocket launcher) in my tests. Debatable, but fair enough.
|
| Is this debatable? Hermetic tests give you a lot of things for
| free and I don't really see a reason why you would default to
| making all of your tests hermetic.
|
| The real thing that this article is touching on is that your
| tests should test all of the code _you_ write but not code
| _others_ have written. This sounds wild at first but: do you need
| to test that Postgres knows how to parse a query? Probably not. I
| think the postgres team knows how to test and release their
| database and I don 't really need to spend time doing that. Then,
| the next layer of abstraction: if you use an ORM or some middle
| layer that abstracts the database do you need to test that the
| ORM knows how to talk to postgres? In a unit test, likely not.
| The ORM people have provided you an API and you should use their
| API to fake/stub/mock/whatever that system so you can focus on
| testing your business logic. After you have that system built you
| should then build integration/E2E tests that actually talk to
| hermetic copies of the real systems. An easy way to do this is to
| build a troubleshooting cli tool that you can run against your
| backend services/dbs/etc that can be used in CI against a copy of
| your backend or in prod to debug configuration issues.
| zestyping wrote:
| You don't need to test that Postgres knows how to parse a
| query. But you do need to test that the query you wrote means
| what you think it means to Postgres. The only way to know that
| for sure is to hand the query to Postgres.
|
| Maybe you don't need to test that the ORM is correct. But you
| do need to test that you are using the ORM in the way that the
| ORM's designer expected, which is often non-trivial. And so on.
| agentultra wrote:
| The mocks are the symptom. The problem is that your code doesn't
| restrict side effects in any way. And so you end up with
| integration tests for everything and setting up a single test
| requires recreating the universe from scratch and slightly
| tweaking it on every run.
|
| _But that 's what our program does, it talks to databases and
| file systems and HTTP servers!_
|
| Sure, and the effect of doing those things is moving data around.
| What does your program do with the results of these side-effects?
| Does it parse it? Transform it in any way? Decide whether to run
| effect A next with the result or effect B? This is the code that,
| if extracted, can be tested in isolation of databases and HTTP
| servers. You want as much of your code to be in this place as
| possible. It's a thousand times easier to test and the tests are
| thousands of times more reliable because they don't perform any
| effects.
|
| Mocks have their places but if you can't test your code without
| mocking out the universe then the problem is that your code is
| interleaving too many effects with the core logic of the program.
| The cure is to refactor effects to the edges of your program and
| run effects in one place in your code. Make the rest just plain,
| pure code as much as possible.
| Spivak wrote:
| But what you're describing creates tight coupling between your
| integration points and your internal logic that doesn't look
| like tight coupling. You're not feeding you code real data,
| you're only feeding it what _you_ think that external service
| will provide. And to validate that your tests are correct you
| need integration tests again and to enforce that your
| integration point code always produces consistent valid output.
|
| So yeah, the advice to avoid side effects is good but still
| exercise your integration points with real data and services.
| ItsMonkk wrote:
| I'm confused as to where any mention of integration tests
| are? Integration tests can't use Mocks, as then they wouldn't
| be testing the integration. Unit tests can use Mocks, but as
| the article and GP and Mark Seemann[0] points out, that is
| always worse than writing your logic in a pure way, if you
| can.
|
| [0]: https://blog.ploeh.dk/2017/02/02/dependency-rejection/
| Spivak wrote:
| Because you don't gain as much as you think when you do
| this.
|
| Before your external external service touches your
| integration point and then runs some logic. You mock the
| external service to unit test the internal logic. Then you
| run integration tests to exercise the integration point.
|
| In the new world you have an external service talking to an
| integration point which then passes the data to your
| internal logic. You don't need the mocks anymore because
| you can just call your logic function with your data that
| would have come from the mock. Great! You run integration
| tests again to test the integration point. But now you have
| another linkage at the call site. Your internal logic
| function now has a dependency on the output of your
| integration point and that's an invariant that has to be
| tested to catch someone modifying one but not the other.
|
| With enough discipline you might be able to make the type
| system do these kinds of checks for you but IRL very few
| people do enough to say catch an external string field
| changing format slightly.
| ItsMonkk wrote:
| I think I was confused because I naturally do what you
| say few people do. My external calls all parse into an
| object that you can then pass into the unit. It's the
| integration tests job to ensure that the parse works.
| Then its the unit tests job to ensure that you run enough
| that you can be reasonably sure that you cover all cases.
| u801e wrote:
| > You mock the external service to unit test the internal
| logic
|
| You only need the output produced by that external
| service or the the input expected by that service. For
| example, I can pass in a data structure or object
| representing the HTTP request to the method that I'm unit
| testing, but I don't need to mock a client to generate
| those requests to test that logic.
| pydry wrote:
| >Sure, and the effect of doing those things is moving data
| around. What does your program do with the results of these
| side-effects? Does it parse it? Transform it in any way? Decide
| whether to run effect A next with the result or effect B? This
| is the code that, if extracted, can be tested in isolation of
| databases and HTTP servers.
|
| It's very frequent that this "decision making/calculation"
| code:
|
| * Doesn't do very much.
|
| * Isn't even in the top 5 source of bugs for your app.
|
| * Integration tests caught those bugs just fine anyway.
|
| * The process of extraction balloons the size of the code base
| (all those yummy extra layers of indirection), sometimes
| _introducing_ fun bugs.
|
| It's certainly the right thing to do if you have a self
| contained chunk of complex decision making/calculation code
| (e.g. a parser or pricing engine).
|
| However, if you do this as a matter of _principle_ (and _far_
| too many do) then this advice isn 't just wrong, it's
| _dangerously_ wrong.
| ehutch79 wrote:
| Serious question. Do most test suites not allow you to run a
| subset of tests?
| gjmacd wrote:
| Docker has made most mocking nonsensical considering how easy it
| is now to use the real thing... But I would disagree with the
| premise of mocking as being non-starter. Often times you want to
| unit test and don't really care if you're using the real "thing"
| but want to hit code that's got zero to do with that dependency.
| Good example, we use Okta to authenticate... We want to run unit
| tests that test how a component in our UI works within our
| application, we mock Okta to get around our authentication for
| testing that very thing. When we want to test authentication with
| Okta, that's what we do.
| mdoms wrote:
| Absolutely terrible advice. No, I won't have my CI rely on every
| integration point in my application.
| astuyvenberg wrote:
| Strongly agree, especially when it comes to things like AWS
| services. Their APIs and services evolve so quickly that things
| like local mocking (or emulation for local development) is an
| anti-pattern.
|
| Where possible, I prefer to utilize short-term, pay-per-use
| infrastructure for development and testing.
| NBJack wrote:
| This isn't a terribly practical article. I don't disagree with
| mocks being an "alternate reality". The author is entitled to
| their opinion on whether this is a good or bad thing. This
| said...what is the alternative? Integration testing all the way
| down?
|
| The implication here is to work with stubs over mocks (i.e. I
| need to work with S3; I would then abstract that to provide a
| StubObjectStore to replace the S3ObjectStore used by other pieces
| of my code during tests). Great; I know they work now. But at
| some point, I need confidence my S3ObjectStore handles everything
| correctly. Do I give everyone on my team a bucket? Perhaps their
| own test AWS account? Test it, but only in the pipeline on its
| way to an intermediate stage? I can't control how AWS writes
| their SDKs (spoiler alert: they don't stub well), but I need some
| confidence that I can handle their well-understood behavior that
| scales. Likewise, I often can't control the libraries and
| integration points with other systems, and mocking offers a
| "cheap" (if imperfect) way to emulate behavior.
| astuyvenberg wrote:
| For AWS specifically, I prefer to have an AWS account
| specifically dedicated to each service + stage. For example, if
| I have an image service that handles s3 uploads (say Lambda,
| S3, Cloudfront and API Gateway), then I'd deploy a "test"
| environment to a dedicated AWS account and run tests against
| that. Since it's fully serverless, it only costs a few pennies
| to test (or free).
|
| I try not to develop locally at all anymore. If you're looking
| for more practical advice, perhaps this will help:
| https://dev.to/aws-builders/developing-against-the-cloud-55o...
| cortesoft wrote:
| That means that every person who runs the tests needs
| credentials for that AWS account. That obviously won't work
| for an open source project. Even for a company project, how
| do you distribute those secrets? It adds friction for
| developers getting their local dev environment setup.
|
| Not only that, but you now need network access to run tests.
| A network blip or a third party service outage now makes your
| tests fail.
|
| There is also the possibility that an aborted test run might
| leave state in s3 that you are now paying for. Someone hits
| Ctrl-c during a test run and now you have a huge AWS bill.
| astuyvenberg wrote:
| > That obviously won't work for an open source project
|
| On the contrary - I was an employee at Serverless Inc,
| working on the Serverless Framework for the last two years,
| we used this pattern extensively (and very successfully) in
| our open source repos.
|
| You can even find an example here which provisions real
| live AWS infrastructure:
| https://github.com/serverless/dashboard-
| plugin/tree/master/i...
|
| We used part of our enterprise SaaS product to provision
| temporary credentials via STS and an assumable role, and it
| works great. You could do the same thing with something
| like HC Vault.
|
| For Lambda, S3, DynamoDB, the perpetual free tier means
| we've never paid to run our own tests. API Gateway isn't
| free (after 1 year), but it's still pennies per month.
| We've had several cases where tests stuck around a long
| time, but a billing alert and occasionally some
| CloudFormation stack cleanup takes care of that.
|
| We still have offline unit tests which test business logic,
| but everything else runs against the cloud - even our
| development environments ship code straight to lambda.
| Supermancho wrote:
| Why spend money on AWS you don't have to? Use Minio for S3
| locally (or on your build server).
|
| Local development is the easiest way to avoid wasting money
| and resources on debugging/development.
| [deleted]
| astuyvenberg wrote:
| Speaking from personal experience, our team wasted far more
| money tinkering with local dev environments and trying to
| replicate the cloud than we ever did simply using it to
| develop.
|
| The blog post in the parent comment lays out our experience
| and my thoughts, but because of the pretty generous free
| tier, I don't think we've ever paid a penny for a
| build/dev/test AWS account.
| pydry wrote:
| I find services like this often work really well with hermetic
| integration tests:
|
| https://github.com/adobe/S3Mock
|
| It's more realistic than using mock objects/function calls and
| requires less maintenance.
| wffurr wrote:
| Yes, use stubs. But then _also_ have integration tests.
|
| The point is to have easier to write lower overhead unit tests,
| and then have a few full fat integration tests that put
| everything together.
|
| Mocking is a terrible middle ground.
| NBJack wrote:
| That still doesn't address my concern. I cannot test my S3
| implementation without either mocks or a very specific
| emulator of the protocol. AWS happens to be popular enough
| that some libraries exist to do the latter, but I assert this
| is the exception rather than the rule for external
| integrations. You shouldn't be checking in code without at
| least some unit testing along for the ride, mocks or
| otherwise. It is indeed no substitute for integration
| testing, but it can certainly help catch bugs sooner rather
| than later.
| devit wrote:
| If your external integration has no local alternative, you
| are getting locked-in to its provider, so you should either
| not use it, or have an abstraction layer and implement an
| alternative backend.
| wffurr wrote:
| It's simple enough take to extract the interface of the S3
| calls you make into a definition that you can write a test
| stub that unit tests can pipe fake data into.
| fiddlerwoaroof wrote:
| I agree you need some integration tests, but, in my
| experience, if you define an interface the defines what you
| need from third parties, you can make 90% of the code you
| care about unit testable.
|
| For me, this isn't just theory: I've worked at a place that
| trained its employees to write code this way, and the
| benefit was obvious.
| tuxie_ wrote:
| Mocking has a place, which is not unit testing. If you find
| yourself mocking a dependency in a unit test you are not unit
| testing any more.
|
| Those points of contact with 3rd parties should be clearly
| defined and encapsulated at the perimeter of your system. Mock at
| that level. Not when testing business logic.
| 8note wrote:
| something else calls that encapsulation.
|
| So you can also mock your encapsulation in the next layer, than
| that one in the next, etc
| catwind7 wrote:
| what does unit testing have to do with whether or not you
| instrument the test with fake responses? those points of
| contact that you're mocking out at the perimeter, that data
| will sometimes need to reach a particular function through a
| collaborator...which you may want to mock?
|
| sometimes the dependency is not a third party, but it may be
| code that requires a ton of setup (as mentioned in article)
| that's not worth the cost. it may make sense to just mock at
| that point to actually test the rest of the business logic in a
| function. I don't think i'd say "well, that's no longer a unit
| test!". You can argue that it's a more brittle test, sure.
|
| update: also, i'll be honest that comments like this really rub
| me the wrong way. This type of dogmatism around what is or
| isn't unit testing (which is a pretty ill-defined phrase in
| industry) is something that needs to stop. I think it hurts new
| practioners in the field who are mislead into black and white
| thinking.
| tuxie_ wrote:
| I'm sorry, I did not intend to offend anyone obviously.
| Needless to say, this is just my opinion condensed in a
| sentence (therefore lacking a lot of context, which I should
| have provided).
|
| I was not aiming to define what a unit test is, more like
| when it stops being a unit test (which I thought it would be
| an easier agreement to reach than a definition of what is,
| but I guess I underestimated the task).
|
| My point was that if you have to mock, for example, a DB call
| inside your business logic, well you are writing an
| integration test at that point, whether or not you mock the
| DB out. If you design your code so that you only have those
| dependencies at the edge of the system then you get, in my
| opinion, a much cleaner design and much more testable code.
|
| Too much mocking (and/or more like mocking in the wrong
| places) is a code smell in my opinion.
| detaro wrote:
| So components that talk to the outside can't be unit-tested
| at all?
| tuxie_ wrote:
| Hi detaro. Good question. If all my module does is make
| an api call, and I mock the API, what am I testing? I
| would rather leave that out of the unit tests because the
| added value is, in my opinion, close to none if you are
| relying on mocked data.
|
| Now, if the module does more than just call the API then
| I would argue that it's breaking the single
| responsibility principle and would prefer to split it
| into a module that does only the call and another that
| does the rest.
| detaro wrote:
| Ok, then go one level in: If a component uses the "only
| makes an API call module", how do I unit test it? I can't
| let it use the module to make the API call (because the
| API might not be available in testing/that's an
| integration test/...), and I can't mock it, because using
| a mock would make it a not-unit test? I guess this gets
| at the line between mocks and stubs, but I never found
| that all that convincing.
| tuxie_ wrote:
| Yes, you just don't unit test it, because if you mock out
| the only dependency it has you are left with nothing. So
| you are not unit testing anything anyway. You know what I
| mean? That code can (and should) be tested, through an
| integration test.
| [deleted]
| bluepizza wrote:
| Why?
| tuxie_ wrote:
| Hey bluepizza. As I wrote in response to another comment, I
| think that too much mocking (and/or mocking in the wrong
| places) is a code smell.
|
| I was not aiming to define what a unit test is, more like
| when it stops being a unit test, and I think that if you have
| to mock, for example, a DB call inside your business logic,
| well you are writing an integration test at that point,
| whether or not you mock the DB out. If you design your code
| so that you only have those dependencies at the edge of the
| system then you get, in my opinion, a much cleaner design and
| much more testable code.
| bluepizza wrote:
| But if you include a real DB on your business logic
| testing, that's not unit testing anymore, is it,
| tuxie_ wrote:
| Exactly. It may sound ridiculous but it happens all the
| time. You dig into the "unit tests" of the app and you
| find mocks for a db call, or an API call, or the system's
| date, or environment variables that some other part of
| the system will break if they are not defined (or all of
| the above). That's what I mean, these are all code smells
| that the code is highly coupled.
| ipaddr wrote:
| Because he has reached a conclusion. Anything further can be
| safely ignored if it doesn't fit that conclusion.
|
| Never let someone tell you what unit tests are.
| tuxie_ wrote:
| Hi, I don't really understand who do you refer to when you
| say that "anything further can be safely ignored". By me?
| What exactly can be ignored?
|
| Sorry, I would like to answer your comment since it seems
| that it upset you, and nothing was further from my
| intention. But I honestly don't understand what you mean.
| [deleted]
| aliasEli wrote:
| The term "unit test" can mean a lot of things (which is very
| unfortunate).
|
| Does it mean:
|
| - A test of some of the requirements that can be done very
| fast. (That's my preferred form when involves a large part of
| the code)
|
| - A test of a small piece of code that makes a lot of
| assumptions about other code by mocking it. (Not a good idea,
| but it might work in your organization)
|
| P.S. I did not read your post, while I was writing mine.
| drewcoo wrote:
| This blog post has tell us that we shouldn't use mocks but should
| instead "send data" to the code we are testing. I think this is
| supposed to mean using dependency injection instead. But the case
| isn't really made. Instead, with a waving of hands and wild
| assertions such as that I'm lying to myself, I'm left wondering
| what I just read.
|
| The previous post was on cargo cult programming. It warns that
| CCG is bad but can't seem to define it. There is "how to spot"
| advice but no evidence to show that the advice works (except for
| a minor appeal to authority).
|
| Yes, English is probably a second language, but the writing has
| more fundamental problems than that. The author needs to consider
| what a thesis is and how to support it. And after that, who the
| audience is. Etc.
| pydry wrote:
| >I think this is supposed to mean using dependency injection
| instead.
|
| I'm pretty sure he means "just use integration tests".
| ramesh31 wrote:
| >I'm pretty sure he means "just use integration tests".
|
| Really he means e2e testing. You can drive yourself mad
| writing integrations that break constantly or provide false
| positives. Better to unit test what is truly unit testable,
| and then rely on e2e to ensure your integrations are fine.
| Ultimately all that matters is the system running as expected
| at the user level.
| nicoburns wrote:
| I think that depends on the scope of what you're testing.
| If you have a database, a set of backend services and 1 or
| more clients (websites, mobile apps, etc), then I think it
| makes a lot of sense to test the backend services in
| isolation from the frontends (which would mean not e2e
| test), but backend by an actual database (so integration
| tests, not unit tests)
| EdSharkey wrote:
| I think so too.
|
| Unfortunately, integration tests are too slow, so the
| practice doesn't scale if one is trying to TDD.
|
| Insinuating integration testing into every user story will
| lead to friction. Test run times will balloon, cycle times
| will get extended and resentment will grow for the test suite
| and the team's testing regime.
|
| If your test suites cannot complete quickly (seconds), then
| they cost more than they're worth. I've learned this about
| outside-in TDD at-scale. Our code quality is glorious. But
| our test run times are untenable.
|
| I'm experimenting with sociable tests to curb my appetite for
| integration tests or at the very least keep on writing them
| but make it safe/productive to run the vast majority of them
| on the CI/CD server only.
| AlphaSite wrote:
| Why run your whole test suite every time? Chances are
| you're only really interested in a small set of them, just
| let those run continuously.
| pydry wrote:
| I find I can do ITDD effectively with integration test
| times of < 30 seconds. I've even done it with tests of up
| to 5 minutes.
|
| What makes it work is pairing it with a REPL. That way I
| can have an "outer" loop that triggers the REPL and then an
| "inner" loop where I can experiment in the area where I'm
| writing the code and get feedback quickly.
|
| I might run the outer loop just a few times:
|
| * At the beginning of the ticket
|
| * When I messed up the state with the REPL and when I want
| a fresh slate.
|
| * When I've pasted code I think will hypothetically make
| the test pass from the REPL and I want final verification.
|
| * One or two more times after that coz I missed something
| stupid.
|
| Often the waiting times are a good time to get up and go
| for a walk/make a coffee/check my messages.
|
| >I'm experimenting with sociable test
|
| What's this? I'm unfamiliar with the term.
| asimpletune wrote:
| Generally you shouldn't really be using integration tests
| for TDD, but it's totally possible to write your tests in a
| way where the dependency supplied is talking to a real
| system in one scenario and in another scenario is talking
| to a system with mocked responses - or any sort of level of
| depth in between.
|
| However, I wouldn't start out writing a system like this
| (aka TDD). From my experience, the best tested software
| looks like this as the end result and the design of the
| software itself has been forged in the same furnace.
|
| Not sure if that completely makes sense, since some of
| these concepts are from functional programming and I never
| know what is totally foreign or totally obvious.
| Chris_Newton wrote:
| _Unfortunately, integration tests are too slow, so the
| practice doesn 't scale if one is trying to TDD._
|
| If integration tests get more useful outcomes than unit
| tests in some situation but TDD only works well with unit
| tests, maybe that means TDD isn't the best process to use
| in that situation. Isn't the essence of agile development
| to recognise what is working well and what is not, so you
| can make changes where they are needed?
|
| _If your test suites cannot complete quickly (seconds),
| then they cost more than they 're worth._
|
| I disagree with this. The goal of testing in software
| development is to improve results. Any testing methodology
| should be evaluated accordingly. Fast feedback is good,
| other things being equal. However, if going slower means
| getting more valuable feedback -- identifying more defects,
| providing more actionable information about how to correct
| them, checking for a broader range of potential problems --
| then maybe going slower is the right choice.
| xiamx wrote:
| Yeah, a blog post like this is too casual and generalizes way
| too much
| bhawks wrote:
| Mocks make me sad. They are so often misused by people with the
| best of intentions. I have seen so many tests that literally only
| assert the mock expectations and say nothing about the code under
| test. I have watched upgrades and refactors take 5x longer
| because someone steps on a landmine of overmocking. It is so
| common to see people mocking dumb data objects, containers, and
| pure functions - creating Frankenstein widgets that break all
| contracts the original had except for the 1 code path the
| original developer used 6 months ago. I've listened to DRY/SRP
| fanatics defend mocks until they realized that their tests were
| littered with unreusable, low fidelity copies of their production
| system in the form of mocks.
|
| Mocks are a power tool and you can use them to do things that
| they are not the right tool for the job.
___________________________________________________________________
(page generated 2021-07-04 23:01 UTC)