[HN Gopher] Scaling Go Testing with Contract and Scenario Mocks
___________________________________________________________________
Scaling Go Testing with Contract and Scenario Mocks
Author : preetamjinka
Score : 37 points
Date : 2025-12-18 18:09 UTC (6 days ago)
(HTM) web link (funnelstory.ai)
(TXT) w3m dump (funnelstory.ai)
| teeray wrote:
| > Mocks are static, but reality evolves.
|
| I learned "test your mocks" long ago from Sandi Metz, and that
| advice has paid off well for me. Have some set of behavioral
| conformance tests for the kind of thing you expect (e.g. any
| database worth its salt should be able write and read back the
| same record). Then stick your mock right under that same battery
| of tests alongside your implementation(s). If either deviate from
| the behavior you depend on, you'll know about it.
| zingar wrote:
| Bang on Sandi! I hadn't heard that quote but she's my favorite
| speaker on testing and OO.
|
| Another way of looking at this advice is that every time
| there's a mock there needs to be a test that shows that the
| real code can be used in the same way that the mock is used.
| mzi wrote:
| > any database worth its salt should be able write and read
| back the same record
|
| This excludes a lot of cases, like just a simple postgres where
| reads are done from a replica.
| teeray wrote:
| You're free to come up with a better example. The point is
| that dependencies have behavioral properties that software
| depends upon. We can write tests for those behaviors. Mocks
| that are correct and remain so should implement those same
| behaviors we expect from the prod implementations of those
| dependencies.
| SPascareli13 wrote:
| I really dislike this idea of testing in go: only ever use an
| interface, never the real implementation + mockgen the mocks
| based on this interface + use the mocks to assert that a function
| is called, with exactly this parameters and in this exact order.
|
| I find this types of tests incredibly coupled with the
| implementation, since any chance require you to chance your
| interfaces + mocks + tests, also very brittle and many times it
| ends up not even testing the thing that actually matters.
|
| I try to make integration test whenever possible now, even if
| they are costly I find that the flexibility of being able to
| change my implementation and not break a thousand tests for no
| reason much better to work with.
| esafak wrote:
| Don't you mean testing the interface of the implementation? I
| see nothing wrong with that, if so.
| et1337 wrote:
| They mean the dependencies. If you're testing system A whose
| sole purpose is to call functions in systems B and C, one
| approach is to replace B and C with mocks. The test simply
| checks that A calls the right functions.
|
| The pain comes when system B changes. Oftentimes you can't
| even make a benign change (like renaming a function) without
| updating a million tests.
| 9rx wrote:
| Tests are only concerned with the user interface, not the
| implementation. If System B changes, that means that you
| only have to change your implementation around using System
| B to reflect it. The user interface remains the same, and
| thus the tests can remain the same, and therefore so can
| the mocks.
| et1337 wrote:
| I think we're in agreement. Mocks are usually all about
| reaching inside the implementation and checking things. I
| prefer highly accurate "fakes" - for example running
| queries against a real ephemeral Postgres instance in a
| Docker container instead of mocking out every SQL query
| and checking that query.Execute was called with the
| correct arguments.
| 9rx wrote:
| _> Mocks are usually all about reaching inside the
| implementation and checking things._
|
| Unfortunately there is no consistency in the nomenclature
| used around testing. Testing is, after all, the least
| understood aspect of computer science. However, the
| dictionary suggests that a "mock" is something that is
| not authentic, but does not deceive (i.e. not the real
| thing, but behaves like the real thing). That is what I
| consider a "mock", but I'm gathering that is what you
| call a "fake".
|
| Sticking with your example, a mock data provider to me is
| something that, for example, uses in-memory data
| structures instead of SQL. Tested with the same test
| suite as the SQL implementation. It is not the datastore
| intended to be used, but behaves the same way (as proven
| by the shared tests).
|
| _> checking that query.Execute was called with the
| correct arguments._
|
| That sounds ridiculous and I am not sure why anyone would
| ever do such a thing. I'm not sure that even needs a
| name.
| tonyhb wrote:
| If you're testing the interface, changing the implementation
| internals won't create any churn (as the mocks and tests don't
| change).
|
| If you are changing the interface, though, that would mean a
| contract change. And if you're changing the contract, surely
| you wouldn't be able to even use the old tests?
|
| This isn't really a go problem at all. Any contract change
| means changing tests.
| preetamjinka wrote:
| Yes, agreed. What the parent is saying about
|
| > only ever use an interface, never the real implementation +
| mockgen the mocks based on this interface + use the mocks to
| assert that a function is called, with exactly this
| parameters and in this exact order.
|
| is not ideal, and that's what we don't do. We test the real
| implementation, then that becomes the contract. We assume the
| contract when we write the mocks.
| aranw wrote:
| > I really dislike this idea of testing in go: only ever use an
| interface, never the real implementation + mockgen the mocks
| based on this interface + use the mocks to assert that a
| function is called, with exactly this parameters and in this
| exact order.
|
| Same I have zero confidence in these tests and the article even
| states that the tests will fail if a contract for a external
| service/system changes
| rhines wrote:
| I see this kind of testing as more for regression prevention
| than anything. The tests pass if the code handles all
| possible return values of the dependencies correctly, so if
| someone goes and changes your code such that the tests fail
| they have to either fix the errors they've introduced or go
| change the tests if the desired code functionality has really
| changed.
|
| These tests won't detect if a dependency has changed, but
| that's not what they're meant for. You want infrastructure to
| monitor that as well.
| Phlebsy wrote:
| I'm a fan of writing tests that can be either. Write your tests
| first such that the real dependencies can be run against.
| Snapshot the results to feed into integration test mocks for
| those dependencies so that you can maintain the speed benefit
| of limited test scope. Re-run against the real dependencies at
| intervals you feel is right to ensure that your contracts
| remain satisfied, or just dedicate a test per external endpoint
| on top of this to validate the response shape hasn't changed.
|
| The fundamental point of tests should be to check that your
| assumptions about a system's behavior hold true over time. If
| your tests break _that is a good thing._ Your tests breaking
| should mean that your users will have a degraded experience at
| best if you try to deploy your changes. If your tests break for
| any other reason then what the hell are they even doing?
| gethly wrote:
| Testing in Go is great, for unit testing. But anything above that
| should be done manually. "Mocking" never made sense to me and I
| never used it in my entire life. If I want to test a
| database/repository, I will spin up a real database with real
| data and data persistence instead of relying on in-memory storage
| and try to abstract away things that make no sense to do in the
| real world, as we're not living in a theoretical world of
| interfaces and adapters and whatnot.
| aranw wrote:
| The problem with mocks is that they test your assumptions, not
| reality...
|
| When you mock a CRM client to return one account, you're assuming
| it always returns one account, that IDs have a particular format,
| that there's no pagination, that all fields are populated. Each
| assumption is a place where production could behave differently
| whilst your tests stay green
|
| Your contract tests use cached JSON fixtures. Salesforce changes
| a field type, your contract test still passes (old fixture), your
| mocks return the wrong type, production breaks. You've now got
| three test layers (contract, mock scenarios, E2E) where two can
| lie to you. All your contract and mock tests won't save you.
| Production will still go down
|
| I have zero confidence in these types of tests. Integration tests
| and E2E tests against real infrastructure give me actual
| confidence. They're slower, but they tell you the truth. Want to
| test rate limiting? Use real rate limits. Want to test missing
| data? Delete the data.
|
| Slow tests that tell the truth beat fast tests that lie. That
| said, fast tests are valuable for developer productivity. The
| trade-off is whether you want speed or confidence
| Thaxll wrote:
| Testing code is usually testing your code not that third party
| contract changed.
|
| You make a lot of assumption about contract change which in
| reality should rarely happen.
| isuckatcoding wrote:
| Wow a full post about contract testing without mentioning pact
|
| https://pactflow.io/
| konart wrote:
| I feel like https://pact.io is more appropriate link, no?
| fleahunter wrote:
| Interesting point about mocks being seen as a bad word. I've
| been in situations where relying solely on integration tests
| led to some really frustrating moments. It's like every time we
| thought we had everything covered, some edge case would pop up
| out of nowhere due to an API behaving differently than we
| expected. I remember one time we spent hours debugging a
| production incident, only to realize a mock that hadn't been
| updated was the culprit--definitely felt like we'd fallen into
| that "mock drift" trap.
|
| I've also started to appreciate the idea of contract tests more
| and more, especially as our system scales. It kind of feels
| like setting a solid foundation before building on top. I
| haven't used Pact or anything similar yet, but it's been on my
| mind.
|
| I wonder if there's a way to combine the benefits of mocks and
| contracts more seamlessly, maybe some hybrid approach where you
| can get the speed of mocks but with the assurance of
| contracts... What do you think?
| kardianos wrote:
| My DB heavy app, when I run a Go unit test, it spins up a DB
| instance, populates it, runs the tests, and drops the DB. Never
| ever any mocks. The best part about unit tests isn't testing the
| Go code. It is testing the SQL.
|
| Yes, it takes longer to run your tests. So be it.
| preetamjinka wrote:
| We do that too. We have hundreds of such tests. That
| establishes contracts.
|
| We also have mocks. It's not one way or the other. This post is
| talking about the mocking side of things.
| bikelang wrote:
| We do the same thing. The core of our service is ingesting data
| and applying transformations to it. There's so many
| permutations and complex interactions in here that the only way
| to ensure a refactor hasn't broken one of these interactions is
| to document those edge cases by piping data through the system
| and reading it back out. We have thousands of these tests and
| it's all tidy controlled via docker compose. It takes about 15
| minutes to run the test suite. Sure I wish it was faster - but
| the real unlock is that we can make big sweeping refactors
| without breaking behaviors of the system. The organizational
| speed unlock this kind of safety net is well worth a bit of
| slowness in CI.
|
| We also have mocks/stubs/spies in our unit tests. Those are
| great for producing hard-to-trigger edge cases. But contract
| testing? The contract is the data flow. In the end it's all
| about using the right tool for the right test. There is no one-
| size-fits-all.
| 9rx wrote:
| If your SQL is isolated to one place then you only need to test
| that single package with a real DB and can use mocks everywhere
| else. Your mocks can be tested with the exact same test suite
| as the SQL package, so you know it conforms to the same
| contract.
|
| If you have SQL scattered all over the place... Leave the
| spaghetti for dinner.
| fireflash38 wrote:
| I'm of the opinion that mocks should be provided by the thing
| that you're mocking. That is, if you are wanting to mock out a
| service, the mock should be owned by the service that is being
| mocked.
|
| And then it should be part of that service's test suite, to
| verify it's own mock.
|
| You update your service? Then you must update the mock.
|
| I guess that's more of a fake, but the naming doesn't matter as
| much as the behavior.
___________________________________________________________________
(page generated 2025-12-24 23:01 UTC)