[HN Gopher] Did I break you? Reverse dependency verification
___________________________________________________________________
Did I break you? Reverse dependency verification
Author : r4um
Score : 60 points
Date : 2021-06-10 06:37 UTC (16 hours ago)
(HTM) web link (developers.soundcloud.com)
(TXT) w3m dump (developers.soundcloud.com)
| pavel_lishin wrote:
| Skimming this, because I'm in a meeting, but it reminds me of
| contract testing, which I'm working on implementing at work today
| with Pact.
| ltbarcly3 wrote:
| Ok, if I understand this properly:
|
| They have a core library that other projects use.
|
| They want to release a new version of this core library, and
| upgrade all the projects to use the newer version.
|
| If the CI breaks for the projects using the library, they want to
| catch it.
|
| This still depends on the projects using the library having
| enough and good enough testing for something to break in the CI
| when this core library does something bad. That is a huge
| investment (having those high quality tests). The rest of this is
| just a script that runs the tests from those projects when people
| merge code into the core library, which seems like very much not
| a big deal.
| treffer wrote:
| The former approach was release + upgrade all projects. Now if
| one of those fails post-update you have to build a new release
| and upgrade all projects _again_. Multiply that by hundreds of
| services and it can become pretty annoying.
|
| By pulling all the reverse dependencies and testing those
| _before_ releasing & upgrading you reduce the chance that you
| need another bugfix release.
|
| And with hundreds of projects the average quality of tests does
| not matter that much. Some projects will happily pass, while
| some may fail. Those that fail will provide valuable insights.
| __alexs wrote:
| > The former approach was release + upgrade all projects. Now
| if one of those fails post-update you have to build a new
| release and upgrade all projects _again_. Multiply that by
| hundreds of services and it can become pretty annoying.
|
| I think there are actually at least 3 possibilities when you
| find a new version of a component breaks something that
| depends on it? 1) Continue using the old
| version for however long it still meets it's requirements and
| then do (2). 2) Make the local changes required to
| be compatible with the new version. 3) There is
| actually a bug in the new version and you need to fix the
| dependency.
|
| If (3) is happening a lot it suggests that your test suite
| isn't good enough or you have some design problems that are
| making it impossible to provide a stable API.
| Mathnerd314 wrote:
| What it sounds like is that they don't have many unit tests
| for jvmkit. So rather than write extensive unit tests,
| they're using integration testing with all the
| dependencies. I guess it makes sense, network stacks are
| hard to unit test.
| treffer wrote:
| Yes, and the whole solution allows you to plan / decide on
| (2) and (3) - both need that awareness.
|
| And (1) is common in the open source and cross organization
| world. But it is less desirable if both the consumers and
| producers of the library work in the same place. Version
| spread causes maintenance cost. And at one point it was a
| common theme that "an upgrade of JVMKit would have avoided
| this incident".
| catern wrote:
| This is a good technique, and it's preferable to investing a
| bunch of effort into building a comprehensive suite of unit
| tests. But this technique is applicable to more than just core
| libraries; you can apply this to your entire stack, and to your
| services: Test that you don't break your reverse dependencies, by
| running those reverse dependencies, like in
| http://catern.com/usertests.html
| masklinn wrote:
| For other projects doing something similar, the Rust project has
| Crater (https://crater.rust-lang.org), which I believe is
| automatically run on betas, and can be run explicitly on PRs. But
| is not by default. Because it builds tens of thousands of crates
| using the specified compiler, and takes 3 days to complete a run.
| dec0dedab0de wrote:
| This might be unpopular, but I don't think libraries should ever
| break backwards compatibility, at least not with enough
| regularity to require all of this. Sure if there is a major
| security thing, or something outside of your control then you
| might have to. But if you just want to change a function's
| signature, or return value, because you think it will help "clean
| things up" could you please just make a new function, or make it
| work both ways if your language allows it. I understand the
| desire to tear out a messy bit of code and make it beautiful, but
| it always leads to a bunch of headaches for little gain.
| failwhaleshark wrote:
| Semver is one way, but yes.
|
| It annoys me when libraries "clean up" (churn) their codebase
| when they should've thought more carefully about designing the
| interface (the contract) before publishing it. Code churn often
| creates more pointless work for N library users. I swear that
| most code churn is to make a project look "alive" or to give
| people something to do / job security.
|
| Deprecate slowly and have semi-automated user code fixups and
| notes.
| Mathnerd314 wrote:
| There's no way to entirely avoid breaking compatibility. If you
| allow small API breakages then at least all of the team is
| aware of the possibility and designs their processes to handle
| backwards incompatible changes. If you design your processes
| assuming backwards compatibility then you'll run into the
| 1-in-1000 case where it's not possible and have to stop
| everything to handle the major update.
| jnxx wrote:
| > There's no way to entirely avoid breaking compatibility.
|
| Why?
| masklinn wrote:
| Because you've got no idea how your code's being used in
| the wild an one person's bug is an other's feature. So any
| _observable_ change in behaviour is a possibly breaking
| change.
|
| And then people go and muck with your internals even if you
| protected them, because that's the only way they found to
| do what they needed to do, and they were much more
| interested in doing the thing than following your
| guidelines.
|
| And that's before even mentioning broken features which
| requires changes which will invariably break other stuff.
|
| The only way to _know for sure_ you 're never breaking
| compatibility is to never release any update. Otherwise
| you're playing the odds.
| simonw wrote:
| Something that here a lot here is having comprehensive,
| trustworthy documentation.
|
| This gives you the moral high ground to adopt semantic
| versioning: the correct behavior is the documented
| behavior. If someone has code that relies on an
| undocumented method or undocumented behavior, it's fair
| game to break that with only a minor version bump.
|
| If your documentation isn't any good, SemVer is almost a
| fiction.
| dmoy wrote:
| https://www.hyrumslaw.com/
|
| Also more facetiously, xkcd/1172
| dec0dedab0de wrote:
| _There 's no way to entirely avoid breaking compatibility. If
| you allow small API breakages then at least all of the team
| is aware of the possibility and designs their processes to
| handle backwards incompatible changes. If you design your
| processes assuming backwards compatibility then you'll run
| into the 1-in-1000 case where it's not possible and have to
| stop everything to handle the major update._
|
| Are you saying that it is better to have to deal with a bunch
| of tiny changes, so that when you have to deal with a big
| change you'll be used to it? If so, I strongly disagree.
|
| I would much rather have to deal with a big annoying change
| every 10 years, than a small annoying change every 6 months
| (which would still likely need the big one every 10 years).
| simonw wrote:
| Expecting library authors to get their design entirely correct
| the first time round isn't reasonable. When libraries make
| backwards-incompatible changes it's usually because the
| original design was flawed in a way that made continuing to
| support it, while adding new features, infeasible.
|
| You're welcome to evaluate and select libraries with the best
| possible track record for avoiding backwards incompatible
| changes - I do that myself. But, especially for libraries that
| are relatively early in their development process, having a
| hard "no breakages" policy isn't realistic.
|
| I still think semver is the best fix for this. If you make a
| backwards-incompatible change, you bump the major version
| number and you extensively document the change.
|
| If you're consuming libraries, the best way to address this
| problem is with solid test coverage. That way any time there's
| a new library version you can lean on your test suite to help
| you understand if it's going to break anything for you or not.
| edejong wrote:
| Is Linux unreasonable? The kernel has been backwards-
| compatible to user-land from the start.
|
| Or TeX, C, RS232?
| rot13xor wrote:
| Same with x86. Even modern x86 CPUs boots in real mode so
| you can run a baremetal 8086 binary from the 1980s.
| wahern wrote:
| > Expecting library authors to get their design entirely
| correct the first time round isn't reasonable.
|
| People shouldn't be publishing such libraries, nor should
| people be using them. With the exception of my first few FOSS
| projects, all of my open source libraries represent the 2nd,
| 3rd, or even 4th incarnation of a concept.
|
| That's just being responsible--don't knowingly litter the
| ecosystem with junk.
|
| I take a similar view at work. People are too quick to break
| out components into modules and libraries, long before they
| have a firm notion of what a proper interface should look
| like. If you can't commit to a stable API, then often times
| it's better to just copy whole functions and files between
| projects and let them evolve naturally. It takes time to see
| where that evolution goes, and which commonalities emerge in
| the implementation and surrounding application.
|
| But what about sharing bug and security fixes, you ask? Well,
| without a stable API you'll invariably end up with different
| projects stuck using different versions, anyhow. That's why
| people invented Docker :( What you end up with isn't an
| actually shared library, just the pretense of sharing, plus a
| bunch of unnecessary build complexity.
| simonw wrote:
| > People shouldn't be publishing such libraries, nor should
| people be using them.
|
| That's why we have pre-1.0 version numbers (and alphas and
| betas) - they let us release code early for people to
| provide feedback before we've committed to a non-breaking
| API.
|
| Even with a stable 1.0 release things change. People using
| the library may come up with logical feature requests that
| the original design didn't consider, and which require
| backwards incompatible changes. That's when you bump to a
| 2.0 release.
|
| My dream project would have a version number of something
| like 1.237 - meaning there have been 237 new features
| releases since the 1.0 release without backwards
| compatibility breaks. My hat is off to any project that
| pulls that off though, it's a rare accomplishment!
| renewiltord wrote:
| I actually genuinely think library authors should choose
| whatever trade-offs they want, and consumers should choose
| between those libraries based on the trade-offs they're
| comfortable accepting.
|
| The "always backwards compatibility" and "never backwards
| compatibility" schools of thought might just as well be "no
| programmer should ever use a garbage collector" or "all
| programmers should only ever use garbage collectors".
|
| Tools like OP help people who want to accept backwards
| incompatibility.
| zachrip wrote:
| People don't understand how their code can mess up others in
| unexpected ways. A few weeks ago my youtube just stopped
| working. Turns out, a change made to the apollo-graphql
| extension caused window.process to be readonly and youtube's
| code was not expecting that:
| https://github.com/apollographql/apollo-client-devtools/issu...
|
| I know this isn't a dependency issue, but it's a great example
| of why things like semver don't work perfectly in practice.
| It's hard to figure out what the effects of your code really
| are in the end. You might assume a change is minor when it
| really is major. I wish programming languages could figure out
| what the version change should be for a given code change.
| dec0dedab0de wrote:
| Sure, but that's just a bug in an application introduced by
| accident, and fixed when they found out about it. That is
| going to happen no matter what. The same thing goes for
| libraries. That is the best part of this tool, since they
| control the library and all the clients they can
| automatically run their tests across everything. Which would
| help you catch more of these types of problems, but it will
| never be perfect.
|
| My point is that if a user of your library is failing after a
| change, you should treat that as a bug in the library.
| Backwards incompatible changes should only be on purpose, and
| only when absolutely necessary.
|
| Of course, it's only their own code they're affecting here,
| so it doesn't really matter as much as a library for users
| you don't know. Plus it is a pretty cool solution, I just
| don't think it should be necessary.
| jesseryoung wrote:
| "Don't change the function signature" and semver are things
| that are absolutely hammered into devs who work on library
| code. I'd be hard pressed to find somebody even tangentially
| involved in software development who doesn't understand this.
|
| Semver doesn't cover changes in behavior. What a "breaking
| change" is when it comes to behavior changes is up to the
| person who implemented it.
| lumost wrote:
| At big companies this attitude eventually yields a stagnant
| product. No one is confidant that any impactful change wouldn't
| break _something_ and no one knows if that _something_ even
| matters to customers. Even worse there may be teams that will
| make 0 changes as they've been marked as KTLO.
|
| Having tooling to acknowledge that in fact no one will be
| affected by a change is a huge productivity boost.
| dec0dedab0de wrote:
| _Having tooling to acknowledge that in fact no one will be
| affected by a change is a huge productivity boost._
|
| Yes I do agree with that part
| nradov wrote:
| Years ago I was blown away when I saw a presentation by a
| Google Java library developer. He was able to make changes to
| his library, then automatically build every Java application
| throughout the enterprise which used that library to check
| for automated test failures. This allowed the library team to
| move fast with confidence.
| wikibob wrote:
| And you can too! http://Bazel.build
| underwater wrote:
| KTLO = Keeping the lights on, for anyone else who hasn't
| heard the acronym before.
| ebiester wrote:
| /orders
|
| /orders2
|
| /orders3
|
| /orders4
|
| /neworders
|
| /getordersbutonlythistype
| earthboundkid wrote:
| If you're changing the endpoint that same amount but keeping
| the URL, you've just imposed a ton of churn on everyone
| downstream.
|
| It's a tradeoff, like everything. Try to get the API right
| the first time. Try to avoid introducing new APIs. Try to
| avoid breaking the old API. There's no silver bullet.
| underwater wrote:
| And /orders uses an old data layer in a way that is
| incompatible with the new one, so the team can't deprecate
| the internal DB abstractions. And they can't introduce a new
| caching layer without making it work across the new and old
| abstractions. Etc, etc.
| samsquire wrote:
| I agree with you. Churn for churn's sake to make things
| marginally better. One man's refactoring is another man's
| refucktoring.
|
| Depending on a library could also insert a record on the
| library's CI system to indicate the dependency. The library CI
| will ping a URL with a version number to indicate that "you
| need to re-test with this version". It would certainly make
| software more robust and evolvable.
|
| I've thought about it before and called it library mesh:
| https://github.com/samsquire/ideas2#10-library-mesh
| m33k44 wrote:
| I remember reading an article on MSDN(a developer info system -
| I call it a system because it was also available on CDs as
| internet was not available everywhere) by a Microsoft employee.
| The article was titled some what on the lines of "How the
| marketing teams won over the engineering teams!" The article
| mentioned a tussle within Microsoft between the marketing and
| the engineering teams. The engineers were pushing for providing
| backwards compatibility which slowed them down and took time in
| providing new things/features that were the next sought after
| tech. The marketing teams on the otherhand were very frustrated
| that new sought after tech was not coming out faster so that
| they can then market it to the world. That tugh-off war was
| ultimately won by the marketing teams at Microsoft as per the
| article.
| matheusmoreira wrote:
| Reminds me of this article:
|
| https://www.joelonsoftware.com/2004/06/13/how-microsoft-
| lost...
| jnxx wrote:
| I've come to the same conclusion. Infrastructure and libraries
| breaking backward compatibility should not happen, it is not a
| sign of technical quality. It is almost always possible to
| provide wrapper interfaces which contain the old behavior and
| function signature as a special case. And if that does not
| work, it is better to release a different library.
|
| There is also one interesting insight I had when looking at
| error codes and exceptions: Normally, backwards compatibility
| is broken when you take things away from an interface -
| functions from a library, values from an enumeration type, and
| so on (Rich Hickey has made a brilliant talk about this).
| However, the general thing is that backward-compatible change
| must never _narrow pre-conditions_ , nor _widen post-
| conditions_.
|
| This means that it is _not_ OK to just add extra error codes to
| the return values of a function interface, or to add new
| classes of exceptions. This breaks backward compatibility.
|
| You can of course add enumeration values, optional arguments,
| keyword arguments and such to function call _arguments_ , and
| these are great ways to make interface changes backwards-
| compatible.
|
| Another insight is that backward-incompatible changes tend to
| cascade up larger dependency graphs. The broader and deeper the
| dependency graphs becomes, the more probable it is that a
| backward-compatible change (e.g. requiring a newer dependency)
| becomes itself a breaking change up the dependency chain. The
| philosophy in this age seems to be that of coourse _the other_
| library authors should "just fix their stuff", they get the
| responsibility of upgrading to the new version of the
| dependency. I do not agree to this at all - I think if
| something changes in a component, and this breaks the software,
| the responsibility is _always_ within that component, and
| nowhere else. If some change in the component causes breakage,
| this is a breaking change, by definition. And this is also true
| if it is an upgrade from Qt4 to Qt5, or OpenCV, or imageio-py2
| to imageio-py3 or such.
|
| One could increment the mayor version number but this does
| _not_ fix the problem. What fixes the problem is to _not break
| stuff in the first place_.
|
| I am quite convinced that as in the future, ever more
| programming will be done in libraries and components, there
| will be ecosystems which do not accept that kind of breakage.
| Linux is already a good example, and it is wildly successful.
| asdfasgasdgasdg wrote:
| I think widening post conditions may have to be possible in
| some cases. In my experience, programs should only depend on
| three possible error states: ok, eagain, and everything else.
| I guess it's fine for library authors to promise more, but
| I'd only do so in the case where I think that calling code
| can respond with more specificity to other conditions, which
| I think is pretty rare.
| nawgz wrote:
| > And if that does not work, it is better to release a
| different library
|
| Is that really a solution though? You're just going to end up
| with an unsupported old version and a supported yet breaking
| new version which probably doesn't even have upgrade
| guidance... Not sure I see it. I think libraries should work
| hard as you say to not break backwards compat, but I also
| think the library author has the right to change some
| function signatures when they change major behaviors too
| stuaxo wrote:
| Nice - I've been thinking stuff should work like this for a
| decade+
| BugsJustFindMe wrote:
| Following reverse dependencies on github (trivial with the search
| api for enough cases to make a difference) is something that I
| very very strongly wish that Pypa had done for the pip 20
| resolver change. It was even suggested to them by several people,
| but the suggestions were dismissed because breaking countless CI
| toolchains across the globe was considered fine because the new
| resolution behavior was "correct".
|
| Instead we got https://github.com/pypa/pip/issues/8713 back in
| August and then https://github.com/pypa/pip/issues/9187 in
| November which is pinned and still open to this day.
___________________________________________________________________
(page generated 2021-06-10 23:01 UTC)