[HN Gopher] Go Upgrade Checklist
___________________________________________________________________
Go Upgrade Checklist
Author : hakann
Score : 29 points
Date : 2025-01-02 19:13 UTC (3 hours ago)
(HTM) web link (hakann.substack.com)
(TXT) w3m dump (hakann.substack.com)
| peterldowns wrote:
| Out of curiosity, were you dealing with microservices defined
| within a monorepo, or microservices each in their own repo? The
| steps here:
|
| > Build your binaries with the new version. Go through the build
| errors if any.
|
| > Run all the unit tests with the new version. Go through the
| test failures.
|
| are a lot easier in a monorepo.
|
| Separately, I've experienced frequent breaking changes in the
| golangci-lint configuration file. I can't point to a specific
| instance of this happening but one thing I'd suggest is pinning
| your version of golangci-lint in development and in CI rather
| than using "latest".
|
| Golang's backwards compatibility and simplified toolchain is one
| of my favorite parts about it. Bumping go.mod and downloading the
| new version of go is usually all it takes!
| crockeo wrote:
| Not Hakan, but I was working closely with him at the time. Lyft
| was on a microservice many-repo setup, and we did pin the
| version of golangci-lint.
|
| I've found it's actually not so bad to do this kind of work
| across many repos, as long as you have the tooling to apply the
| same change to any number of codebases all at once. Our
| strategy was typically:
|
| - Write an idempotent codemod to do an upgrade. This is easy as
| long as your configuration is in a declarative language.
|
| - Regularly apply it or update it on all of the applicable
| repos.
|
| - Merge upgrades incrementally until you've upgraded 100%.
| peterldowns wrote:
| Thanks for the insight, that makes sense. What tool did you
| use for codemods? And do you have any other advice for
| dealing with many (50+) repos?
| mseepgood wrote:
| > Bumping go.mod and downloading the new version of go is
| usually all it takes!
|
| Doesn't it auto-download when you bump go.mod nowadays?
| physicles wrote:
| If you're working at a large company and downtime is extremely
| expensive, this checklist is a good guide. Otherwise, if you have
| good test coverage, you can get by with something simpler. It's
| super rare to have a breaking change in go.
|
| We do quarterly upgrades of all services in a monorepo (about
| 20-30). The steps are basically this:
|
| - Upgrade all dependencies to their latest versions, fixing build
| and test breaks (I read release notes for Go, but not for
| dependencies)
|
| - Look for deprecated packages and replace them
|
| - Upgrade all toolchains, including CI/CD containers, go.mod,
| etc.
|
| - Run all tests
|
| - Deploy to the test environment and make sure everything is
| green
|
| - Deploy to staging and do some sanity checks
|
| - Deploy to prod, keeping an eye on metrics for an hour or two
|
| We're on k8s and the state of all clusters (i.e. which images are
| running) is tracked in git, so a rollback is just git revert +
| apply.
|
| In practice, after about four years of this, we've seen maybe a
| dozen build breaks, and I can only remember one regression caused
| by a breaking change in a library[1].
|
| [1] https://github.com/golang/go/issues/24211
| nine_k wrote:
| Quarterly upgrades, four years: 16 upgrades. A dozen build
| breaks means that 75% upgrades face a build break.
|
| Since it's over the total number of builds for 20-30 services,
| it should not be that bad; instead, sometimes there happened a
| completely uneventful upgrade of everything!
| physicles wrote:
| You're right. Going back over some notes (including go.mod's
| history), we've been at this for six years, not four. And a
| dozen build breaks is probably an overestimate -- it's more
| like one every 12-18 months. Most upgrades are uneventful,
| everything just continues working.
| et1337 wrote:
| I'll add an item that is not yet on our checklist but has already
| bitten us several times: check your code generation. Since code
| generation is so popular in the Go ecosystem, we've got 5 or 6
| different codegen tools that update on various timelines. Twice
| now we've gone through a checklist similar to this article,
| patted ourselves on the back, and a week later found out no one
| can regenerate any code.
| jzwinck wrote:
| This is one reason why code generation should run as part of
| the build process, every time. Even if you decide to check-in
| the generated code for visibility.
| skunkworker wrote:
| The internal tools I've used regenerate the code as part of
| the CI process, and will fail the pipeline if the regenerated
| code has dirtied the git tree.
| timewizard wrote:
| > With the introduction of generics at 1.18, many linters lacked
| support for generics for months. We delayed the upgrade due to
| this issue.
|
| I wouldn't plan on using a new feature in production in the
| release that introduced it. Why would you plan to be using
| generics on day one?
|
| > There was talk of trying to solve this issue in the upstream
| ourselves.
|
| Was there a genuine business case that would make Lyft more
| profit if they used generics? If not then why would you even
| consider this?
|
| > Fortunately, by the time we seriously started exploring this
| option, linter support was added and go 1.19 was also released.
| We eventually upgraded directly to 1.19 from 1.17 but we were
| around 10 months late.
|
| You weren't late. You were precisely on time. This is some odd
| project mentality.
___________________________________________________________________
(page generated 2025-01-02 23:00 UTC)