https://github.com/golang/go/discussions/55090 Skip to content Toggle navigation Sign up * Product + Actions Automate any workflow + Packages Host and manage packages + Security Find and fix vulnerabilities + Codespaces Instant dev environments + Copilot Write better code with AI + Code review Manage code changes + Issues Plan and track work + Discussions Collaborate outside of code + Explore + All features + Documentation + GitHub Skills + Changelog * Solutions + By Size + Enterprise + Teams + Compare all + By Solution + CI/CD & Automation + DevOps + DevSecOps + Case Studies + Customer Stories + Resources * Open Source + GitHub Sponsors Fund open source developers + The ReadME Project GitHub community articles + Repositories + Topics + Trending + Collections * Pricing [ ] * # In this repository All GitHub | Jump to | * No suggested jump to results * # In this repository All GitHub | Jump to | * # In this organization All GitHub | Jump to | * # In this repository All GitHub | Jump to | Sign in Sign up {{ message }} golang / go Public * Notifications * Fork 15.4k * Star 104k * Code * Issues 5k+ * Pull requests 293 * Discussions * Actions * Projects 2 * Wiki * Security * Insights More * Code * Issues * Pull requests * Discussions * Actions * Projects * Wiki * Security * Insights extending Go backward compatibility #55090 rsc announced in Discussions extending Go backward compatibility #55090 @rsc rsc Sep 15, 2022 * 8 comments * 1 reply Return to top [104] rsc Sep 15, 2022 Maintainer edited - This discussion is about backward compatibility, meaning new versions of Go compiling older Go code. For the problem of old versions of Go compiling newer Go code, see this other discussion about forward compatibility. Go 1 introduced Go's compatibility promise, which says that old programs will by and large continue to run correctly in new versions of Go. There is an exception for security problems and certain other implementation overfitting. For example, code that depends on a given type not implementing a particular interface may change behavior when the type adds a new method, which we are allowed to do. We now have about ten years of experience with Go 1 compatibility. In general it works very well for the Go team and for users. However, there are also practices we've developed since then that it doesn't capture (specifically GODEBUG settings), and there are still times when users' programs break. I think it is worth extending our approach to try to break programs even less often, as well as to explicitly codify GODEBUG settings and clarify when they are and are not appropriate. As background, I've been talking to the Kubernetes team about their experiences with Go. It turns out that Go's been averaging about one Kubernetes-breaking change per year for the past few years. I don't think Kubernetes is an outlier here: I expect most large projects have similar experiences. Once per year is not high, but it's not zero either, and our goal with Go 1 compatibility is zero. Here are some examples of Kubernetes-breaking changes that we've made: * Go 1.17 changed net.ParseIP to reject addresses with leading zeros, like 0127.0000.0000.0001. Go interpreted them as decimal, following some RFCs, while all BSD-derived systems interpret them as octal. Rejecting them avoids taking part in parser misalignment bugs. (Here is an arguably exaggerated security report.) Kubernetes clusters may have stored configs using such addresses, so this bug required them to make a copy of the parsers in order to keep accessing old data. In the interim, they were blocked from updating to Go 1.17. * Go 1.15 changed crypto/x509 not to fall back to a certificate's CN field to find a host name when the SAN field was omitted. The old behavior was preserved when using GODEBUG=x509ignoreCN=0. Go 1.17 removed support for that setting. The Go 1.15 change broke a Kubernetes test and required a warning to users in Kubernetes 1.19 release notes. The Kubernetes 1.23 release notes warned users who were using the GODEBUG override that it was gone. * Go 1.18 dropped support for SHA1 certificates, with a GODEBUG= x509sha1=1 override. We announced removal of that setting for Go 1.19 but changed plans on request from Kubernetes. SHA1 certificates are apparently still used by some enterprise CAs for on-prem Kubernetes installations. * Go 1.19 changed LookPath behavior to remove an important class of security bugs, but the change may also break existing programs, so we included a GODEBUG=execerrdot=0 override. The impact of this change on Kubernetes is still uncertain: the Kubernetes developers flagged it as risky enough to warrant further investigation. These kinds of behavioral changes don't only cause pain for Kubernetes developers and users. They also make it impossible to update older, long-term-supported versions of Kubernetes to a newer version of Go. Those older versions don't have the same access to performance improvements and bug fixes. Again, this is not specific to Kubernetes. I am sure lots of projects are in similar situations. As the examples show, over time we've adopted a practice of being able to opt out of these risky changes using GODEBUG settings. The examples also show that we have probably been too aggressive about removing those settings. But the settings themselves have clearly become an important part of Go's compatibility story. Other important compatibility-related GODEBUG settings include: * GODEBUG=asyncpreemptoff=1 disables signal-based goroutine preemption, which occasionally uncovers operating system bugs. * GODEBUG=cgocheck=0 disables the runtime's cgo pointer checks. * GODEBUG=cpu.=off disables use of a particular CPU extension at run time. * GODEBUG=http2client=0 disables client-side HTTP/2. * GODEBUG=http2server=0 disables server-side HTTP/2. * GODEBUG=netdns=cgo forces use of the cgo resolver. * GODEBUG=netdns=go forces use of the Go DNS resolver Programs that need one to use these can usually set the GODEBUG variable in func init of package main, but for runtime variables, that's too late: the runtime reads the variable early in Go program startup, before any of the user program has run yet. For those programs, the environment variable must be set in the execution environment. It cannot be "carried with" the program. Another problem with the GODEBUGs is that you have to know they exist. If you have a large system written for Go 1.17 and want to update to Go 1.18's toolchain, you need to know which settings to flip to keep as close to Go 1.17 semantics as possible. I believe that we should make it even easier and safer for large projects like Kubernetes to update to new Go releases. In particular, I think we should probably: * Commit to gating these kinds of allowed but potentially-breaking runtime behavior changes by GODEBUG settings. Some settings may persist forever (execerrdot=0 seems like one). Others may be retired after a multiyear window (x509sha1=1 seems like one), instead of the "one release" window we've attempted in the past. * Export a counter of the number of code executions that behaved differently from standard Go due to a GODEBUG setting, through the runtime/metrics interface. (For example, /godebug/execerrdot= 0 would increment each time that setting suppresses an ErrDot error.) Programs can then watch that counter themselves. * Provide some easy way for main packages to set their default GODEBUG in source code. (That's already partly possible today with os.Setenv at the start of main, but that approach fails for runtime settings and for settings consulted at package init time. Note that main's init runs after all other package inits.) * Provide some easy way for programs to say "set the GODEBUGs to look like as much like Go 1.X as possible" for some earlier release of Go. * Note that none of this lowers the bar on making incompatible changes: we still avoid as much as possible, following https:// go.dev/doc/go1compat. GODEBUG may not be the mechanism I'd design today, but it's what we have and it doesn't seem bad enough to be worth adding a second way, so I'm going to assume it stays. Then the two things we need are (1) a way to set individual GODEBUG defaults in package main, and (2) a way to make the default GODEBUGs match an earlier version of Go. For (1), I am thinking about something like //go:debug x509sha1=1 in any package main source file. These would be pulled out by the go command and linked into the binary for processing at startup (before any Go code runs). The GODEBUG environment variable would still override these, of course. For (2), I am thinking about having the go line in the go.mod of package main's module, which already defines the exact language semantics of package main's Go source files, also define the default GODEBUG settings. So if package main's go.mod says go 1.17, SHA1 certificates still work, and os/exec does not generate ErrDot. As noted above, some GODEBUG settings will stick around forever (for example, execerrdot=1, netdns=go), while we will want to retire others. When a newer version of Go is compiling code written for an older version, if a GODEBUG has been retired, the behavior will depend on whether it is named explicitly (as in (1)) or implicitly (as in (2)). If a retired GODEBUG is mentioned explicitly, the build should fail. If it is only implied by the earlier Go version, then build should succeed, on the assumption that the vast majority of programs that say "go 1.17" are saying it because they were written in that era, not because they require support for SHA1 certificates. I think these two changes would go a long way toward making it even easier and safer to update to new Go toolchains, because it separates the update from the riskiest behavior changes and makes those changes easy to temporarily opt out of and also to debug. Note that this mechanism would be inappropriate to use for new, incompatible features, because the settings in package main are affecting the entire binary, and it is unlikely that all the packages in a large program would agree on which version of a large feature they want. In contrast, for "extra-backwards compatibility shims" like these, especially in the context where you're keeping older code running, affecting the whole binary is appropriate and does not cause problems. One thing people ask occasionally is whether Go will ever add LTS (long-term support) releases. I've always thought of Go 1 as the LTS release of Go, but subtle, necessary breaking changes like the ones listed above contradicted that idea. Being able to use a new Go toolchain but still get more faithful "old" semantics in these cases brings us much closer to Go 1 as LTS. There is a question about what to do if go.mod says a newer version of Go than the toolchain being used for the build. In that case the older toolchain does not know what the newer version does differently. I am filing a separate discussion about this forward compatibility problem. This is a discussion, not a proposal. I haven't implemented this nor even worked out all the implications. I'm curious what people think and what concerns they have. Thanks! Beta Was this translation helpful? Give feedback. 9 You must be logged in to vote 11 1 [?] 3 Replies: 8 comments * 1 reply Oldest Newest Top This comment was marked as spam. Sign in to view [980] liggitt Sep 15, 2022 - For (2), I am thinking about having the go line in the go.mod of package main's module, which already defines the exact language semantics of package main's Go source files, also define the default GODEBUG settings. So if package main's go.mod says go 1.17, SHA1 certificates still work, and os/exec does not generate ErrDot. I like the simplicity of this a lot. For what it's worth, "behave like go1.x" is actually what I expected the go directive to do when I first learned about modules, and is what it already accomplishes for the main module w.r.t. language features and go command behavior. Expanding that to include whatever runtime compatibility switches exist seems sensible to me. Beta Was this translation helpful? Give feedback. 3 You must be logged in to vote 6 0 replies [688] randall77 Sep 15, 2022 Maintainer - The GODEBUG environment variable would still override these, of course. We'd need to have the GODEBUG settings override field-by-field, probably. So if you did //go:debug: x509sha1=1 In front of main, and then ran your program by setting GODEBUG= asyncpreemptoff=1, we'd want the x509sha1 setting to still survive. Only if you set GODEBUG=x509sha1=off would the default setting be overridden. Beta Was this translation helpful? Give feedback. 2 You must be logged in to vote 1 reply @rsc rsc Sep 15, 2022 Maintainer Author - Indeed. That's exactly right: the //go:debug lines would set the default values for specific settings, which could then be overridden individually by $GODEBUG. Beta Was this translation helpful? Give feedback. [674] josharian Sep 15, 2022 Collaborator - Overall, a hearty +1. One downside to the implicit go.mod behavior is that getting all security improvements from a new Go version now requires actively editing go.mod. OTOH, this matches MVS pretty well, so maybe that's a point in its favor. Beta Was this translation helpful? Give feedback. 1 You must be logged in to vote 0 replies [762] xdg Sep 15, 2022 - FWIW, the go.mod approach is similar to what Perl (which also strongly prioritizes backwards compatibility) is doing. See the middle of this blog post from the Perl Steering Council for some examples. Beta Was this translation helpful? Give feedback. 1 You must be logged in to vote 0 replies [674] josharian Sep 15, 2022 Collaborator edited - Right now, if a dependency has a higher go.mod Go version number than package main, but the code compiles, everything is fine. Would that change? If not, how does a dependency indicate that it depends on new behavior? If so, it seems like this might rapidly cause dependency management pain, because everything has a Go toolchain version dependency. (I know this is your other discussion topic, I just wanted to point out that the answer there might turn out to be very important to assessing the viability here.) Beta Was this translation helpful? Give feedback. 2 You must be logged in to vote 0 replies [129] prattmic Sep 15, 2022 Maintainer - Should we add a new name for GODEBUG / //go:debug (as an alias, probably)? These uses of GODEBUG are clearly not for debugging (as many of the initial GODEBUG options are), so it seems potentially confusing, especially having the go.mod version set default GODEBUG values. This is very minor, but perhaps a chance to make a small improvement. Beta Was this translation helpful? Give feedback. 5 You must be logged in to vote 4 0 replies [638] willfaught Sep 15, 2022 - This seems like adding another "knob" to build configurations. As @josharian pointed out, there seem to be complexity in resolving how multiple modules with conflicting GODEBUG settings are merged together. Plus, it means that GODEBUG becomes a real, official part of the module system. Just saying "go 1.x" is so simple; I would hate to give that up. Let me push back and ask whether we should even have GODEBUG in the first place. Various settings are listed above, and they seem to relate to security decisions, like SHA1, or just deciding to break things, like making HTTP/2 the default (if I understand that correctly). In the former case, those breaking changes are part of the compat promise; old programs breaking with newer Go versions that make those kinds of changes is unfortunate, but expected, and even desired. Broken programs in that context reveal security flaws in them. I'm glad Kubernetes broke in those cases. If they want the benefits of newer Go versions, then they can pay the cost of avoiding those security flaws. This very well may mean that Kubernetes has to adopt a similar breaking change policy for security concerns that Go has. I suppose it's worth debating whether Go should impose that on package makers, but in my opinion, it should. No one should be entitled to upgrade without paying a price in this context. In the latter case, perhaps those changes shouldn't have been made at all. Instead of making HTTP/2 the default, we could have instead added an http2 package, or added an optional HTTP/2 mode to clients and servers. I can't recall whether the DNS resolver change was security-related, but if it wasn't, that also could have been an opt-in setting in the runtime or net packages. ParseIP probably shouldn't have been changed, since it was working as intended without a security flaw. Perhaps GODEBUG shouldn't exist at all. If behavior is changed, and it's compatible with the compat promise, then there should be a really good reason for it, and Kubernetes or whoever should be happy that it happened. Perhaps the reason why @rsc was so surprised by how often Kubernetes was broken was because the Kube team knew that, in most cases, most of those breaking changes happened for good reasons, and they accepted it. (Just conjecture. I don't claim to know or speak for them.) Beta Was this translation helpful? Give feedback. 1 You must be logged in to vote 0 replies Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment Category Discussions Labels None yet 8 participants @rsc @dims @willfaught @josharian @liggitt @prattmic @randall77 @xdg Add heading text Add bold text, Add italic text, Add a quote, Add code, Insert Link Link Text [ ] URL [ ] Add Add a link, Add a bulleted list, Add a numbered list, Add a task list, Directly mention a user or team Reference an issue or pull request Add heading text Add bold text, Add italic text, Add a bulleted list, Add a numbered list, Add a task list, 1 reacted with thumbs up emoji 1 reacted with thumbs down emoji 1 reacted with laugh emoji 1 reacted with hooray emoji 1 reacted with confused emoji [?] 1 reacted with heart emoji 1 reacted with rocket emoji 1 reacted with eyes emoji Footer (c) 2022 GitHub, Inc. Footer navigation * Terms * Privacy * Security * Status * Docs * Contact GitHub * Pricing * API * Training * Blog * About You can't perform that action at this time. You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.