[HN Gopher] The perils of transition to 64-bit time_t
___________________________________________________________________
The perils of transition to 64-bit time_t
Author : todsacerdoti
Score : 171 points
Date : 2024-09-28 16:31 UTC (6 hours ago)
(HTM) web link (blogs.gentoo.org)
(TXT) w3m dump (blogs.gentoo.org)
| kccqzy wrote:
| My biggest takeaway (and perhaps besides-the-point) is this:
|
| > Musl has already switched to that, glibc supports it as an
| option. A number of other distributions such as Debian have taken
| the leap and switched. Unfortunately, source-based distributions
| such as Gentoo don't have it that easy.
|
| While I applaud their efforts I just think as a user I want to be
| over and done with this problem by switching to a non source-
| based distribution such as Debian.
| wtallis wrote:
| It sounds like the difficulty for source-based distributions
| comes from trying to do an in-place upgrade that makes
| incompatible changes to the ABI. So changing to an entirely
| different distribution would be at least as disruptive (though
| possibly less time consuming) as doing a clean install of
| Gentoo using a new ABI.
| akira2501 wrote:
| Starting a few years ago I partition all my drives with two
| root partitions. Once is used and the other is blank. For
| precisely this reason. Sometimes it's easier to just roll out
| a brand new stage3 onto the unused partition, build an
| entirely new root, and then just move over to that once it's
| finished.
|
| The bonus is you can build your new system from your old
| system using just chroot. Very convenient.
| jasomill wrote:
| Several OSes I use on a daily basis (FreeBSD, Fedora
| CoreOS, and Fedora Kinoite) adopt related strategies as
| part of their regular (binary) update processes:
|
| https://wiki.freebsd.org/BootEnvironments
|
| https://coreos.github.io/rpm-ostree/
| ordu wrote:
| There is an easy way to deal with this in Gentoo. Boot from usb
| or something, run mkfs.ext4 (or .whatever fs you use) on your /
| and /usr partitions, mount them, unpack stage3 on them, chroot
| into them and run `emerge $all-my-packages-that-where-
| installed-before-mkfs`.
|
| You can install new copy of Gentoo instead of upgrading it
| incrementally.
| robin_reala wrote:
| "easy"
| ordu wrote:
| It is easy. It is just a tl;dr version of "how to install
| gentoo". :)
| lnxg33k1 wrote:
| I hope I will get a day where I have the will to
| recompile everything in the next 14 years, I think I had
| the same install for the past 10 or so on the gentoo
| desktop. The only think I reinstalled recently has been
| the laptop to switch from arch to sid
| viraptor wrote:
| > by switching to a non source-based distribution such as
| Debian.
|
| The distinction has more nuance. Source based distros like
| nixos don't have the same issue. The problem is more in how
| Gentoo builds/installs the packages than in building from
| source.
|
| Also with third party closed source software, you're still
| going to have issues, even on binary systems. Actually you
| could even have issues with the first party packages if they're
| installed in separate independent steps.
| n_plus_1_acc wrote:
| I'm no expert on C, but I was under the impression that type
| aliases like off_t are introduced to have the possibility to
| change then later. This clearly doesn't work. Am I wrong?
| beached_whale wrote:
| at the source level it does, but when you have compiled
| libraries it breaks.
| wtallis wrote:
| Source vs binary compatibility. Using typedefs like off_t mean
| you usually don't have to re-write code, but you do have to re-
| compile _everything_ that uses that type.
| rblatz wrote:
| But isn't the point of a source only distribution like
| Gentoo, to build everything yourself? Who is running gentoo
| but also lugging around old precompiled stuff they don't have
| the source for?
| mananaysiempre wrote:
| As the post describes, the problem is that on Gentoo you
| can't really build everything then switch binaries for
| everything, or at least that's not what happens when you
| update things the usual way.
|
| Instead, dependencies are built and then installed, then
| dependents are built against the installed versions and
| then installed, etc. In the middle of this process, the
| system can technically be half-broken, because it could be
| attempting to run older dependents against newer
| dependencies. Usually this is not a real problem. Because
| of how pervasive time_t is, though, the half-broken state
| here is potentially _very_ broken to the point that you may
| be unable to resume the update process if anything breaks
| for any reason.
| progbits wrote:
| Edit: nevermind others raise better points.
| cataphract wrote:
| That's not the problem. If time_t was a struct with a single
| int32_t, you'd be in the same situation when you changed it
| to int64_t (ABI incompatibility: you need more space to store
| the value now).
|
| In order not to have this problem you'd need the API to use
| opaque struts, for which only pointers would be exchanged.
| jonathrg wrote:
| It does work, the problem with ABI changes is that when you
| change it, you have to change it everywhere at the same time.
| By default there's nothing stopping you from linking one
| library built using 32-bit off_t with another library built
| using 64-bit off_t, and the resulting behaviour can be
| incredibly unpredictable.
| smueller1234 wrote:
| They make it easier, but just at a source code level. They're
| not a real (and certainly not full) abstraction. An example
| that'll be making it obvious: if you replace the underlying
| type with a floating point type, the semantics would change
| dramatically, fully visible to the user code.
|
| With larger types that otherwise have similar semantics, you
| can still have breakage. A straightforward one would be padding
| in structs. Another one is that a lot of use cases convert
| pointers to integers and back, so if you change the underlying
| representation, that's guaranteed to break. Whether that's a
| good or not is another question, but it's certainly not
| uncommon.
|
| (Edit: sibling comments make the same point much more
| succinctly: ABI compatibility!)
| SkiFire13 wrote:
| Yeah, the problem in general with type aliases is that they're
| just that, aliases, not proper types. This means that they leak
| all the details of the underlying type, and don't work for
| proper encapsulation, which is what's needed for being able to
| change what should be implementation details.
| tsimionescu wrote:
| The problem here would be exactly the same regardless of what
| form this type took. The issue is fundamental to any low
| level language: the size of a type is a part of its public
| API (well, ABI), by definition. This is true in C++ with
| private fields as well, for example: if you add a private
| field to a class, or change the type of a private field with
| one of a different size, all code using that class has to be
| re-built. The only way to abstract this is to use types
| strictly through pointers, the way Java does.
| ordu wrote:
| It kinda work, but not in a source based distro. If you can
| atomically rebuild @world with changed definition of off_t,
| then there will be no problem. But source based distro doesn't
| rebuild @world atomically. It rebuild one package at time, so
| there would be inconveniences like libc.so has 64-bit off_t,
| while gcc was build for 32-bit off_t, so gcc stops working. Or
| maybe bash, coreutils, make, binutils or any other package that
| is needed for rebuilding @world. At this point you are stuck.
|
| So such upgrade needs care.
| codys wrote:
| > there would be inconveniences like libc.so has 64-bit off_t
|
| glibc specifically has support for the 32-bit and 64-bit
| time_t abi simultaneously.
|
| From the post:
|
| > What's important here is that a single glibc build remains
| compatible with all three variants. However, libraries that
| use these types in their API are not.
| ordu wrote:
| Yeah, glibc was a bad example. Probably libz.so or
| something would be better.
| codys wrote:
| Yep, agreed. Though it does expose an option here which
| would be "have glibc provide a mechanism for other
| libraries (likely more core/widely used libs) to support
| both ABIs simultaneously.
|
| Presumably if that had been done in glibc when 64-bit
| time_t support was added, we could have had multi-size-
| time ABI support in things like zlib by now. Seems like a
| mistake on glibc's part not to create that initially
| (years ago).
|
| Though if other distros have already switched, I'd posit
| that perhaps Gentoo needs to rethink its design a bit so
| it doesn't run into this issue instead.
| jasomill wrote:
| This sounds like something similar to NeXTSTEP/macOS fat
| binaries, only with the possibility of code sharing
| between "architectures".
|
| I like it, though it sounds like something that'd be
| unlikely to see adoption in the Linux world without the
| endorsement of multiple major distros.
| o11c wrote:
| > have glibc provide a mechanism for other libraries
| (likely more core/widely used libs) to support both ABIs
| simultaneously
|
| The linked article is wrong to imply this isn't possible
| - and it really doesn't depend on "GLIBC provide a
| mechanism". All you have to do is:
|
| * Compile the library itself with traditional time_t
| (32-bit or 64-bit depending on platform), but convert and
| call time64 APIs internally.
|
| * Make a copy of every public structure that embeds a
| time_t (directly or indirectly), using time64_t (or
| whatever contains it) instead.
|
| * Make a copy of every function that takes any time-using
| type, to use the time64 types.
|
| * In the public headers, check if everything is compiled
| with 64-bit time_t, and if so make all the traditional
| types/functions aliases for the time64 versions.
|
| * Disable most of this on platforms that already used
| 64-bit time_t. Instead (for convenience of external
| callers) make all the time64 names aliases for the
| traditional names.
|
| It's just that this is a lot of work, and little benefit
| to any particular library. the GCC 5 std::string
| transition is probably a better story than LFS, in
| particular the compiler-supported `abi_tag` to help
| detect errors (but I think that's only for C++, ugh -
| language without room for automatic mangling suck).
|
| (minor note: using typedefs rather than struct tags for
| your API makes this easier)
| Athas wrote:
| Why not essentially treat it as a cross compilation scenario?
| NixOS is also source based, but I don't think such a
| migration would be particularly difficult. You'd use the
| 32-bit off_t gcc to compile a glibc with 64-bit off_t, then
| compile a 64-bit off_t gcc linked against that new glibc, and
| so on. The host compiler shouldn't matter.
|
| I always understood the challenge as _binary_ compatibility,
| when you can 't just switch the entire world at once.
| codys wrote:
| Nixos has it easier here because they don't require
| packages to be "installed" before building code against
| them. For Gentoo, none of their build scripts (ebuilds) are
| written to support that. It's plausible that they might
| change the embuild machinery so that this kind of build
| (against non-installed packages) could work, but it would
| need investigation and might be a difficult lift to get it
| working for all packages.
|
| "treat it as a cross compilation scenario" is essentially
| what the post discusses when they mention "use a different
| CHOST". A CHOST is a unique name identifying a system
| configuration, like "x86_64-unknown-linux-gnu" (etc).
| Gentoo treats building for different CHOSTs as cross
| compiling.
| andrewaylett wrote:
| NixOS isn't the same kind of source-based. At some level,
| even Debian could be said to be source based: there's
| nothing stopping you from deciding to build every package
| from source before installing it, and obviously the
| packages are themselves built from source at some point.
|
| NixOS sits between Debian and Gentoo, as it maintains an
| output that's capable of existing independently of the rest
| of the system (like Debian) but is designed to use the
| current host as a builder (like Gentoo). Gentoo doesn't
| have any way to keep individual builds separate from the
| system as a whole, as intimated in the article, so you need
| to work out how to keep the two worlds separate while you
| do the build.
|
| I think what they're suggesting winds up being pretty
| similar to what you suggest, just with the right plumbing
| to make it work in a Gentoo system. NixOS would need
| different plumbing, I'm not sure whether they've done it
| yet or how but I can easily imagine it being more
| straightforward than what Gentoo is needing to do.
| tadfisher wrote:
| There absolutely will be problems with different Nix
| profiles that aren't updated together; for example, if
| you update some packages installed in your user's profile
| but not the running system profile. But this is common
| enough with other glibc ABI breakage that folks tend to
| update home and system profiles together, or know that
| they need to reboot.
|
| Where it will be hell is running Nix-built packages on a
| non-NixOS system with non-ABI-compatible glibc. That is
| something that desperately needs fixing on the glibc
| side, mostly from the design of nss and networking, that
| prevent linking against glibc statically.
| ArsenArsen wrote:
| this is just as much of a problem on binary distros mind you.
| just because there might be a smaller time delta does not
| mean that the problem is solved
| codys wrote:
| nixos is an example of a distro that is source based and can
| do the atomic rebuild here because it has a way to build
| packages against other packages that aren't "installed". In
| nixos, this is because there are very few things that get
| "installed" as the global, only thing to use. But one could
| imagine that Gentoo could build something that would allow
| them to at least build up one set of new packages without
| installing them and then install the files all at once.
| jasomill wrote:
| As someone whose nearest exposure to a "source based distro"
| is FreeBSD, this sounds like madness, as it means a broken
| build could not only impair attempts to repair the build, but
| render the system unbootable and/or unusable.
|
| And as someone who regularly uses traditional Linux distros
| like Debian and Fedora, the idea that a package management
| system would allow a set of packages known to be incompatible
| with one another to be installed without force, or that core
| package maintainers would knowingly specify incorrect
| requirements, is terrifying.
|
| While I'm not familiar with Gentoo, my reading of this
| article suggests that its maintainers are well aware of these
| sorts of problems and that Gentoo does not, in fact, suffer
| from them (intentionally; inevitably mistakes happen just as
| they occasionally do on the bleeding edge of FreeBSD and
| other Linux distros).
| raldi wrote:
| I think the main reason they're introduced is to provide a hint
| to the programmer and maybe some typesafety against
| accidentally passing, say, a file descriptor.
| jandrese wrote:
| It's only the first step of the puzzle. And arguably only a
| half step. As the article points out anytime that off_t is
| stuffed into a struct, or used in a function call, or
| integrated into a protocol, the abstraction is lost and the
| actual size matters. Mixing old and new code, either by loading
| a library or communicating over a protocol, means you get
| offsets wrong and things start crashing. Ultimately the
| changeover requires everybody to segregate their programs
| between "legacy" and "has been ported or at least looked over",
| which is incredibly painful.
| devit wrote:
| The C library uses macros so that the referenced symbols are
| different depending on ABI (e.g. open becomes open64, etc.),
| but most 3rd party libraries don't bother with that, so they
| break if their API uses time_t/off_t.
| BoingBoomTschak wrote:
| A very thoughtful way of handling a problem much trickier than
| the earlier /usr merge. Had thought about 2 but not 1 and 3. I
| also had kinda forgotten that 2038 thing, time sure is flying!
|
| I must say, mgorny's posts are always a treat for those who like
| to peer under the hood! (The fact that Gentoo has remained my
| happy place for years doesn't influence this position, though
| there's probably some correlation)
| fred_is_fred wrote:
| Besides epoch time and the LFS support mentioned, are there any
| other 32-bit bombs waiting for Linux systems like this?
| fph wrote:
| IPv4, technically, is another 32-bit bomb, but that's not
| Linux-specific.
| samatman wrote:
| I wouldn't call this a bomb at all. Bombs are events, not
| processes.
|
| Resource contention for IPv4 has been around for a long time,
| with a number of workarounds and the ultimate out of
| supporting IPv6. There has been, to date, no moment of
| crisis, nor do I expect one in the future.
|
| It will just get steadily more annoying/expensive to use IPv4
| and IPv6 will relieve that pressure incrementally. We're at
| least two decades into that process already.
| poincaredisk wrote:
| Two decades in, and IPv6 is still the more annoying option.
|
| I wish they were less ambitious and just increased address
| sizes when designing ipv6.
| tredre3 wrote:
| The ext file system uses many 32bit counters. Admittedly,
| version 4 fixed most of that (when formatted with the correct
| options).
| kbolino wrote:
| In a similar vein, inodes can run out. On most conventional
| Linux file systems, inode numbers are 32 bits.
|
| For many, this is not going to be a practical problem yet, as
| real volumes will run out of usable space before exhausting
| 2^32 inodes. However, it is theoretically possible with a
| volume as small as ~18 TiB (using 16 TiB for 2^32 4096-byte
| or smaller files, 1-2 TiB for 2^32 256- or 512-byte inodes,
| plus file system overheads).
|
| Anticipating this problem, most newer file systems use 64-bit
| inode numbers, and some older ones have been retrofitted
| (e.g. inode64 option in XFS). I don't think ext4 is one of
| them, though.
| lclarkmichalek wrote:
| It does happen in prod. Usually due to virtual FSes that
| rely on get_next_ino: https://lkml.org/lkml/2020/7/13/1078
| loeg wrote:
| The C standard does not require time_t to be signed (nor does
| POSIX). Just changing the 32-bit type to unsigned would (in some
| senses) extend the lifetime of the type out to 2106. You could at
| least avoid some classes of ABI breakage in this way. (On the
| other hand, Glibc has explicitly documented that its time_t is
| always signed. So they do not have this option.)
| umanwizard wrote:
| You would then lose the ability to represent times before Jan.
| 1st, 1970. Which is not just a theoretical concern; those times
| appear e.g. in databases with people's date of birth.
| nobluster wrote:
| And a signed 32 bit time_t with an epoch of 1970 cannot
| represent dates before 1902. Using time_t to store legacy
| dates is not advisable - even if you ignore all the issues
| with time zones and changing local laws pertaining to offsets
| from UTC and daylight saving time.
| pwg wrote:
| While true, that limitation has always existed, so everyone
| has already implemented whatever was necessary to represent
| dates earlier than that.
|
| Changing 32-bit time_t to unsigned suddenly makes all dates
| from 1902 to Jan 1 1970 which were stored using time_t
| (even if it was non-advisable, it still will have occurred)
| appear to teleport into the future beyond 2038.
| cryptonector wrote:
| That's alright, see, because I have no filesystems, no
| tarballs, no backups, no files on any kind of media with
| timestamps before 1970, and indeed, no one can except by
| manually setting those timestamps -- and why bother doing
| that?!
|
| So any pre-1970 32-bit signed timestamps will be in...
| not spreadsheets, in what? In databases? No, not either.
| So in some sort of documents, so software consuming those
| will need fixing, but we're not going to see much
| teleporting of 1902-1970 timestamps to 2038-2106. I'm not
| concerned.
| jasomill wrote:
| Many important software projects predate UNIX. Perhaps
| you want to create a Git repository for one of them, with
| historically accurate commit timestamps?
| loeg wrote:
| Git represents timestamps in ASCII strings, like "12345."
| Not time_t. (I am not sure if it even allows negative
| integers in this format.)
| cryptonector wrote:
| Well, Git doesn't seem to set file timestamps when
| cloning. And as the sibling comment says, Git doesn't use
| 32-bit signed integers to store timestamps. So this is
| not a problem.
|
| If, however, Git were to some day get an option to set
| file mtimes and atimes at clone time to the last-modified
| time of the files based on commit history, then you could
| always just use a 64-bit system where `time_t` is 64-bit.
| bloak wrote:
| Serious question: do people use time_t for representing a
| date of birth?
|
| To me that wouldn't seem right: a date of birth isn't a
| timestamp and you typically receive it without a
| corresponding place or time zone so there's no reasonable way
| to convert it into a timestamp.
|
| (The other problem is that a signed 32-bit time_t only goes
| back to 1901. You might not have to deal with a date of birth
| before 1901 today, unless you're doing genealogy, of course,
| but until fairly recently it's something you'd probably want
| to be able to handle.)
| nomel wrote:
| My birth certificate has location, day, year, hour, and
| minute of birth. Birth is an event in time, perfectly
| represented with a (UTC) timestamp.
| Merad wrote:
| Your birth is an event that happened at an instant in
| time, but very few systems concern themselves with that
| detail. The vast majority need to store birth _date_ and
| have no interest in the time or location.
| nomel wrote:
| > have no interest in the time
|
| Setting the bottom couple of bytes to zero achieves this,
| while maintaining nearly universal consistency with all
| other events in time that might need to be related with a
| packing of bits. People do it because it's what's being
| used at nearly every other level of the stack.
| cryptonector wrote:
| > Birth is an event in time, perfectly represented with a
| (UTC) timestamp.
|
| That's not the same thing as `time_t` though. `time_t` is
| UTC. UTC is not `time_t`.
| umanwizard wrote:
| > Serious question: do people use time_t for representing a
| date of birth?
|
| I have seen people in real life use seconds since the UNIX
| epoch to represent DOB, yes.
| loeg wrote:
| Databases do not and can not use system time_t. Consider how
| their on-disk state would be impacted by a change from 32-bit
| time_t to 64-bit! Instead they use specific or variable size
| integer types.
| cryptonector wrote:
| Those databases might not even use `time_t`. It's their
| problem anyways, not the OS's.
| pezezin wrote:
| Does any database actually use time_t? PostgreSQL uses its
| own datatype, the number of microseconds since 4713 BC. I am
| sure that other databases do the same.
|
| https://www.postgresql.org/docs/current/datatype-
| datetime.ht...
| scheme271 wrote:
| If time_t is unsigned, how do times before the unix epoch get
| represented?
| loeg wrote:
| They don't, much like they already do not. 32-bit time_t has
| always been finite, and 1970 was a long, long time ago. (See
| "in some senses" in my earlier comment.)
| pwg wrote:
| While that avoids 2038 as a "drop dead" date for 32-bit time_t,
| it also removes the ability to represent any date/time prior to
| 00:00:00 UTC on 1 January 1970 using 32-bit time_t because you
| lose the ability to represent negative values.
|
| Having all existing stored date/times that are currently prior
| to the epoch suddenly become dates post 2038 is also not a good
| scenario.
| loeg wrote:
| > it also removes the ability to represent any date/time
| prior to 00:00:00 UTC on 1 January 1970 using 32-bit time_t
|
| Yes, of course. This is probably not the main use of negative
| values with signed time_t, though -- which is just
| representing the result of subtraction when the operand
| happened before the subtrahend.
|
| > Having all existing stored date/times that are currently
| prior to the epoch suddenly become dates post 2038 is also
| not a good scenario.
|
| In practice, there are ~zero of these on systems with 32-bit
| time_t and a challenging migration path as we approach 2038.
| cryptonector wrote:
| > Yes, of course. This is probably not the main use of
| negative values with signed time_t, though -- which is just
| representing the result of subtraction when the operand
| happened before the subtrahend.
|
| This is definitely a bigger concern, yes. One has to be
| very careful with subtraction of timestamps. But to be fair
| one already had to be very careful before because POSIX
| doesn't say what the size or signedness of `time_t` is to
| begin with.
|
| Indeed, in POSIX `time_t` can even be `float` or
| `double`[0]! time_t and clock_t shall be
| integer or real-floating types.
|
| Though on all Unix, BSD, Linux, and any Unix-like systems
| thankfully `time_t` is always integral. It's really only
| size and signedness that one has to be careful with.
|
| Thus one should always subtract only the smaller value from
| the larger, and cast the result to a signed integer. And
| one has to be careful with overflow. Fortunately
| `difftime()` exists in POSIX. And there's a reason that
| `difftime()` returns a `double`: to avoid having the caller
| have to deal with overflows.
|
| Basically working safely with `time_t` arithmetic is a real
| PITA. [0] https://pubs.opengroup.org/online
| pubs/009696799/basedefs/sys/types.h.html
| loeg wrote:
| > Indeed, in POSIX `time_t` can even be `float` or
| `double`[0]!
|
| Standard C, yes. Newer POSIX (your link is to the 2004
| version) requires time_t be an integer type: https://pubs
| .opengroup.org/onlinepubs/9699919799/basedefs/sy...
|
| > Though on all Unix, BSD, Linux, and any Unix-like
| systems thankfully `time_t` is always integral. It's
| really only size and signedness that one has to be
| careful with. Thus one should always subtract only the
| smaller value from the larger, and cast the result to a
| signed integer. And one has to be careful with overflow.
| Fortunately `difftime()` exists in POSIX. And there's a
| reason that `difftime()` returns a `double`: to avoid
| having the caller have to deal with overflows.
|
| > Basically working safely with `time_t` arithmetic is a
| real PITA.
|
| Yes.
| cryptonector wrote:
| Yes, I know that was older POSIX. But we're talking about
| old code, the unstated context is portability over a long
| period of time, and I wanted to make a point :)
|
| So use `difftime()`, don't assume signedness or size, but
| do assume that it's an integral type.
| cryptonector wrote:
| Stored _where_ exactly?
| petee wrote:
| Openbsd did just this for 32bit compat when they changed to
| 64bit time 12 years ago, and seems to have worked out fine.
| brynet wrote:
| No, time_t is a signed 64-bit type on all architectures,
| 64-bit architectures have no "32bit compat".
|
| https://www.openbsd.org/55.html
| petee wrote:
| From the notes, this is what i was referring to, I think I
| just mistook the meaning -
|
| _Parts of the system that could not use 64-bit time_t were
| converted to use unsigned 32-bit instead, so they are good
| till the year 2106_
| codys wrote:
| There are a few options for Gentoo not discussed in the post,
| possibly because for Gentoo they would be a larger amount of work
| due to the design of their system:
|
| 1. Allow building against packages without installing them. The
| core issue here is that Gentoo package build and install happen
| as a single step: one can't "build a bunch of things that depend
| on one another" and then "atomically install all the build items
| into place". This means that Gentoo can easily be partially
| broken when one is doing updates when an ABI change occurs
| (modulo so versioning, see next option). This issue with 64-bit
| time_t is an example of an ABI change that folks are very aware
| of and is very widespread. It's also an example of something that
| causes an ABI change that isn't handled by the normal `.so`
| versioning scheme (see next option).
|
| 2. Extend the normal `.so` versioning to capture changes to the
| ABI of packages caused by packages they depend on. Normally,
| every `.so` (shared object/library) embeds a version number
| within it, and is also installed with that version number in the
| file name (`libfoo.so.1.0.0`, for example, would be the real
| `.so` file, and would have a symlink from `libfoo.so` to tell the
| linker which `.so` to use). This shared object version is
| normally managed by the package itself internally (iow: every
| package decides on their ABI version number to enable them to
| track their own internal ABI breakages). This allows Gentoo to
| upgrade without breaking everything while an update is going on
| as long as every package manages their `.so` version perfectly
| correctly (not a given, but does help in many cases). There is a
| process in Gentoo to remove old `.so.x.y.z` files that are no
| longer used after an install completes. What we'd need to do to
| support 64-bit time_t is add another component to this version
| that can be controlled by the inherited ABI of dependencies of
| each `.so`. This is very similar in result to the "use a
| different libdir" option from the post, but while it has the
| potential to set things up to enable the same kinds of ABI
| changes to be made in the future, it's likely that fixing this
| would be more invasive than using a different libdir.
| akira2501 wrote:
| > Allow building against packages without installing them.
|
| A partial staged update system would work the best. I should be
| able to schedule the build of several new packages, have them
| built into a sandbox, and have new compilations use the sandbox
| first and then fallback to the system through a union, and then
| once everything is build finally package up all the pieces and
| then move them out of the sandbox and into the system at large.
|
| You could make all gentoo updates transactional that way which
| would be a huge boon in many other ways.
| aaronmdjones wrote:
| Gentoo already allows option 1 by specifying a directory to
| ultimately install the finished image to (normally it is set to
| /) [1]. One can do a complete rebuild of everything in @system
| and @world, install it to the specified subdirectory, and then
| sync it all over in one shot. Preferably you would do this from
| a live session, although in theory you could also bind-mount /
| onto a subdirectory of the place you reinstalled everything to,
| chroot into it, and then sync it (what is now /) into the bind-
| mounted real upper /.
|
| [1] https://devmanual.gentoo.org/ebuild-writing/variables/#root
| codys wrote:
| It's true that Gentoo has some pieces of what they need to
| build an implementation of option 1 (build packages that
| depend on other packages before completeing an install), but
| it's currently the case that that is not what happens when
| one runs `emerge` (the gentoo packaging build/installing
| tool), as you've noted one would need to write scripts that
| wrap emerge (or do the same work manually) to attempt to
| accomplish this today.
|
| I suspect that using bind mounts and overlays (to allow the
| "building packages" chroot a view of the "installed" root)
| could be used to accomplish this, or alternately some
| filesystem snapshotting features if we're thinking about this
| from the "external to emerge" angle. (It's my understanding
| that for Gentoo to do this ABI change, though, they need some
| solution integrated into emerge).
|
| To some extent, this kind of potential model also reminds me
| of systems that integrate their package updating with
| filesystem snapshotting to allow rollbacks and actually-
| atomic upgrades of many files. I think one or more of the
| solaris distributions did this?
| rini17 wrote:
| The third paragraph says following, which means it's not
| suitable solution to build whole system:
|
| When building a package, ROOT should not be used to satisfy
| the required dependencies on libraries, headers files etc.
| Instead, the files on the build system should be specified
| using /.
| nobluster wrote:
| For a large legacy 32 bit unix system dealing with forward dates
| I replaced all the signed 32 bit time_t libc functions with
| unsigned 32 bit time_t equivalents. This bought the system
| another 68 years beyond 2038 - long after I'll be gone. The
| downside is that it cannot represent dates before the unix epoch,
| 1970, but as it was a scheduling system it wasn't an issue.
|
| If legacy dates were a concern one could shift the epoch by a
| couple of decades, or even reduce the time granularity from 1
| second to 2 seconds. Each alternative has subtle problems of
| their own. It depends on the use case.
| suprjami wrote:
| If you can change the whole system from signed to unsigned, why
| not change to 64-bit?
| dark-star wrote:
| Are there other distros where the switch to 64-bit time_t has
| already happened? What's the easiest way to figure out whether
| $distro uses 32-bit or 64-bit time_t? Is there something
| easier/quicker than writing a program to print `sizeof(struct
| stat)` and check if that's 88, 96 or 1098 bytes (as hinted in the
| article)?
| teddyh wrote:
| Like the article says, Debian has already switched.
| jonathrg wrote:
| printf '#include <stdio.h>\n#include <time.h>\nint main() {
| printf("time_t is %%zu-bit\\\n", sizeof(time_t)*8); }\n' | gcc
| -x c -o timesize - && ./timesize && rm ./timesize
|
| If you're wondering about a specific distro that you're not
| using right now - just look it up.
| SAI_Peregrinus wrote:
| NixOS switched. And they're source-based, but with actual
| dependency tracking and all the (insanely complex) machinery
| needed to allow different programs to use different C library
| versions simultaneously.
|
| `printf("sizeof (time_t) = %zu, %zu bits", sizeof (time_t),
| sizeof (time_t) * CHAR_BIT);` gives you the size in bytes and
| in bits. Needs time.h, stddef.h, and stdio.h.
| grantla wrote:
| > A number of other distributions such as Debian have taken the
| leap and switched. Unfortunately, source-based distributions such
| as Gentoo don't have it that easy.
|
| For Debian it was extremely painful. A few people probably burned
| out. Lots of people pointed to source-based distributions and
| said "they will have it very easy".
| Denvercoder9 wrote:
| > For Debian it was extremely painful.
|
| Do you have any references that elaborate on that? From an
| outsider perspective, the time64 transition in Debian seemed to
| have been relatively uncontroversial and smooth. Way better
| than e.g. the /usr-merge.
| jlarocco wrote:
| https://lwn.net/Articles/812767/
|
| https://wiki.debian.org/ReleaseGoals/64bit-time
|
| I'm continually impressed by Debian. My Debian systems are
| pretty boring, so maybe it's to be expected, but as an end-
| user neither the /usr merge or time64 transition affected me
| much.
| ajsnigrutin wrote:
| > For Debian it was extremely painful. A few people probably
| burned out. Lots of people pointed to source-based
| distributions and said "they will have it very easy".
|
| 'easy' in a way "just tell the user to rebuild everything in
| one go"? :)
| cryptonector wrote:
| The simplest thing to do is to make any 32-bit `time_t`s be
| unsigned. That buys another 68 years to get the transition done.
| Not that that's exactly easy, but it's easier than switching to
| 64-bit time_t.
| jeffbee wrote:
| Are we still keeping shared libraries? They are a complex
| solution to a problem that arguably stopped existing 20 years
| ago. Might be time to rethink the entire scheme.
| filmor wrote:
| On Gentoo, definitely. I really don't want to rebuild my whole
| system whenever some foundational library fixes a bug. It
| already annoys me quite a bit that I need to set up sccache to
| get half-way reasonable compile times out of Rust projects (and
| I'm saying that as someone who enjoys gradually replacing
| significant portions of the userspace with Rust tools).
| otabdeveloper4 wrote:
| The Docker image layers you so dearly love are an
| implementation of shared libraries, except done in a broken way
| that's a thousand times less performant and more insecure.
| wpollock wrote:
| I'm probably naive, but I see another way forward. If all the t64
| binaries are static linked, they have no danger of mixing abis.
| After 2038, all the t32 code is broken anyway so there's no
| _additional_ risk for going back to dynamic linking then. I feel
| if this was a solution the author would have mentioned it but I
| 'm willing to look foolish to hear what others will say.
| layer8 wrote:
| Static linking is probably impractical for applying security
| updates, as often you'd have to recompile/relink basically the
| whole system. In some cases it could also significantly
| increase memory usage.
| GeorgeTirebiter wrote:
| with fast networks, huge disks, fast processors --- it seems
| wasteful to me to even consider shared libraries. Shared
| libraries is a technology that was useful when we were memory
| starved. We are no longer memory starved. So you replace the
| static binary? Big deal, size is not an issue (for 99% of the
| cases) given what we have today.
|
| Recall, too, that the "link" step of those .o files is
| actually a "link / edit" step, where routines in the
| libraries not used are not linked.
| layer8 wrote:
| It's much more straightforward to ensure consistency with
| shared libraries, and not having to rebuild stuff. Wasting
| disk space, RAM, network bandwidth and processing time is
| what seems wasteful to me.
| gotoeleven wrote:
| Are you by chance a javascript programmer?
| jiggawatts wrote:
| Static linking is the root cause for "modern" apps based on
| Electron taking minutes to start up and be useful. They're
| statically linking almost an entire operating system
| (Chromium), an entire web server, server runtime framework,
| _and_ an equivalent client framework for good measure.
|
| On the fastest PC that money can buy this is somewhere
| between "slow" and "molasses".
|
| I miss the good old days when useful GUI programs were mere
| kilobytes in size and launched instantly.
| layer8 wrote:
| > The second part is much harder. Obviously, as soon as we're
| past the 2038 cutoff date, all 32-bit programs -- using system
| libraries or not -- will simply start failing in horrible ways.
| One possibility is to work with faketime to control the system
| clock. Another is to run a whole VM that's moved back in time.
|
| Yet another is to freeze the time for those programs at the last
| 32-bit POSIX second. They would just appear to execute incredibly
| fast :). Of course some will still break, and it's obviously not
| suitable for many use cases (but neither is running in the past),
| but some might be just fine.
| ndesaulniers wrote:
| I think we should start putting details of the ABI in ELF (not
| the compiler flags as a string). Wait.. Who owns the ELF spec??
|
| Then the linker and loader could error if two incompatible
| objects with different ABIs were attempted to be linked together.
|
| For instance, I suspect you could have fields to denote the size
| of certain types. I guess DWARF has that... But DWARF is optional
| and sucks to parse.
| ddoolin wrote:
| Why was it 32 bits to begin with? Wasn't it known that 2038 would
| be the cutoff?
| NegativeLatency wrote:
| Ultimately probably hardware? I suspect it's also been like
| this for a long time.
| jdndhdhd wrote:
| Why is it 64 bit now? Isn't it know that 292277026596 would be
| the cutoff?
| int_19h wrote:
| It was not exactly a big concern when this was designed in
| early 70s, especially when you consider that Unix itself was
| kind of a hack at the time.
| wongarsu wrote:
| It has been that way since Unix V4 which released in 1973. Back
| then moving to something that breaks after 65 years was a
| massive upgrade over the old format that wrapped after two and
| a half years. And at least from the standpoint of Unix
| engineers 32 bits was more than enough: Nobody is using Unix V4
| in production in 2024, never mind 2038.
|
| Why it made it into Posix and wasn't updated is a different
| question that's a bit more difficult to answer
| modeless wrote:
| In the 1970s people probably would have laughed at the idea
| that Unix would still be running in the year 2038.
| eschaton wrote:
| The way this was handled on Mac OS X for `off_t` and `ino_t`
| might provide some insight: The existing calls and structures
| using the types retained their behavior, new calls and types with
| `64` suffixes were added, and you could use a preprocessor macro
| to choose which calls and structs were actually referenced--but
| they were hardly ever used directly.
|
| Instead, the OS and its SDK are versioned, and at build time you
| can _also_ specify the _earliest_ OS version your compiled binary
| needes to run on. So using this, the headers ensured the proper
| macros were selected automatically. (This is the same mechanism
| by which new /deprecated-in-some-version annotations would get
| set to enable weak linking for a symbol or to generate warnings
| for it respectively.)
|
| And it was all handled initially via the preprocessor, though now
| the compilers have a much more sophisticated understanding of
| what Apple refers to as "API availability." So it should be
| feasible to use the same mechanisms on any other platform too.
| db48x wrote:
| Lol, that only works if you can force everyone on the platform
| to go along with it. It is a nice solution, but it requires you
| to control the c library. Gentoo doesn't control what libc
| does; that's either GNU libc or MUSL or some other thing that
| the user wants to use.
| dmitrygr wrote:
| > requires you to control the c library
|
| Which is why basically every other same operating system does
| that. BSDs, macOS, WinNT. Having the stable boundary be the
| kernel system call interface is fucking insane. And somehow
| the Linux userspace people keep failing to learn this lesson,
| no matter how many times they get clobbered in the face by
| the consequences of not learning it.
| eschaton wrote:
| And yet it would still work out for Linux if musl, glibc,
| et al just adopted `API_VERSION_MIN` and `API_VERSION_MAX`
| macros themselves, it doesn't actually have to be handled
| entirely at the `-isysroot` level.
| db48x wrote:
| It's extra work, but I don't know that it is necessarily
| insane. If libc was under the complete control of the
| kernel developers, then that gives other languages fewer
| options. Go famously (or infamously) uses certain syscalls
| without going through libc, for example. Sometimes the
| choices made for the C library just aren't compatible with
| other languages. Frankly the C library, as it exists today,
| is insane. Maybe the solution is to split it in half: one
| for the syscalls and another for things like strings and
| environment variables and locales and all the other junk.
| eschaton wrote:
| It's entirely opt-in, Apple doesn't force it. If you just do
| `cc mything.c -o mything` you get a binary whose minimum
| required OS is the version of the SDK you built it against,
| just as with any other UNIX-like OS. It's just giving the
| developer the option to build something they know will run on
| an earlier version too.
|
| And since it was all initially done with the preprocessor
| rather than adding knowledge to the compilers, there's no
| reason individual libraries can't handle API versioning in
| exactly this way, including things like differing `ino_t` and
| `off_t` sizes.
| panzi wrote:
| The only place where this is relevant for me is running old
| Windows games via wine. Wonder how wine is handling this? Might
| as well re-map the date for 32bit wine to the late 90s/early
| 2000s, where my games are from. Heck, with faketime I can do that
| already, but don't need it yet.
| ok123456 wrote:
| What about making time_t 64-bit across all profiles and
| incrementing all the profile versions by one? Gentoo users are
| used to breaking changes when upgrading profiles.
___________________________________________________________________
(page generated 2024-09-28 23:00 UTC)