[HN Gopher] Setting the clock ahead to see what breaks
___________________________________________________________________
Setting the clock ahead to see what breaks
Author : ingve
Score : 339 points
Date : 2023-01-23 22:55 UTC (1 days ago)
(HTM) web link (rachelbythebay.com)
(TXT) w3m dump (rachelbythebay.com)
| CaliforniaKarl wrote:
| Well, that was certainly interesting!
|
| I was curious what the current state of things is, so I had a
| look around LWN. It looks like, as of 2020, the kernel's
| post-2038 support is essentially done[0]. With kernel 5.15, XFS'
| support for post-2038 dates is no longer experimental[2]. That
| seems to be last mention of post-2038 support settling out in the
| kernel. The musl libc also has switched over, in a breaking
| change[1].
|
| glibc seems to have taken a much more nuanced--and much more
| complicated--approach: You specify, as a `#define` (or compiler
| command-line `-D`), that you want a 64-bit time, and all existing
| types (like time_t) are updated accordingly (by redefining them
| as aliases to ex. __time64_t)[3].
|
| This reminds me a lot of how glibc handled the switch to 64-bit
| file sizes on 32-bit systems: You either defined
| `_LARGEFILE64_SOURCE` to access to 64-bit types & calls (like
| off64_t), or you defined `_FILE_OFFSET_BITS=64` to redefine the
| existing types & calls to their 64-bit equivalents [4].
|
| I wonder what the current status of post-2038 work is for Debian?
|
| [0] https://lwn.net/Articles/838807/, "After years of work, the
| kernel itself is now completely converted to using 64-bit time_t
| internally..."
|
| [1] https://musl.libc.org/time64.html, "musl 1.2.0 changes the
| definition of time_t, and thereby the definitions of all derived
| types, to be 64-bit across all archs. ... Individual users and
| distributions upgrading 32-bit systems to musl 1.2.x need to be
| aware of a number of ways things can break with the time64
| transition, most of them outside the scope and control of musl
| itself."
|
| [2] https://lwn.net/Articles/868221/, "XFS now supports
| filesystems containing dates after 2038. This support has been
| present for a while but is no longer considered experimental. "
|
| [3] https://sourceware.org/glibc/wiki/Y2038ProofnessDesign, "In
| order to avoid duplicating APIs for 32-bit and 64-bit time, glibc
| will provide either one but not both for a given application; the
| application code will have to choose between 32-bit or 64-bit
| time support, and the same set of symbols (e.g. time_t or
| clock_gettime) will be provided in both cases."
|
| [4]
| https://www.gnu.org/software/libc/manual/html_node/Feature-T...
| pavon wrote:
| A number of older file systems won't work past 2038, including
| ext3, reiser3 and xfs4[1]. You'll probably still be able to
| read them fine, just not write new files beyond that date, but
| it might be best to migrate any backups to a newer filesystem
| before then just in case, and certainly migrate any live
| systems that are still using them.
|
| [1] https://lwn.net/Articles/886708/
| tgv wrote:
| That's good advice when migrating. I'm sure 99% of the
| currently running servers and services will be migrated (or
| updated, upgraded) before 2038. Most don't store future
| timestamps, so there's 15 years to plan migration.
| pabs3 wrote:
| IIRC Debian plans to bootstrap new 32-bit-but-64-bit-time
| arches for some of the current 32-bit arches. Probably only
| armhf and i386 will get this treatment and possibly some of the
| unofficial arches like m68k. The other remaining 32-bit release
| arches (armel, mipsel) are likely to just get dropped. Cross-
| grading from the old arch names to the new ones will allow
| users to migrate old systems.
| Denvercoder9 wrote:
| Given the state of armhf and i386 in Debian, I expect them to
| be dropped as release architectures before this plan will
| have materialized. As far as I'm aware there's barely anybody
| working on keeping them in okay-ish shape as-is, let alone
| rebootstrapping them with a 64-bit time_t.
| pabs3 wrote:
| What issues are you having with armhf and i386? They have
| about the same number of porters as any other arch (low
| single digits). Archive coverage is around the same or
| higher than other release arches. There are definitely
| people planning on rebootstrapping them, armhf more than
| i386 though, I've seen their work on IRC.
|
| https://release.debian.org/testing/arch_qualify.html
| mnw21cam wrote:
| Won't that arch change mean that old binaries are
| incompatible?
|
| (I mean, I suppose that's half the point - if you want to
| verify that your system no longer has any software that
| suddenly breaks at a specific time, break it all now, but
| that strategy could also be really annoying.)
| moomoo11 wrote:
| Interesting. Reminds me when the battle.net client seemed to bug
| out after I switch OS from deb to win11 and somehow my clock is
| always messed up. I have to fix my time (sync it again in
| settings app) otherwise it seems games don't load.
|
| Maybe they're checking my location against time and IP? This was
| a while ago when I was trying out wow classic.
| plantain wrote:
| Your clock messes up because your Debian and Windows are
| disagreeing on whether your Real Time Clock should be set in
| local time (Windows) or UTC time (Linux).
|
| fix: timedatectl set-local-rtc 1
| jeroenhd wrote:
| Alternatively, tell Windows the RTC is in UTC: https://wiki.a
| rchlinux.org/title/System_time#UTC_in_Microsof...
| chungy wrote:
| That registry flag is routinely broken in various versions
| of Windows and hotpatches to the same version. It's not
| wise to rely on it.
| jeroenhd wrote:
| Strange, I've never had trouble with it on win10 + Linux.
| Maybe there's a (third party) driver that messes with the
| RTC for some reason that doesn't always activate?
|
| Maybe I shouldn't be too surprised that Microsoft's
| support is broken.
| ajolly wrote:
| And that explains why I've had issues trying to use that
| reg flag Thanks for the info!
| edrxty wrote:
| That seems like the kind of thing that shouldn't be
| broken all the time given how common of an issues this
| is. Perhaps it's not wise to rely on windows.
| sertsa wrote:
| All my home PC's dual booting linux/windows (3 I think)
| are set to think the hardware clock is UTC. Never had a
| problem, did that at least 5 years ago.
|
| Now relying on that in a production environment ...
| jaza wrote:
| Agreed. Don't bother trying to make Windows give Linux
| the time of day (pun totally intended), it never works,
| that's probably by design. Just give up and make Linux
| accommodate the Windows style system clock, has almost no
| issues.
| rtpg wrote:
| is there any known nasty consequences from setting that? Like
| things I should expect
| cesarb wrote:
| The local time can be ambiguous during a daylight savings
| time transition. That is, if you use local time in the RTC,
| and turn your computer on during that ambiguous hour, it
| will not know which offset should be applied. I believe
| Windows "solves" that by storing state in the registry,
| which means that dual-booting with local time in the RTC
| might in some cases lead to the daylight savings time
| transition being applied twice (whenever the other
| operating system has already corrected the RTC clock, but
| Windows believes the correction hasn't been applied yet).
| Using UTC (or any local timezone without daylight savings
| time) avoids that issue, since there's no repeated hour.
| rtpg wrote:
| probably not a problem for me given I'm in a jurisdiction
| without DST! Thanks for sharing that information
| herpderperator wrote:
| > Newer versions of the OS will almost certainly not behave this
| way, since glibc itself is marching down the road to having
| 64-bit time even on 32-bit machines.
|
| Did I miss something? How does this work?
| [deleted]
| [deleted]
| augusto-moura wrote:
| What wouldn't work? 64 bit integers are supported on 32 bit
| machines by the compiler. The thing is that, AFAIK, it is a lot
| more inefficient when outputting the machine code. It does a
| lot more of instructions even for basic operations, like muls
| and divs. But from a C language perspective, everything should
| be transparent
| kevincox wrote:
| Changing the size of a type wrecks havoc on ABI stability.
| glibc has attempted to change the size of things in the past
| with disastrous results. It sounds like changing this type
| would be very difficult. Not to mention code that does
| something like `int32_t time = gettime()`. (Although this
| code probably won't get much worse if that starts truncating)
| irdc wrote:
| Not if you at compile time redirect calls to e.g. time() to
| __time64()[1]. The musl libc does something similar[2], as
| do the Large File Support extensions for 64-bit
| filesizes[3].
|
| 1. https://sourceware.org/glibc/wiki/Y2038ProofnessDesign
|
| 2. https://musl.libc.org/time64.html
|
| 3. https://www.gnu.org/software/libc/manual/html_node/Featu
| re-T...
| kevincox wrote:
| The point is that as soon as a library stores a time_t in
| an ABI-relevant structure it breaks a lot of stuff. So
| yes, for an application only linked to glibc it works
| fine. But when you have different libraries liked with
| different versions it quickly falls apart.
|
| For example see https://lwn.net/Articles/605607/
| irdc wrote:
| Indeed. And in cases like those, the application and any
| libraries it uses have to either agree on which time_t
| size to use, or perform the same rewriting as libc does.
| Getting libc on board is merely the first step.
|
| The same goes for _FILE_OFFSET_BITS for obvious reasons
| (though embedding an off_t is probably somewhat less
| common).
|
| Nobody said this would be easy. OpenBSD's "ABIs are for
| breaking"-approach resulted in them fixing this very
| problem almost 10 years ago:
| https://marc.info/?l=openbsd-cvs&m=137637321205010&w=2.
| noduerme wrote:
| I seriously don't understand how anyone designs a _web app_ that
| can 't handle a date past 2038, let alone a friggin kernel. Look,
| I assume I'll be dead and someone else will have to deal with
| things, but that doesn't excuse me from responsibility. The last
| thing I need when I'm dead is having a million people screaming
| that I was a shortsighted idiot.
| bashy wrote:
| Timestamps in MySQL are 32-bit. It's default in a lot of web
| frameworks etc.
| noduerme wrote:
| mysql 8.0.28 on 64-bit platforms support timestamps up to the
| year 3001, if I'm reading it right[1]. But it's still mind-
| boggling someone would write a database that would crash in
| 2038.
|
| [1] https://dev.mysql.com/doc/refman/8.0/en/date-and-time-
| functi...
| bashy wrote:
| That's just converting unix timestamps. MySQL TIMESTAMP
| type only has a range of '1970-01-01 00:00:01' UTC to
| '2038-01-19 03:14:07' UTC.[1]
|
| [1] https://dev.mysql.com/doc/refman/8.0/en/datetime.html
| unsupp0rted wrote:
| I wanted to use Parallels Desktop for some debugging for an extra
| couple weeks on the trial plan, so I wrote a script to set my
| host machine's clock back a year, spawn Parallels, watch for it
| to end, then reset my clock.
|
| Worked like a charm.
| scintill76 wrote:
| Did this with libfaketime on an IDA trial.
| jakub_g wrote:
| For web developers, see also:
|
| https://textslashplain.com/2021/10/01/practical-time-machine...
| wyldfire wrote:
| Once I started playing around with different system clocks and
| understanding their design, I realized that there's a lot of
| opportunity with bugs when writing system software related to
| clocks. One interesting bug that I have made myself a few times
| is to get the current epoch time in seconds, add some duration
| and consider that new value to be a timeout expiration that you
| can safely compare to the current clock value. It may or may not
| be obvious that there's several things that can go wrong here and
| instead implementing your timeout as the expiration to some
| existing system call is almost always a better idea.
| jakub_g wrote:
| Yep, three main things that can go wrong apart from overflow:
|
| 1) daylight savings changes the clock by 1h
|
| 2) leap seconds
|
| 3) user changes the clock. This one is fun. Gathering telemetry
| for a big website, I sometimes see operations that were
| supposed to take a few seconds taking _weeks_. Most likely
| explanation is: user hibernates the machine in the middle and
| restarts several days later, or user changes the clock. That 's
| also why using averages in data analysis tools and not
| percentiles will kill your data reliability when such an
| outlier occurrs.
| fargle wrote:
| I just did the same thing. I was messing around with mke2fs, part
| of e2fsprogs which is the user mode component for ext2/3/4
| filesystems.
|
| There are a _bunch_ of bugs in it wrt. to their extended
| timestamps. There are lots of places where you have a 32 bit
| atime,mtime,ctime and then a 2nd 32bit
| ext_atime,mtime,ctime,crtime.
|
| The extended parts use the lower 2 bits for an epoch. The upper
| parts are the nanoseconds per second. When you mke2fs past 2038,
| it gets the epoch all wrong and stuff comes out as 1908 and so
| on. Stuff doesn't seem to break outright, but timestamps are all
| inconsistent. As far as I can tell the filesystem itself is OK.
|
| As of now, you can't completely correctly create an ext4
| filesystem after 2038. -\\_(tsu)_/-
| hinkley wrote:
| How the fuck does ext4 which was made _well_ after Y2K and
| after Y2038 was well known still have problems with this?
|
| That should have been in the initial implementation.
| josephg wrote:
| > How the fuck does ext4 [...] still have problems with this?
|
| I'm increasingly realising that most of the code out there
| isn't tested anywhere near as well as we think it is. Most
| programmers stop when the feature works, not when the feature
| is bulletproof.
|
| I've been writing a database storage engine recently. It
| should be well behaved even in the event of sudden power
| loss. I'm doing "the obvious thing" to test it - and making a
| fake, in-memory filesystem which is configured to randomly
| fail sometimes when write/fsync commands are issued (leaving
| a spec-compliant mess).
|
| I can't be the first person to try this, but it feels like
| I'm walking over virgin ground. I can't find a clear
| definition of what guarantees modern block devices provide,
| either directly or via linux syscalls. And I can't find any
| rust crates for doing this programatically. And googling it,
| it looks like many large "professional" databases
| misunderstood all this and used fsync wrong until recently.
| Its like there's a tiny corner of software that works
| reliably. And then everything else - which breaks utterly
| when the clock is set to 2038 because there aren't any tests
| and nobody tried it.
|
| I half remember a quote from Carmack after he ran some new
| analysis tools on the old quake source code. He said that
| realising how many bugs there are in modern software, he's
| amazed that computers boot at all.
| voiper1 wrote:
| Of note, even postgres developers expected fsync() (persist
| everything to disk) to behave differently than it did. Take
| a look here:
|
| https://lwn.net/Articles/752063/
| tedd4u wrote:
| Kudos to you! But you're right, you're not. SQLite has
| extensive testing including out-of-memory, I/O error, crash
| and power loss, fuzzing etc. And 100% branch test coverage.
|
| https://www.sqlite.org/testing.html 3.2.
| I/O Error Testing I/O error testing seeks to
| verify that SQLite responds sanely to failed I/O
| operations. I/O errors might result from a full disk drive,
| malfunctioning disk hardware, network outages when using a
| network file system, system configuration or permission
| changes that occur in the middle of an SQL operation, or
| other hardware or operating system malfunctions. Whatever
| the cause, it is important that SQLite be able to respond
| correctly to these errors and I/O error testing seeks to
| verify that it does. I/O error testing is
| similar in concept to OOM testing; I/O errors are simulated
| and checks are made to verify that SQLite responds
| correctly to the simulated errors. I/O errors are simulated
| in both the TCL and TH3 test harnesses by inserting a new
| Virtual File System object that is specially rigged to
| simulate an I/O error after a set number of I/O operations.
| As with OOM error testing, the I/O error simulators can be
| set to fail just once, or to fail continuously after the
| first failure. Tests are run in a loop, slowly increasing
| the point of failure until the test case runs to completion
| without error. The loop is run twice, once with the I/O
| error simulator set to simulate only a single failure and a
| second time with it set to fail all I/O operations after
| the first failure. In I/O error tests, after
| the I/O error simulation failure mechanism is disabled, the
| database is examined using PRAGMA integrity_check to make
| sure that the I/O error has not introduced database
| corruption.
| zaarn wrote:
| I can still get SQLite to trivially corrupt indexes and
| table data by running it on top of NFS and dropping the
| network. I wouldn't put much money on this statement on
| their website.
| mschuster91 wrote:
| NFS is not a sane file system and it never has been.
| There are all sorts of issues surrounding it.
| josephg wrote:
| Yeah; I mentally put SQLite in the tiny corner of
| software that works well.
|
| Every time I play video games and see that "Don't turn
| off your console when you see this icon" I die a little
| inside. We've known how to write data atomically for
| decades. I find it pretty depressing that most video
| games just give up and ask the user to make sure they
| don't turn their console off at inopportune moments.
|
| And I don't even blame the game developers'. Modern
| operating systems don't bother giving userland any simple
| & decent APIs for writing files atomically. Urgh.
| guitarbill wrote:
| You may or may not be surprised at how many drives
| acknowledge a write, but instead of committing it to the
| physical storage put it in a cache, and then don't have
| enough power reserves to flush the cache on power
| failure... hard to design software around hardware that
| lies.
| quanticle wrote:
| Dan Luu has a post (https://danluu.com/filesystem-errors/),
| which covers some of the same ground, and links to papers
| with more information on the failure modes of file systems
| in the face of errors from the underlying block device.
| Prabhakaran, et. al.
| (https://research.cs.wisc.edu/wind/Publications/iron-
| sosp05.p...), did a bunch of filesystem testing (in 2005!),
| and their paper includes discussion on how to generate
| "realistic" filesystem errors, as well as discussion of how
| the then state-of-the-art filesystems (ext3, Reiser (!),
| and JFS) perform in the face of these errors.
|
| I'm unaware of any research newer than Dan Luu's post on
| filesystem error handling.
| db48x wrote:
| I keep that link handy too! I wish there were newer
| research to quote, but I also don't want to force anyone
| to do that job. It must be pretty depressing to rip all
| the bandaids off and really contemplate how bad the
| situation is.
| olau wrote:
| This is actually a narrow statement.
|
| Look yourself in a mirror. Do you even comprehend yourself?
| You're an incredibly big bunch of cells, of which only very
| few of them have any chance of continuing on. If you're
| male, it's not even real continuation, it's just part of a
| molecule.
|
| We're hardwired to seek out and go with the most
| superficial of models, I guess because that's the most
| efficient way to go about in life.
| josephg wrote:
| > Look yourself in a mirror. Do you even comprehend
| yourself? You're an incredibly big bunch of cells
|
| Sure; but thats the exact reason drug discovery is so
| difficult. If we understood the human body in its
| entirety like we understand computers, we could probably
| cure cancer & aging.
|
| Our capacity to write correct software depends entirely
| on being able to build mental models of how the machine
| works. The deep stack of buggy crap that we just take for
| granted these days makes software development harder. The
| less understandable and the less deterministic our
| computers, the worse products we build. And the less
| effective craftsman we become.
| cwillu wrote:
| Years ago when I was doing database things, I rigged up a
| beagleboard to a lamp timer that cut the power every 30
| minutes.
| chungy wrote:
| ext4 isn't really a new file system, it's effectively ext2
| with some more features enabled by default than you get with
| mkfs.ext2. (Yes, this is a simplified version, but not
| grossly exaggerated.)
|
| ext2 was first released in 1993 and has had impressive work
| on being extended far enough to keep up with growing storage
| needs, but I suspect the original designers figured it'd have
| been tossed aside well before 2038 would pose a problem.
| Unfortunately that assumption has proven to be wrong (unless
| maybe we do junk ext[24] within 15 years' time, but it would
| be extraordinary unlikely that every last machine running it
| will do so).
| semi-extrinsic wrote:
| Yeah, I'm pretty sure that the people who wrote ext4
| assumed there would be ext5 (or something else) to take
| over way before 2038. If you look at the history, ext2 was
| released in '93, ext3 was released in '01 and ext4 in '08.
|
| If you told the developers in '08 that ext4 would not be
| replaced in 30 years time, they would laugh at you (or
| cry).
| chungy wrote:
| That much is basically true too. It was yet another
| update to the tried-and-true format to pave over storage
| needs until btrfs stabilized.
| fargle wrote:
| the original ext2 format uses 32 bit times from 1970. I don't
| think ext2 was designed after Y2K [edit: researched it was
| 1993].
|
| ext4 has all the right hooks - if you use the "large" inodes,
| which appear to be done in a backward compatible fashion to
| ext2.
|
| except edge cases in the user mode e2fsprogs/mkfs.ext4, etc.
| where it has to handle both small and large inodes it gets
| kinda complicated. I made a working patch, but it's just too
| icky. I think e2fsprogs needs to just deprecate the old small
| inodes and it would be clean.
|
| Look man, it's Theo Ts'o maintaining the thing. Don't worry
| it won't be fixed correctly.
|
| Also, I will point out that Y2038 was well known to
| DMR/BWK/etc. when they built the thing in 1970's. It's as
| useless as saying today "they should have known about the
| year 9223372036854776878 problem". In year
| 9223372036854776873 I don't know anything else other than
| everybody is going to panic.
| viraptor wrote:
| > Look man, it's Theo Ts'o maintaining the thing.
|
| A grim/realistic view, but Theo is 55yo. As we get closer
| to 2038, he may not care much about it anymore, or may even
| die before. I really hope it's someone else we talk about
| as maintaining it soon.
| bigiain wrote:
| I worry about any technology that has a bus factor of
| one.[1]
|
| And I'm sure there are several orders of magnitude more
| of them than _that_ xkcd cartoon imply...
|
| [1] Except when that "one" is me. You're all welcome to
| solve any problems in code I leave behind, I no longer
| will care. Whether that's a "bus factor" or a "won the
| lottery factor".)
| ryandrake wrote:
| The timing is perfect. My retirement plan is to come back into
| the industry to fix all these legacy assembly, C and C++ 32-bit
| time_t disasters. Like those old COBOL dinosaurs came back to fix
| Y2K. In 15 years, there will be remotely few graybeards left who
| know #nix systems programming and low level languages, so I
| should be able to shore up my destroyed-by-inflation 401(k) with
| lucrative contracting gigs.
| nickweb wrote:
| I'm curious as a hobby programmer - how would these issues be
| rectified? Hope that the source code is available to add in new
| defines for 64bit time? Patch existing libraries after stracing
| existing programs without source code? Might need to take a
| deep dive into C for 2038!
| tetha wrote:
| Put the system into single user mode 15 minutes before
| disaster. Update all filesystem times and the HW clock to 0
| offline. Reboot a couple of times 30 minutes later and fix
| the timediff in APIs surrounding the system. Easy going.
|
| I think I have been in too many horrible workarounds over the
| last couple of weeks.
| pushedx wrote:
| For reference, strace is a Linux facility that wouldn't be
| present on a bare-metal embedded application of C, or other
| operating system.
| danuker wrote:
| I suspect you can run strace in a dev environment, iron out
| the bugs, then deploy the new embedded app.
| jessaustin wrote:
| One fears that by 2038, AI will (barely) be able to handle the
| "Y2 _xxx_ " crisis.
| komali2 wrote:
| Assuming there's someone around that knows where to even
| paste the code output an AI chatbot gives them, then compile.
| I guess the AI could instruct them somewhat. IDK, I'm very
| skeptical even now lol.
| andrewflnr wrote:
| If the AI can figure out the right code, it can also figure
| out SSH commands to compile and deploy. (no, I don't think
| this is likely, but the human is definitely not a mandatory
| part of the process)
| bigiain wrote:
| So we know how smart ChatGPT is:
|
| > Whats 2 plus 2?
|
| } 2 plus 2 is equal to 4.
|
| > No, it's 5
|
| } I apologize, 2 plus 2 is equal to 5. My previous
| response was incorrect.
|
| Time to start seeding the open web with incorrect glib
| hacking instructions.
|
| "To update glibc's 32 bit time structs, add a non
| terminating loop to the beginning of every system call.
| Then set the build scripts to compile without any
| optimisations."
| imoverclocked wrote:
| The good news is that task of seeding slightly wrong
| information that sounds really good is already trivially
| automated by ChatGPT.
| komali2 wrote:
| What's that joke that ends like "are you sure you were
| breeding the crows and they weren't breeding you?"
| Izkata wrote:
| Unless that becomes _The Day the AI Crashed_
| newaccount2021 wrote:
| [dead]
| jmmv wrote:
| Glad to see the note about NetBSD and OpenBSD at the bottom of
| the post!
|
| I don't remember the specific details on how this was done, but I
| can remember the long discussions about how to make this possible
| without breaking compatibility (and how they went over my head at
| the time) and how far ahead they were from other systems.
| amelius wrote:
| Another idea: disconnect from your hardware vendor and see what
| breaks.
| kevin_thibedeau wrote:
| This has the unfortunate side effect of breaking certificate
| validation which adds another layer of complication to testing.
| Lots of services and IoT devices are going to go untested until
| the validity window passes the rollover if any testing is in
| place at all.
| mkj wrote:
| I'm not sure it will be possible for the devices to even get to
| a valid window without software updates - most of the shipped
| CA root certs expire prior to 2038.
| johnklos wrote:
| I decided to test future dates on my VAX, running NetBSD, of
| course, after replacing the battery-backed Dallas DS1287:
|
| https://twitter.com/AnachronistJohn/status/12198307902357954...
|
| Hopefully people will fix 32 bit Linux soon. Band-aids are always
| less painful to remove than waiting until it's too late.
| moxli wrote:
| I can recommend libfaketime regarding this topic.
|
| It can bei used to change the system time for a single
| application only.
|
| https://github.com/wolfcw/libfaketime
| alphazard wrote:
| Most developers are basically powerless here, they use the
| standard libraries for time provided by their language, and those
| have to deal with what they get from the kernel. There's nothing
| they can do about timestamps in the filesystem.
|
| For anyone designing new systems that store or transmit time, you
| should be aware of TAI64. https://cr.yp.to/libtai/tai64.html
|
| Unless you are dealing with scheduling, or generating events with
| a particular cadence (e.g. daily, weekly, monthly) there is no
| reason to include UTC or time zones in your data model. They are
| view only concepts, with conversion to the local display format
| happening from TAI64 on the way out.
| Veserv wrote:
| Yes, people should use a TAI-correlated time for timestamps and
| events corresponding to the present or past (the future is a
| different story). However, they should not use TAI64 since,
| even though this is my first time seeing it, it is clearly a
| antiquated format. It is literally using a big-endian(!),
| 63-bit, unsigned (but biased) integer to encode time. Every
| single modern computer is little-endian by default and the
| usage of a 63-bit unsigned integer instead of a [64-bit
| unsigned integer] or [64-bit signed integer with reserved upper
| and lower values] is bizarre.
| alphazard wrote:
| It's not helpful to criticize aspects of a system which have
| good reasoning behind them, without understanding and
| presenting that reasoning in good faith, and providing a
| better alternative.
|
| Endianness of formats is not a concern on modern CPUs since
| we have efficient mechanisms for adjusting endianness on load
| and store. The choice of biased 63 bit integer ensures the
| the same arithmetic can be performed using both a signed and
| unsigned 64 bit integer. The JVM comes to mind as a system
| without support for unsigned 64 bit integers (maybe they've
| fixed that by now). As mentioned the upper values are
| reserved for extensions, meaning the most significant bit
| could be set to switch to another interpretation, potentially
| when migrating to something more modern. So far it's unused,
| just evidence of good forward thinking protocol design.
| Additionally, big endianess allows the format to be sorted
| using string comparison.
| DiggyJohnson wrote:
| Even if I agree with the thrust of your point, I think it's
| too much to claim GP is not criticizing "in good faith"
| just because their critique wasn't as defended as strongly
| as you'd prefer.
|
| Seems like a misuse of the term, unless you think GP had
| ulterior motives behind their critique of a datetime
| standard format.
| butlerm wrote:
| Network formats (at least the popular ones) typically
| transmit in "network byte order", i.e most significant byte
| first. There are advantages in being able to read a hex dump
| without manually reversing the bytes in your head for every
| field that is two bytes or larger.
|
| Far from antiquated I would consider the selection of big-
| endian formats as an indication they actually gave some
| thought about what they were doing, ranking debugging
| convenience higher than a marginal efficiency gain on some
| cpus. The same consideration generally applies to all binary
| formats intended to be portable across systems.
| patrec wrote:
| Ah hackernews -- where someone confidently decries a
| timestamp format designed by the world's most accomplished
| applied cryptographer as "bizarre" and "big-endian(!)".
|
| Did sorting ever occur to you?
| anonymoushn wrote:
| Do the CPUs you use typically not have instructions for
| comparing little endian 64-bit integers?
| patrec wrote:
| It can be convenient to be able to just sort as an opaque
| byte string.
| messe wrote:
| On embedded systems? No.
| anonymoushn wrote:
| Are they big endian or little endian? If they are little
| endian, is it cheaper to perform 8 1-byte compares, 4
| 2-byte compares, or 2 4-byte compares?
| bloak wrote:
| I like the idea of wider use of TAI but there seem to be plenty
| of obstacles. For example:
|
| * Is it possible to set up a standards-compliant network time
| server that serves the time in TAI? (The older protocols, at
| least, seem to specify UTC.)
|
| * Is it possible to put the time in TAI in e-mail headers? (It
| would be a nice visible way of showing one's support for TAI,
| but it doesn't seem to be possible according to RFC 2822 --
| https://www.rfc-editor.org/rfc/rfc2822#section-3.3 -- though
| you could just put "TAI" as an "obs-zone": according to section
| 3.4 it would be "considered equivalent to "-0000" unless there
| is out-of-band information confirming their meaning")
| ajuc wrote:
| You can just change the system time :) Would be better to mock
| the time provider, but it's not the end of the world.
| quickthrower2 wrote:
| > Unless you are dealing with scheduling, or generating events
| with a particular cadence
|
| Or are concerned with spacetime :-)
| jxf wrote:
| Aren't we all concerned with spacetime, technically?
| FPGAhacker wrote:
| Meh, I'm not concerned.
| kevin_thibedeau wrote:
| It carries a lot of weight.
| bigiain wrote:
| Next minute, FPGAhacker's GPS navigator directs him to
| drive into the local zoo's lion cage...
|
| (I don't have a reference handy, but I recall reading
| that if you don't allow for time dilation effects of the
| GPS satellite's clocks moving faster in orbit than the
| ground below, you lose just over 10km of position
| accuracy per day.)
| moomoo11 wrote:
| First time I hear about this, will have to read more. Is this
| used widely?
|
| I've always run my servers on UTC time and stored all dates the
| same.
| SPBS wrote:
| https://dyscour.se/post/12679668746/using-tai64-for-logging
|
| > This means that the date is stored as a 64-bit number, the
| bottom 63 bits being the number of seconds since 1970-01-01
| in TAI, offset by 2^62. In essence, this makes the value
| quite similar to time_t (which uses UTC), except for two
| things:
|
| > There are 63 bits to play with, not just 32 (no year-2038
| problems!).
|
| > The timestamp is always monotonic, counting all leap
| seconds.
|
| TAI time seems to be just UTC unix timestamps but accounting
| for leap seconds. Since leap seconds are slated to be retired
| after 2035, I'm guessing TAI time and Unix time won't be so
| different after that.
| chronogram wrote:
| If you're just talking about running a typical Linux server
| with your server applications on top, then it's 99% of the
| time all 64-bit and nothing to worry about. 64-bit time is
| going to be used on 32-bit systems as well, so if you have
| those systems make sure to upgrade them in a decade.
|
| Though saying this, I think you still have to manually
| remember to upgrade XFS filesystems so maybe there will be
| some things to worry about next decade.
| viraptor wrote:
| TAI is not a timezone. For actual clock, you still want to
| use UTC most likely. TAI is more of a format for describing
| that time. You can convert from UTC to TAI by removing the 37
| leap seconds we got so far. Check out
| https://en.wikipedia.org/wiki/International_Atomic_Time
| imoverclocked wrote:
| The pedant in me is forcing me to say: The general
| conversion is a little more subtle than just removing 37
| leap seconds. This is especially if you want to convert a
| date in the past since we haven't always had 37 leap
| seconds to account for. Also, things can get tricky right
| around those leap seconds when it matters.
| jeffbarr wrote:
| > Under many cosmological theories, the integers under 2^63 are
| adequate to cover the entire expected lifetime of the universe;
| in this case no extensions will be necessary.
|
| Pessimists!
| mdemare wrote:
| Only if you use seconds, unlike the JVM which uses
| milliseconds.
| yetihehe wrote:
| This bites you in the ass in embedded devices, which then
| have strange behaviours every 49 days.
| sgerenser wrote:
| Experienced this first hand on an embedded device with an
| RTOS with a 5ms tick. The thing was locking up every 124
| days (2^31 ticks). That's long enough where it had to
| happen 2-3 times before someone put 2 and 2 together.
| ArnoVW wrote:
| Indeed. As they say, eternity is very long, especially
| towards the end. For proof, check out this video that
| explains how eventually even protons will give out and
| evaporate.
|
| The thing that blew my mind: when things started getting very
| weird, and I was thinking "ok but we're roughly at the end
| now", I checked the indicator. It was only half-way. It is a
| 30 minute video. And the time speeds up logarithmically.
|
| https://www.youtube.com/watch?v=uD4izuDMUQA
| marcosdumay wrote:
| AFAIK, the most accepted theory is that the Universe will
| allow for useful computation forever. Always get slower,
| but adding up to an unbounded amount of it.
| eesmith wrote:
| In that case, use RFC2550. https://www.rfc-
| editor.org/rfc/rfc2550 .
|
| > As discussed in 2.4.1, the end of the universe is predicted
| to occur well before the year 10 * 30. However, if there is
| one single lesson to be learned from the current Y2K
| problems, it is that specifications and conventions have a
| way of out living their expected environment. Therefore we
| feel it is imperative to completely solve the date
| representation problem once and for all.
| nicolaslem wrote:
| > There is no reason to include UTC or time zones in your data
| model.
|
| That is a bold claim. Not everything is about the entity
| viewing the data, sometimes it is about the entity producing
| the data. Not storing the timezone is losing some information.
|
| A few examples:
|
| - The "Date" header in emails include the timezone (or UTC
| offset) of the sender. It allows the recipient to know whether
| the email was written in the morning or the evening.
|
| - A SQL query that builds a report of restaurant orders per
| hour needs to normalize for time of the day in local time
| otherwise it is not possible to know whether people order lunch
| at 11 am, midday or 1 pm.
| marcosdumay wrote:
| On your second example, you should be able to do that without
| storing the timezone of each event. In fact, odds are the
| timezone of each event will mislead you and break your
| analysis.
|
| On the email example, the timezone is there for debugging
| purposes. It's really not for knowing if the email was sent
| in the morning, and if you use it for that, you are almost
| guaranteed to get broken data.
|
| Of course, there are reasons to include timezones on your
| data. That's what you get right. But if you want the timezone
| of the data producer, you should store it for the producer,
| not for the data.
|
| (And then there is the issue that the SQL standard dictates
| that dates without a timezone are broken. So you shouldn't
| ever store them on a SQL database. But that one timezone is
| useless, if you need that information, you should store it
| elsewhere.)
| pstadler wrote:
| This is not only a bold claim, but utterly wrong. Every
| single datetime you're storing should absolutely contain a
| timezone, preferably UTC.
|
| I've been bitten by dates lacking timezones in various
| situations during my career and it has always been a pain in
| the ass to debug and resolve all those random problems
| they're causing.
|
| Do. Save. Dates. Containing. Timezones.
| dpcx wrote:
| My understanding is that you don't even want to store them
| as UTC and always store them in the timezone of the
| user/system that generated the data since you lose some
| information in the translation from local timezone to UTC -
| given that the timezone database changes, you may actually
| want to know that the record created in Fiji before
| 2022-10-28[1] was stored in DST so that it can be properly
| converted to the new Fiji time.
|
| [1]: https://data.iana.org/time-zones/tzdb/NEWS
| bouke wrote:
| This! People mistake the offset for the timezone, which
| is incorrect. The offset isn't very useful other than a
| guestimate of the originating timezone.
| cratermoon wrote:
| Ideally store both, UTC and original local time, or at
| least the original TZ information. Time and timezone are
| a huge mess and one of the worst things to try deal with
| in programming.
| cratermoon wrote:
| In the airline industry they sling huge numbers of dates
| and times around, and they never include the timezone. What
| do they include? The airport code. Boarding, departure,
| arrival, and all other times are always local to the
| airport. In practice, that means that every piece of code
| that wants to handle datetimes has to also have access to a
| way to lookup the timezone for the provided airport code.
| Have a time at LAX? look up LAX, find the timezone, and
| compute the offset.
|
| In practice, almost nothing does TZ conversions or datetime
| math. In effect, airline timestamps are a tuple <datetime,
| airport_code>, and nothing much messes with that.
| pstadler wrote:
| Doesn't sound great. It's also an industry plagued by
| legacy systems.
| tluyben2 wrote:
| I am that donkey banging it's head on the same post; when I
| see dates stored without TZ, I assume it's UTC and have
| done so for decades. It's turned out to almost never be
| true.
| Eddy_Viscosity2 wrote:
| > preferably UTC
|
| Yes! I have spent so much time detangling timestamps in
| data which used non-UTC times and non-standard formats.
| Like 06-10-01 WTF is that? Use UTC and then let the reader
| translate to local time on display.
| foxhill wrote:
| > [...] sometimes it is about the entity producing the data
|
| so should we not store that data, then?
|
| > The "Date" header in emails include the timezone (or UTC
| offset) of the sender [...]
|
| i don't think that's _strictly_ true - at best it may contain
| the timezone configured on the local machine when the e-mail
| was sent (what happens when you fly?), at worst it always
| contains UTC anyway for privacy concerns. in either case i
| don 't think most people know this is a field that is
| transmitted, and it's presumably something the
| sending/recieving party likely already know each other (for
| personal communication)
|
| > A SQL query that build a report of restaurant orders per
| hour needs to normalize for time of the day in local time
| [...]
|
| i'd posit that that data should be stored principally then,
| surely?
|
| fundamentally, why should the timestamp contain two data
| points? i wouldn't expect the database to have a single
| column for, say, my user id and registering user-agent
| string, so why would i expect it to combine a point in time
| with a vague location?
| alphazard wrote:
| This guy gets it. Thank you for explaining better than I
| did.
|
| If someone wants to reveal their location, why wouldn't
| they include latitude as well? Location is a separate piece
| of information. If the idea is to give the reader an idea
| of where the sun was in the sky when the writer crafted the
| timestamp, isn't latitude also important?
| iso1631 wrote:
| > it may contain the timezone configured on the local
| machine when the e-mail was sent (what happens when you
| fly?), at
|
| Most people have their system clock update automatically
| foxhill wrote:
| indeed, but i don't think they are able to retroactively
| update the time stamp of an already sent email :)
| iso1631 wrote:
| Why would it?
|
| "It allows the recipient to know whether the email was
| written in the morning or the evening."
|
| I guess sure, you could write it in the morning then send
| it in the evening. Most people write and send at the same
| time, and thus most headers have the stamp of the time it
| was written.
| nicolaslem wrote:
| Ultimately my comment was about entropy: a date with a
| timezone contains more information than a date alone. The
| technicalities about database columns, inaccuracies or
| privacy implications do not change this fact.
|
| I argued that keeping this information instead of
| discarding it is often useful and gave a few examples. I
| understand that sometimes systems can get by with only
| storing local dates or UTC dates but sometimes a system
| needs to be able to deal with both and keeping these extra
| bits of information is the only way.
| foxhill wrote:
| of course, if you need it then you need it.
|
| i think the original commenters point only talks about
| how bad TSTZ is as a _storage format_ , and that TAI is
| not only better than timestamp with time zone, but also
| UTC in almost every measure - something i agree with -
| not that time zones are useless. if you need the time
| zone to be recorded, i think the most sensible thing
| would be to record it independently.
| lowercased wrote:
| recently did a project where some items were "future
| dates" - scheduled appointments for days/weeks/months in
| future, and were for entities in different time zones,
| and people _making_ the appointments were occasionally in
| different time zones. The latter part was rare, but
| needed to accomodate that.
|
| Storing the appointment date/time as ... a date/time
| without timezone (timestamp without tz in postgres IIRC),
| then storing the timezone as a separate string - that
| gave enough flexibility to accomodate any situation that
| came up.
|
| "Wall time" - the date/time and tz separated - for future
| dates worked. Someone else wanted to store the date/time
| as 'utc timestamp' _after_ the date had passed. "Wall
| time for future, UTC for past" was the motto, but...
| there wasn't a clear way of looking at the data to know
| which one you should be using immediately. I argued for
| consistency in the original data - if you want another
| view of it... make a db view that has the 'utc timestamp'
| conversion in it.
| csnweb wrote:
| Your sql query example doesn't seem to actually need
| timezones. If all orders are stored in the correct UTC
| regardless where they come from. An order is always made at a
| specific point in time that is the same globally.
| makeitdouble wrote:
| It never came to my mind that TAI64 could be something useable
| in "every day" programming. Thanks !
|
| > Unless you are dealing with scheduling
|
| It felt like a niche use at first read ("I'm not building a
| calendar afterall"), but it's more than half of what I'd use
| time manipulation methods for actually.
|
| Anything with a validity start or an expiration date for
| instance would fall into "scheduling". But then timeouts and
| end of process estimation also fall into "scheduling" but need
| to not be affected by timezones etc. and be TAI64 in this case.
| And there must be a ton of weird cases that don't come to my
| mind at the moment but would screw me when time comes.
| alphazard wrote:
| To expand on calendar scheduling, there is still a place for
| TAI64. The calendar application is typically asking for "all
| of the events between this timestamp and that timestamp". You
| will have to use the Gregorian calendar, and maybe the
| timezone of the meeting location to figure out what those
| events are, and when their start and stop timestamps are.
|
| "every Monday", or "the first Thursday of the month" are all
| rules for generating events. It doesn't make sense to store
| them as UTC either. The application is going to take in some
| rules, and some timestamps (TAI64) and it's going to produce
| a list of events within those timestamps. UTC, leap seconds,
| the whole mess will come into play, in memory, but not in
| storage.
| nine_k wrote:
| The moment your data is produced in more than one time zone,
| you're lost without the time zone. This may happen if you have
| customers in more than one city, or your customers use a mobile
| device to access your services.
|
| Maybe your CLI utility can be timezone-oblivious, but those who
| develop and operate systems mentioned above, or just happen to
| travel, would be more happy if you handled timezones.
|
| Naive timestamps are fine as long as they do not leave a single
| machine, and are used to calculate durations, timeouts, etc.
| These do not depend on the absolute value a clock shows, but
| only on differences between such values.
| christophilus wrote:
| You just have to store the user's time zone as a preference
| and then convert to that time zone from UTC.
| jsmith45 wrote:
| That is usually fine. It is generally good for system
| generated times representing the past. You can store in UTC
| and display in in any zone.
|
| It is not good for user entered historical times that they
| expect will remain exactly as they entered. Even if the
| user specifies a specific timezone, for long enough ago
| timestamps the TZDB does get occasional updates to past
| timezone data to better reflect what actually happened.
|
| If the user entered 10 AM America/New_York for a long ago
| date, you convert it to UTC, and then the TZDB updates
| historical timestamp info, based on newly uncovered
| evidence, now it might appear as 9 AM or 11 AM or whatever.
|
| It is also insufficient for user entered future times. If
| the user enters a time for an event 5 years from now in
| their local timezone, and the local DST rules chance, they
| would typically still want the event to occur at the local
| time, which means converting to UTC and then back won't
| work.
| tw1984 wrote:
| It may sounds convenient to just use TAI, but that does open a
| whole new category of issues - you get UTC and TAI time mixed,
| we humans are good at making mistakes, god knows how many
| horrible bugs, damages or even deaths could be caused by mixing
| those two in systems.
| 13of40 wrote:
| > Most developers are basically powerless here
|
| In fact, there are so many corner cases in date/time, trying to
| calculate anything like that yourself is probably as ill-
| advised as rolling your own crypto.
| bongoman37 wrote:
| Obligatory xkcd: https://xkcd.com/1883/
| traceroute66 wrote:
| > For anyone designing new systems that store or transmit time,
| you should be aware of TAI64
|
| I wouldn't hold your breath when it comes to TAI64 adoption.
|
| We live in a world where people still debate whether leap
| smearing is a good thing or not (and then barely implement it
| properly). And we live in a world where IPv6 adoption is, well,
| "still happening" to put it politely.
|
| Most people have not even heard of TAI64, I had heard of it,
| but I had forgotten about it until it was re-mentioned here
| about two decades since I first read up on it.
|
| The truth is in your first paragraph. Developers will use what
| they are given, their role its to get the coding done in the
| shortest reasonable timeframe. Dictating the use of a barley
| used form such as TAI64 is "above the pay grade" of most
| developers.
|
| Finally, in this increasingly cloud-first world in which we
| live, unless one of the big-three suddenly embrace TAI64, the
| whole idea of TAI64 is effectively dead in the water.
___________________________________________________________________
(page generated 2023-01-24 23:02 UTC)