https://www.netmeister.org/blog/epoch.html
Signs of Triviality
Opinions, mostly my own, on the importance of being and other things.
---------------------------------------------------------------------
[homepage] [blog] [jschauma@netmeister.org] [@jschauma] [RSS]
---------------------------------------------------------------------
Time is an illusion, Unix time doubly so...
October 23rd, 2022
UnixEpoch clock As you well know, on Unix systems we measure time as
the number of seconds since "the epoch": 00:00:00 UTC on January 1st,
1970. This has made a lot of people very angry and been widely
regarded as a bad move.
For starters, this definition is not based on something sensical such
as, say, the objective frequency of vibration of a Cesium-133 atom,
but on a convenient fraction of the time it takes a particular large
rock to complete a full rotation around its own axis.
You see, in Unix time each day is guaranteed to have exactly 86400
seconds, and we pretend that this number monotonically increases.
When it turns out that because our reference rock actually took a bit
longer than what was convenient for us to complete its rotation and
we need to add a leap second, we simply pretend that didn't happen,
and our timestamp mechanism ceases to identify a unique point in
time.
The other thing that continues to cause problems here is that as we
try to count seconds, we run into data storage and representation
issues, because honestly, computers aren't really all that good at
numbers, it turns out. Talk about "epoch fail".
So how did we get here? It all began back in 1971, when the First
Edition Unix Programmer's Manual defined Unix time as "the time since
00:00:00, 1 January 1971, measured in sixtieths of a second":
Excerpt from the Unix Programmer's Manual showingthe definition of
Unix time
That's right, the original Unix epoch was 1971-01-01T00:00:00. What
timezone, you ask? Well, it sure wasn't "UTC", because that didn't
replace GMT as the standard time until 1972. Secondly, note that time
was measured in 1/60ths of a second, not in seconds. Why would that
be?
Remember that at that time, Unix was being developed in the US on a
PDP-11. These systems had a Line-Time Clock (LTC), which takes the AC
power frequency to generate an interrupt for the processor. This
interrupt is then used to update the system clock.
The funny thing here is that this interrupt frequency thus depends on
the utility frequency of its power supply. In much of Asia and
Europe, this frequency is 50 Hertz, but in the US, the Westinghouse
Electric Corporation observed that the carbon arc lamps in use at the
time had less flicker at 60 Hz and standardized on that frequency.
(Fun fact: Westinghouse, an early competitor to Thomas Edison and
General Electric, would later buy CBS and was then bought by Viacom.)
A weird side effect of this US/Europe split is that in Japan, both
are used: in the East, where the first generators were purchased from
the German AEG and installed in Tokyo, the utility frequency is 50
Hz; in the West, they were installed in Osaka from G.E., using 60 Hz.
As a result, the country now uses several High-Voltage, Direct
Current (HVDC) electric power transmission systems to convert the
elecitricity between the two regions!
The Power Grid of Japan
Image by Callum Aitchison on Wikipedia
Aaaaanyway... so yeah, the 1st Edition UNIX measured time at those 60
Hz using a 32-bit integer, thus only being capable of counting 2^32 /
60 ticks/second * 60 s/m * 60 m/h * 24 h/d * 365 d/y = 2.3 years, as
noted in the original manual. Some time in 1971, the time measurement
was redefined to count seconds, now capable of representing 136 years
of time, and the Unix epoch date was -- contrary to the common
misconception that it represents the birthdate of Unix -- somewhat
arbitrarily picked to be 1970:
"At the time we didn't have tapes and we had a couple
of file-systems running and we kept changing the
origin of time," he said. "So finally we said, 'Let's
pick one thing that's not going to overflow for a
while.' 1970 seemed to be as good as any. "
-- Dennis Ritchie, Wired
(The actual birthdate of Unix lands somewhere in 1969, when Ken
Thompson ported "Space Travel" to the PDP-7, so it's easy to see why
the Unix epoch seems like it would represent the birth of Unix.)
Leap Seconds
Ok, so now we have our 32-bit seconds-since-the-epoch counter, which
monotonically increases at a rate of 1 Hz, and promising us to count
exactly 86400 seconds for any 24 hour period. But our silly reference
rock appears to be slowing down in its rotation, making it necessary
to fudge this number a bit every so often.
To do that, the International Earth Rotation Service (IERS) (now "
International Earth Rotation and Reference Systems Service") sends
out an email to all Time Lords indicating whether or not a leap
second will be introduced:
Screenshot of an email from the IERS announcing aLeap Second in 2016
IERS Bulletin C
Now we can't really blame the Unix epoch time for not accounting for
leap seconds initially, since those did not exist up until 1972.
Since then, we've had 27 positive leap seconds (as of 2022), the last
one being introduced at the end of 2016. Whenever a positive leap
second occurs, our Unix epoch time simply pretends it didn't, with
the result being that two dates map to the same epoch timestamp (
epoch-time.c):
From epoch to time, via gmtime(3) to strftime(3):
1483228798 2016-12-31T23:59:58
1483228799 2016-12-31T23:59:59
missing leap second here
1483228800 2017-01-01T00:00:00
From time to epoch, via strptime(3) to mktime(3):
2016-12-31T23:59:58 is 1483228798
2016-12-31T23:59:59 is 1483228799
2016-12-31T23:59:60 is 1483228800
2017-01-01T00:00:00 is 1483228800 duplicate timestamp here
(And let's forget all about negative leap seconds, or the fact that
some Unix system define the range of tm_sec from [0-61], accounting
for a mystical "double leap second" that never actually existed.)
Good thing this is all in line with what POSIX requires:
4.16 Seconds Since the Epoch
A value that approximates the number of seconds that have elapsed since
the Epoch.
The relationship between the actual time of day and the current value
for seconds since the Epoch is unspecified.
How any changes to the value of seconds since the Epoch are made to
align to a desired relationship with the current actual time is
implementation-defined. As represented in seconds since the Epoch, each
and every day shall be accounted for by exactly 86400 seconds.
Year 2038 Problem
Now even counting monotonically forward and ignoring leap seconds,
the time_t data type used to measure seconds since the epoch is
eventually going to overflow. Good thing we have standards to save us
here! Sure, POSIX is quiet on this, but the good thing about
standards is that there are so many to choose from, right? Oh wait
*taps ear*, what's that? I see.
Bad news everybody: standards are not going to save us. For example,
the C standard says:
The range and precision of times representable in
clock_t and time_t are implementation-defined.
Well, ok then. With a 32-bit time_t, we can then account for roughly
136 years starting in 1970. Except, time_t is a signed 32-bit
integer, so we only get 68 years in either direction, leaving us with
what you all know as the "Year 2038 Problem".
That is, the largest date that can be represented as a time_t using a
32-bit signed integer is epoch 2^31 - 1 = 2147483647, or
2038-01-19T03:14:07Z:
Illustration of the Year 2038 Problem showing thedate wrapping.
Image by Monaneko on Wikipedia
Easy to fix, right? We "just" change the data type of a time_t to be
a 64-bit signed integer, yielding a theoretical maximum epoch date of
2^63-1 = 9223372036854775807 after January 1st, 1970. As people like
to point out, that represents a date around 292 billion years in the
future, or roughly 22 times the estimated age of the universe, and
thus officially Somebody Else's Problem.
But -- there's always a "but", isn't there? -- is that really what
happens? Why don't we give it a try and see how different systems
behave when we hand them not-quite-so-reasonable times to chew on.
mktime(3) wrapping
While time is represented as a time_t, the other common format in use
to represent broken down time is a struct tm, from which you can then
get a time_t by calling mktime(3). The distinct elements of the
struct tm are noted by POSIX to be:
int tm_sec Seconds [0,60].
int tm_min Minutes [0,59].
int tm_hour Hour [0,23].
int tm_mday Day of month [1,31].
int tm_mon Month of year [0,11].
int tm_year Years since 1900.
int tm_wday Day of week [0,6] (Sunday =0).
int tm_yday Day of year [0,365].
int tm_isdst Daylight Savings flag.
The funny thing about the ranges noted there are that they are...
advisory at best. POSIX specifically notes:
...the original values of the [...] components shall not
be restricted to the ranges described in .
So what exactly happens when you feed mktime(2) a struct tm with
values outside of these ranges? Consider the following program:
int main() {
struct tm t;
time_t epoch;
/* 2022-12-31 */
t.tm_year = 122; t.tm_mon = 11; t.tm_mday = 31;
/* 22:57 */
t.tm_hour = 22; t.tm_min = 57;
t.tm_sec = 3785;
if ((epoch = mktime(&t)) == -1) {
err(EXIT_FAILURE, "mktime");
}
(void)printf("%s", ctime(&epoch));
return 0;
}
$ cc -Wall -Werror -Wextra t.c
$ ./a.out
Sun Jan 1 00:00:05 2023
$
Here, before we set tm_sec, our struct tm represented December 31st,
22:57:00, 2022. But then we set tm_sec to 3785, i.e., 1 hour, 3
minutes, and 5 seconds. This then leads to tm_min to be incremented
by 3 minutes, which leads to tm_hour to be bumped up by one. Then we
increment tm_hour once more (from the remaining 3600 seconds from
tm_sec), which wraps that value over to 00 and leads to tm_mday
needing to be incremented. tm_mday now wraps to 01 with tm_year
incrementing, ultimately leading to the final date being
2023-01-01T00:00:05.
This normalization of timestamps gets even more confusing once you
add negative values (e.g., tm_mday = -1 means the last day of tm_mon
- 1) or (again) with leap seconds in play and can make your head
hurt. It's best to avoid this situation.
Funny date(1)
Now let's look at epoch dates at or around the epoch. We'll ignore
the fact that epoch times prior to 1972 are not well defined, because
our epoch time is defined to be in UTC, but (see above) UTC had not
yet been standardized before 1972. We'll just be like Unix with leap
seconds and pretend that doesn't concern us.
Displaying an arbitrary date using the date(1) command is easy
enough. You just give it the date in the format CCyymmddHHMM.SS.
Unless you're using Linux, where GNU date(1) wants to use what is
perhaps the most infuriating format (MMDDhhmmCCYY.ss), even using
inconsistent capitalization of the date fields compared to the
strftime(3) % sequences. Why would you stick the year in between
minutes and seconds?
But using either format is not useful for our purposes anyway,
because at some point we want to go beyond four-digit years, so
instead let's specify seconds since the epoch directly ("-r
" for BSD date(1), "--date @" for GNU date(1)).
Displaying dates at or around the epoch is straight forward:
NetBSD / FreeBSD / macOS: Linux: OmniOS (using GNU date)
$ date -r 0 $ date --date @0 $ date -r 0
Thu Jan 1 00:00:00 UTC 1970 Thu Jan 1 12:00:00 AM UTC 1970 January 1, 1970 at 12:00:00 AM UTC
$ date -r -1 $ date --date @-1 $ date -r -1
Wed Dec 31 23:59:59 UTC 1969 Wed Dec 31 11:59:59 PM UTC 1969 December 31, 1969 at 11:59:59 PM UTC
$ date -r 1 $ date --date @1 $ date -r 1
Thu Jan 1 00:00:01 UTC 1970 Thu Jan 1 12:00:01 AM UTC 1970 January 1, 1970 at 12:00:01 AM UTC
(GNU date(1) using AM/PM instead of a 24-hour clock here is annoying,
but ok.)
Let's see what happens if we try to probe for 32-bit time_t s. As
discussed above, the Year 2038 Problem would be triggered at epoch
2147483648 / -2147483649:
netbsd$ date -r 2147483647 linux$ date --date @2147483647
Tue Jan 19 03:14:07 UTC 2038 Tue Jan 19 03:14:07 AM UTC 2038
netbsd$ date -r 2147483648 linux$ date --date @2147483648
Tue Jan 19 03:14:08 UTC 2038 Tue Jan 19 03:14:08 AM UTC 2038
netbsd$ date -r -2147483648 linux$ date --date @-2147483648
Fri Dec 13 20:45:52 UTC 1901 Fri Dec 13 08:45:52 PM UTC 1901
netbsd$ date -r -2147483649 linux$ date --date @-2147483649
Fri Dec 13 20:45:51 UTC 1901 Fri Dec 13 08:45:51 PM UTC 1901
omnios$ date -r -2147483648
December 13, 1901 at 08:45:52 PM UTC
omnios$ date -r -2147483649
date: failed to parse -r argument: -2147483649
omnios$ date -r 2147483647
January 19, 2038 at 03:14:07 AM UTC
omnios$ date -r 2147483648
date: failed to parse -r argument: 2147483648
Oh, hey, look at that. OmniOS seems to have problems with epoch dates
beyond 2^31. I wonder what's going to happen if we not only display
the date, but set it? Let's loop, set the date, and print the date:
omnios$ for s in 5 6 7 8; do sudo date -u 011903142038.0$s; date; done
January 19, 2038 at 03:14:05 AM UTC
January 19, 2038 at 03:14:05 AM UTC
January 19, 2038 at 03:14:06 AM UTC
January 19, 2038 at 03:14:06 AM UTC
January 19, 2038 at 03:14:07 AM UTC
December 13, 1901 at 08:45:52 PM UTC
ld.so.1: date: fatal: /lib/libc.so.1: Value too large for defined data type
Killed
omnios$ date
ld.so.1: date: fatal: /lib/libc.so.1: Value too large for defined data type
Killed
omnios$ ls
ld.so.1: ls: fatal: /lib/libc.so.1: Value too large for defined data type
Killed
omnios$ sudo reboot
sudo: unknown uid 100
sudo: error initializing audit plugin sudoers_audit
omnios$
Nice. Note how the date actually wraps around before the OS freaks
out. Now the other systems should not have any problems setting a
date. Right?
netbsd# date 197001010000; date +%s linux$ sudo date -s @0
Thu Jan 1 00:00:00 UTC 1970 date: cannot set date: Invalid argument
0 Thu Jan 1 12:00:00 AM UTC 1970
netbsd# date 196912312359 linux$ uptime; sudo date -s @7080
date: settimeofday: Invalid argument 03:34:27 up 1:57, 1 user, load average: 0.04
netbsd$ Thu Jan 1 01:58:00 AM UTC 1970
That is, on NetBSD we can't set the date to before the epoch, while
on Linux (since kernel version 4.3) we can't set the date before the
current uptime. This is, settimeofday(2) will return EINVAL, because
of guarantees made by its "CLOCK_MONOTONIC" clock that, according to
POSIX "represents the amount of time since an unspecified point in
the past":
All CLOCK_MONOTONIC variants guarantee that the time returned by
consecutive calls will not go backwards, but successive calls
may--depending on the architecture--return identical (not-increased) time
values.
Along those lines, it looks like OmniOS calculates uptime based on
boot time relative to the system date, while NetBSD and Linux, for
example, keep separate counters:
omnios$ sudo date -u 010100001970.00
January 1, 1970 at 12:00:00 AM GMT
omnios$ uptime
00:01:46 up -49 min(s), 1 user, load average: 0.00, 0.00, 0.02
omnios$ sudo date -u 011803142038.00
January 18, 2038 at 03:14:00 AM GMT
omnios$ uptime
03:14:11 up 5567 day(s), 7 min(s), 1 user, load average: 0.24, 0.16, 0.07
Trying to set the date to before the epoch yields different results
as well. On NetBSD and Linux, it'll simply fail, but on OmniOS, it
will set the month, day, hour, minute, and second, but pin the year
to 1970:
omnios$ sudo date -u 020100001969.00
February 1, 1970 at 12:00:00 AM GMT
omnios$ date
February 1, 1970 at 12:00:01 AM UTC
omnios$ sudo date -u 121308451901.52
December 13, 1970 at 08:45:52 AM GMT
omnios$ date
December 13, 1970 at 08:45:54 AM UTC
But so what is the largest date we can set our clock to on systems
that use a 64-bit time_t? As noted above, we'd expect to be able to
set it to epoch 2^63 - 1 = 9223372036854775807. Let's first display
the date, then try to set it:
netbsd$ date -r 9223372036854775807
date: 9223372036854775807: localtime: Value too large to be stored in data type
netbsd$ date -r 67768036191676799
Wed Dec 31 23:59:59 UTC 2147485547
netbsd$ date -r 67768036191676800
date: 67768036191676800: localtime: Value too large to be stored in data type
linux$ date --date @9223372036854775807 freebsd$ date -r 9223372036854775807
date: time '9223372036854775807' is out of range date: invalid time
linux$ date --date @67768036191676799 freebsd$ date -r 67768036191676799
Wed Dec 31 11:59:59 PM UTC 2147485547 date: invalid time
linux$ date --date @67768036191676800 freebsd$ date -r 67767976233532799
date: time '67768036191676800' is out of range Tue Dec 31 23:59:59 UTC 2147483647
linux$ freebsd$ date -r 67767976233532800
date: invalid time
So this is interesting. Even though we were promised a 64-bit time_t,
we can't set the time to 9223372036854775807. We seem to max out at
67768036191676799 in the year 2,147,485,547. While that seems
initially arbitrary, you may notice that 2147485547 is 2^31 -1, and
suddenly this makes sense: even though the time_t is 64-bit, the
struct tm's tm_year is still 32-bit, and so the maximum value that
can be represented is thus the last second of the year 2,147,485,547.
But what's the deal with FreeBSD? Somehow there the epoch of
67768036191676799 is invalid, but epoch 67767976233532799 represents
what on the other platforms is 67768036191676799? If you do the math,
you'll notice that the difference between these two epoch times is
1900 years -- so apparently FreeBSD bases it's tm_year not on the
year 1900 (as the struct tm in claims), but on the year 0?
Since I couldn't figure this out, I filed a bug report. -\_(tsu)_/-
When we try to set the date, we first observe that using NetBSD date
(1) it's impossible to set a date beyond the year 9999 (another bug
report), but, fun fact, that doesn't really matter because it turns
out we can't go higher than the year 4147 anyway:
netbsd# date 414708200732.17
date: settimeofday: Invalid argument
netbsd# date 414708200732.16; date +%s
Sun Aug 20 07:32:16 UTC 4147
68719476736
The reason for that is that NetBSD has a hardcoded limit of 2^36 =
68719476736 on the tv_sec value it will accept when setting the time,
because larger value made KUBSAN unhappy (code):
/*
* The time being set to an unreasonable value
* will cause unreasonable system behaviour.
*/
if (ts->tv_sec < 0 || ts->tv_sec > (1LL << 36))
return EINVAL;
On Linux, we have a different practical maximum date, set in 2232:
linux$ sudo date -s @8277292036
date: cannot set date: Invalid argument
Wed Apr 18 11:47:16 PM UTC 2232
linux$ sudo date -s '@8277292035'
Wed Apr 18 11:47:15 PM UTC 2232
linux$ sleep 10; date +%s
8277292045
linux$ sudo date -s @$(date +%s)
date: cannot set date: Invalid argument
Wed Apr 18 11:47:25 PM UTC 2232
The reason for this limitation is found in the source as to be such
that it can accommodate a 30-year uptime before the counter wraps:
#define NSEC_PER_SEC 1000000000L
#define KTIME_MAX ((s64)~((u64)1 << 63))
#define KTIME_SEC_MAX (KTIME_MAX / NSEC_PER_SEC)
/*
* Limits for settimeofday():
*
* To prevent setting the time close to the wraparound point time setting
* is limited so a reasonable uptime can be accomodated. Uptime of 30 years
* should be really sufficient, which means the cutoff is 2232. At that
* point the cutoff is just a small part of the larger * problem.
*/
#define TIME_UPTIME_SEC_MAX (30LL * 365 * 24 *3600)
#define TIME_SETTOD_SEC_MAX (KTIME_SEC_MAX - TIME_UPTIME_SEC_MAX)
Interestingly, this means that on Linux, the signed 64-bit max time
value (2^63 - 1 = 9223372036854775807) does not represent the seconds
since the epoch, but counts the nanoseconds since the epoch, bringing
Linux's theoretical max date (KTIME_SEC_MAX) to a mere 9223372036
epoch seconds or the date 2262-04-23T11:47:16 -- a fair bit away from
the "22 times the estimated age of the universe" and a bit more in
the realm of possibly becoming an actual problem.
Lastly, when trying to see what value FreeBSD lets us set the clock
to, I found that the value differed between the two latest releases,
but either had the unpleasant problem of leading to a spontaneous
reboot of the system:
freebsd# date -u -f "%s" 49282253052249598
Fri Dec 31 23:59:58 UTC 1561694399
49282253052249598
freebsd# sleep 1; date; date +%s
Fri Dec 31 23:59:59 UTC 1561694399
49282253052249599
freebsd# sleep 1; date; date +%s
Sat Jan 1 00:00:00 UTC 1561694400
49282253052249600
[ system reboots ]
Note: I'm able to initially set the date to values larger than
49282253052249598, but about three seconds later, the system reboots.
If I set the date to one second earlier, i.e., 49282253052249597,
then the system will not reboot, even as the system time progresses
beyond the next value. Aren't computers great?
Oh, and one more thing...
We all love how macOS is a UNIX -- one of only 6 operating systems
currently registered (the others being AIX, EulerOS (a commercial
Linux distribution made by Huawei), HP-UX, Xinuos (previously
UnixWare, via the old AT&T Unix System Laboratories + Novell, SCO,
Caldera, and UnXis), and z/OS). But Apple's Core Foundation framework
does not use the Unix epoch as it's basis for time. Instead, it uses
2001-01-01T00:00:00 GMT as it's reference date:
All CLOCK_MONOTONIC variants guarantee that the time returned by Core
Foundation measures time in units of seconds. The base data type is the
CFTimeInterval, which measures the difference in seconds between two
times. Fixed times, or dates, are defined by the CFAbsoluteTime data
type, which measures the time interval between a particular date and the
absolute reference date of Jan 1 2001 00:00:00 GMT.
So if you want to convert those timestamps into Unix epoch, you need
to add 978307200.
---------------------------------------------------
As I said, time is an illusion, and Unix time doubly so. And while
the Year 2038 Problem may not concern you (unless you happen to use
OmniOS, I suppose), perhaps I was able to show you that there are
plenty of other surprises lurking. And we haven't even touched upon
the insanity that is timezones and Daylight Saving Time...
October 23rd, 2022
-------------------------------------------------------
Links:
* This blog post as a Twitter thread
* Discussion on HackerNews
---------------------------------------------------------------------
Previous: [The Sender Policy Framework (SPF)]
---------------------------------------------------------------------
[homepage] [blog] [jschauma@netmeister.org] [@jschauma] [RSS]
---------------------------------------------------------------------