[HN Gopher] Clock synchronization is a nightmare
___________________________________________________________________
Clock synchronization is a nightmare
Author : grep_it
Score : 87 points
Date : 2025-12-23 18:59 UTC (4 days ago)
(HTM) web link (arpitbhayani.me)
(TXT) w3m dump (arpitbhayani.me)
| koudelka wrote:
| the Huygens algorithm is also worth a look
|
| https://www.usenix.org/system/files/conference/nsdi18/nsdi18...
| jeffbee wrote:
| PTP requires support not only on your network, but also on your
| peripheral bus and inside your CPU. It can't achieve better-than-
| NTP results without disabling PCI power saving features and deep
| CPU sleep states.
| pclmulqdq wrote:
| You can if you just run PTP (almost) entirely on your NIC. The
| best PTP implementations take their packet timestamps at the
| MAC on the NIC and keep time based on that. Nothing about CPU
| processing is time-critical in that case.
| jeffbee wrote:
| Well, if the goal is for software running on the host CPU to
| know the time accurately, then it does matter. The control
| loop for host PTP benefits from regularity. Anyway NICs that
| support PTP hardware timestamping may also use PCI LTR
| (latency tolerance reporting) to instruct the host operating
| system to disable high-exit-latency sleep features, and
| popular operating systems respect that.
| rcxdude wrote:
| How so? If the NIC is processing the timestamps as it
| arrives/leaves on the wire, the latency and jitter in the rest
| of the system shouldn't matter.
| emptybits wrote:
| Normally I would nod at the title. Having lived it.
|
| But I just watched/listened to a Richard Feynmann talk on the
| nature of time and clocks and the futility of "synchronizing"
| clocks. So I'm chuckling a bit. In the general sense, I mean. Yes
| yes, for practical purposes in the same reference frame on earth,
| it's difficult but there's hope. Now, in general ...
| synchronizing two clocks is ... meaningless?
|
| https://www.youtube.com/watch?v=zUHtlXA1f-w
| hinkley wrote:
| Einstein was worried about whether people in two different
| relativistic frames would see cause and effect reversed.
| emptybits wrote:
| Wild. My layperson mind goes to a simple example, which may
| or may not be possible, but please tell me if this is the
| gist:
|
| Alice and Bob, in different reference frames, both witness
| events C and D occurring. Alice says C happened before D. Bob
| says D happened before C. They're both correct. (And good
| luck synchronizing your watches, Alice and Bob!)
| mrkeen wrote:
| That will be the case when Alice stands close to where C
| happens, and Bob stands close to where D happens.
|
| It's a little trickier to imagine introducing cause-and-
| effect though. (Alice sees that C caused D to happen, Bob
| sees that D caused C to happen).
|
| I think a "light cone" is the thought-experiment to look up
| here.
| hinkley wrote:
| If Bob and Alice are moving at half the speed of light in
| opposite directions.
| ianburrell wrote:
| There is distinction between seeing when events happened,
| and when they really happened. The latter can be
| reconstructed by an observer.
|
| In special relativity, time is relative and when things
| actually happened can be different in different frames.
| Casually linked events are always really in the same
| order. But disconnected events can be seen in different
| orders depending on speed of observer.
| emptybits wrote:
| > But disconnected events can be seen in different orders
| depending on speed of observer.
|
| What are "disconnected events"? In a subtle but still
| real sense, are not all events causally linked? e.g.
| gravitationally, magnetically, subatomically or
| quantumly?
|
| I can understand that our simple minds and computational
| abilities lead us to consider events "far away" from each
| other as "disconnected" for practical reasons. But are
| they really not causally connected in a subtle way?
|
| There are pieces of space time that are clearly,
| obviously causally connected to each other. And there are
| far away regions of the universe that are, practically
| speaking, causally disconnected from things "around
| here". But wouldn't these causally disjoint regions
| overlap with each other, stringing together a chain of
| causality from anywhere to anywhere?
|
| Or is there a complete vacuum of insulation between some
| truly disconnected events that don't overlap with any
| other observational light cone or frame of reference at
| all?
| hinkley wrote:
| We now know that gravity moves at the speed of light.
| Imagie that you aretwo supernovas that for some unknown
| reason, explode at essentially the same time. Just before
| you die from radiation exposure, you will see the light
| pulse from each supernova before each supernova can 'see'
| the gravitational disruption caused by the other. Maybe a
| gravity wave can push a chain reaction on the verge of
| happening into either a) happening or b) being delayed
| for a brief time, but the second explosion happened
| before the pulse from the first could have arrived. So
| you're pretty sure they aren't causally linked.
|
| However if they were both triggered by a binary black
| hole merger, then they're dependent events but not on
| each other.
|
| But I think the general discussion is more of a 'Han shot
| first' sort. One intelligent system reacting to an action
| of another intelligent system, and not being able to
| discern as a person from a different reference frame as
| to who started it and who reacted. So I suppose when we
| have relativistic duels we will have to preserve the role
| of the 'second' as a witness to the events. Or we will
| have to just shrug and find something else to worry
| about.
| ianburrell wrote:
| Causality moves at the speed of light. Events that are
| farther apart are called spacelike and aren't causally
| connected.
|
| I think you might be confusing events that have some
| history between them, and those are influence each other.
| Like say right now, Martian rover sends message to Earth
| and Earth sends message to them, those aren't causally
| connected cause don't know about the other message until
| light speed delay has passed.
| hinkley wrote:
| Yes that definitely happens. People orbiting Polaris would
| be seeing two supernovas explode at different times than us
| due to the speed of light. Polaris is 400 light years away
| so the gap could be large.
|
| But when you are moving you may see very closely spaced
| events in different order, because you're moving toward
| Carol but at an angle to Doug. Versus someone else moving
| toward Doug at an angle to Carol.
| m463 wrote:
| it might be meaningless, but in practical terms just don't
| check util.c from the gravity well into the git repo in orbit.
| yapyap wrote:
| Love learning new things. This also explains why my casio clock
| sync starts skewing over time
| maximinus_thrax wrote:
| I wouldn't say it's a 'nightmare'. It's just more complicated
| than what regular folk think computers work when it comes to time
| sync. There's nothing nightmareish or scary about this, it's just
| using the best solution for your scenario, understanding
| limitations and adjusting expectations/requirements accordingly,
| perhaps relaxing consistency requirements.
|
| I worked on the NTP infra for a very large organization some time
| ago and the starriest thing I found was just how bad some of the
| clocks were on 'commodity hardware' but this just added a new
| parameter for triaging hardware for manufacturer replacement.
|
| This is an ok article but it's just so very superficial. It goes
| too wide for such a deep subject matter.
| blibble wrote:
| PTP isn't even that much more difficult, as long as you planned
| for it form the start
|
| you buy the hardware, plug it all in, and it works
| geerlingguy wrote:
| Sometimes hardware that has PTP support in the specs doesn't
| perform very well though, so if you do things at scale, being
| able to validate things like switches and network card
| drivers is useful too!
|
| It's to the point timing server vendors I've spoken to have
| their own test labs where they have to validate network gear
| and then publish lists of recommended and tested
| configurations.
|
| Even some older cards where you'd think the PTP issues would
| be solved still have weird driver quirks in Linux!
| hinkley wrote:
| I took to distributed systems like a duck to water. It was only
| much later that I figured out that while there are things I can
| figure out in one minute that took other people five, there
| were a lot of others that you will have to walk them through
| step by step or they would _never_ get there. That really
| explained some interactions I'd had when I was younger.
|
| In particular I don't think the intuitions necessary to do
| distributed computing well would come to someone who snoozed
| through physics, who never took intro to computer engineering.
| libraryofbabel wrote:
| > I don't think the intuitions necessary to do distributed
| computing well would come to someone who snoozed through
| physics
|
| Yeah. I was a physics major and it really helped to have had
| my naive assumptions about time and clocks completely
| demolished early on by taking classes in special and general
| relatively. When I eventually found my way into tech a lot of
| distributed systems concepts that are difficult to other
| people (clock sync, indeterminate ordering of events,
| consensus) came quite naturally because of all that early
| training.
|
| I think it's no accident that distributed systems theory guru
| Leslie Lamport had written an unpublished book on General
| Relativity before he wrote the famous _Time, Clocks and the
| Ordering of Events in a Distributed System_ paper and the
| Paxos paper. In the former in particular the analogy to
| special relatively is quite plain to see.
| mgaunard wrote:
| Another protocol that's not mentioned is PPS and its variants,
| such as WhiteRabbit.
|
| A regular pulse is emitted from a specialized high-precision
| device, possibly over a specialized high-precision network.
|
| Enables picosecond accuracy (or at least sub-nano).
| kobieps wrote:
| Even just a single accurate clock is a nightmare...
| https://www.npr.org/2025/12/21/nx-s1-5651317/colorado-us-off...
| georgelyon wrote:
| Unfortunate that the author doesn't bring up FoundationDB version
| stamps, which to me feel like the right solution to the problem.
| Essentially, you can write a value you can't read until after the
| transaction is committed and the synchronization infrastructure
| guarantees that value ends up being monotonically increasing per
| transaction. They use similar "write only" operations for atomic
| operations like increment.
| lll-o-lll wrote:
| Yes. A consistent total ordering is what you need (want) in
| distributed computing. Ultimately, causality is what is
| important, but consistent ordering of concurrent operations
| makes things _much_ easier to work with.
| hinkley wrote:
| Vector clocks are one of the other things Barbara Liskov is known
| for.
| hinkley wrote:
| > Google faced the clock synchronization problem at an
| unprecedented scale with Spanner, its globally distributed
| database. They needed strong consistency guarantees across data
| centers spanning continents, which requires knowing the order of
| transactions.
|
| > Here's a video of me explaining this.
|
| Do you need a video? Do we need a 42 minute video to explain
| this?
|
| I generally agree with Feynman on this stuff. We let explanations
| be far more complex than they need to be for most things, and it
| makes the hunt for accidental complexity harder because
| everything looks almost as complex as the problems that need more
| study to divine what is actually going on there.
|
| For Spanner to be useful they needed a high transaction rate and
| in a distributed system that requires very tight grace periods
| for First Writer Wins. Tighter than you can achieve with NTP or
| system clocks. That's it. That's why they invented a new clock.
|
| Google puts it this way:
|
| Under external consistency, the system behaves as if all
| transactions run sequentially, even though Spanner actually runs
| them across multiple servers (and possibly in multiple
| datacenters) for higher performance and availability.
|
| But that's a bit thick for people who don't spend weeks or years
| thinking about distributed systems.
| j_seigh wrote:
| Ok,so people use NTP to "synchronize" their clocks and then write
| applications that assume the clocks are in exact sync and can use
| timestamps for synchronization, even though NTP can see the
| clocks aren't always in sync. Do I have that right?
| kccqzy wrote:
| If you are an engineer at Google dealing with Spanner, then you
| _can_ in fact assume clocks are well synchronized and can use
| timestamps for synchronization. If you get commit timestamps
| from Spanner you can compare them to determine exactly which
| commit happened first. That's a stronger guarantee than the
| typical Serializable database like postgresql:
| https://www.postgresql.org/docs/current/transaction-iso.html...
|
| That's the radical developer simplicity promised by TrueTime
| mentioned in the article.
| awesome_dude wrote:
| Isn't that because Google has its own atomic clocks, rather
| than NTP which is (generally) using publicly available atomic
| clocks?
| vlovich123 wrote:
| More that they use GPS to synchronize the clocks. Having
| your own atomic clock doesn't really improve your accuracy
| except for within the single data center you have it
| deployed (although I'm sure there's techniques for
| synchronizing with low bounds against nearby atomic clocks
| + GPS to get really tight bound so they don't need one in
| every data center)
| idorosen wrote:
| Alternatively, you could guarantee the same synchronization
| using PPS and PTP to each host's DCD pin of their serial port
| or to specialized hardware such as modern PTP-enabled smart
| NICs/FPGAs that can accept PPS input. GPS+PPS gets you to
| within 20-80ns global synchronization depending on
| implementation (assuming you're all mostly in the same
| inertial frame), and allows you to make much stronger
| guarantees than TrueTime (due to higher precision distributed
| ordering guarantees, which translate to lower latency and
| higher throughput distributed writes).
| AlotOfReading wrote:
| Truetime is based on GPS and local atomic clocks. Google's
| latest timemasters are even better, around 10ns average.
| vlovich123 wrote:
| > and allows you to make much stronger guarantees than
| TrueTime (due to higher precision distributed ordering
| guarantees, which translate to lower latency and higher
| throughput distributed writes).
|
| TrueTime is the software algorithm for managing the
| timestamps. It's agnostic to the accuracy of the underlying
| time source. If it was inaccurate then you get looser
| bounds and as you note higher latency. Google already does
| everything you suggest for TrueTime while also having
| atomic clocks in places.
| vlovich123 wrote:
| That's actually not at all what TrueTime guarantees and
| assuming they've solved a physical impossibility is dangerous
| technically as a founding assumption for higher level tech
| (which thankfully Spanner does not do).
|
| What TrueTime says is that clocks are synchronized within
| some delta just like NTP, but that delta is significantly
| smaller thanks to GPS time sync. That enables applications to
| have tighter bounds on waiting to see if a conflict may exist
| before committing which is why Spanner is fast. CockroachDB
| works similarly but given the logistical challenge of getting
| GPS receivers into data centers, they worked to achieve a
| smaller delta through better NTP-like timestamps and
| generally get fairly close performance.
|
| https://programmingappliedai.substack.com/p/what-is-true-
| tim...
|
| > Bounded Uncertainty: TrueTime provides a time interval,
| [earliest, latest], rather than a single timestamp. This
| interval represents the possible range of the current time
| with bounded uncertainty. The uncertainty is caused by clock
| drift, synchronization delays, and other factors in
| distributed systems.
| kccqzy wrote:
| That's exactly what I'm saying but you simply provided more
| details. TrueTime guarantees clocks are well synchronized:
| and of course that means synchronized to a reasonable upper
| bound. It's no more possible for clocks to be absolutely
| synchronized, than for two line segments drawn
| independently to have absolutely the same length.
| vlovich123 wrote:
| > you can compare them to determine exactly which commit
| happened first
|
| This is the part I was referring to. You cannot just
| compare timestamps and know which happened first. You
| have to actually handle the case where you don't know if
| there's a happens before relationship between the
| timestamps. Thats a very important distinction
| TrainedMonkey wrote:
| Depending on the application you would generally use PTP to get
| sub-microsecond accuracy. The real trick is that architecture
| should tolerate various clocks starting or jumping out of sync
| and self correct.
| forrestthewoods wrote:
| Timesync isn't a nightmare at all. But it is a deep rabbit hole.
|
| The best approach, imho, is to abandon the concept of a global
| time. All timestamps are wrt a specific clock. That clock will
| skew at a rate that varies with time. You can, hopefully, rely on
| any particular clock being monotonous!
|
| My mental model is that you form a connected graph of clocks and
| this allows you to convert arbitrary timestamps from any clock to
| any clock. This is a lossy conversion that has jitter and can
| change with time. The fewer stops the better.
|
| I kinda don't like PTP. Too complicated and requires specialized
| hardware.
|
| This article only touches on one class of timesync. An entirely
| separate class is timesync within a device. Your phone is a
| highly distributed compute system with many chips each of which
| has their own independent clock source. It's a pain in the ass.
|
| You also have local timesync across devices such as wearables or
| robotics. Connecting to a PTP system with GPS and atomic clocks
| is not ideal (or necessary).
|
| TicSync is cool and useful. https://sci-
| hub.se/10.1109/icra.2011.5980112
| a_t48 wrote:
| Clock sync is such a nightmare in robotics. Most OSes happily
| will skew/jump to get the time correct. Time jumps (especially
| backwards) will crash most robotics stacks. You might decide to
| ensure that you have synced time before starting the stack.
| Great, now your timestamps are mostly accurate, except what
| happens when you've used GPS as your time source, and you start
| indoors? Robot hangs forever.
|
| Hot take: I've seen this and enough other badly configured time
| sync settings that I want to ban system time from robotics
| systems - time from startup only! If you want to know what the
| real world time was for a piece of data after, write what your
| epoch is once you have a time sync, and add epoch+start time.
| awesome_dude wrote:
| THIS is what will save us from the robot uprising!
| michaelt wrote:
| If your requirements are "must have accurate time, must start
| with an inaccurate time, must not step time during operation,
| no atomic clocks, must not require a network connection, or a
| WWVB signal, must work without a GPS signal" then yes, you need
| to relax your requirements.
|
| But it doesn't have to be the first requirement you relax.
| Asmod4n wrote:
| Does wall clock time matter for anything but logging? For
| everything else one could just create any form ,,time" to keep
| stuff in sync, no?
| tbrownaw wrote:
| Isn't it also useful for checking validity periods for stuff
| like TLS certs or JWTs or Kerberos tickets?
| Asmod4n wrote:
| We could use some made up ,,time" for that since it's not
| made for human consumption just sync between different
| systems.
| NelsonMinar wrote:
| On the flipside, clock sync for civilians has never been easier.
| Thanks to NTP any device with an Internet connection can pretty
| easily get time accurate to 1 second, often as little as 10 ms.
| All major consumer computers are preconfigured to sync time to
| one of several reliable NTP pools.
|
| This post is about more complicated synchronization for more
| demanding applications. And it's very good. I'm just marveling at
| how in my lifetime I from "no clock is ever set right" to
| assuming most anything was within a second of true time.
| jasonwatkinspdx wrote:
| At this point the only clock in my life that doesn't auto set
| is the one on my stove, and that's because I abhor internet
| connected kitchen appliances.
| shomp wrote:
| Absolute synchronization impossible?? Challenge accepted.
| josephg wrote:
| > When two transactions happen at nearly the same time on
| different nodes, the database must determine which happened
| first. If clocks are out of sync, the database might order them
| incorrectly, violating consistency guarantees.
|
| This is only true if you use wall clock time as part of your
| database's consistency algorithm. Generally I think this is a
| huge mistake. It's almost always much easier to swap to a logical
| clock - which doesn't care about wall time. And then you don't
| have to worry about ntp.
|
| The basic idea is this: event A happened before event B iff A (or
| something that happened after A) was observed by the node that
| generated B before B was generated. This means you end up with a
| dag of events - kind of like git. Some events aren't ordered
| relative to one another. (We say, they happened concurrently). If
| you ever need a global order for all events, you can
| deterministically pick an arbitrary order for concurrent events
| by comparing node ids or something. And this will give you the
| same order across all peers.
|
| If you make database events work like this, time is a little more
| complex. (It's a graph traversal rather than simple numbers). But
| as a result the system clock doesn't matter. No need to worry
| about atomic clocks, skew, drift, monotonicity, and all of that
| junk. It massively simplifies your system design.
___________________________________________________________________
(page generated 2025-12-27 23:00 UTC)