[HN Gopher] Evolving clock sync for distributed databases
___________________________________________________________________
Evolving clock sync for distributed databases
Author : def-
Score : 64 points
Date : 2022-02-02 16:24 UTC (6 hours ago)
(HTM) web link (blog.yugabyte.com)
(TXT) w3m dump (blog.yugabyte.com)
| rdtsc wrote:
| > YugabyteDB is the first--and only--100% open-source, hybrid,
| multi-cloud, distributed SQL database on the planet.
|
| It might be worth pointing out that HCL are patented by Cloudera
| in 2014 (with a provisional patent application filed in 2013)
| https://www.freepatentsonline.com/y2015/0156262.html
|
| So while it is 100% open source, depending on your jurisdiction,
| you could be sued by Cloudera for patent infringement.
|
| It is interesting, also how the referenced paper
| https://cse.buffalo.edu/tech-reports/2014-04.pdf is from 2014.
| Wonder if they knew about it each other? It doesn't seem like the
| researchers from the paper are mentioned in the patent at all.
| jeffbee wrote:
| I find these articles (and there are dozens of them) somewhat
| frustrating because they fundamentally misrepresent the role of
| time sync in distributed databases like Spanner. Spanner doesn't
| get _correctness_ from tight synchronization, it gets _low
| latency_ from that. Spanner 's _correctness_ comes from
| estimating the clock error of cluster participants and evicting
| them when necessary. You can have the latter property without the
| former, and having the former property without the latter is
| useless.
| rkarthik007 wrote:
| Spanner gets both correctness and low latency from tight
| synchronization. They do COMMIT_WAIT, meaning wait for the max
| clock skew to pass. But the max clock skew without TrueTime
| will be around 500ms (which it impractical to wait out). So,
| the 7ms is lower latency, but 500ms is impractical (more so
| than just calling it higher latency). And any other technique
| to drop latency (without HLC) will violate correctness.
|
| Disclosure: co-founder/CTO of YugabyteDB project
| anonymousDan wrote:
| Interesting. How does this differ from HLC in practice - in
| the article you say you use a 500ms max skew for HLC?
| rkarthik007 wrote:
| The difference is in how the regular path (exercised most
| of the time) vs an edge case when there is a conflict
| (typically in larger clusters with a pathological access
| pattern) works. With TrueTime, the latency is always 7ms
| and no issues in the pathological cases. With HLC, the
| latency is lower in most cases, but high in the
| pathological cases (when it can be 500ms), but these should
| not matter for many use cases.
| vlovich123 wrote:
| What do you think of RIFL/TAPIR?
| rkarthik007 wrote:
| Thanks for pointing this out, was not aware of TAPIR. Will
| take a look, seems pretty interesting.
| dastbe wrote:
| fwiw i think you're both saying something very similar. true
| time has to be correct about max skew in order for it to not
| break the assumption spanner is built upon. you could also
| use a looser time bound and still have correctness, but end
| up with a database that is useless to most/all customers
| traceroute66 wrote:
| Oh joys, the sound of re-inventing the wheel. The old "not
| invented here" syndrome.
|
| Per the blog, they state they want max 7ms difference between
| nodes.
|
| I'm no timesync guru, but AFAIK can do that off-the-shelf today
| using chrony with hardware timestamping or PTP. No need to invent
| your own.
|
| And if you still insist on doing something different, why not
| work on improving the existing work being done under the the Open
| Compute Time Appliance Project
| (https://www.opencompute.org/projects/time-appliances-
| project...).
| rkarthik007 wrote:
| > but AFAIK can do that off-the-shelf today using chrony with
| hardware timestamping or PTP. No need to invent your own.
|
| Actually, the issue is about the max clock skew guarantees (as
| opposed to the average or median). Even a single violation of
| this breaks the ACID semantics. So, we do use chrony, but need
| all this to ensure there is a max guarantee. We would totally
| have adopted an existing solution - we did look at all
| alternatives available.
|
| Disclosure: one of the founders of the YugabyteDB project
| traceroute66 wrote:
| How would you say your solution compares to the CockroachDB
| solution[1] ?
|
| [1] https://www.cockroachlabs.com/docs/stable/architecture/tr
| ans...
| rkarthik007 wrote:
| This blog post covers the various techniques to sync clocks
| in general (NTP, GPS clocks / TrueTime, Timestamp Oracle,
| HLC, etc). CockroachDB uses HLC (hybrid logical clocks),
| which is the same clock sync mechanism as YugabyteDB uses
| (and Apache Kudu as well).
|
| The page you linked to talks about how HLC is used to get
| distributed transactions to work, and there are differences
| here between the different databases (Spanner, CRDB,
| YugabyteDB, etc).
| jeffbee wrote:
| Good luck using PTP in EC2.
| yangosoft wrote:
| Aren't they providing NIC with hardware timestamp? That's
| bad.
| jeffbee wrote:
| ENA does not have this capability, or the capability is not
| exposed to the instance. Perhaps some of the older instance
| types with different network adapters have it, but I never
| tried them.
| dbarale108 wrote:
| Yugabyte ensures transactions consistency for a ACID model in a
| linear clock synchronization between nodes. Great article!
___________________________________________________________________
(page generated 2022-02-02 23:02 UTC)