[HN Gopher] Going in circles without a real-time clock
___________________________________________________________________
Going in circles without a real-time clock
Author : X-Cubed
Score : 205 points
Date : 2024-04-11 04:22 UTC (18 hours ago)
(HTM) web link (rachelbythebay.com)
(TXT) w3m dump (rachelbythebay.com)
| koala_man wrote:
| tl;dr: machine is unable synchronize its clock because it won't
| trust the NTP server's DNSSEC certificate because its clock isn't
| synchronized. Oof.
| iforgotpassword wrote:
| Yeah in hindsight it might not be the best idea to enable this
| for ntp.org, especially when the ntp protocol itself isn't
| secure.
| kakkoko wrote:
| https://github.com/systemd/systemd/commit/abf4e5c1d3ad767bc0...
|
| While systemd-timesyncd contains a workaround for this, chrony
| does not seem to.
| jandrese wrote:
| This is the perennial problem with trying to secure NTP. I've
| also seen people suggest that we just use HTTPS for time sync,
| not realizing that if your clocks are too far off you can't
| make a HTTPS connection.
| toast0 wrote:
| It takes more creativity, but you can start to make a
| narrowing range of times if you're careful.
|
| You'll have to ignore DNSSEC time based validation while you
| start up, of course... Start with the last time you were
| properly synced: it's most likely less than a month behind
| then.
|
| Then take a sampling of certificates offered from well known
| sites. If the certificates validate, other than time, no
| reasonable CA that you trust issues not-before dates in the
| future, so it's not before the most recent not-before date in
| a certificate. It's probably not much after the not-after
| dates either, well know sites tend to replace their
| certificates (but maybe you're getting MITMed with a
| compromised, old certificate).
|
| Maybe ask for OSCP stapling, which should get you a more
| narrow range of times. I think OSCP responses are valid for a
| week? But if you get several from different https servers,
| chances are good you'll narrow the range.
|
| In the case as suggested that the time was only off by about
| a day, IMHO, something is seriously wrong if it can't get to
| sync from there, but I guess I'm not really surprised either.
| creeble wrote:
| You can use plain HTTP for time sync. Almost all HTTP servers
| respond with a time header.
| jandrese wrote:
| The whole point of the exercise was to make it secure
| though. If you don't care about MITM attackers then NTP
| works great.
| numpad0 wrote:
| I don't get the supposed security aspect of getting false
| time. Hacker gives you 1980-01-01 or 4096-13-32 and mess
| up CRL and ruin your day...how.
|
| Years ago I've tried privilege escalation exploit to play
| with a phone and it involved rolling back date to
| unexpire signature, so I know there is exploit potential,
| but it... it just feels like RTC bootstrap problem should
| be something solvable.
| creeble wrote:
| Isn't that a provably pointless exercise though?
|
| Security protocols (at least the ones in common use)
| require certificates or keys that eventually expire,
| because of the risk of a permanent key being compromised.
| If they expire, the protocol needs time. QED.
| lifthrasiir wrote:
| > By "everything else", I also mean WireGuard. Did you know that
| if your machine gets far enough out of sync, that'll stop
| working, too? I had no idea that it apparently includes time in
| its crypto stuff, but what other explanation is there?
|
| WireGuard only requires a monotonic clock, because it
| periodically rotates keys to provide the forward secrecy. Peer
| clocks are otherwise not required to be synchronized [1]. I guess
| the clock had a higher rate than usual and it couldn't be
| corrected due to the lack of RTC?
|
| [1] https://www.wireguard.com/papers/wireguard.pdf#page=7 "In
| fact, it does not even have to be an accurate timestamp; it
| simply must be a per-peer monotonically increasing 96-bit
| number."
| josephg wrote:
| Why does it use the time then? Why not just increment its own
| 96 bit number whenever you use it?
| lifthrasiir wrote:
| Because it is required to be monotonic _per peer_. WireGuard
| has no intrinsic states, so multiple machines with the same
| peer key will be seen as a single peer and that is actually a
| legitimate use of WireGuard. These machines would have to be
| synchronized to each other (but not necessarily to the
| external clock), and using a time is a straightforward and
| reasonable way to ensure this.
| apexalpha wrote:
| So could you share a single set of keys on multiple
| machines and Wireguard will work, as long as all machines
| use the same ntp server?
| lifthrasiir wrote:
| Technically yes, but I don't think (but haven't exactly
| confirmed) that you can initiate a new session to itself.
| wodenokoto wrote:
| What is she referring to by "THE ONE"?
| semireg wrote:
| The know it all. This "wouldn't have happened to them because
| they are" THE ONE. Also known as, "that guy" as in, "don't be
| THAT GUY."
| k1t wrote:
| As sibling comment somewhat points out, she is literally
| referencing The One (2001)
|
| https://m.imdb.com/title/tt0267804/
| bjackman wrote:
| It's pre-empting the sibling comment saying "RTC module is $1"
| dmvdoug wrote:
| See, e.g., https://rachelbythebay.com/w/2018/04/28/meta/
| Joker_vD wrote:
| Amazing. So when you read about some weird (and mostly self-
| inflicted) problem you've never had in your life, you are
| _not_ allowed to express your surprise because by doing that
| you 're implying you're better than other people, and that
| would be horrible (the implying part is what is horrible, I
| think? Or maybe actually being better is also wrong? Can you
| even be better--yeah, I guess it's possible to be better than
| someone else at something).
| Ensorceled wrote:
| You're totally _allowed_ to express your surprise, you 're
| totally _allowed_ to be a condescending jerk while doing
| it, you 're totally _allowed_ to call it "self-inflicted".
|
| The author is _allowed_ to ask you to shut up and go away
| if you do.
| smallmancontrov wrote:
| You're allowed to. We are then allowed to think you are an
| asshat for doing it.
|
| Sharing cautionary stories is good. Learning from others'
| mistakes is good. Jumping on these as an opportunity to put
| someone down for the purpose of self-promotion ("mostly
| self-inflicted", express your surprise) is unkind, probably
| bullshit (I have seen soooo many people pretend to be above
| problems and then run smack into them), but most
| importantly: it shits up the dynamic where people are
| willing to share their mistakes, allowing group-learning.
| It would have been the easiest thing in the world for
| rachelbythebay to _not_ make this blogpost and thereby
| avoid providing an opening for asshats like yourself, but I
| 'm glad she made it anyway, even knowing that you would
| take your shot, so that everyone else could learn.
| dblohm7 wrote:
| Found the one!
| shadowgovt wrote:
| Sorry about wasting your time. This is not for you.
| caseyy wrote:
| You can be better and nice.
| M95D wrote:
| RTC module for Raspberry is ~ $1.
| icehawk wrote:
| and you certaintly get what you pay for, DS3231s fry almost as
| easily as MAX3232s,
|
| (And you can do the same thing were you write the time to the
| RTC before you have the correct time)
| Joel_Mckay wrote:
| I have used these in product designs before with zero
| failures. Where exactly did you buy your samples?
|
| We found they drift less than 28s a year in standalone mode.
| Also, were repeatable within +-18ms when using a NTP client.
|
| Microchip RTCs are harder to setup, but also seem to work
| just as well (ignoring the goofy epoch).
|
| Chip shortage caused people to do unspeakable things.... ;-)
| hcfman wrote:
| You could buy an rtc and put up with that kind of clock
| drift, or.... for the same money or less you could be a GPS
| and have none of that drift.
| r2_pilot wrote:
| (a bit more money for the GPS to be fair) not even
| mentioning the fact that you should have clear-ish access
| to the sky, the time taken for you to receive the time
| data, and the possibility of jamming. I could go on but
| the two methods have differing modes of operation and
| failure despite ultimately being capable of fulfilling
| the need for a reasonably synchronized clock
| M95D wrote:
| I have a DS1307 installed ever since I bought a RPi3B+. It
| didn't fail and it's accurate enough until NTP updates the
| time.
|
| Are you sure you connected it correctly?
| michaelt wrote:
| Right, it's only $1.
|
| Plus the time to learn a few modprobe commands. Perhaps run a
| few commands echoing numbers to random sysfs files. Perhaps
| learn WTF a 'Device Tree Overlay' is - a concept that's
| completely absent on x86 systems. You see, this is an 'I2C'
| device, so it doesn't plug and play like USB and PCIe devices
| do. It is much simpler, and hence for some reason more
| difficult. Perhaps your device tree overlay needs to be
| complied? Or maybe not. Perhaps you'll edit your modules file
| and your kernel command line. Of course that won't involve
| update-grub, why would you think that?
|
| Don't forget to periodically check on your settings, just in
| case an OS update or something has disabled one of those
| settings.
|
| Oh and of course you won't be able to fully test this over SSH,
| if the network is up NTP will have set your clock whether the
| RTC is set up right or not. As we've learned from the article,
| if you run into clock problems you might not be able to connect
| remotely.
|
| Then simply invent a time machine so you can do all of this in
| the past, _before_ the pi SD card fails and it drops off the
| network.
| M95D wrote:
| If the user has issues with all of that, then RPi is not the
| rigth tool for them.
| michaelt wrote:
| Ah yes the Raspberry Pi, a charitable project intended to
| get young kids into coding, which comes with kid-friendly
| projects, brightly coloured beginners' guide books, and
| pre-installed with Scratch, is truly only a tool for
| experienced professionals.
|
| /s
| jlubawy wrote:
| My raspberry pi is also currently dead, something it didn't like
| about updating the OS caused it to be bricked. All updates were
| done within the OS, so didn't expect anything bad to happen.
|
| I could reflash the OS, but I also "don't have the time". I'll
| just use my laptop until I do have the time someday to resurrect
| it.
| hcfman wrote:
| Quite likely installing the sbts-aru as a base project would
| have prevented this as well. Loads of SD cards get corrupted
| historically without apparent good reason. Using a memory-based
| overlayFS file system solves so many of these problems. sbts-
| aru, as well as setting up a stratum #1 time server sets up a
| resilient memory overlayFS as a base.
| riobard wrote:
| Speaking of RTC battery, I've recently come to the realization
| that I have to make sure that BIOS battery is not absolutely dead
| in always-on PC boxes.
|
| Background: I use an x86 box as home router. I've changed the
| configuration in the BIOS that it should automatically boot up on
| power. However if the BIOS battery is dead, the config will be
| lost and it will revert to default settings, which is not to boot
| on power.
|
| But there is no way to know how much juice is left in the BIOS
| battery, thus there is no way to issue warnings that the battery
| should be replaced soon. If there is power loss AND the battery
| is dead, when the power comes back up again, the router will just
| stay off indefinitely until it's manually turned on.
|
| That's when I realized why most consumer routers do not feature
| RTC nor do they require batteries.
| qmarchi wrote:
| Most newer systems persist EFI options to NVRAM now days as
| well. Pulling the battery isn't generally enough to reset them
| anymore.
| progbits wrote:
| My desktop remembers all settings, but after power loss still
| goes into a warning where I have to press F1 to enter
| settings, then exit them without making changes. I couldn't
| find an option to not do that.
| riobard wrote:
| Yes, this is very annoying.
| qiqitori wrote:
| Some unsolicited solutions, maybe not for you but somebody in a
| similar situation: maybe replace it with a supercapacitor
| (needs some wiring changes, otherwise it won't ever be charged,
| should last long enough for most power outages), or use a stack
| of coin cells in parallel (difficult due to physical
| dimensions, and they'll still go dead at some point). You can
| also short two wires on the ATX supply to automatically turn on
| the computer when power is supplied (which may not be good
| enough if you have to press a key during boot to dismiss the
| low battery warning). Or hack the BIOS :)
| riobard wrote:
| Somehow I feel reliability might take a hit... :|
| mkesper wrote:
| If using supercapacitors, use some known not for leaking else
| you're back to square one (do a search for capacitor recap).
| Hrnrurj wrote:
| Voltage check on battery is part of regular PC maintenance.
| Like cleaning dust, checking all fans are spinning, capacitors
| are not getting bigger, checking for weird sounds in PSU,
| overnight memtest...
|
| You should do it every year or two...
| riobard wrote:
| Cannot do that if it acts as the always-on router...
| r2_pilot wrote:
| Companies have regularly scheduled (weekly in some cases I
| know) maintenance windows affecting thousands of people. A
| home environment with an "always-on" router can surely find
| a 10 min window (say, 3 AM on a Sunday morning) where
| nobody cares.
| shadowgovt wrote:
| Funny enough, instead of doing that every year, my policy has
| been "replace the machine if it breaks" and in general, I've
| gotten a good five-year cycle out of all my PCs without
| having to do disassemble-maintenance with an air can and a
| static strip. This has been good enough because I do enough
| high-graphics-demand gaming that five years is about the
| cycle on which some new-shiny has come out that renders my
| machine too old to play modern games.
|
| Maybe I'm just lucky.
| guenthert wrote:
| > But there is no way to know how much juice is left in the
| BIOS battery,
|
| True, but its voltage can be used as an estimate. Most PC MB
| still around monitor the supply voltages, some also of the
| battery feeding the RTC. If you have the (Debian) package 'lm-
| sensors' installed, look for sensors[<pid>]:
| Vbat: +2.91 V (min = +2.70 V, max = +3.63 V)
| ALARM
|
| in /var/log/syslog.
| riobard wrote:
| Ran `sensors` and it only shows CPU core temps. I guess my MB
| does not report RTC battery voltage?
| guenthert wrote:
| One thing are the sensors, 'nother thing is the software.
| Have you configured the lm-sensors package (by running
| `sensors-detect`) yet?
|
| https://hwmon.wiki.kernel.org/lm_sensors also has a FAQ.
| riobard wrote:
| Yes I did run `sensors-detect` previously and just to be
| sure ran it again. No voltage sensors on this box
| unfortunately :(
| mplaisier wrote:
| I ran into a similar problem on a Raspberry Pi, where my program
| would ignore data because it was deemed too old. It turns out a
| Pi uses fake-hwclock. This tool stores a timestamp regularly in a
| file. On boot the Pi reads from that file to set the time. When
| the Pi has been turned off for longer then the initial time can
| be hours or days out of sync. The Pi then takes a couple of
| seconds to obtain a valid time via NTP, but my program ran
| quickly enough to work with the old timestamp. After a couple of
| minutes it would typically work again.
| Joel_Mckay wrote:
| If you are running Bind or SSL/x.509 certs, than an RTC is pretty
| much a requirement on the raspberry pi. Network time leaps over
| NTP even when working (first big leap was allowed etc.) can still
| bork a lot of services.
|
| Local stratum 1 GPS time (even with PPS pin) servers based on
| Pi's can suffer similar issues when they lose satellite lock
| (weather etc.)
|
| Have a great day =)
| hcfman wrote:
| Having said that, I've never personally seen my stratum #1 Pi
| loose sync due to weather and I run about 4 of em here. I also
| use a cheap 7 euro ublox 7m. So long as you position it or an
| antennae with a reasonable view of the sky that shouldn't
| happen I reckon.
|
| Why all the complex solutions when it's so easy and cheap now
| to install a stratum #1 time server, even a Pi 3 will suffice.
| It works on a Pi Zero, but then you only have a wifi link.
|
| The sbts-au project make's it incredibly easy to setup. git
| lone and then install. One command, it's that simple.
| teruakohatu wrote:
| Any advice for a cheap USB GPS for a stratum #1 time server?
|
| EDIT: Any any advice on how essential PPS is?
| Joel_Mckay wrote:
| The PPS can be good (pi kernel support is stable), but one
| needs to be careful as some modules PPS signal has a great
| deal of instability.
|
| Some U-Blox modules support the additional ground based
| navigation signal and cellular beacons. This is recommended
| in addition to the 3 satellite networks classes, but note
| some modules will not work with some antennas (get the
| unified hardware with built in antenna/lna if unsure about
| RF notes).
|
| Best regards, =)
| hcfman wrote:
| If you have good sky visibility, then I don't need an
| antenna. Though this often tends to mean putting a Pi
| outside. I'm also using just the first Antenna I find
| actually.
| hcfman wrote:
| I have good results with cheap ublox 6m's and 7m's from
| aliexpress. For some reason the 8m's I got were poor in
| sensitivity.
|
| Adafruit ones also work great, but are a lot mor expensive,
| I do have several though.
|
| PPS is criticial to proper timing. However... I was using
| the time synched systems for sound localization. But as
| they only cost around 7 euros on aliexpress, why not.
| zokier wrote:
| It's unfortunate that using DHCP to get _rough_ time isn 't more
| popular, especially for devices without RTC. Sure DHCP can be
| spoofed and should be considered more of a last resort, but it
| sure would be better than nothing and ending up in this kind of a
| situation where nothing works anymore.
|
| Previous discussion on similar thing: "Encrypted DNS + NTP =
| Deadlock" https://news.ycombinator.com/item?id=34177331
| CaliforniaKarl wrote:
| I know I've brought this up before, but I can't remember exactly
| where.
|
| It would be nice if DHCP could provide the current time as one of
| the pieces of information it provides with a lease. If you're
| already trusting it for your IP address, you might as well trust
| it with the current UTC time: The current 64-bit timestamp would
| be enough to get timestamp-sensitive things working.
| citrin_ru wrote:
| I trust a free wifi hotspot to give me an IP (traffic will be
| protected by TLS), but not with the time. E. g. time from the
| past can be used to trick into accepting an expired TLS
| certificate. But for most wifi hotspots it will be
| unintentionally wrong because such devices often unmonitored
| and not maintained until they completely break.
| mapreduce wrote:
| > If you're already trusting it for your IP address, you might
| as well trust it with the current UTC time
|
| I don't follow! How does trusting DHCP with IP address
| automatically mean I should trust it with the current UTC time?
|
| Time requires higher degree of trust than IP.
|
| I may not care what my IP address looks like but I might care a
| lot about what the current UTC time is and I might want this to
| come from a more trustworthy device.
| Terr_ wrote:
| To offer an example, if an attacker can manipulate the time
| then they can make the target accept expired or revoked
| cryptographic certificates, potentially enabling
| impersonation or man-in-the-middle attacks.
|
| In contrast, a different IP than expected isn't such a big
| deal... Although it might break the collaboration of two
| computers as crude denial of service
| mikepurvis wrote:
| DHCP servers can command your dns config and hostname too.
| It's not a total mitm story since the certs are still on the
| machine itself but it's definitely more than just a local IP
| address.
| zokier wrote:
| You don't need to trust the dhcp time. But it would be useful
| for bootstrapping, especially devices without rtc. This does
| not really apply to your average PC which probably already
| has good guess on current time, those can simply just ignore
| the dhcp provided value.
|
| The DHCP server provided value can not be much worse than
| 1970-01-01T00:00:00Z default value if you don't have any
| other data. And if you have some other data, e.g. ext4
| superblock timestamps, you can pretty trivially protect
| against DHCP providing time from the past (i.e. use the
| maximum of different sources).
|
| Finally, you can restrict the use of the dhcp provided time
| to the initial bootstrapping process only; it's not necessary
| to use it for system-wide clock
| simoncion wrote:
| > It would be nice if DHCP could provide the current time as
| one of the pieces of information it provides with a lease. [0]
|
| DHCP can provide the IP address of one or more NTP servers to
| consult for time queries. I use this feature so that all of the
| machines on my LAN (at least the ones that can be configured to
| ask for and/or use this information) have the same general idea
| of what time it is.
|
| [0] See subsection 8.3 here: <https://www.rfc-
| editor.org/rfc/rfc2132#section-8>
| magicalhippo wrote:
| I do that, with a local GSP-synced Pi. Very cheap and easy to
| set up with IPv4 at least.
|
| That said, if you don't trust the DHCP server because some
| evil actor might give you the wrong time, why should you
| trust the NTP server it tells you about?
| simoncion wrote:
| I agree that if you distrust the network operator (but must
| use the operated network for some reason) that you should
| accept as little configuration information provided by that
| operator as is reasonably possible.
|
| That goes without saying.
|
| I mentioned the fact that DHCP can provide NTP server
| information as a way to explain why DHCP wouldn't just
| straight-up provide "current time" information. Why do that
| when you can point people to a server running a far
| superior purpose-built timekeeping protocol? It'd just be
| silly to do the worse thing.
|
| (Notice also that RFC2132 was published in _1997_. In my
| mind, this moots any retorts that a good reason for
| providing time directly would be because the DHCP server 's
| supplicant is too underpowered to run an NTP client. In
| 2024? On a consumer-grade device? No way.)
| magicalhippo wrote:
| Oh yeah, certainly agree. Especially since devices can
| use SNTP[1] to talk to a NTP server if they just want to
| know the current time and don't need all the complexity
| of NTP.
|
| [1]: https://timetoolsltd.com/ntp/sntp-overview/
| pja wrote:
| If your DHCP server supports bulk queries (RFC6926:
| https://www.rfc-editor.org/rfc/rfc6926.html ) then responses to
| such a query from a dhcp client include a timestamp in the form
| of an 32-bit integer count of seconds since the epoch.
|
| I don't know how widely this DHCP RFC is supported though.
| flemhans wrote:
| It would be nice if DHCP (or maybe wifi?) could provide the
| location as well, so that you could relay the airplane's geo
| position to the connected clients.
| cjbillington wrote:
| The wireguard problem is a pain in the neck. Even if you're happy
| for your system to not have a realtime clock, when it does come
| online you'd want wireguard to not start until after the clock
| has synced to a timeserver. The below is what I'm doing at the
| moment, but I can't say I'm sure it's working - haven't seen
| wireguard start before the time is synced since doing it, but it
| could be probabilistic: systemctl enable
| systemd-time-wait-sync.service mkdir -p
| /etc/systemd/system/wg-quick@wg0.service.d/ echo "[Unit]
| After=time-sync.target Wants=time-sync.target " >
| /etc/systemd/system/wg-quick@wg0.service.d/override.conf
|
| I'd be interested if anyone could let me know if they think this
| is likely to be achieving what I want or not.
| hcfman wrote:
| Well... I see where you are going with this but is it
| necessary? What happens if the clock gets it's correct time
| sync back, will not wireguard come back online by itself ?
|
| Then the real solution might involve ensuring good clock sync.
| Using a GPS synched time source is good for this, such as this
| one :) https://github.com/hcfman/sbts-aru
| yjftsjthsd-h wrote:
| That was my thought as well - if we know certain things need
| time to be in sync, can we use systemd dependencies to enforce
| that? I'm not sure it would help with OP's problem because that
| was more or less a circular dependency (time sync needed name
| lookup which needed time to be in sync), but it could mitigate
| some of the effects or at worst make the problem more
| explicit/obvious.
| silasdavis wrote:
| Tangential: I have a pi 4 running openwrt and a wireguard
| interface that is routed over the WiFi access point. If you
| connect to this network it's as if you are connecting in another
| country. I have outbound traffic only going over wireguard. If
| wireguard is down then no internet (to avoid any leakage).
| However if pi loses power the clock resets, wireguard can't
| handshake with that much clock skew, NTP can't connect without
| wireguard. So I need to manually update clock after a power
| failure.
|
| I guess I need requests to the NTP server to go over WAN
| directly. Always seemed like a hassle to get this to work with
| openwrt zones and stuff. My eternal gratitude to anyone who
| shoves me in the right direction...
| pynne wrote:
| Yeah you got it. Set the listening interface to eth0 or
| whatever the wan interface is. Might need to get full ntpd
| instead of the busy box version. The Interface option here:
| https://www.ntp.org/documentation/4.2.8-series/miscopt/
|
| I'm actually setting up a router with a rpi4 right now, only
| using Fedora IoT instead of OpenWrt. It's a bit more assembly
| required lol
| silasdavis wrote:
| I need to have a play with it, but the problem is i
| deliberately have a 'kill switch' style setup to stop any
| requests accidentally going not over wireguard (for reasons)
| so I assume I'll need to do some other firewall type thing.
| Maybe something something tagging.
| hcfman wrote:
| I also use wireguard a lot. I didn't know that clock
| sensitivity could be an issue here. I've never seen an issue
| before. Now that I know it could be one I'll make a point of
| deploying all cellular based systems with a GPS synched time
| source as well and avoid that sort of problem.
| nix0n wrote:
| > I guess I need requests to the NTP server to go over WAN
| directly. ... My eternal gratitude to anyone who shoves me in
| the right direction...
|
| Do you have something on the LAN side that could run an NTP
| server? I'm thinking you could add your laptop to your Pi's
| list of NTP servers, then just start that service up
| temporarily as-needed to give the Pi a time that's good enough,
| then it can get actual correct time sync from somewhere else
| once Wireguard is working.
| ogurechny wrote:
| If you "always" have Internet access, you should really just
| place ntpd onto the physical network where it belongs.
| Otherwise, the options are all mentioned in the article (and
| might be used in your distribution).
|
| -- Restoring more sane shutdown timestamp from file instead of
| just using 1980.
|
| -- Making blind single use ntpdate request to some public
| server to set local time after the network is available, but
| before any other application uses it. There might be
| appropriate hooks in your init system if you don't want to make
| it an independent step.
|
| -- The same, but using some local time source. It is not widely
| known that even plain old Windows desktops, while not having a
| complete NTP server, can be trivially configured to announce
| time over some form of SNTP. If you have any computer with a
| real clock running nearby, you can probably rely on it. Next
| step router is also a good potential candidate.
|
| As for compartments and access control, you can try to use
| basic VLANs (one for ntp traffic, and only it, and one for VPN)
| or maybe some two-step network configuration process (get time
| with one configuration, reset and proceed with other services).
|
| All of that is not a problem, the problem is what to do when
| something fails. Time server becomes unavailable, intermittent
| internet connectivity, server you use for pings is available,
| but others are not, etc. You can choose to stop completely if
| there is no time source, and wait, but that might render the
| system unavailable for remote configuration. You can try to
| restart the device or restart the network router automatically
| (though only a fixed number of times, restart loop will surely
| break something), but that won't help if the problem gets fixed
| some time later. You can also use a fallback configuration that
| doesn't depend on correct time (e. g. only start an ssh server
| so you can connect to it).
| hcfman wrote:
| Install sbts-aru as a base system and it will keep your Pi
| accuracy to sub-microsecond error.
|
| https://github.com/hcfman/sbts-aru
|
| Problem solved.
| hcfman wrote:
| Or... Install one of these on a Pi somewhere in your network
| and use that as a time source for all of your devices. Problem
| solved for all your machines.
| hcfman wrote:
| So you know, sbts-aru is an audio recording program. However,
| in one command it sets an an overlayFS protected SD card
| setup and chrony GPS based time synchronization. You can turn
| off all the recording related stuff and then you have a
| stratum #1 time server for your network. Real easy, real
| cheap. No more time problems.
|
| There's no reason now for all geeks not to have at least one
| stratum #1 time server in their networks. Really none.
| mkesper wrote:
| IF you get a DCF77 (stratum 0) signal, there are receivers for
| Pi below 10EUR.
| ndsipa_pomu wrote:
| Getting time from DCF77 seems like the best solution for
| timekeeping on a lot of computers, but there don't seem to be
| nicely packaged ones over USB etc. I had a quick look for Pi
| receivers and none of them looked to be nicely packaged, but
| with an external antenna.
| shawabawa3 wrote:
| I've personally had so many issues with SD card corruption on
| raspberry pi's that I've decided it was a mistake to even allow
| SD card disks
|
| I only ever use USB storage on my pi's now and not had a single
| issue since
| hcfman wrote:
| Indeed, except if you run an in-memory overlay filesystem this
| might change dramatically for you. I have several Pi's that run
| for years and years without SD card corruption this way.
| Normally, however, as in my sbts-aru project, I don't just do
| that but also create separate data partitions that are read-
| write. It appears that even though there are some partitions on
| the card that are read-write, if they are not the system one it
| is also much more resilient to fatal corruption.
| yabones wrote:
| Same with "real" servers. For a while there, it was trendy to
| boot ESXi from an SD card stuck to the system board and use all
| of the 2.5" disks for just data storage, no OS. It worked
| great... until it didn't. It turns out that writing logs and
| swap to an SD card roasts it after 2-4 years pretty
| consistently.
| simoncion wrote:
| The smart ESXi admins I knew used Compact Flash cards as the
| system boot disk. (It has been too long since I've been in
| touch with those guys to remember if they offloaded logs to
| local disks (or maybe a datastore on a SAN) or if they also
| wrote them to the CF device. I bet they did the former, but I
| no longer clearly remember.)
|
| Using an SD card is just nuts.
| yjftsjthsd-h wrote:
| I wouldn't rule out running the OS off of that kind of
| storage if you had a read only root file system, but what
| madness would inspire you to write logs to it? And _swap!?_
| Moru wrote:
| I heard that this is caused by not supplying enough power to
| the Pi, did you use the official powerbrick? I have had
| corruptions happen a couple of times but I have never tried
| with a good powersource if it actually does a difference.
| constantcrying wrote:
| SD cards are not made to run an OS from. They are quite
| limited in read/write cycles .
| constantcrying wrote:
| Using SD cards for an OS is a really bad idea in general. You
| may make it work with a read only filesystem, but if you don't
| sooner or later you will have issues.
| bityard wrote:
| Most SD cards for sale are not suitable for write-intensive
| random-access workloads. However, they do make SD cards that
| are intended for this, I've had very good luck with cards
| marked "high-endurance." They cost more, but they work.
|
| I've always been amazed that the Foundation never mandated the
| use of validated brands or types of SD cards.
|
| Not that it matters much to me, I've had to retire all of my
| Pis earlier than the 4. It seems that all of my Raspberry Pis
| prior to the 4 all had some sort of design issue with their
| power regulation. After a couple years of constant operation,
| they just start locking up randomly around once every day or
| so. I've had everything from a Pi 1 through 3 exhibit this
| behavior. I spent WEEKS troubleshooting this and conclusively
| ruled out wonky SD cards, power supplies, connected devices,
| proximity to other equipment, and every other factor I can
| think of. I found many threads with other people reporting the
| same issues but the Foundation refuses to acknowledge it as a
| problem. I only run Pi 4's now, and am gradually switching over
| to Intel N100-based systems for low-power operation.
| lightswitch05 wrote:
| I had this happen before. I used it as an excuse to learn how to
| setup NTP with a GPS receiver. I made a little blog on it if
| anyone is interested in the results. Be sure to click the
| sandwich menu for some real-time data:
| https://www.developerdan.com/ntp/
| hcfman wrote:
| Quick question, because I've followed a number of how tos
| before that don't, if you disconnected all network
| connectivity, would it sync to the PPS source? Second, have you
| tested this, I only ask because 1# I found a lot of time sync
| was really slow, and #2 some of them reported that they had
| sub-microsecond time sync but when tested they were several
| milliseconds out.
|
| Anyway, I solved these problems with sbts-aru, but maybe it's
| interesting for you to test if you haven't already done this.
| lightswitch05 wrote:
| Good question, it absolutely syncs without any network
| connectivity. For better or for worse, I have it setup NOT to
| sync to anything else- being a stratum 1 source is enough.
| Now... I will admit I'm just a hobbyist with this stuff.
| Anyways, the way I got around it trying to sync to another
| source when using the PPS was to define the GPS as two
| separate sources. One for the PPS signal and another for the
| NMEA sentences: refclock pps ppspath
| /dev/gpspps0 prefer time1 0.004 minpoll 3 maxpoll 4 iburst
| refclock nmea baud 57600 time2 0.068 minpoll 3 maxpoll 4
| prefer iburst
|
| To get around issues with sync being slow, I have NTPD
| configured with some extra flags to allow large time jumps at
| startup: NTPD_OPTS="-ggg -N"
|
| I have two raspberry pies setup this way- which I don't use
| for much else. My primary computers have these two servers
| configured as time sources- but also have external sources-
| so as long as network connections are available, they can
| determine if they are providing a sane time or are false
| tickers. As for accuracy, you can view the ntpviz output for
| each setup in the hamburger menu. For catwoman, its regularly
| within 4 microseconds.
| hcfman wrote:
| Ah okay. Because I found that the default start order of
| gpsmon and chrony was incorrect meaning that the shared
| memory communications wasn't working but there were no
| error messages stating this. So sometimes, it would sync
| (slowly) and then chrony sources would say that it was
| having microsecond sync time but when I connected a switch
| (On/Off type) to the same gpio on two separate Pi's with
| linked earth connections and had a python program output
| the system time in nano seconds I found that despite
| telling you it was keeping time accurate to microseconds it
| was in fact many milliseconds out. The only thing I can
| conclude is that despite the setup indicating that it
| should use the PPS source that use the interrupt, this was
| in fact not happening and it was somehow getting it's time
| from the serial line only. Without indicating that this was
| happening.
|
| When I switched the startup of gpsmon and the chrony daemon
| to the other way around time sync was achieved within 1/2
| minute to a minute (Real fast) and the reported time on two
| separate Pi's with the joined up gpio's (So you can get
| very close to triggering a time check at the same time) was
| then sometimes as low as 30 ns's from each other, which is
| damn good considering that most of the difference would be
| jitter.
|
| And none of the how-to's that I read talked about changing
| the start order of chronyd and gpsmon so I can only assume
| they all might be suffering from the same problem. It's
| important to note that the diagnostic commands from chrony
| will not indicate this and the only way you will notice if
| this is happening if you setup a rig like I mentioned
| above. I noticed because my sound localization calculations
| were showing errors that didn't make sense.
| lightswitch05 wrote:
| I must admit I've never used chrony, so I'm unfamiliar
| with how to configure it. I've read a lot say saying
| gpsmon's shared-memory-segment is a really great
| interface and to go with it instead of NTP's own GPS
| drivers, but I cannot say that lines up with my
| experience. I do have one computer setup this way, and it
| works, but I found on the PPS configurations, it seemed
| to me to have better accuracy and less jitter using NTP's
| own GPS drivers and taking gpsmon out of the equation.
| Its quite possible I just don't know the right
| configuration combination, but ntpsec's own GPS drivers
| work great for me.
| hcfman wrote:
| Ah okay. Conversely I've never tried that setup :) Nice
| to hear about it though.
| lightswitch05 wrote:
| This is off-topic, but I was looking at your sbts-aru
| project and remembered having read the hackaday post on
| it about fireworks. I'm curious if you've ever seen the
| RaspberryShake BOOM sensor? If so, any thoughts on how
| your project and it differ?
|
| https://shop.raspberryshake.org/product/turnkey-iot-
| atmosphe...
| hcfman wrote:
| I hadn't seen that. It uses a pi so in principle it could
| sync the time the same way. How well related to
| localizable recordings would also depend on how it
| establishes the actual time arrival with the clock and
| how it deals with jitter.
|
| Currently I only support USB mics at 16 bits that can
| talk to jackd. This was a conscious decision because it
| was all that was needed but also because jacks makes it
| essentially have multiple source sinks. Such as as well
| as recording a real time audio processing pipeline that
| potentially leads to real time gunshot and other sound
| event localization.
|
| But it would be cool if the infrasound sensor had a USB
| sound interface. As then it could be useful in localizing
| elephants.
| hcfman wrote:
| If it syncs time accurately by any method maybe you are
| able to localize some very interesting events! The
| localization code I provide with my project should fine
| with times obtained from this project. I seriously doubt
| whether that project uses a memory overlayFS but it's to
| add as an improvement.
| boneitis wrote:
| Interesting! What is the accuracy like when lacking "PPS
| precision"/Clayface? I'm wondering about the magnitude by which
| it is off (seconds, minutes, or other).
|
| PPS turned up quickly on Wikipedia, so that one is readily
| answered
|
| > PPS signals have an accuracy ranging from 12 picoseconds to a
| few microseconds per second, or 2.0 nanoseconds to a few
| milliseconds per day based on the resolution and accuracy of
| the device generating the signal.
| lightswitch05 wrote:
| You can see for yourself the level of accuracy in the ntpviz
| output. Notice the units of measurement on the graphs:
|
| * Clayface:
| https://www.developerdan.com/ntp/#./clayface/7-days/
|
| * Catwoman:
| https://www.developerdan.com/ntp/#./catwoman/7-days/
|
| That Wikipedia quote should mention temperature! Temperature
| variations have a big impact at this level of accuracy. These
| really cheap GPS receivers do not have temperature adjusted
| clocks. Unfortunately my server closet (this is just a hobby)
| does not have well regulated temperature, so you can see the
| impact of temperature on the clock accuracy. Also, I found if
| I start running a bunch of stuff on these computers - that
| makes the CPU heat up, which also affects the jitter. If you
| really want high-precision, you'll have to shell out some
| extra cash then I did:
| https://www.sparkfun.com/products/18774
| hcfman wrote:
| rtk gnsses really rock. I would to save up for one for next
| year :)
| antx wrote:
| As for the GT-U7, this software can be used to avoid Windows,
| it seems:
|
| https://github.com/semuconsulting/PyGPSClient
| lightswitch05 wrote:
| > While not intended to be a direct replacement, the
| application supports most of the UBX configuration
| functionality in u-blox's Windows-only u-center (c) tool
| (only public-domain features are supported).
|
| I'm going to have to check this out! Thank you for sharing!
| the-kenny wrote:
| Note that the Pi 5 has a real time clock - you just need to add a
| small battery and it will keep the time when powered off:
| https://picockpit.com/raspberry-pi/raspberry-pi-5-has-a-real...
| billsmithaustin wrote:
| I guess that's what the article meant here: "This particular Pi
| doesn't have a real-time clock. The very newest ones (5B) _do_
| , but you have to actually buy a battery and connect it. "
| doubloon wrote:
| Its weird that people work OK without knowing exactly what time
| it is. Sure it helps us but if a person lost track of the date
| for a few days in a city they would still be able to perform
| basic functions. We only have had timekeeping devices for a tiny
| part of our history but we invented alot of stuff without precise
| synchrony.
| hcfman wrote:
| How did they do sound localization then in those times ???
| avianlyric wrote:
| Isn't that just because individual human activities aren't very
| precise, and for the most part, the position of the sun in the
| sky is enough for an individual to broadly synchronise
| themselves with the rest of society.
|
| Unless someone is actually aiming to do a time coordinated task
| with others, you really need to know the time, or even the day-
| of-week to do stuff, because most human things in world aren't
| actually occurring in small time sensitive windows.
| malivvan wrote:
| Unfortunately a Raspberry Pi is a bit ill suited for production
| environments. Id recommend an RTC module. Otherwise this might be
| helpful: https://github.com/hcfman/sbts-aru
| benmmurphy wrote:
| I'm surprised tptacek hasn't shown up to defend DNSSEC from this
| libel
| tptacek wrote:
| DNSSEC is doing a fine job of representing itself here.
| smashed wrote:
| > I figured they must be running DNSSEC on that zone (or some
| part of it), and it must have a "not-before" constraint
|
| Since clients will attempt to resolve ntp.org in order to
| actually sync their clock, there is a good probability that some
| clients will be way off.
|
| Enabling dnssec on that zone was probably not without important
| drawbacks? I wonder if the operators thought about that potential
| pitfall. Seems like they might be doing a disservice to their
| core mission of allowing devices to sync their clock.
| growse wrote:
| It feels like enabling DNSSEC on any zone is inviting a bunch
| of fun service risks and unexpected failures.
|
| Is my clock more likely to be accurate after NTP.org got
| signed? It looks like it's less likely, on average.
| cbhl wrote:
| If you _don't_ enable dnssec on ntp.org, then a mitm can
| intercept the dns request to redirect to an attacker-owned
| timeserver with a time in the past. Then the host can have old
| and expired (without loss of generality) keys/certificates
| replayed against it.
|
| If memory serves though, Raspbian used to not even have `fake-
| hwclock` by default and even more Pis would end up with this
| "wildly wrong time near the epoch and can't bootstrap DNS/NTP"
| failure mode. You'd sometimes also see it with VMs too (esp if
| they were doing a pure-software clock that ran slower than
| realtime, instead of patching clock calls to the real hardware
| clock on the host).
| the__alchemist wrote:
| It's surprising that it was left out. Ie, the workaround makes
| sense, but this is a peripheral that is sometimes built into MCUs
| themselves. For example, most (all?) STM32s include an RTC...
| with the caveat that if you are using them for canonical use
| cases, you will need external hardware in the form of a dedicated
| 32kHz oscillator, and possibly a battery.
|
| For a lot of micro-controller timing uses, they aren't required;
| for example, hardware timers based on the primary clock source,
| or the Cortex-M systick, but for maintaining accurate dates and
| times over long periods, the RTC is the right tool. It can also
| output the dates and times in a convenient format as well, as
| long as you find named register fields convenient!
| 01HNNWZ0MV43FF wrote:
| I had to support an IoT device once that lacked an RTC.
|
| That was actually the easier hardware platform, because it
| either had good NTP sync or it didn't reach our C&C servers.
|
| The difficult one was the platform that had an RTC. Those would
| slooowly drift out of sync if NTP was broken. Slowly and
| _silently_.
| AdamH12113 wrote:
| An RTC really needs a battery, and even a smaller battery like
| a 1220 takes up a fair bit of board space -- space that could
| be used for another connector or more functionality. And even
| if you have an RTC, you still have to handle the edge case
| where the battery dies or is removed and the clock resets.
| Critical low-level internet services probably shouldn't require
| clients to already have the correct time.
|
| I use an external RTC with a Raspberry Pi for some sensor
| systems I work with. Its purpose is to give a reasonable chance
| of having the correct time for recording data in remote
| locations where no internet connection is available at startup.
| For a Pi that only provides network services (and thus always
| needs a working internet connection), I probably wouldn't think
| to bother.
| numpad0 wrote:
| RTC is just an electronically readable Casio watch, it's not
| even responsible for keeping real time while OS is running. All
| it does is always incrementing regardless of motherboard power
| state so to provide time as OS loads.
| aimor wrote:
| Two nights ago I rebooted my router and it wasn't able to update
| its clock. The logs were filled with "ntp: start NTP update"
| every 5 seconds.
|
| Well, it was something related to using AdGuard's (94.140.14.14)
| DNS and pool.ntp.org. I added Google's (8.8.8.8) to resolv.conf
| temporarily to get the clock synced. But I'm still not sure what
| the root cause was.
| cbhl wrote:
| I think the last paragraph was my favorite part of this article.
|
| For those of you who didn't read the fine article, I've quoted it
| here:
|
| > As usual, this post is not a request for THE ONE to show up. If
| you are THE ONE, you don't make mistakes. We know. Shut up and go
| away.
| yjftsjthsd-h wrote:
| I dunno, I've always found that to be a... less impressive part
| of the blog. When the author hits problems in _her_ stack,
| anyone who would have caught it should shut up and stop being
| THE ONE. Fine, we 've all run into people with big skills and
| bigger egos. But... the other half of her blog posts are her
| making snide remarks about how stupidly other people have set
| stuff up. To her (ex-)coworkers, _she_ is THE ONE. So it leaves
| a sour taste in my mouth.
| dclowd9901 wrote:
| I'm not sure this is related or not but if there's one thing I've
| learned about complex objects (engines, computers, software,
| sewing machines, etc), it's that every _custom_ thing you do to
| it has some kind of order of magnitude impact on the overall
| complexity of the machine, and thus, you should avoid customizing
| it. At least if your primary concern is usability vs solving
| specific problems.
|
| It's definitely served me well in a wide variety of disciplines.
| fuzzfactor wrote:
| There is not much of an allowance any more for devices that are
| not on the internet 24/7 without fail.
|
| Multi-booting PCs with Windows and Linux over the years, I have
| seen the time sync problem go from nonexistent to show-stopper.
|
| For devices that _you_ only _need_ to connect to the internet
| occasionally or sporadically (so that 's what you do), that's
| where I noticed it most.
|
| Linux sets the RTC to UTC, then when you reboot to Windows,
| Windows uses the RTC as local time. Which for me is 5 or 6
| hours different from UTC, depending on Daylight Savings Time.
|
| Plus when Daylight Savings comes around, each OS wants to make
| a 1 hour correction the first time you boot it after that date.
| With multi-booting this can add up to more than one hour
| difference too.
|
| Didn't used to be so bad, if you were a few hours (days or
| longer for many PCs) away from the actual time, things did not
| fail for this reason.
|
| Eventually only one hour off was OK for a while, now nope.
|
| There are just so many more obstacles to smooth reliable
| operation, and weak links in a more extensive chain that must
| remain perfectly strong. Otherwise the chain is broken, you are
| disconnected, and the weak link is too many sections away from
| you now to be within reach.
|
| Complete perfection is required more so than ever, while at the
| same time, your efforts to approach perfection are being made
| more difficult.
| gorkish wrote:
| The Pi ecosystem is coming apart at the seams. Talk about falling
| off a pedestal.
|
| By the time you buy everything you need to make a Pi 5 even
| remotely usable and reliable today, you'll have spent as much as
| a small form factor PC. What do you need? 1.
| Special power supply - Yes, it's USB-C but it requires a high
| amperage 5V supply instead of accepting a higher voltage like
| most USB-C stuff 2. Active cooler - Pi 5 throttles
| immediately without it; it's not an optional component 3.
| NVMe Expansion HAT - Seriously why the fiddly little PCIe header?
| Put an NVMe slot on the bottom and let people adapt *that*
| connector 4. RTC Battery - At least put a damn capacitor on
| the board 5. Enclosure - I'm halfway expecting the pi
| foundation to print some dashed cut lines on the cardboard box so
| they can say it comes with a case =]
|
| At the end of this you get a system that can run ... raspbian
|
| Not like the Pi Foundation couldn't have contributed to uboot or
| the upstream kernel or anything before thier hardware was
| released. Pretty much nothing released for "Raspberry Pi" runs on
| a Pi 5 at this point in time.
|
| They better try to salvage something before putting out the
| expected CM5, but I'm not expecting much.
| champtar wrote:
| In OpenWrt (Linux distro for routers that often don't have RTC)
| dnsmasq starts with `--dnssec-no-timecheck` until the ntp client
| gets a first sync
| (https://github.com/openwrt/openwrt/commit/5acfe55d7139a52941...)
| raggi wrote:
| If you don't have an RTC I'd recommend having a tlsdate with some
| bounding heuristics to prevent extreme clock fixation from a
| mitm. You can relatively cheaply hit a large number of public
| servers that are likely to have good times available over TLS and
| trust the common result. You validate the certs without
| considering the notbefore stamps and then if you're feeling
| aggressive validate them after you've managed to approximate a
| date from the cohort. I know there are commercial packages that
| do this, I'm not sure about OSS ones.
|
| Roughtime would be far better, but essentially there's no broad
| deployment of it yet.
|
| Ideally something good would be picked by Raspbian and delivered
| in the distro as standard.
| cyounkins wrote:
| I ran into this before! https://medium.com/@cyounkins/encrypted-
| dns-ntp-deadlock-9e3...
___________________________________________________________________
(page generated 2024-04-11 23:02 UTC)