[HN Gopher] Broken VPNs, the Year 2038, and certs that expired 1...
___________________________________________________________________
Broken VPNs, the Year 2038, and certs that expired 100 years ago
Author : kdp747
Score : 100 points
Date : 2024-02-10 08:10 UTC (1 days ago)
(HTM) web link (www.theregister.com)
(TXT) w3m dump (www.theregister.com)
| eszed wrote:
| This is a great mystery story, with a satisfying ending. And this
|
| > I generally start troubleshooting an issue by asking the system
| what it is doing," explained Zimmie. "Packet captures, poking
| through logs, and so on. After a few rounds of this, I start
| hypothesizing a reason, and testing my hypothesis. Basic
| scientific method stuff. Most are simple to check. When they're
| wrong, I just move on. As I start narrowing down the
| possibilities, and the hypotheses are proven, it's electric.
| Making an educated guess and proving it's right is incredibly
| satisfying.
|
| is an approach every every one of us should internalize.
| stouset wrote:
| Binary search (or bisecting) is also an incredibly valuable
| approach that I don't see junior and intermediate engineers
| reach for nearly as often as they should.
|
| When some thing is failing, find a midpoint between where
| things are working and where the bug is manifesting. Do you see
| evidence of the bug? If not, look earlier in the pipeline. If
| so, look later. Repeat.
|
| In my experience this process is the primary distinguisher
| between those who flail around looking for a root cause and the
| people who can rapidly come to an answer.
| eszed wrote:
| Good call. When you've got no idea where to start, that's how
| to start.
|
| Mostly, though, I think people "flail" because they don't
| know the pipeline well enough to even do that. I know I've
| been in that position before, when approaching completely new
| (to me) systems. (Sometimes there isn't someone more
| knowledgeable you can ask!) That's where I find hypothesis ->
| test -> refine particularly useful. You're still wrong far,
| far more often than you're right, but it stops feeling like
| flailing, and more like making progress towards understanding
| the system well enough to apply other techniques (whatever
| they might be) more smartly.
| ta1243 wrote:
| > I don't see junior and intermediate engineers reach for
| nearly as often as they should
|
| Or senior engineers
| numtel wrote:
| I like that term. I always called this "divide and conquer."
| drewzero1 wrote:
| I think I kind of internalized that idea from my early soft
| eng courses; after seeing how efficiently a computer can find
| a result by cutting the set in half repeatedly, I've tried to
| apply that approach elsewhere when it fits.
| marcus0x62 wrote:
| Once someone understands a complex system well enough to find
| a good midpoint, are they still a junior engineer?
| EnigmaFlare wrote:
| I use this technique all the time to help people who are
| stuck with problems using software - often cause by bugs.
| Divide-and-conquer quickly isolates the issue. I try to share
| the technique when I use it, or just offer it as a
| suggestion.
|
| Part of why it's so useful is you hardly have understand
| anything about the system internally. Just reduce the
| complexity of what you're doing until it works to find the
| lower bound if you don't already have a working case.
|
| That random guessing is like gambling - you hope for a big
| quick payout but when your hypothesis fails, you end up worse
| off than before. Wasted time and no closer to the solution.
| devjab wrote:
| How do you debug if this isn't what you're doing? I'm genuinely
| curious... are you using some sort of advanced tools like a
| psychopath?
| recursive wrote:
| 1. Internet search 2. Make random changes 3. Test 4. Start
| over.
| Espressosaurus wrote:
| I can't even imagine how that would work with any
| complicated system.
| recursive wrote:
| Not well. It's pretty frustrating to observe its
| practitioners in the wild.
| Espressosaurus wrote:
| That's just how you debug any system.
|
| If you're on this site and haven't already internalized
| it...how do you debug?
| 8organicbits wrote:
| I'm not sure I see why it was revoking the certificates, when you
| renew a certificate that's about to expire you can just let the
| old one expire, right?
| tialaramex wrote:
| I'd say that more often than not people building this sort of
| stuff in-house have no idea what they're doing. So although
| that part of the design doesn't make much sense it's not
| astonishing to see it.
|
| A PKI provides a deeply technical solution to a hard problem
| you probably don't have. This technology is most often deployed
| when somebody has a different, easy problem, but they don't
| like the relatively easy non-technical solution.
| pixl97 wrote:
| This can go back to your old buddy NTP, specifically DHCP
| assigning this on untrusted networks. If you control the
| network (time?) and you manage to get the full expired
| certificate you may be able to MITM the victim successfully. If
| you force the CRL check first then things won't match up. I
| have no ideas on the feasibility of faking the CRL though, so
| it might be a wash.
| nijave wrote:
| Seems like it'd be fairly difficult in practice to change
| time on a host such that you can use an expired certificate
| without breaking a bunch of other stuff
| dancemethis wrote:
| I have a very soft spot for this kind of "campfire story". Open
| Office not printing on Tuesdays comes to mind. Anyone got some
| more?
| jrlocke wrote:
| I'm partial to the 500 mile email:
| https://www.ibiblio.org/harris/500milemail.html
| WildGreenLeave wrote:
| This story comes to mind:
| https://web.mit.edu/jemorris/humor/500-miles
| indrora wrote:
| A tale I use in interviews is "The Homesick Laptop's
| Replacement Desktop That Ate Hard Drives in Summer".
|
| long story short, Dell ship-of-theseus'd an entire machine
| looking for an issue that only happened on cloudy-hot days when
| the disks were under high load. It was an air conditioner out
| of phase with the rest of the system causing EMI that the power
| supply just let on through.
| bdw5204 wrote:
| The solution to the year 2038 problem is to upgrade your time
| since the Unix epoch fields to 64 bit integers. Hopefully this
| won't be an actual issue 14 years from now because it's such a
| simple fix.
| ooterness wrote:
| It's just like Y2K, and just as pervasive, but harder to
| explain to upper management. My guess is that it won't go
| smoothly.
| ta1243 wrote:
| There's a perception that y2k was overblown because we spent
| tons of money and didn't have the problems that were
| suggested in the media
|
| Of course the fact that the problems were overhyped, but
| importantly FIXED by all that money, doesn't come into it, it
| was a cry-wolf situation.
| EnigmaFlare wrote:
| The media wasn't telling us that the problems were getting
| fixed though. It kept on hyping the doomsday in whatever
| way it could. We were supposed to wake up in 2000 to find
| our fridge door open and melted ice all over the floor.
| Why? Fridges didn't have clocks that told them to turn
| themselves off, or clocks at all.
| wongarsu wrote:
| About a decade ago I was involved in the development of an
| embedded product for industrial use cases. The kind of stuff
| you install once and use for 20-40 years. The library we used
| for displaying the time breaks around 2036 (so a bit ahead of
| y2k38). But the person responsible would long be in retirement
| by then and the issue doesn't impact critical functions, so it
| was decided not to do anything about it. This version of the
| product is still sold today. I doubt this story is uncommon.
| pxeger1 wrote:
| > the CRL size for the median certificate is 51KB and that half
| of all CRLs are under 900B.
|
| What? So there are no CRLs between 900B and 51KB, and the first
| one larger than 51KB just happened to be the median one??
| hcs wrote:
| Not sure, but: median certificate (so each CRL has a
| multiplicity of however many certificates would use it, or
| perhaps of how many times it is actively retrieved) vs median
| CRL size (each CRL listed once)
|
| Or they meant mean for the first one, I guess.
|
| Edit: it's the former, from the paper:
|
| > We immediately observe that half of all CRLs are under 900 B.
| However, this statistic is deceiving: if you select a
| certificate at random from the Leaf Set, it is unlikely to
| point to a tiny CRL, since the tiny CRLs cover very few
| certificates.
| denton-scratch wrote:
| > I suspect the NTP server had a badly faulty internal clock
| which ran very fast.
|
| A time server with a defective clock seems to be a serious
| problem. Zimmie says the time server was an appliance; so someone
| is selling as an appliance a time server that can't tell the
| time.
| m3047 wrote:
| Ran into a case where a whole datacenter became untethered from
| its NTP upstream and drifted off into a timezone of its own
| creation. Customer was failing authentication for a data product
| we sold them (TSIG was failing). I was on the phone with them for
| an hour, reassuring them constantly that everything was working
| for our other customers, tailing logs, and reporting what I saw.
|
| More datacenter stakeholders kept joining the call, most of whom
| had nothing to do with our data product. Many times I heard
| people ask "have they found the problem yet" as though.. what? We
| were the best tech support they had for an entire data center
| going dark? After an hour somebody noticed that the clocks on
| servers in the datacenter didn't match up with their laptop;
| shortly after that I was able to extricate myself from the
| call... still watching the logs, their downloads started working
| again a short while later.
| yjftsjthsd-h wrote:
| > Many times I heard people ask "have they found the problem
| yet" as though.. what? We were the best tech support they had
| for an entire data center going dark?
|
| Possible. Some companies are mostly lacking in competent
| technical people, so anyone who knows what they're doing will
| quickly find themselves pulled into every possible task; I see
| no reason why this shouldn't include external parties.
| tetha wrote:
| Especially if things enter the very strange territory, like
| NTP running wild.
|
| I pretty much remember some time ago, one of our customers
| had trouble with our on-prem installation. Eventually it
| seemed that the database had been corrupted. At that point I
| could tell the poor guy on the other side was running what I
| was doing with my colleague on a couple of other, similar
| systems, but I noted he was getting pretty nervous so I
| figured as long as the clock runs, whatever. It's kinda what
| I do a lot at work and I don't like letting people in the
| rain like that.
|
| And eventually we could confirm that large amounts of VMDKs
| had been corrupted in various ways. Seemed like another
| vendor hat let the SAN they were managing run full or into
| some other catastrophic situation. And their backup appliance
| also didn't work.
| macintux wrote:
| Not nearly as interesting a story: in 1996 I visited a customer
| who was using up for dialup services, but reported some of their
| Windows desktops couldn't connect.
|
| It didn't take me long to figure out that the computers that
| weren't working had their clocks set well into the 21st century.
| The shell couldn't even display the year properly, I assumed a
| Y2K incompatibility, but after so many years now I can't remember
| exactly what I saw.
|
| Anyway, easy fix, but I never did find out what caused such a
| weird glitch in their environment. It's small wonder that many
| people aren't fluent with computers: they misbehave in such a
| wide variety of ways.
| kro wrote:
| Last year I had a (of many) freshly provisioned Linux VMs clock
| change to the year 2257 2 nights in a row. Never figured that
| out sadly, reprovisioning "fixed" that.
| frereubu wrote:
| Related and fascinating article that came up on HN recently after
| the originator of NTP, David Mills, died:
| https://www.newyorker.com/tech/annals-of-technology/the-thor...
|
| (Just turn off JavaScript to read it if you hit a paywall).
| arter4 wrote:
| They 2038 thing I get, but the clock drift of BILLIONS of seconds
| really scares me. What kind of fucked up setup can lead to
| something like this?
| vdaea wrote:
| Why does this NTP implementation accept a sudden change of 4
| billion seconds? For example, the NTP implementation in Windows
| refuses to change the clock by more than 54,000 seconds.
| cesarb wrote:
| This reminded me of this article from last year:
| https://arstechnica.com/security/2023/08/windows-feature-tha...
| (HN discussion: https://news.ycombinator.com/item?id=37151220)
___________________________________________________________________
(page generated 2024-02-11 23:00 UTC)