[HN Gopher] Working around expired root certificates
___________________________________________________________________
Working around expired root certificates
Author : tomwas54
Score : 68 points
Date : 2021-10-11 21:27 UTC (3 days ago)
(HTM) web link (scotthelme.co.uk)
(TXT) w3m dump (scotthelme.co.uk)
| jlgaddis wrote:
| I've got an old iPad that's still running iOS 10.0.1 (yes, I
| know). It's used very infrequently and mostly just for reading
| (PDF documentation, primarily).
|
| Coincidentally, I happened to be in the middle of looking
| something up right as the clock struck "NotAfter".
|
| My first clue that "something was up" came when clicking a link
| to another page on one of the sites I had opened... I switched
| tabs, clicked a link to some other page on that site... nope.
| Then, I tried visiting a few random sites that are bookmarked.
| Some loaded just fine, others through errors.
|
| Out of habit, I glanced at the clock -- my ISP's maintenance
| windows typically start at 2am and they are _very_ good at
| starting at exactly that time! It wasn 't 2am, but it was _just_
| past the top of the hour at this point.
|
| I don't recall now if the errors displayed were certificate-
| related specifically or just "generic" in nature, but I suddenly
| recalled seeing an article in the past day or two reminding us
| all of the impending doom and destruction that was about to be
| unleashed upon us when the root certificate expired.
|
| Using the Firefox browser (which _was_ up-to-date) on my
| workstation, I tried loading a couple of the sites that were
| refusing to load on the iPad but I encountered no issues
| whatsoever -- everything was fine. A quick check of the
| certificate chain on the "affected" sites confirmed the cause of
| my problems.
|
| Back on the iPad, I downloaded a copy of the "new" root
| certificate from my workstation, confirmed, confirmed, confirmed,
| and was back to whatever it was I was looking into by 10 minutes
| past the hour.
|
| Somehow I managed to escape all of the Y2K-level atrocities that
| were supposed to occur and had totally forgotten about the entire
| "incident" until seeing this here -- and, as you might guess,
| haven't ran into any other issues related to this since then.
|
| Apocalypse averted (for now), I suppose.
| tialaramex wrote:
| Yes. The procedure you describe could also have been done any
| time prior to expiry, and more easily of course since the web
| still worked back then.
|
| The time is completely arbitrary by the way, the people who
| minted the certificate could of course choose something else,
| but by default it's likely if it happens to be 14:26:39 UTC
| when you make the certificate, it's also going to expire at
| 14:26:39 UTC however many years in the future.
|
| It's a little disappointing that sites catering to people who
| run old stuff (e.g. both old iPads and old Macs) didn't put
| much work into e.g. low friction ways to make this happen.
| Perhaps that's something where Let's Encrypt should have
| reached out to the right people and made sure this was on their
| agenda back in the summer.
|
| But perhaps most people in your situation don't pay any
| attention to such things anyway and would still have been
| blind-sided.
| paol wrote:
| This cert expiration semi-broke my home backup setup. (The
| backups work fine but the reporting broke, which is subtly more
| dangerous.)
|
| Duplicati, which is a .NET app, doesn't seem to have the latest
| certificates when running on Fedora. The same software on Ubuntu
| works fine.
| zinekeller wrote:
| That's a very good workaround on OpenSSL-based systems, although
| it's kind-of moot now since if you can update the IdenTrust/DST
| root, you could simply also remove it and add ISRG's roots.
| [deleted]
| forgotmypw17 wrote:
| This is why keeping HTTP open is an accessibility must-have,
| despite the low risk of ISP MITM and such.
|
| There are millions of devices out there already which are usable
| for many tasks like reading, searching, and non-critical writing,
| as long as HTTP is allowed.
|
| HTTPS/SSL introduces many accessibility issues, such as for
| people with older devices, those who do not know how to set their
| clock correctly, and many other scenarios.
| sevenf0ur wrote:
| PKI (public key infrastructure) is much more fragile than
| people realize. Each browser vendor has its own store of
| trusted certificates. Java, Node.JS, .NET/Mono have their own
| stores. These are completely separate from the operating
| system's store for good and bad reasons. Also, certificate
| revocations are not even handled like you'd expect:
| https://www.ssls.com/blog/why-ssl-certificate-revocation-che...
| forgotmypw17 wrote:
| Thank you for sharing this. Despite suffering many issues
| with older devices and browsers, I never realized it is this
| bad.
| mcspiff wrote:
| Who decides what's non-critical? I'd prefer my news articles
| not be rewritten by the government for example. Catering
| security to the absolute lowest common denominator ("can't set
| a clock") is not a path to success.
| nonameiguess wrote:
| I don't see how this kind of argument generalizes. Short of
| armored truck deliveries, cryptographically signed digital
| transmissions are about the only form of information delivery
| where it's even possible to get this kind of assurance. Yet
| civilizations have operated for centuries if not millennia on
| the premise that we deliver things nonetheless. There is no
| guarantee at all that the government isn't rewriting the New
| York Times before it hits newsstands, intercepting and
| changing television signals, swapping out or reading your
| mail, injecting mind control serum into foods before they
| reach the grocery store, or that fluoride isn't used to
| subdue the population (other than science, but you don't know
| the government isn't intercepting and rewriting published
| research). The only assurance you get is all these things are
| illegal. Governments definitely don't universally follow
| their own laws, but at some point, the existence of some
| system of laws is either enough for you, or you go live in
| the woods with a bunker full of seeds and ammo, or start a
| revolution to replace the government with a new one.
| CamperBob2 wrote:
| _I_ decide what 's non-critical. What a concept, huh. If I
| consider an online transaction to be critical, I'll check for
| https:// in the address bar.
|
| Usually I don't.
|
| Case in point: Firefox 93 now issues gratuitous scary
| warnings when a .PDF is downloaded over a non-https
| connection. [1] Right now it only seems to happen if you
| arrive at the link via a search engine, but it would be silly
| to pretend they'll stop there. There is nothing OK about
| this. It literally breaks the whole idea of decentralized
| Internet protocols.
|
| The obsession with "https everywhere" needs to stop, _now_.
| Otherwise, not only will our future landfills groan under the
| weight of megatons of e-waste that didn 't need to stop
| working when it did, but our collective cultural history
| online will eventually consist of nothing but undecodable
| random numbers.
|
| Not everything needs unbreakable encryption. The vast
| majority of online content doesn't.
|
| 1: https://i.imgur.com/NwXeyGx.png
| yjftsjthsd-h wrote:
| > The obsession with "https everywhere" needs to stop, now.
| Otherwise, not only will our future landfills groan under
| the weight of megatons of e-waste that didn't need to stop
| working when it did, but our collective cultural history
| online will eventually consist of nothing but undecodable
| random numbers.
|
| If those devices can't even update certs, they absolutely
| should not be online because they're solid blocks of
| vulnerable software that will just contribute to botnets.
|
| And the only way for this to contribute to losing history
| is if you're somehow archiving content by grabbing it off
| the wire, which seems inefficient anyways.
| CamperBob2 wrote:
| _If those devices can 't even update certs, they
| absolutely should not be online because they're solid
| blocks of vulnerable software that will just contribute
| to botnets._
|
| Ah, yes, the presumption of guilt. "You're going to do a
| bad thing at some point, I just know it. This is probably
| because you're a moron, while I'm not. Fortunately, I
| have the solution."
|
| We hear that a lot these days.
| forgotmypw17 wrote:
| Suppose the likelihood of the government or ISP rewriting my
| requests is very low, as it is in the U.S.
|
| Suppose also that I am an underprivileged individual whose
| only device is a 10-year-old tablet I happened to find in the
| closet.
|
| I'm searching for a crucial piece of information, e.g. how to
| receive a subsidy benefit or health symptoms.
|
| This is a real-world scenario I have myself witnessed, and
| have also read about online many times.
|
| Are you REALLY arguing for denying this human being access to
| the information in the name of "security"?
| ArchOversight wrote:
| > Suppose the likelihood of the government or ISP rewriting
| my requests is very low, as it is in the U.S.
|
| Comcast used to (and maybe still does) inject JavaScript
| into HTTP requests when users were approaching their
| transfer caps, so that a warning banner would be shown to
| let them know they are almost at their terabyte limit.
|
| ISP's have been and continue to be caught playing tricks
| like this. T-Mobile for instance used to rewrite the
| playlists that YouTube would send down for HSL to remove
| the higher bitrates to reduce network traffic.
|
| Just because you are in the US does not mean you are immune
| from this sort of shenanigans, currently it is used mostly
| in the name of network management, but it could also be
| used for nefarious purposes.
|
| I also fully believe this is why public libraries should
| exist. The library near me has new computers, help that can
| help the person navigate those sites and even has tablets
| available for loan.
| forgotmypw17 wrote:
| So, you have a choice between:
|
| ISP may inject bandwidth cap limit
|
| -or-
|
| No access to information at all
|
| What do you think the person looking for critical
| information would choose?
|
| By the way, libraries are scarce in the U.S., and library
| computers have been some of the most out-of-date and
| poorly maintained machines I've used. This is a problem I
| have no chance of solving, while allowing HTTP access is
| something I can do today.
| Dylan16807 wrote:
| You act like "no access" can't happen because _other_
| things break. If that was the only choice, I 'd consider
| it, but I'm not going to give up everything in tiny
| incremental bits to maximize access in every edge case.
|
| Also you could disable expiration as a much safer
| measure.
| enzanki_ars wrote:
| ISP MITM is not low risk, and has been already done by Comcast
| at the very least in the US to inject warnings about bandwidth
| caps [0]. And with the horrible security of home routers, ISP
| provided or not, MITM attacks are highly likely on home
| connections sourced from botnets and malware attacks on
| routers.
|
| [0]: https://rietta.com/blog/comcast-insecure-injection/
| forgotmypw17 wrote:
| If you had to choose between "ISP may inject a notice" and
| "no access to critical information at all", which would you
| go for?
| enzanki_ars wrote:
| In terms of the topic in this thread, it's not the
| customers responsibility to deal with expired root
| certificates. That said, I understand that there is a large
| issue with devices that drop support way too soon, even
| though the hardware is good. But the solution is not the
| weaken the security, but instead to force better standards
| for how hardware is maintained, and ensuring that there is
| a long lifetime where either the manufacture supports the
| device, or the device is completely unlocked and allowed
| for easy community sourced modifications and updates. That
| sort of critical information should be secure, because
| taking an example from elsewhere in this thread, I'd rather
| be confident I was reading the exact page as intended from
| the government source regarding how to apply for
| unemployment benefits and not have to worry that malware in
| the router is modifying the information on the page to
| steal information and use it to redirect those unemployment
| benefits.
|
| And this theoretical attack on home routers is not out of
| the question at all. How many unmaintained unpatched IoT
| devices have been abused with malware/botnets. The clock is
| ticking on mass exploitation of home routers being attacked
| and it's firmware replaced with one injecting/stealing
| information from insecure webpages. If the devices can't be
| updated, we should make sure there are _safe_ alternatives
| to accessing the information, rather than hoping that no
| actor is doing things they should not.
|
| I can list so many scenarios with critical information and
| attacks that could be made if the webpage was not HTTPS
| with proper certificates. Including school districts in the
| United States being force to block inappropriate content in
| order to receive federal funding, and those firewalls
| abusing non-http content to decide what to block, and
| school districts abusing that capability to block anything
| and everything they want. How about a student trying to
| understand more about LGBTQ+ individuals and the school
| district inspecting and censoring the exact content inside
| the pages to remove words like "lesbian" or "gay" because
| the school considers them "questionable." Or a school
| blocking articles pertaining to hacking. I have seen that
| exact last scenario in fact, where certain articles posted
| here were blocked in my highschool years ago because they
| were considered hacking and that was apparently not
| appropriate to view in school. These are real scenarios and
| not hypotheticals.
|
| For more info on some other various types of fun attacks,
| see https://www.troyhunt.com/heres-why-your-static-website-
| needs...
| forgotmypw17 wrote:
| > That said, I understand that there is a large issue
| with devices that drop support way too soon, even though
| the hardware is good. But the solution is not the weaken
| the security, but instead to force better standards for
| how hardware is maintained, and ensuring that there is a
| long lifetime where either the manufacture supports the
| device, or the device is completely unlocked and allowed
| for easy community sourced modifications and updates.
|
| Yes, that would be nice.
|
| Unfortunately, I have exactly ZERO control over this, and
| it won't happen for years or decades to come, if at all.
| And almost certainly not for existing devices.
|
| What I DO have control over is supporting all those
| devices today, right now, with some MITM risk in certain
| scenarios.
| lousken wrote:
| Kinda related to this - anyone tried to fix their xamarin app so
| that it'd work with letsencrypt? Developers told me xamarin tries
| to validate both chains so the hack breaks the validation. Not
| sure if they've come up with a workaround
| kadonoishi wrote:
| Possibly related to that - I just fixed my letsencrypt problem
| on an old Mac by following these instructions [0], which were
| referenced here [1].
|
| [0]
| https://docs.certifytheweb.com/docs/kb/kb-202109-letsencrypt...
|
| [1] https://apple.stackexchange.com/questions/428728/why-are-
| my-...
| a3w wrote:
| Yes. Android <5.1 or s.th. like that failed due to not having
| chrome available I think. 6.0 and 7.0 seem to have a different
| issue with the one of the letsencrypt certificate chains.
|
| OIDC for us seems to work on iOS 15, or not, depending on
| iPadOS 15 , iOS (Simulator), or iOS-out-in-the-wild .
| cmeacham98 wrote:
| This won't fix the legacy devices that already exist, but I
| wonder if there should be some way for root certificates to sign
| a replacement of themselves. All other signatures would be
| treated as invalid after the expiration, but this special "this
| is my replacement" signature would allow non-updated clients to
| continue to work.
| Semaphor wrote:
| If you can do that, what would be the point of root expiry?
| eli wrote:
| I think that's a valid question
| shadowgovt wrote:
| Out of curiosity, does anybody know how Apple laptops got updated
| with the replacement for the let's encrypt certificate?
|
| I was surprised to discover that my wife's machine had not
| automatically updated, several of her websites went inaccessible
| on September 30th, and I had to do the update manually. I'm not
| sure what step we missed that and updated certificate wasn't
| automatically sent to her.
| jeromegv wrote:
| Which OS? For El Capitan for example, no updates were released
| by Apple.
|
| I will be using those instructions instead to do it manually
|
| https://apple.stackexchange.com/questions/428460/how-to-upda...
| singlow wrote:
| My iMac was still on El Capitan and I updated it to Big Sur to
| resolve it.
| bradstewart wrote:
| Same thing happened to my wife's laptop as well. As far as I
| know, Apple included the new certs in an OS update.
|
| They did not issue such an update for older versions of macOS
| (I forget which version of macOS she's running, but its
| relatively old).
| njx wrote:
| This thing broke my development and testing environment.
|
| I am in the middle of developing a plugin
| (https://www.crawlspider.com) that catches SEO changes,
| certificate changes and errors (crawlspider)
|
| Suddenly the plugin was catching SSL errors for all the domains I
| was monitoring. Upon checking the domains in the browser
| everything look fine. There was a green secure icon on all of
| them.
|
| It was only after several tests and chat with my admin, it became
| clear that this is a root certificate issue and somehow connected
| to letsencrypt based certificates.
| TazeTSchnitzel wrote:
| Enforcing expiry of root certificates on a device which no longer
| gets root certificate store updates is essentially creating
| "planned obsolescence". I know it's not the most comfortable
| thing, but for the sake of avoiding things becoming e-waste too
| early, it's important that devices allow ignoring root cert
| expiration.
| tialaramex wrote:
| Right.
|
| Fusing things like this makes sense. If you have a year old
| Chrome that's still running but is blocked from updating
| itself, it doesn't do some of its security checks, "Huh," it
| says to itself, "I am out-of-date" and some checks it would
| normally require are disabled because it has no way to know
| what should be checked a year later.
|
| This makes it less safe than a brand-new Chrome. But, duh, of
| course it is, it also doesn't have the last twelve months of
| security fixes, meaning there are twelve months worth of
| publicly known security flaws in your browser.
|
| Android's choice in this makes a lot of sense. If you get root
| trust store updates, then you stop trusting old roots and trust
| new ones instead but if you're no longer getting updates then
| just do the best you can with what you have.
| jlgaddis wrote:
| * Once you think about this more though, you realise that it does
| make sense and that it just doesn't seem logical upon first
| thought. In order to make this change and it actually have an
| effect on the client, you need to have access to that client with
| administrative privileges to change the root store. If your
| concern is that an attacker might do something like this, well,
| I'd suggest you have far bigger issues to concern yourself with
| given that the attacker has admin/root and is on your device!*
|
| Yes, it's true that not just anyone can modify your local trust
| store (and if someone does, yes, you've likely got bigger
| issues).
|
| When I started to "think about this more though", my first
| thoughts were more along the lines of ...
|
| _" What assertions have the CAs made -- whether implicitly or
| explicitly -- regarding their security measures / practices (WRT
| private key material, of course) once 'NotAfter' has passed?"_
|
| ... and similarly ...
|
| _" What obligations do the CAs have -- either morally,
| ethically, contractually, and/or in accordance with the Baseline
| Requirements -- to continue to protect the 'C-I-A' of the private
| key material after 'NotAfter'?"_
|
| I'm fairly familiar with the BR's, having spent more time reading
| through (portions of) them than I'd have preferred -- and I keep
| a hard copy here in one of my desk drawers -- I'll happily admit
| that don't know the answers off the top of my head.
|
| To be clear, I trust [0] that no ("responsible") CA would ever do
| something like publicly share the private key used to sign a
| "trusted" root certificate -- or even an intermediate /
| subordinate, for that matter -- regardless of how much time has
| passed since 'NotAfter' ...
|
| However, when a CA generates such a certificate (and/or the
| private key, for the pedants), requests that it be included in
| one or more of the various root programs, and agrees to the
| conditions / requirements of such programs -- including,
| obviously, the BR's -- they're promising that they will keep
| those certificates (that is, again, the key material) secure.
| This isn't an "eternal promise", though, and the 'NotAfter' date
| that they include in the certificate is how they tell everyone
| how long that promise is valid for -- that is, it is, literally,
| an expiration date [1].
|
| Once that expiration date arrives, though, their obligation has
| been fulfilled. I may very well be forgetting some relevant
| provisions of the BR's at the moment [2] but once the CA's
| promise has expired, is it appropriate to expect them to continue
| keeping their promises (i.e., keep trusting the certificate) if
| they haven't asserted that they _will_ continue to keep those
| promises (by renewing the certificates)? [3]
|
| --
|
| [0]: Although "expect" (perhaps just "hope") is probably a more
| fitting word choice here.
|
| [1]: Yes, they can extend -- "renew" -- that promise later, if
| they so choose.
|
| [2]: And this is the point at which I'd typically go review them
| to clarify the matter (just for myself) but this entire comment
| is mostly rhetorical (a "thought exercise", if you will).
|
| [3]: Personally, I think that "trusted" root certificates should
| automatically become "untrusted" (and, of course, get flagged as
| such) immediately upon expiration but that's a whole 'nother
| argument involving operating system vendors, browser makers, and
| many other parties. Luckily, the operating system and software I
| use allows _me_ to be the ultimate authority over matters such as
| these!
| NovemberWhiskey wrote:
| > _I 'm fairly familiar with the BR's, having spent more time
| reading through (portions of) them than I'd have preferred --
| and I keep a hard copy here in one of my desk drawers -- I'll
| happily admit that don't know the answers off the top of my
| head._
|
| CAs are required to have a statement on practices for
| deactivation and destruction of private keys per the BRs; it's
| 6.2.9 and 6.2.10 respectively.
|
| e.g. for Let's Encrypt: https://letsencrypt.org/documents/isrg-
| cps-v2.6/#6-2-9-metho...
| 51Cards wrote:
| We got bit by the Let's Encrypt root cert expiry. I knew it was
| coming but based on what I had read and the stats of our users I
| expected the impact to be slim to none. We used it for some of
| our support domains, but not the primary ones. Then Sept 30th hit
| and everything else hit the fan. Thankfully we were able to get
| another cert provider to issue a replacement within 8 hours but
| our customer service team hated us for most of that day.
___________________________________________________________________
(page generated 2021-10-14 23:02 UTC)