[HN Gopher] Intent to end OCSP service
___________________________________________________________________
Intent to end OCSP service
Author : soheilpro
Score : 199 points
Date : 2024-07-23 15:25 UTC (7 hours ago)
(HTM) web link (letsencrypt.org)
(TXT) w3m dump (letsencrypt.org)
| c0balt wrote:
| Interesting, how does the support for CRLs in web servers look?
|
| Afaik, NGINX and Apache only have OCSP stapling support.
| c0balt wrote:
| Note: for nginx there is
| https://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_...
| which needs to be pointed to a pem-encoded file with the CRL
| list (list of revoked certs).
|
| So no API-based configuration, like OCSP stapling, that just
| works. I can probably try to configure this with a
| cronjob/systemd timer but this is significantly less ergonomic.
| Asmod4n wrote:
| Why would CRL require web server support?
| iso8859-1 wrote:
| So that admins find out that the certificate they have
| configured is no longer valid.
| Asmod4n wrote:
| You'd need a monitoring solution for that, your web server
| won't just send you a mail when it's cert got revoked.
|
| The client does that for you, by checking back at the CA.
| No need for any configuration server side.
| mcpherrinm wrote:
| For webservers, our recommendation is that your ACME client
| queries the ACME Renewal Information endpoint to find out
| if the certificate needs to be renewed:
|
| https://letsencrypt.org/2023/03/23/improving-resliiency-
| and-...
| eqvinox wrote:
| If your certificate has been revoked and you don't know
| that (e.g. from a very alarming e-mail), something has
| _seriously_ gone wrong. In the general case, it 'd be
| yourself requesting the revocation to begin with. But even
| if it isn't, I'm pretty sure the CA is supposed to tell you
| expeditiously.
| phasmantistes wrote:
| Not all subscribers provide email addresses through which
| they could be informed of a revocation.
| mcpherrinm wrote:
| Ideally your server handles this automatically, which is
| why Let's Encrypt is working on extending ACME to help
| with this case.
|
| Some good implementations of ocsp stapling can already
| automatically get a new certificate if they receive a
| "revoked" ocsp response.
|
| Requiring humans to read their email and be looped in is
| work I want to avoid needing at all.
| c0balt wrote:
| To verify client and server certificates. For certificate-
| based auth this is quite important, but it's also nice to,
| e.g., know that your server cert is not revoked.
| Asmod4n wrote:
| Your client does the checking of the server cert, no need
| for any configuration server side.
|
| If you use certs for client auth it's unlikely it's used on
| the web and only in a company setting where you control the
| PCs, where you could just use something more suited for
| that case.
| eqvinox wrote:
| It's the client that is supposed to fetch the CRL. The
| server doesn't care unless it does client certificate
| authentication (which is incredibly rare due to poor UX.)
| hunter2_ wrote:
| A server (nginx / httpd) might very well make outbound
| HTTPS calls and want to verify the origin's certificate
| to the fullest extent possible. This is known as a
| reverse proxy. However, many -- certainly not all --
| reverse proxy configurations use origin servers that
| reside on the same exact network, and therefore don't
| need TLS connections (or use them with `SSLProxyVerify
| none` for e.g. self-signed certs to simplify things [0]),
| but I digress.
|
| [0] https://httpd.apache.org/docs/current/mod/mod_ssl.htm
| l#sslpr...
| arsome wrote:
| Disappointing to hear considering the limitations of CRLs - is
| there any intention to go forward with OCSP stapling or is that
| completely abandoned at this point?
| tialaramex wrote:
| My understanding is that stapling is the victim of the usual
| incompetence and laziness that infects a lot of systems where
| if one in a billion fail closed that would be considered a
| disaster but one in ten fail open is considered fine. You can't
| achieve meaningful security this way.
|
| The browser vendors have learned that you have to do it
| yourself or it won't be done well enough to be useful. So you
| pull every CRL, do a bunch of compression or other tricks, then
| give your users that data and now they have working revocation.
|
| When Bob's CA and Kebab Shop breaks their revocation stack,
| instead of dozens of poor individual users or web site owners
| confused and calling Bob's outsourced call centre in Pakistan
| with no sign of a fix, now a Google account exec asks Bob's CTO
| whether they forgot to say they were getting out of the CA
| business...
|
| I agree this isn't a desirable outcome, but it might be all we
| have.
| eqvinox wrote:
| > The browser vendors have learned that you have to do it
| yourself
|
| Cool. We already got the internet ossified on TCP + UDP,
| other L4 protocols just get stuck in firewalls and whatnot.
| Now we're progressing in ossification of HTTP. <insert
| expletives here>
|
| To be clear: this OCSP decision seems to be driven directly
| and only by web/HTTP consumers. Anything else is just not
| considered.
| tialaramex wrote:
| It is called the Web PKI after all. If somebody else
| actually wants to do all the hard work they're welcome, but
| my impression is that there's only enthusiasm for bitching
| and whining which won't get the work done.
| eqvinox wrote:
| > It is called the Web PKI after all.
|
| Let's Encrypt issues X.509 certificates, not "Web PKI
| certificates".
| codegeek wrote:
| Can someone ELI5 what does this mean for people using LetsEncrypt
| today with servers like Nginx or Caddy ? Do we need to make any
| changes to adjust ?
| mcpherrinm wrote:
| For regular webserver users, accessed by web browsers, no
| changes are needed.
|
| Note this is still a long ways out. At this time, only people
| who are writing code against OCSP need to be aware of the
| future roadmap.
| candiddevmike wrote:
| This mostly only impacts verification of Lets Encrypt
| certificates. No changes for ACME users.
| eqvinox wrote:
| I... er... what?
|
| First of all, privacy was one of the points of OCSP stapling.
|
| Second, this breaks all non-http applications in the sense that
| they could previously work through OCSP stapling which would be
| communicated in-line. CRLs need to be fetched by the client,
| generally over HTTP.
|
| Third, most non-browser TLS clients simply do not fetch CRLs, the
| implementation hurdle was too high.
|
| ... I'm left seriously befuddled by this decision by Let's
| Encrypt [edit: or rather the CA/B forum] :(
| andrewaylett wrote:
| OCSP pretty much has to fail open, and stapling doesn't fix
| this.
|
| If you're in a position to MITM a client using an exfiltrated
| certificate that no longer passes OSCP, you're probably also in
| a position to block the request to the OSCP server. And you're
| not going to staple while you MITM.
|
| As a client, you can't really tell the difference between "this
| certificate is valid but not stapled and the OSCP server is
| down" and "this certificate isn't stapled because I'm being
| MITM'd, and the OSCP server is blocked".
|
| For those who could successfully staple, _really_ short-lived
| certificates might be a suitable answer -- that 's effectively
| what OSCP gave you, only without actually ensuring that the
| certificate would cleanly expire.
| datadrivenangel wrote:
| "As soon as the Microsoft Root Program also makes OCSP optional,
| which we are optimistic will happen within the next six to twelve
| months, Let's Encrypt intends to announce a specific and rapid
| timeline for shutting down our OCSP services. We hope to serve
| our last OCSP response between three and six months after that
| announcement. The best way to stay apprised of updates on these
| plans is to subscribe to our API Announcements category on
| Discourse."
|
| Interesting to see Microsoft dragging here.
| nick238 wrote:
| I don't see anything about OCSP policy changing in the Mozilla
| CA Program Policy release notes [1] or Chrome's [2], so I
| wonder if Microsoft was more proactive (for better or worse) in
| requiring it vs. other popular root programs.
|
| [1]: https://wiki.mozilla.org/CA/Root_Store_Policy_Archive [2]:
| https://www.chromium.org/Home/chromium-security/root-ca-poli...
| agwa wrote:
| Mozilla and Chrome were requiring it via the incorporation of
| the Baseline Requirements into their policies. The removal of
| the requirement from the BRs won't appear in the release
| notes of their policies.
| agwa wrote:
| After I created OCSP Watch[1], I regularly detected CAs returning
| an OCSP response of unknown or unauthorized for certificates
| found in CT logs, indicating that they had basically forgotten
| that they had issued a certificate. I find that rather troubling.
| Indeed, OCSP Watch is currently reporting several forgotten
| NETLOCK certificates. (The certificates from other CAs are
| recently issued and will probably have OCSP responses provisioned
| in the near future.)
|
| CRLs can't be used to detect this, because they only list revoked
| certificates rather than providing a definitive status for every
| issued certificate.
|
| I do wish the root programs had merely removed the requirement to
| include an OCSP URL in certificates, but required OCSP URLs for
| every issuer to be disclosed in the CCADB. That would have solved
| the privacy problems and made OCSP responders much cheaper to
| operate, while continuing to provide transparency into the status
| of certificates.
|
| [1] https://sslmate.com/labs/ocsp_watch/
| nine_k wrote:
| BTW what is the need for a CA to remember issued certificates?
|
| Certificates should work as long as they are not expired, not
| on a CRL, and while the same applies to their.CA's certificate
| used to sign it.
|
| The only good use for remembering issued certificates that I
| see is that a CA could detect a compromise if it's handed a
| certificate it does not remember issuing. It looks far-fetched
| to me, but I know nothing about CA operation.
| phasmantistes wrote:
| CAs need to know every certificate they've issued so that
|
| a) if they receive a request to revoke that certificate, they
| can actually do so; and
|
| b) if they need to revoke _all_ of their certs (e.g. due to
| discovering a validation process failure) they don 't miss
| any.
| masklinn wrote:
| Isn't it more "a lot" of certs? Surely if they need to
| revoke all certs they can just revoke the intermediate.
| patrakov wrote:
| There is a middle ground between "all certs" and "a few
| certs". An example would be
| https://community.letsencrypt.org/t/revoking-certain-
| certifi... (2.4% of all certificates). This needs a full
| list of affected certificates but is too small to revoke
| the intermediate.
| masklinn wrote:
| > There is a middle ground between "all certs" and "a few
| certs".
|
| And the GP used the word "all" which is what I was
| replying to.
| agwa wrote:
| When a CA has an incident, such as learning that one of their
| domain validation methods is flawed, they need to be able
| revoke all certificates impacted by the incident. Without a
| reliable database of certificates, there's no guarantee
| they'll find all of the impacted certificates. OCSP could
| fail closed in this situation. CRLs always fail open.
|
| Aside: it was in fact envisioned that OCSP could be used to
| detect CA compromises, and the BRs used to say "The CA SHOULD
| monitor the OCSP responder for requests for 'unused' serial
| numbers as part of its security response procedures." I'm not
| sure how many CAs actually implemented that, and in any case
| I don't think it ever detected any compromises.
| nashashmi wrote:
| This central authority for certificate validation seems
| like extra infrastructure without which the internet fails.
|
| Once upon a time, internet communication was between two
| computers. Now there is a third computer to verify that the
| communication between two computers is legitimate.
|
| Is there another communication design that works without
| the need for a third computer?
|
| Edit: I don't think so. Identity validation of a public
| computer should be done by another well-trusted computer.
| devman0 wrote:
| Could dust off DANE here, the connection probably already
| included a DNS lookup, why not get the public key info
| from the same.
| nine_k wrote:
| Between two distrusting parties, a third mediating party
| is needed. But that party does not need to be
| centralized. Indeed, there are many TLS certificate
| registrars.
| jesboat wrote:
| require sufficient info to identify the validation method
| to be included in an extension in the precert?
| agwa wrote:
| Yeah, that would be a good idea.
|
| Though sometimes the CA needs to know more than just the
| validation method to determine if a certificate should be
| revoked, and it's not practical to stuff it all in an
| extension (e.g. this recent GoDaddy issue which required
| examining past CAA queries:
| https://bugzilla.mozilla.org/show_bug.cgi?id=1904748).
| fragmede wrote:
| it's exactly that. the thing is that if you're a CA, that's
| not farfetched at all.
| jaas wrote:
| OCSP systems at scale are complex. At the core there is an on-
| demand or pre-signed OCSP response generation system, then
| there is usually an internal caching layer (redis or similar),
| then there is an external CDN layer. Just because the outer
| layer appears not to know about a certificate (a bug, to be
| sure), doesn't necessarily mean that other CA systems don't
| know about it.
|
| Requiring CAs to run OCSP makes running a CA more complex and
| expensive, which has considerable downsides like it does for
| any system, particularly as it is a zero sum game. Every dollar
| or hour of engineering time that Let's Encrypt spends on OCSP
| (and it is a considerable amount) is a dollar or hour not being
| spent on other parts of the CA. From the outside that may not
| be very visible because it's not obvious what isn't getting
| done instead but there is a real and considerable cost that is
| not worth it IMO.
| agwa wrote:
| Do you disagree that OCSP would be significantly less costly
| and complex if the responder URL were not included in
| certificates, freeing the responders from having to serve
| hundreds of millions of clients?
| jaas wrote:
| Yes, I disagree. Best case scenario I think it would just
| allow us to get rid of the CDN layer. We'd still have to
| build and manage the rest with utmost care.
|
| Even that really depends on what the "SLA" expectation is.
| How many QPS would we be required to support before we're
| allowed to start dropping queries? If there's any
| possibility of a burst beyond what we can handle locally
| we'd keep the CDN, so maybe the CDN bill would be
| significantly smaller on account of lower volume but all
| the complexity and care of operation would still be there.
| lokar wrote:
| They are so complex, and in practice unreliable, that my
| employer runs a caching proxy for our (non-browser) users
| (they mostly don't want fail open).
|
| IMO it is unfixable and should go
| kmeisthax wrote:
| The CA did not forget the certificate if it's in their CT logs.
| The whole point of CT (unlike OCSP) is to make it impossible
| for a CA to forget what they've signed. If they forget to put a
| certificate in their CT logs, then the cert isn't valid. Anyone
| can download and retain copies of the logs, because they're
| signed and hash treed. Removing a cert after-the-fact is about
| as disruptive as editing Git history on Linus Torvalds' tree,
| meaning it will be incredibly obvious that the CA is trying to
| lie about what they signed.
|
| The reason why OCSP forgets about certificates is that it was
| never intended to make CAs permanently remember every cert they
| ever signed. It was intended to replace CRLs with something
| less resource-intensive, except they screwed up the design, so
| it's now just a data siphon that provides little protection.
| agwa wrote:
| CT doesn't tell you if a certificate is revoked, so the CA
| still has to operate a revocation system. That revocation
| system can forget about certificates.
| dambi0 wrote:
| A unique per certificate CRL might work in theory. Theory is
| not the same as practice
| mcpherrinm wrote:
| Per-certificate CRLs are possible, though have about the same
| privacy problems as OCSP (without any of the possible
| advantages like OCSP pinning)
| dambi0 wrote:
| Internal / private certificate distribution monitoring, but
| yes privacy beyond that.
| trillic wrote:
| Are there any free or very inexpensive certificate providers
| which support ACME, DNS-01 challenges, and OCSP other than
| ZeroSSL?
| psz wrote:
| acme.sh has a list in their Wiki: https://github.com/acmesh-
| official/acme.sh/wiki/CA
| agwa wrote:
| Let's Encrypt will surely not be the only CA to discontinue
| OCSP after Microsoft removes their requirement.
| hlieberman wrote:
| Why not simply add the Must Staple restriction unconditionally to
| all certificates (aka "status_request")?
|
| That will eliminate privacy concerns -- no compliant TLS
| implementation should fetch a OCSP ticket given a stapled
| response -- while still allowing for broad non-browser support.
| agwa wrote:
| Many web servers have terrible to nonexistent OCSP stapling
| implementations.
| eqvinox wrote:
| This competes against terrible to nonexistent CRL checking on
| clients, but there are generally more clients than servers.
| darkhelmet wrote:
| _cough_ nginx. Nginx would start up and serve TLS on must-
| staple certs .. before doing the staple setup. ie: any client
| that validated that a must-staple cert had a stapled ocsp
| ticket would fail for the first few queries after nginx
| startup.
|
| I don't know if they've fixed it yet. I doubt it though -
| they were pretty aggressive in their assertion that violating
| must-staple wasn't a concern.
| agwa wrote:
| Yeah, I looked into nginx's stapling implementation almost
| a decade ago. I fixed some simpler bugs (I submitted a
| patch which was mostly rewritten and then merged) but
| fixing the problem you mention would have required major
| re-architecting. I doubt it has changed.
| mcpherrinm wrote:
| While it would eliminate privacy concerns, it would also likely
| break an enormous number of websites which don't implement OCSP
| stapling, or don't implement it correctly.
| eqvinox wrote:
| I'd suggest making browsers soft-enforce it with e.g. a
| yellow URL bar, giving server operators a window of time to
| fix things.
| nick238 wrote:
| I think with the whole menagerie of green locks, green bars
| & EV sites replacing/cluttering the address bar (meh [0]),
| only showing PSL+1 [1] (which _actually does help security_
| , but vocal power-users hate [2])...UI/UX designers are
| probably loathe to mess with it again.
|
| [0]: https://www.troyhunt.com/extended-validation-
| certificates-ar... [1]:
| https://blog.chromium.org/2020/08/helping-people-spot-
| spoofs... [2]: https://www.reddit.com/r/chrome/comments/h11
| gde/how_do_i_sto... [3]:
| https://www.wired.com/story/google-chrome-kill-url-first-
| ste...
| mcpherrinm wrote:
| The browsers, generally, are not onboard with improving
| OCSP and supporting OCSP stapling. Chrome already doesn't
| check OCSP at all.
|
| Apple and Firefox are all working on out-of-band revocation
| information based on CRL data. Presumably Chrome and its
| family are going to go down that path too, but I'm not sure
| I've heard commitments from them yet.
|
| Firefox has the best documentation on what's happening:
| https://blog.mozilla.org/security/2020/01/09/crlite-
| part-1-a...
| doctorpangloss wrote:
| You talk about it like there's a vibrant community of
| websites out there. There's streaming sites and social media,
| and then a bunch of apps that are delivered through the web.
| All of them are sophisticated web operations.
| fragmede wrote:
| in practice I think you're right, but oof.
| mcpherrinm wrote:
| Let's Encrypt's goal has always been to support the vibrant
| community of websites, extending TLS to the long tail.
|
| The handful of streaming and social media sites can always
| go to a big commercial CA and spend a few hundred bucks a
| year on certs that do whatever they need.
| blahgeek wrote:
| I don't get the OCSP Stampling protocol: if the server needs /
| is able to to contact the CA frequently anyway, why not simply
| use very short-lived certificates?
| eqvinox wrote:
| Issuing a certificate has significantly more overhead (e.g.
| Certificate Transparency logs) than signing a statement that
| an existing certificate is still valid.
| briansmith wrote:
| Which CA's will issue short-lived certificates without
| negotiating a custom ($$$) contract with them?
| agwa wrote:
| Google Trust Services lets you go down to 1 day using ACME.
| No payment required.
| kmeisthax wrote:
| Let's Encrypt, they said a while back they want to allow
| issuing certs for fewer than 90 days.
| citrin_ru wrote:
| In addition to listed below performance is another concern -
| OCSP stapling make TLS handshake large and slower. While it
| removes need for the client to query OCSP server itself in
| practice it such queries happen infrequently, asynchronously
| and in some may not happen at all. In theory a clients could
| include Certificate Status Request into the Client Hello only
| when a local cache expires but IIRC a few years ago Firefox did
| request OCSP on every TLS handshake making every handshake
| large if stapling is enabled on the server.
| fiddlerwoaroof wrote:
| OCSP sounds to me like a really bad idea: certificate
| expiration is already a significant cause of issues and now we
| add a whole new way for certificates to break your site?
| eduction wrote:
| It would be nice if one didn't need to be a TLS expert to
| understand the post -- particularly since the whole point of
| Let's Encrypt was to democratize TLS access.
|
| I have no idea if this means my setup will break even after
| consulting the docs of my ACME client.
|
| Can I still use ACME Tiny[1] with nginx? Any reason to think that
| will break? How can I tell if I'm using OCSP or CRL?
|
| Totally incomprehensible blog post.
|
| [1] https://github.com/diafygi/acme-tiny
| mcpherrinm wrote:
| For regular webserver users, accessed by web browsers, no
| changes are needed.
|
| Note this is probably at least a year, if not more, away.
|
| I'm sorry this post wasn't accessible enough, and we'll have
| more communications in the coming years as this gets closer.
|
| (I work at Let's Encrypt and proofread this post before
| publishing)
| phasmantistes wrote:
| Nothing will change for you, and nothing will break. The point
| of this post is to give a maximum-lead-time heads-up to the
| folks who _do_ need to care (the folks writing revocation-
| checking code in clients) so that later, more specific
| announcements don't come as a surprise.
| Sohcahtoa82 wrote:
| I'm curious how much of a factor this is. How often do
| certificates get revoked before expiration? I would think the
| only reason to revoke a certificate is if it was compromised.
| agwa wrote:
| Currently only 20,830,034 revoked certificates per my daily CRL
| download, versus ~1 billion active certificates. But the number
| of revoked certs could balloon if there's another Heartbleed-
| like event, or a mass misissuance.
| mintplant wrote:
| So are embedded devices just expected to not check for
| certificate revocation?
| woodruffw wrote:
| Compromised, or mis-issued. In practice, currently, mis-
| issuance is significantly more common (in terms of # of
| certificates revoked) than compromise is.
| jrochkind1 wrote:
| letsencrypt is the nonprofit community-focused infrastruture that
| we imagined the internet would consist of 20 years ago. I love
| you letsencrypt.
| klysm wrote:
| Certificate management is an interesting problem along the
| intersection of human behavior and computer science that feels
| similar to BGP. In theory, it's simple, but when met with reality
| things get messy really really fast.
| remram wrote:
| So if I'm not using Chrome or Firefox, how do I check for revoked
| certificates? This just makes the web less open again.
| agwa wrote:
| To check if a certificate is revoked, you can look up its
| issuer in the CCADB (https://ccadb.my.salesforce-
| sites.com/ccadb/AllCertificateRe...) to find the URL(s) of its
| CRL(s), and then check the CRL(s) for the serial number.
|
| It's true this isn't an IETF standard (sometimes the IETF
| standards just don't work in practice) but the data is still
| very much in the open.
| mcpherrinm wrote:
| Let's Encrypt will put CRL URLs in certificates before or
| when removing OCSP URLs, so no CCADB lookup should be
| required
| notepad0x90 wrote:
| CRLs don't scale, that's their problem and they take too long to
| update.
|
| Why isn't there a standard binary format for CRLs that is a
| cuckoo filter or a similar data structure? That way you don't
| have to worry about a CRL ballooning to gigabytes and you can
| expect clients to fetch the latest binary blob frequently.
___________________________________________________________________
(page generated 2024-07-23 23:01 UTC)