[HN Gopher] DigiCert Revocation Incident (CNAME Domain Validation)
___________________________________________________________________
DigiCert Revocation Incident (CNAME Domain Validation)
Author : vitaliyf
Score : 119 points
Date : 2024-07-29 22:50 UTC (1 days ago)
(HTM) web link (www.digicert.com)
(TXT) w3m dump (www.digicert.com)
| kevinday wrote:
| https://bugzilla.mozilla.org/show_bug.cgi?id=1910322
|
| for more background. The short story is that when doing CNAME
| based validation, they were supposed to put an underscore at the
| start of the random string for you to add to your DNS records.
| They still generated sufficiently random strings but didn't
| include a _ before it which is in violation of the RFC. The
| rationale is that some sites might do something like give you
| control of yourusername.example.com and they don't want to make
| it possible for random users to register the random string and be
| able to manipulate it. If you don't allow users to generate
| anything that causes a hostname to appear with a leading
| underscore, they can't pass the domain validation.
| tialaramex wrote:
| Also, while a DNS name can have an underscore a _host_ name,
| even in DNS, cannot have this character. So if you have a user
| named "haha_funny" you already aren't allowed to give them the
| hostname "haha_funny.somesite.example" - and on some system it
| will just silently not work because it's invalid.
|
| So even if you are completely oblivious to this work, and don't
| care about security at all, your "Give everybody a hostname"
| code should already avoid underscore characters as desired
| because otherwise stuff breaks.
|
| Several current systems use DNS names (but not hostnames) which
| feature underscores but it's pretty unlikely that you've got
| (for example) a service where users can pick their own TCP/IP
| service name and port and issued appropriate records for it in
| DNS. If you _have_ done this weird thing you probably want to
| use the existing mechanism (in DNS of course, the CAA record)
| to tell most CAs that they should not issue for your names even
| if they think they 've received permission. You can then cut a
| suitable deal with a for-profit CA to do whatever crazy extra
| checks you want (e.g. Meta's CA has to contact actual people in
| the appropriate security team at Meta, so that "mistakes" which
| give somebody a certificate for facebook.com never happen
| without some pretty drastic real world errors).
| userbinator wrote:
| _So if you have a user named "haha_funny" you already aren't
| allowed to give them the hostname
| "haha_funny.somesite.example" - and on some system it will
| just silently not work because it's invalid._
|
| Not long ago I actually did come across a site that had an
| underscore in its domain name, and it worked both for me and
| apparently Google, because it indexed and showed a (relevant)
| page from that site. I only remembered it was on a
| *.tripod.com subdomain, and can't find that exact site now
| since I don't remember what I was searching for (it was a
| highly obscure and technical topic), but there do appear to
| be others there with underscores, e.g.:
|
| http://computer_collector.tripod.com/
|
| http://hattori_striker.tripod.com/
|
| http://forgotten_dark_angel.tripod.com/
| tialaramex wrote:
| There are.
|
| There shouldn't be, but there are.
| jesprenj wrote:
| Google also indexed my site http://_.4a.si as seen here
| http://google.com/search?q=site:_.4a.si
| rvnx wrote:
| A live proof that CNAME records starting with _ exist.
| pests wrote:
| In 2019 the CAs agreed not to issue certs to underscored
| subdomains making this less useful.
|
| As evidenced by all your links being http.
|
| (as an aside, it looks really weird seeing a bare http link
| in the wild - crazy that was the old norm!)
| tialaramex wrote:
| My browser is configured to auto-upgrade such links and I
| get a full screen interstitial when the upgrade fails (as
| of course it did for these)
|
| This is now at a place where I'd recommend such
| configuration more broadly, it's not suitable for
| everybody, but many could benefit from just knowing all
| links are secured.
| agwa wrote:
| > _The underscore prefix ensures that the random value cannot
| collide with an actual domain name that uses the same random
| value. While the odds of that happening are practically
| negligible, the validation is still deemed as non-compliant if it
| does not include the underscore prefix._
|
| That's not the rationale for mandating the underscore prefix. The
| actual reason is so services that allow users to create DNS
| records at subdomains (e.g. dynamic DNS services) can block users
| from registering subdomains starting with an underscore. It
| serves the same purpose that /.well-known does.
|
| For example, if an attacker requests a certificate for
| dyndns.example and DigiCert gives them a record without an
| underscore prefix like
| da39a3ee5e6b4b0d3255bfef95601890afd80709.dyndns.example, they can
| register that subdomain with the dynamic DNS provider, publish
| the required record, and get the certificate for dyndns.example.
| It doesn't matter how much entropy DigiCert put in the record
| name.
|
| I definitely commend DigiCert for pledging to revoke the
| certificates within 24 hours and not having a delayed revocation
| or trying to language lawyer their way to a 5 day revocation as
| other CAs have tried. Nevertheless, this post severely minimizes
| the security impact of their mistake, and provides an excellent
| example of why CAs should always be required to strictly adhere
| to the rules and not be permitted to excuse noncompliance based
| on their own security analysis.
| michaelt wrote:
| _> For example, if an attacker requests a certificate for
| dyndns.example_
|
| Shouldn't that get caught by the Public Suffix List?
|
| I would hope DigiCert has checks in place to prevent someone
| domain-validating ownership of the entire of co.uk under any
| circumstances :)
|
| (They should still revoke the mis-issued certificates though)
| remus wrote:
| > Shouldn't that get caught by the Public Suffix List?
|
| PSL is a best-effort sort of thing, so it's good but not
| definitive. It would be dangerous to rely on it when issuing
| certs imo.
| nickf wrote:
| PSL has a couple of sections - ICANN and PRIVATE. PRIVATE can
| be a little more flexible/ignorable. If they implement a hard
| rule, then occasionally they'd have to make exceptions when
| the _real_ Dyn comes along and wants (legitimately) a
| wildcard for their name.
| agwa wrote:
| There's no prohibition against issuing certificates for names
| on the Public Suffix List.
|
| BR 3.2.2.6 prohibits issuing a wildcard certificate for an
| entire public suffix unless the "Applicant proves its
| rightful control of the entire Domain Namespace" (without
| specifying how this should be done - arguably, publishing a
| DNS record would qualify) but also says that CAs should use
| the "ICANN DOMAINS" section of the PSL only, not the "PRIVATE
| DOMAINS" section, so domains for dynamic DNS providers and
| the like wouldn't be included. [https://github.com/cabforum/s
| ervercert/blob/main/docs/BR.md#...]
| agwa wrote:
| > _I definitely commend DigiCert for pledging to revoke the
| certificates within 24 hours and not having a delayed
| revocation or trying to language lawyer their way to a 5 day
| revocation as other CAs have tried._
|
| It seems I spoke too soon:
| https://bugzilla.mozilla.org/show_bug.cgi?id=1910322#c8
| jiggawatts wrote:
| I just want to call out both CrowdStrike and DigiCert for being
| one of "those" companies that insist on publishing critical
| support information behind a login with _the clock ticking_ on a
| global outage of their own making.
|
| There are no polite words that I can use to accurately convey the
| depth of my disappointment at this kind of inconsiderate
| behaviour during a crisis, so I won't say anything more.
| Cyphase wrote:
| Are you referring to the list of affected certificates?
| Brian_K_White wrote:
| What critical support information? What global outage? How are
| these two events or companies remotely equivalent?
|
| If I'm not a Digicert customer, what do I care about the
| details of how to redo a validation on Digicert? If I am a
| Digicert customer I have been emailed already and I will
| obviously have to log in to do anything at all with my domain.
|
| They say this affects 0.4% of Digicert customers who are what %
| of the world? Actually not even 0.4% of Digicert customers, but
| 0.4% of those particular validations. What does that actually
| work out to? Just who all is actually down?
|
| I fail to see any equivalence.
| freeone3000 wrote:
| If you're not a customer, your domain isn't affected.
| Apfel wrote:
| You may be a customer of a customer (e.g. Azure) so you could
| be affected peripherally via that route.
| 256_ wrote:
| > While we had regression testing in place, those tests failed to
| alert us to the change in functionality because the regression
| tests were scoped to workflows and functionality instead of the
| content/structure of the random value. [...]
|
| > Unfortunately, no reviews were done to compare the legacy
| random value implementations with the random value
| implementations in the new system for every scenario.
|
| In other words, they didn't do proper testing. At the bottom of
| the article they suggest they're going to improve it.
| ratg13 wrote:
| 24h notice to change certificates in who knows how many systems,
| at the worlds largest companies, while everyone is on vacation.
|
| This will be interesting.
| devrand wrote:
| Respect to them for actually abiding by the BRs. Most CAs just
| shrug [1] and [2] say [3] it's [4] too [5] complicated [6], or
| just lie and claim planes will start crashing [7]. It's really
| disheartening that publicly trusted CAs just ignore their
| contractual obligations however they see fit.
|
| Ideally these companies should have response plans in place to
| prioritize certificate rotation. They can use this as a fire
| drill for what would happen if there were a key compromise.
|
| Alternatively, if companies cannot handle the rotation, then
| they likely should re-evaluate if WebPKI is even appropriate
| for their use-case.
|
| [1]: https://bugzilla.mozilla.org/show_bug.cgi?id=1885568
|
| [2]: https://bugzilla.mozilla.org/show_bug.cgi?id=1898848
|
| [3]: https://bugzilla.mozilla.org/show_bug.cgi?id=1910237
|
| [4]: https://bugzilla.mozilla.org/show_bug.cgi?id=1896053
|
| [5]: https://bugzilla.mozilla.org/show_bug.cgi?id=1896553
|
| [6]: https://bugzilla.mozilla.org/show_bug.cgi?id=1877388
|
| [7]: https://bugzilla.mozilla.org/show_bug.cgi?id=1903066#c48
| hg35h4 wrote:
| "Alternatively, if companies cannot handle the rotation, then
| they likely should re-evaluate if WebPKI is even appropriate
| for their use-case."
|
| I hate hearing this awful take, as if every IT organization
| has the same neat and tidy systems deployed as they do. Never
| had to deal with 3rd party SaaS vendors certificate pinning
| requiring service tickets to change, don't have any hardware
| devices or appliance based software images each with their
| own web interface to update certs...
|
| Yes companies should have a plan to do their minimum yearly
| certificate rotates. Yes those companies should have a
| security plan to rotate affected certificate issues, but in
| those cases the business users are ok with an outage to
| remediate a real security issue.
|
| But what happened here is that Digicert invalided the entire
| domain's worth of certs. All those service.companyname.com
| certs or duplicates under that domain validation were
| affected in bulk. In some companies there could be thousands
| of certs under that domain. Digicert screwed up their system
| implementation and made their customers suffer.
|
| "It's really disheartening that publicly trusted CAs just
| ignore their contractual obligations however they see fit."
|
| It's also disheartening to see browsers in the CA consortium
| ignore the CA resolutions as well. Like how everyone voted
| for 2 year certs and Apple did their own thing anyways. Any
| punishment for Apple come? So why pick on the others?
| devrand wrote:
| "Never had to deal with 3rd party SaaS vendors certificate
| pinning requiring service tickets to change"
|
| I think this tends to fall into "probably shouldn't have
| been using Web PKI". I can't immediately think of a reason
| why you'd need a publicly trusted certificate if you're
| pinning a specific public key.. at that point who cares who
| signed it?
|
| I do agree that there are real costs with rotating
| certificates that ultimately may make it impossible for an
| organization to complete that work in the revocation
| window. That is very much an area that needs further
| automation developed and more importantly, for it to
| actually be adopted. I believe that's what ACME Renewal
| Information is attempting to address.
|
| "but in those cases the business users are ok with an
| outage to remediate a real security issue"
|
| Ideally yes, but that might be the same point you find out
| the certificate was used in some critical system (let's say
| Air Traffic Control like a previous CA tried to claim).
| They still may very well not be okay with the revocation
| despite the security issue. _Those_ are the people that
| need to stop using these certificates and there's really no
| way to weed them out until a revocation actually needs to
| occur.
|
| "Digicert screwed up their system implementation and made
| their customers suffer."
|
| And those customers are right to be mad at DigiCert. They
| probably don't have a legal basis to challenge as the
| subscriber agreement explicitly permits immediate
| revocation without prior notice, but they can certainly
| take their business elsewhere.
|
| "It's also disheartening to see browsers in the CA
| consortium ignore the CA resolutions as well. Like how
| everyone voted for 2 year certs and Apple did their own
| thing anyways. Any punishment for Apple come? So why pick
| on the others?"
|
| Admittedly I'm not very familiar with the various root
| programs and the obligations they have with CAs, but it
| doesn't seem unreasonable that root programs would be free
| to impose stricter requirements then the BRs.
|
| Though I do find it two-faced for Apple to vote for Ballot
| 193 only to then impose a stricter requirement. At the very
| least they should have abstained.
| crote wrote:
| Stuff like this is why some parties have been calling for
| increasingly-shorter cert validity. When a cert is valid
| for several years it allows companies to develop an
| increasingly complex workflow around deploying them,
| sometimes taking _weeks_ and involving dozens of parties to
| roll them out. This is in turn used as an excuse by CAs to
| completely ignore the industry standards.
|
| Those SaaS vendors probably shouldn't be doing cert pinning
| to begin with. If you don't trust your root store either
| implement support for CAA or DANE, no need to roll out your
| own workflow. Those hardware devices should either 1) not
| use publicly trusted certs, 2) renew their own certs, or 3)
| have an API to automatically update certs.
|
| The _only_ reason they 're still getting away with it is
| because doing it manually once a year isn't horribly
| painful. If 90-day validity becomes the industry standard,
| pain-free certificate renewal turns into a must-have for
| all new contracts.
| devrand wrote:
| Spoke too soon... seems like subscriber(s?) issued DigiCert a
| Temporary Restraining Order to not revoke:
| https://bugzilla.mozilla.org/show_bug.cgi?id=1910322#c8
|
| Bold.
| Apfel wrote:
| Is this a potential cause of the current Azure outages hitting
| western europe? I know DigiCert are used by Azure extensively...
| agwa wrote:
| Unlikely. Microsoft operates their own CAs. Some of their CAs
| have been cross-signed by a DigiCert root, but Microsoft is
| responsible for the domain validation. I don't think they
| extensively use certificates issued directly by a DigiCert CA.
___________________________________________________________________
(page generated 2024-07-30 23:02 UTC)