[HN Gopher] Entrust Certificate Distrust
___________________________________________________________________
Entrust Certificate Distrust
Author : iancarroll
Score : 122 points
Date : 2024-06-27 17:22 UTC (1 days ago)
(HTM) web link (security.googleblog.com)
(TXT) w3m dump (security.googleblog.com)
| crazysim wrote:
| Some popular users:
|
| chase.com aa.com
| dextercd wrote:
| Some more:
|
| washingtonpost.com, cdc.gov, dell.com, jpl.nasa.gov,
| mastercard.com
| theandrewbailey wrote:
| api.cybersource.com
|
| This is gonna cause me some headaches, along with everyone else
| who processes payments through Cybersource, and possibly others
| :(
| qmarchi wrote:
| CYBS Engineer here.
|
| We're already working on it. Keep an eye for merchant
| notifications if you use certificate pinning.
|
| Now, back to rotating certificates....
| crazysim wrote:
| Out of curiosity, is your organization planning to switch
| to Let's Encrypt or just another year long certificate
| provider?
|
| It'll be interesting to see what, if any, organizations
| affected by this switch to: Stick with 1 YR certs or go to
| the future with free 90 days?
| agwa wrote:
| I'm curious why that is. Is your API client using a root
| store that doesn't contain CAs other than Entrust, or pinning
| to an Entrust CA?
| nahikoa wrote:
| The issues identified really show a dumpster fire:
| https://bugzilla.mozilla.org/buglist.cgi?o2=greaterthaneq&sh...
|
| Directly from Entrust: "Yes, there has been ongoing internal
| discussion and reflection on the issues found in this and other
| incidents, which has led to the action items described previously
| and ongoing changes, including the decision to revoke the
| certificates affected by this bug. Exceptional circumstances
| would need to be provided and justified by the Subscribers. Given
| the nature of the feedback we have received to date, we doubt
| that the community has any real interest in anything that Entrust
| could suggest, except to use against Entrust in a destructive,
| not constructive, way. We therefore would like more explicit and
| clear guidelines or a definition of "exceptional circumstances"
| to be adopted and applied equally to all CAs, perhaps through
| updates in the CA/B Forum requirements."
|
| https://bugzilla.mozilla.org/show_bug.cgi?id=1888714
| dextercd wrote:
| I wonder if Entrust can survive this. Even if Web-PKI doesn't
| account for the majority of their income (which it might, I
| genuinely don't know) this is a huge blow to their credibility.
|
| And for a CA, credibility is everything
| kseifried wrote:
| Entrust has BIMI certs which use a different root (CN = Entrust
| Verified Mark Root Certification Authority - VMCR1) and for
| which your choices of a BIMI certificate are: Entrust or
| Digicert. I doubt it makes as much money as their web certs
| (BIMI certs are not super common, and they are expensive to
| issue since there's an actual validation process that typically
| involves a public notary validating the ID of a corporate
| officer).
|
| If you believe https://bimiradar.com/glob
|
| it looks like Entrust is selling on the order of a few dozen
| certs a week to maybe upwards of 100-200.
|
| EDIT: I've asked Google if Gmail will be discontinuing support
| for Entrusts VMC certificate (and thus BIMI logos), I would
| guess not since BIMI has some actual requirements, but
| assumptions are not the best way to make decisions about risk
| (like our BIMI logo not working later this fall).
| Kwpolska wrote:
| BIMI is a CA racket.
| intelVISA wrote:
| Thanks Google...
| farresito wrote:
| Entrust makes a ton of revenue from hardware-related products
| (for example, printing ID cards), so it is far from the end.
| crote wrote:
| Right up until the next contract renewal. "Not trustworthy
| enough to secure a basic website" isn't exactly a great look.
| noname120 wrote:
| The real question is: why didn't they get booted out earlier?
| sieabahlpark wrote:
| My guess is Google wanted to really give them every opportunity
| so that most people come to your conclusion instead of "wow!
| Google is just trying to censor the internet"
| sybercecurity wrote:
| Probably worried that it would break something for a
| significant number of customers, so they took a cautious
| approach. That's the problem with any Internet-wide change: you
| don't always know who will be impacted negatively or how
| severely. No one wants to make a quick change only to find out
| some critical system is now broken.
| rxu wrote:
| Can someone ELI5 what the violations linked in the first line
| are? They seem pretty minor to me but I don't understand certs
| whizzter wrote:
| Since contents of certificates contents sadly often diff there
| was a ballot to streamline the contents to lessen the burden on
| implementations to interpret the differences.
|
| Entrust missed/ignored the updates to how certificates were
| supposed to be formed and when caught declined to revoke the
| incorrectly issued ones (because it's probably a more or less
| manual process for many admins working in a pre-Let's Encrypt
| style of fashion) and they didn't want to inconvenience their
| customers and assumed that they themselves were the important
| party in the equation (CA's was that historically compared to
| site-admins).
|
| The certificate industry has always been quite ad-hoc with CA's
| being entitled middlemen, we have Let's Encrypt and almost
| ubiquitous encryption now because browser makers and other
| internet actors saw security as more important than protecting
| the CA's business and now that LE is established Google,etc
| aren't the slightest interested in pampering CAs if they aren't
| interested in cleaning up the system.
| plorkyeran wrote:
| They are very minor, but because the consequences of an mis-
| issued certificate can be so high, there's an explicit policy
| that misissued certificates _must_ be revoked and reissued
| promptly. The distrust was due to them refusing to comply with
| the policy and outright stating that they did not intend to
| comply in future incidents either.
| crote wrote:
| Correct, the violations are minor and _should_ be trivial to
| deal with.
|
| The problem in this case is that Entrust displayed a complete
| disinterest into actually solving the underlying issues. Doing
| an oopsie is one thing. Doing an oopsie, lying about it,
| refusing to take precautions, and failing to take measures to
| prevent a repeat _despite promising to do so_? Completely
| different story.
|
| If they can't be trusted to respond properly to minor
| administrative issues, why should they be trusted to respond
| adequately during a real security incident?
| olliej wrote:
| There are some issues that are bad, but most of these issues
| with the _certificates_ are minor.
|
| The problem is that to be a CA in the root stores you agree to
| (and entrust voted in support of) a pile of rules, and entrust
| demonstrated a complete disinterest in complying with those
| rules.
|
| The reason for removing trust is not the severity of the
| original errors, it's the the severity of errors in the
| response.
|
| 1. The BRs requires revocation of invalid certificates with 5
| days unless there is an exceptional reason not to. Entrust did
| not.
|
| 2. In the event a CA discovers that they are mis-issuing
| certificates they are required to stop issuing until they have
| resolved the error. In this case not only did entrust not do
| this, but they explicitly stated - after the issues were
| raised, and they were already told they were failing to revoke
| certs in the required time frame - that they were intentionally
| continuing to mis-issue
|
| 3. They repeatedly made errors in the past, promised to correct
| them, and then kept making the same errors
|
| 4. They made claims they were trying to get customers to
| prepare for revocation in the require 5 day window, but then it
| turned out they were telling customers that they had 30+ days
| (which was only discovered when one of the relevant customers
| forwarded info to someone else)
|
| 5. When a CA discovers miss issuance they are required to file
| an incident report, and provide detailed information about how
| it occurred, why it was not caught, what remediating steps are
| being taken, and what mechanisms are being introduced to ensure
| a similar failure cannot occur in future. None of Entrust's
| responses came close to this, until the chrome root store rep
| came in to say "this is unacceptable", and even then their
| "improved" reports were incomplete and lacked sufficient
| detail.
|
| 6. Once they were finally doing the basic steps they were meant
| to have done the moment they learned of the miss-issuance they
| repeatedly failed to produce an accurate set of the impacted
| certificates (as in the provided a list and people outside of
| Entrust were able to immediately turn around and say "but these
| certs are also broken, why aren't you listing those details")
|
| and so on and so forth.
|
| Google's post to CCADB provides more details than the blog
| post:
| https://groups.google.com/a/ccadb.org/g/public/c/29CRLOPM6OM...
| xeeeeeeeeeeenu wrote:
| It always fascinates me when this happens. Don't the CAs
| understand that the browser vendors can and will kill their
| business if they don't comply with the rules? It's not like a
| fine that can be ignored.
|
| How dysfunctional does a company have to be to let this happen?
| skrebbel wrote:
| I'm really impressed. CAs are 100% rent-seeking businesses and
| their position is solely derived from having convinced browser
| and OS vendors to put them in a list. You'd assume their top
| prio would be to stay in the list.
| crote wrote:
| They genuinely believe they are "too big to fail". They've got
| _thousands_ of employees, they 've been around for 30 years,
| they are a critical part of public infrastructure: surely
| something as trivial as a few weirdos in a mailing list
| couldn't instantly kill their entire business?
|
| Stuff like this happens when upper management has _zero clue_
| about the business they are in. They believe they are in the
| business of selling certificates, while in reality they are in
| the business of _selling trust_. They treat things like the CA
| /B Forum and the various Root Programs as more like an optional
| networking event than the combination of judge, jury, and
| executioner that it actually is - with a completely predictable
| outcome.
| hvenev wrote:
| Perhaps they've decided to draw inspiration from
| https://bugzilla.mozilla.org/show_bug.cgi?id=647959.
| darkhorn wrote:
| Software developers say that we have issues in the software
| that needs to be fixed or updated. Managment, who has never
| seen one line of code in their life says "no, make new
| features". And then the software starts to fall.
| 1oooqooq wrote:
| All you need is one single [cc]?.gov as your client and you are
| in business forever.
| vbezhenar wrote:
| I saw company being killed by failed backup system. One
| unfortunate hardware failure, bad backups and company service
| goes offline with no way to recover in timely manner. Big
| clients require big compensation, company goes bankrupt. One
| shell script put in crontab could have prevented that, but
| nobody cared enough. It was not a big company, though. But
| consequences of one simple overlook were dire.
| kseifried wrote:
| Entrust has BIMI certs which use a different root (CN = Entrust
| Verified Mark Root Certification Authority - VMCR1) and for which
| your choices of a BIMI certificate are: Entrust or Digicert. I
| doubt it makes as much money as their web certs (BIMI certs are
| not super common, and they are expensive to issue since there's
| an actual validation process that typically involves a public
| notary validating the ID of a corporate officer). If you believe
| https://bimiradar.com/glob
|
| it looks like Entrust is selling on the order of a few dozen
| certs a week to maybe upwards of 100-200.
|
| EDIT: I've asked Google if Gmail will be discontinuing support
| for Entrusts VMC certificate (and thus BIMI logos), I would guess
| not since BIMI has some actual requirements, but assumptions are
| not the best way to make decisions about risk (like our BIMI logo
| not working later this fall).
| Scaevolus wrote:
| Aren't BIMI certs an even sillier cash grab than EV certs?
| aaomidi wrote:
| I'm one of the people who really went in depth with Entrust (Amir
| on Bugzilla).
|
| I'm also an author on https://webpki.substack.com. I will be
| writing my thoughts on the distrust soon.
|
| I can try to answer any questions folks may have. I can also help
| folks find ways they can also be involved!
|
| Root programs can only do so much and need surveillance of the
| CAs from the community.
| doctorpangloss wrote:
| I am a layperson so I appreciate the attention on the matter.
|
| Regardless of how Entrust is operated, there appears to be
| significant complexity in CA program that the browsers operate.
| On the flip side, Let's Encrypt is basically effortless for me
| to use, as an end user of an LE secured site and as a
| developer. Why misallocate all this toil on root CA compliance
| on the one hand, when LE could redirect that labor towards
| something valuable instead? What is so challenging about giving
| LE full leadership on this issue? Where does the proverbial
| political strength of Entrust and similar entities come from,
| in an ecosystem where there are functionally 5 cooperating,
| more or less transparent entities that decide the trust of
| certificates for 99% of end users? Why does anyone care about
| any of the CAs?
| glzone1 wrote:
| LE has to comply with the root CA standards too. For a
| variety of reasons there haven't been as many problems with
| LE - partly because they don't get paid by folks getting
| certs and issuing certs is a cost to them so their incentives
| are different.
|
| I don't get entrust here. It's not like they weren't told
| what to do.
| aaomidi wrote:
| Let's Encrypt is just a player in the same ecosystem.
| Effectively they're no different from Entrust, GoDaddy,
| Google Trust Services, Digicert, Sectigo, etc etc.
|
| Let's Encrypt started their operations with _automated_
| certificate issuance only. They also do not do OV/EV
| certificates that are much, much harder to automate without
| providing any real benefits.
|
| So, LE's mission is to issue certificates under the rules set
| by CAs and Browsers. (Yes, CAs do participate in setting up
| rules for CAs.
|
| > Where does the proverbial political strength of Entrust and
| similar entities come from
|
| Generally supporting non-automated certificate issuance.
| Effectively, technical debt. A lot of older enterprises have
| done manual certificate issuance, and they don't feel the
| pressure/reason to switch.
| cedws wrote:
| I wonder which root CA the intelligence agencies use to
| selectively MITM TLS traffic a la Crypto AG.
| aaomidi wrote:
| Certificate transparency prevents this style of attack.
| schoen wrote:
| As long as the victims are checking it and know what to look
| for!
| Sateallia wrote:
| Chrom(e/ium) and Safari don't trust certificates that are
| not in public logs [0].
|
| [0] https://en.wikipedia.org/wiki/Certificate_Transparency#
| Manda...
| schoen wrote:
| Right, and that's a fundamental sea change in PKI
| security posture since the Iranian "ComodoHacker" and the
| Soghoian and Stamm compelled issuance paper! My point is
| just that some attackers might be willing to have their
| attacks show up in public logs if their victims are
| unlikely to ever notice that and if nobody else is likely
| to notice it either.
|
| With Let's Encrypt we made a lot of people's certificate
| management a "fire and forget" thing, which is exactly
| what we hoped to do, but if they _completely_ forget
| about it, it may be that there will be lots of targets
| against whom nobody would notice certificate misissuance.
| aaomidi wrote:
| The other argument is, why bother MITMing when you can go
| to Cloudflare and get them to share the data with you :)
| Sateallia wrote:
| I got every self-hosting sysadmin I know to run
| certificate monitors for sites they maintain but it
| certainly isn't a common thing to do. I know Cloudflare
| has a beta certificate monitoring feature which would
| certainly help a lot with this problem considering their
| market share if they enable it by default. (Although one
| problem with this is that they issue backup certificates
| from other CAs so it'd easily trigger warning fatigue!)
|
| (I wasn't aware of your credentials when I made my
| previous comment so I assumed you didn't know about
| mandatory certificate transparency which is a mistake on
| my part, sorry! I'll make sure to check profile about
| sections before I assume again.)
| schoen wrote:
| Yeah, I think it's tricky to know how most sysadmins
| could make good decisions about this information,
| especially when misissuance is likely to be less than 1%
| of 1% of all CA issuance and automated renewal is working
| properly. Warning fatigue is a pretty big deal here!
|
| Also, we made Certbot randomize the subject key by
| default every time it renews, so you have a huge amount
| of churn in subject keys, so you can't just say "oh,
| well, this public key has been used for a long time, so
| it's probably correct!". Every subject key is typically
| new and is unrelated to every previous subject key.
|
| I hope that won't turn out to have been a poor trade-off.
| (We thought it was good to have more turnover of keys in
| order to reduce the impact of successfully stealing or
| cryptographically attacking one.)
| yugcesofni wrote:
| Not just Safari, but all TLS connections instantiated on
| Apple OSes
| aaomidi wrote:
| There's definitely a lot of people watching CT for
| anomalies (I'm one of them), but more surveillance of it is
| also good and something I've been trying to advocate.
|
| It's also why I'm personally against SMIME and think it's a
| bad idea.
| bpfrh wrote:
| There is a more detailed statement in the chrome CCADB Public
| group:
|
| https://groups.google.com/a/ccadb.org/g/public/c/29CRLOPM6OM...
| mikeiz404 wrote:
| > This approach attempts to minimize disruption to existing
| subscribers using a recently announced Chrome feature to remove
| default trust based on the SCTs (signed certificate timestamps)
| in certificates.
|
| I was wondering how Chrome was able to revoke a certificate based
| on time without trusting the CA to not back date certificates and
| it looks like this is due to being able to trust certificate
| transparency logs instead. This is where they get the signed
| certificate timestamps (SCT) from.
|
| See also https://certificate.transparency.dev/howctworks/
| olliej wrote:
| This kind of thing was one of the reasons CT was introduced :D
| agwa wrote:
| CT was introduced to detect misissued certificates, but it
| was never intended to be used as a timestamping system like
| this. The point of the SCT timestamp was to start the clock
| on the deadline for the log to publish the certificate, not
| for use in trust decisions. So when Chrome announced they
| were using SCT timestamps for trust decisions, my first
| question[1] was whether anyone was auditing CT logs to detect
| backdated timestamps. Since then, I have added timestamp
| auditing to my monitor and should be able to detect a CT log
| backdating an SCT timestamp.
|
| [1] https://groups.google.com/a/ccadb.org/g/public/c/wRs-
| zec8w7k...
| olliej wrote:
| Sorry I think my phrasing was not great.
|
| The goal of CT was not to make staged distrust of a CA
| possible (that's just a happy coincidence). It was to make
| it possible to detect CAs misissuing certificates, which
| includes CAs back dating certificates.
| zX41ZdbW wrote:
| The list of affected websites, just in case:
| https://pastila.nl/?000882d6/bed25fdc842914abbc89e528012a961...
| zX41ZdbW wrote:
| Potentially affected. No problem for now, per the linked blog
| post.
| amluto wrote:
| > Additionally, should a Chrome user or enterprise explicitly
| trust any of the above certificates on a platform and version of
| Chrome relying on the Chrome Root Store (e.g., explicit trust is
| conveyed through a Group Policy Object on Windows), the SCT-based
| constraints described above will be overridden and certificates
| will function as they do today.
|
| This continues to annoy me. Chrome (and other browsers) have
| detailed trust constraints, e.g. SCTNotAfter, in their own root
| stores. Why can't administrators do the same thing?
| 1oooqooq wrote:
| All the google root security team's due diligence email are just
| a list of links to firefox's bugzilla who documented and followed
| up on all the issues.
|
| https://groups.google.com/a/ccadb.org/g/public/c/29CRLOPM6OM...
| agwa wrote:
| Representatives from the Chrome and Apple root programs
| participate in the Bugzilla discussions in an official
| capacity. But yes, there is significant help from the community
| in uncovering evidence and grilling CAs.
| tialaramex wrote:
| In practice the Web PKI is overseen by the general public, via
| Mozilla's m.d.s.policy. It makes no sense for the proprietary
| vendors, including Google, to insist on doing something
| themselves badly when Mozilla is the obvious host for this
| work.
|
| The older vendors are even _less_ able to be properly open with
| their customers (let alone the general public) than Google. At
| Apple it 's probably a firing offence to even confirm obvious
| decisions - it seemingly took _months_ to get Apple 's _chosen
| representative_ to confirm that Apple 's new 398 day rule was
| an issuance requirement, rather than just something where Apple
| wouldn't trust longer lived certs in Safari.
| 1oooqooq wrote:
| none of what you list are good excuses for anything. I fail
| to see the point. Is it that marketing trumps technical know
| how and it should be ok?
___________________________________________________________________
(page generated 2024-06-28 23:00 UTC)