[HN Gopher] Health industry company sues to prevent certificate ...
___________________________________________________________________
Health industry company sues to prevent certificate revocation
Author : todsacerdoti
Score : 65 points
Date : 2024-07-31 20:04 UTC (2 hours ago)
(HTM) web link (www.hezmatt.org)
(TXT) w3m dump (www.hezmatt.org)
| loceng wrote:
| Is this only happening to this one company? It seems like it'd be
| a widespread problem?
| kadoban wrote:
| Naw it's a bunch of people/companies. It was discussed on HN at
| least once recently, but best link I have is:
| https://bugzilla.mozilla.org/show_bug.cgi?id=1910322
| deathanatos wrote:
| The complaint:
| https://www.courtlistener.com/docket/68995396/1/alegeus-tech...
|
| IANAL.
|
| > _DigiCert knew or should have known about its failure to
| properly complete the Security Certificates for Alegeus weeks
| ago._
|
| 'You should have known there were bugs in your code!'
|
| > _DigiCert is a voluntary member of the Certification Authority
| Browser Forum (CABF), which has bylaws stating that certificates
| with an issue in their domain validation must be revoked within
| 24 hours_
|
| "voluntary" is _technically_ correct, I suppose. But let 's
| assume they'd like to be a CA?
|
| But then we go
|
| > _despite that DigiCert has arbitrarily chosen this less than
| 24-hour window_
|
| It wasn't arbitrary, it was a requirement, as the complaint
| itself pointed out...
|
| > _Alegeus has hundreds of such Clients. Alegeus is generally
| required by contract to give its clients much longer than 24
| hours' notice before executing such a change regarding
| certification._
|
| Yet another example of that nobody ever thinks about cert
| rotation, ever.
|
| > _Consequently, due to the errors and delays of DigiCert alone,
| Alegeus cannot practically make all the required changes_
|
| Yeah, no, your error was not having a process to account for this
| in place.
|
| The TRO also states,
|
| > _The threatened injury to Alegeus outweighs any injury DigiCert
| might suffer under the injunction._
|
| Well, if DigiCert is removed as a CA because they've violated the
| CA/B BRs (at least, AIUI, they have) -- that could be a pretty
| big injury. They're probably too big to fail, and "we tried but
| we had a TRO" is a pretty good excuse.
|
| The TRO is scoped to only their certs, and only for 7 days
| (barring further orders), so it's at least pretty minimal? So I
| suppose DigiCert could revoke the rest.
| stackskipton wrote:
| My guess is all they want is 7 days since complaints indicates
| they asked for 3 days.
|
| So they filed this suit, got the TRO, will do the work while
| TRO is underway and at hearing, declare it all moot, pay the
| lawyers and move on.
| JoshTriplett wrote:
| What _is_ the legal process for "cannot comply with this TRO
| because doing so would likely cause us to cease to exist"? It
| certainly is the case that "any injury DigiCert might suffer
| under the injunction" has the _possibility_ of including no
| longer being able to be a certificate provider.
| dataflow wrote:
| > What is the legal process for "cannot comply
|
| Appealing, I imagine?
|
| > would likely cause
|
| > has the _possibility_ of
|
| Doesn't sound like it's "likely" in this case, so said
| process probably wouldn't work?
| wbl wrote:
| This is a TRO. Digicert hasn't been heard yet. So it's not
| an appeal just a regular hearing.
| dataflow wrote:
| I thought you can appeal a TRO, at least sometimes? Don't
| know if the states differ on this.
| JoshTriplett wrote:
| Appealing doesn't work if you're being made to comply
| within 24 hours before having any opportunity to prepare
| and make arguments.
|
| I'm wondering what the process is for showing that a TRO
| could potentially cause irreparable harm.
| dataflow wrote:
| Aren't there emergency appeals for stuff that can't wait?
| Like in NY there's a phone number for "Emergency
| Applications during nonbusiness hours":
| https://www.nycourts.gov/courts/ad2/contactus.shtml
| kmeisthax wrote:
| The legal process is contempt of court. If CAFB revokes
| DigiCert than they are interfering with the TRO on DigiCert,
| which puts them in contempt of court.
| Sakos wrote:
| The fact that the TRO was issued seems insane to me. So now
| DigiCert can't revoke the certificates because these jokers
| can't handle rotating certificates and they're making it
| everybody else's problem.
| Hizonner wrote:
| 1. It _does_ look like there 's very strong reason to believe
| that these particular certs were _not_ misissued, in the sense
| that the people who have the private keys really are the people
| who control the domains. Stronger than the sort of verification
| that the CA /BF requires. It's a bug if the system can't deal
| with that.
|
| 2. It's also true that there are rules, they exist for a reason,
| these Alegeus guys aren't special, and dragging this stuff into
| the courts is antisocial in the extreme. So it wouldn't be
| unreasonable for browsers to, say, explicitly blacklist their
| domains for as long as there conceivably might be noncompliant
| certs out there.
| jandrese wrote:
| The issue isn't that someone is taking over this particular
| healthcare industry domain, just that in that time period there
| could be bad certs issued for other subdomains and there is no
| way to verify it. The whole setup where the verification
| depends on a leading underscore because subdomains aren't
| allowed to start with an underscore seems rather flimsy to me,
| but it's how the system is set up.
| michaelt wrote:
| _> It 's also true that there are rules, they exist for a
| reason, [...] dragging this stuff into the courts is antisocial
| in the extreme [...] it wouldn't be unreasonable for browsers
| to, say, explicitly blacklist their domains_
|
| You know who else thinks there are rules, that the rules exist
| for a reason, and that draconian punishments should be applied
| to people who try to derail attempts to enforce the rules?
|
| Judges. Like the judge that granted this temporary restraining
| order.
| Hizonner wrote:
| The browsers don't have any contract with Alegeus, nor any
| relationship at all. They owe Alegeus nothing. They're not
| covered in the TRO. I doubt the TRO could legally be extended
| _to_ cover them.
|
| Browsers _do_ have a relationship with their users, who may
| very well be relying on them to enforce the CA /BF validity
| rules, or if they can't do that to provide equivalent
| security guarantees. And in particular many of the users rely
| on their browsers to protect them by blocking suspicious
| domains in various ways. So blocking the domains wouldn't be
| arbitrary, capricious, discriminatory, or anything like that.
|
| Judges don't have infinite sua sponte powers to order just
| anybody to do just anything. Although I admit they sometimes
| act as though they did, especially when they feel like their
| authoriteh has been disrespecked.
|
| Anyway the browsers aren't gonna do it. There's zero chance.
| So legalities don't really arise. I'm just saying it wouldn't
| be _unreasonable_.
| bastawhiz wrote:
| The judge's TRO doesn't apply to browser vendors, it applies
| to Digi cert. The plaintiff didn't file for an injunction
| against browser vendors, and they probably couldn't even if
| they wanted to because they have no business relationship
| with them. What responsibility do the browsers have to this
| company?
| michaelt wrote:
| The judge has issued an order that the websites not be
| disrupted, just for a few days while the contracts and
| paperwork get straightened out.
|
| If a browser vendor disrupted the websites, because they
| knew of this court order and decided to take deliberate
| action to defy the authority of the court, that's contempt
| of court.
|
| The browser vendor then gets pulled up in front of the
| judge who granted the restraining order. The browser vendor
| says "ah, but the court order only says the certificate
| can't be revoked, we didn't revoke it we blacklisted it"
| and the judge says "that's a great argument so I'm not
| going to send you to jail, I'm going to send you to prison
| instead"
| gaadd33 wrote:
| How does a judge send a company to prison or jail?
| newprint wrote:
| If there is a contract between DigiCert & Alegeus Technologies,
| stating that DigiCert could potentially revoke contracts within
| 24h and Alegeus client contracts state that clients have more
| than 24h to deal with expired certs, then I feel like it's
| Alegeus problem, who over-promised to their clients and should
| have carefully read their contracts.
| jandrese wrote:
| More to the point, Alegeus should have a system in place that
| can re-issue certificates within 24 hours since that is the
| timetable that their registrar works under.
|
| The lawsuit smells like them trying to buy time to fix their IT
| problems. Even if it loses or gets tossed out all they needed
| was that preliminary injunction. It's really sucky for DigiCert
| who have to choose which party to anger, the courts or the
| browser manufacturers, just because this healthcare company
| doesn't have its act together.
| Filligree wrote:
| It would seem that the contact between DigiCert and Alegeus
| said nothing about revocation.
| agwa wrote:
| The Master Services Agreement (https://storage.courtlistener.
| com/recap/gov.uscourts.utd.149...) states:
|
| > The Service Specific Terms are incorporated by reference
| into this Agreement
|
| > "Service Specific Terms" mean additional terms specific to
| certain Services as set forth at www.digicert.com/service-
| specific-terms, which are incorporated herein to the extent
| applicable to any specific Services procured by Customer
| hereunder (which includes the Certificate Terms of Use with
| respect to Customer's use of any Certificates).
|
| https://www.digicert.com/service-specific-terms states:
|
| > The Certificate Terms of Use available at
| www.digicert.com/certificate-terms (the "Certificate Terms of
| Use") apply to Certificates other than Qualified Certificates
| and PKIoverheid Certificates requested or issued by Customer.
| The applicable Certificate Terms of Use are incorporated
| herein by reference.
|
| https://www.digicert.com/certificate-terms states:
|
| > DigiCert may revoke a Certificate without notice for the
| reasons stated in the CPS, including if DigiCert reasonably
| believes that:
|
| > Industry Standards or DigiCert's CPS require Certificate
| revocation, or revocation is necessary to protect the rights,
| confidential information, operations, or reputation of
| DigiCert or a third party.
| hn_throwaway_99 wrote:
| I see way too many comments here that are basically saying "Why
| aren't these thousands of contracted orgs able to drop everything
| at the drop of a hat to update their DNS." Here is my
| understanding of the issue:
|
| 1. Alegeus creates white-labeled sites for their customers, but
| those sites are hosted on customer domains (usually for ease of
| sharing login cookies across subdomains on the customer), e.g.
| alegeus.customerdomain.com.
|
| 2. So the issue isn't necessarily that _Alegeus_ can 't update
| their systems, it's that their many customers can't update their
| DNS records fast enough to point to the new certs.
|
| IMO, if you built a system (and I'm talking about DigiCert here)
| that expects thousands of customers can update everything at the
| drop of a hat to update their code, you built a bad system. I
| don't think it's too different from CrowdStrike, except it's as
| if CrowdStrike gave 24 hours notice instead, saying "Hey, all of
| our customers, just a heads up we're going to break all your
| machines in 24 hours, you really should have planned better if
| you can't handle that."
|
| Alegeus is the one suing here because they've got tons of
| customers, but make no mistake this would cause outtages for
| _tons_ of other companies that use DigiCert.
| Hizonner wrote:
| Um, what are you talking about?
|
| Unless you use DANE (which _nobody_ does, and the browsers don
| 't check it even if they did), you do not have to change any
| DNS records, or touch the DNS at all, to update a certificate.
| There is no DNS involvement. Period.
|
| DANE _should_ be the norm, but it 's not.
| hn_throwaway_99 wrote:
| Perhaps I am misunderstanding what needs to happen, but since
| Alegeus itself doesn't control the domains on which its sites
| are hosted, all of its clients would basically need to rerun
| this procedure,
| https://docs.digicert.com/en/certcentral/manage-
| certificates...
| Hizonner wrote:
| OK, true, that's one of the (inadequate) methods that the
| CA/BF permits for verification at issuance, and I had
| forgotten about that.
|
| You can also verify over HTTP or email (egad...). You would
| _think_ that if Alegeus controlled the HTTP servers and not
| the DNS servers, Alegeus would opt for HTTP verification,
| but I concede that they may have built everything around
| DNS verification.
| temac wrote:
| > that's one of the (inadequate) methods that the CA/BF
| permits for verification at issuance
|
| Why inadequate (in the absolute)? This can be automated
| and let's encrypt allows verification through DNS,
| moreover this allows verification for wildcard
| certificates.
|
| Now in this _particular_ case maybe they should have gone
| through HTTP, and even automated with ACME. But there is
| nothing inadequate in the absolute in DNS verification.
| Besides allowing wildcard it also allows verification
| when you don 't control the web server(s), when you don't
| even have a webserver at all, when the standard ports are
| occupied for something else, etc.
| mynameisvlad wrote:
| > IMO, if you built a system (and I'm talking about DigiCert
| here) that expects thousands of customers can update everything
| at the drop of a hat to update their code, you built a bad
| system.
|
| Per the article, this is _required_ to be a CA. It 's not a
| DigiCert thing, it's a baseline requirement. If you can't
| revoke your certificates quickly, you risk being delisted by
| Google/Mozilla/Microsoft et al. Entrust was just recently
| dropped in part because of this very thing.
| x0x0 wrote:
| Sure, but it's still DigiCert's fault doubly:
|
| 1 - they made the screwup;
|
| 2 - DigiCert appear not to have put language into their MSAs
| and so forth that this was a thing that could happen and
| customers had better be prepared to do it. Why shouldn't
| Alegeus sue DigiCert to force DigiCert to conform to the
| terms of the agreement that DigiCert agreed to with Alegeus?
| Palomides wrote:
| would you say the same thing if these certs were being actively
| misused? how long should they get to run amok?
|
| if you don't have a fast process for replacing this stuff,
| you're going to have a bad day eventually
| bastawhiz wrote:
| > it's that their many customers can't update their DNS records
| fast enough to point to the new certs.
|
| This is almost certainly wrong. With the exception of DNSSEC,
| there's no interaction between DNS and certificates. There's no
| DNS record that points at a certificate. The HTTP server that
| terminates the connection is the server that has the
| certificate.
|
| TXT records can validate a domain, but you can also validate a
| domain by hosting a file on that domain to show that you
| control it.
|
| I find it more likely that Alegeus is giving customers
| appliances or VMs that they run themselves, and those are each
| configured with a baked-in certificate. Which is bad anyway
| because it means the customer needs to be involved in the
| certificate update process.
| plorkyeran wrote:
| If you build a system which relies on WebPKI certificates and
| you don't have the ability to quickly replace those
| certificates, your system is insecure and _will_ break. This is
| fairly central to how the entire concept of a publicly trusted
| CA works.
| michaelt wrote:
| _> In any event, given that Alegeus was aware that DigiCert is
| required to revoke certificates within 24 hours, one wonders why
| Alegeus went ahead and signed agreements with their customers
| that required a lengthy notice period before changing
| certificates._
|
| It seems obvious enough how this would come about:
|
| 1. The healthcare company knows better than to do a Crowdstrike,
| where "just a configuration change" is rushed out with inadequate
| testing, for "security". As even a configuration change could
| trigger a latent bug, they agree configuration changes should be
| rolled out gradually, and not applied during clients' peak
| operating hours.
|
| 2. The healthcare company could have used "Let's Encrypt" to get
| certificates for free, if they wanted short-term certificates.
| But instead they paid $$$$ to DigiCert for certificates with a 12
| month duration.
|
| 3. Even if Alegeus _had_ read the small print saying mis-issued
| certificates can be revoked with only 24 hours of notice - they
| quite reasonably assumed DigiCert would have controls in place to
| _avoid_ mis-issued certificates. Hasn 't DigiCert told the
| browser vendors precisely that?
| mynameisvlad wrote:
| > 3. Even if Alegeus had read the small print saying mis-issued
| certificates can be revoked with only 24 hours of notice - they
| quite reasonably assumed DigiCert would have controls in place
| to avoid mis-issued certificates. Hasn't DigiCert told the
| browser vendors precisely that?
|
| I mean, sure, in theory there should be controls in place, but
| no company or process is perfect. That's why there's a process
| in place to respond to problems with issuance in the first
| place.
|
| DigiCert is responding in the way the browser vendors mandate;
| 24-hour revocation, per the article, is a baseline requirement
| to be a CA.
| bastawhiz wrote:
| > they quite reasonably assumed DigiCert would have controls in
| place to avoid mis-issued certificates
|
| This makes no sense. The revoke certs when there's been an
| incident. Telling customers that they might need to revoke
| certs in the event of an incident should mean that they are
| going to be free from incidents??
|
| If Stripe tells me webhooks might be delivered more than once,
| that does NOT mean I would assume they have controls in place
| to prevent webhooks from being delivered more than once. That's
| just wild to assume.
| TheJoeMan wrote:
| My very naive reading is Alegeus may have assumed this
| certificate revocation would only be from their own screw up
| - such as publishing their private key somewhere - and they
| would of course eat some costs on the SLA's. Instead, we have
| Alegeus's vendor saying there was technically the
| possibility, unconfirmed, that the key could have been
| compromised, and therefore are executing the nuclear option.
| etimberg wrote:
| One thing that i think is clear from the responses here is that
| it's going to be a long time before software engineering is like
| other engineering disciplines.
| wbl wrote:
| You can't sue the health inspector for saying "those rats got
| to go" or the FAA saying "you didn't keep records on this
| plane, it doesn't fly"
| 8organicbits wrote:
| The bug sounds interesting. The underscore is needed to ensure
| the CNAME isn't a valid domain name. For services that offer
| subdomains for customers, an attacker could get a certificate for
| the parent domain. Like if example.com offered subdomains based
| on a name the user chooses, the attacker would start a
| certificate verification process and get a random value from
| Digicert. They'd then create an account with that random value,
| causing the example.com service to create a CNAME for
| <random>.example.com. Digicert then sees the random value and
| issues the certificate for example.com to the attacker. The
| underscore is there as _RANDOM.example.com isn't a valid domain
| name, so you'd expect services to reject these sorts of domains.
|
| Edit:
|
| > Therefore, this is truly a security-critical incident, as there
| is a real risk (not a negligible 2^-150 risk as implied by
| DigiCert) that this flaw could have been exploited to get
| unauthorized certificates. Revocation of the improperly validated
| certificates is security-critical.
|
| Yeah... have they proven that this wasn't exploited? With 83,000
| certificates getting revoked this isn't a small scale mistake.
| psnyder wrote:
| The original email response addressed to Alegeus from the
| DigiCert CEO can be found here
| https://storage.courtlistener.com/recap/gov.uscourts.utd.149...
| CaliforniaKarl wrote:
| For discussion on the DigiCert issue that triggered this, have a
| look at the following HN discussions (which are still active!)
|
| DigiCert's blog post...
|
| DigiCert Revocation Incident (CNAME Domain Validation)
| (digicert.com) -- https://news.ycombinator.com/item?id=41104504
| -- 136 points by vitaliyf 1 day ago
|
| ... and the Bugzilla entry where DigiCert reported this to
| Mozilla:
|
| Digicert pauses mass certificate revocation due to legal action
| from customer (bugzilla.mozilla.org) --
| https://news.ycombinator.com/item?id=41114794 -- 9 points by
| frakkingcylons 23 hours ago
___________________________________________________________________
(page generated 2024-07-31 23:01 UTC)