[HN Gopher] Why Certificate Lifecycle Automation Matters
___________________________________________________________________
Why Certificate Lifecycle Automation Matters
Author : pabs3
Score : 78 points
Date : 2024-01-30 13:52 UTC (9 hours ago)
(HTM) web link (www.hezmatt.org)
(TXT) w3m dump (www.hezmatt.org)
| Macha wrote:
| I do wish the ACME ecosystem was as fire and forget as it's
| always aimed to be. It feels like an annual occurrence where I
| need to fix... Something misfiring with acme cert issuance and
| I've tried certbot, acme.sh, Lego and they've all had issues.
| Like maybe when first issued the tool decided to use ZeroSSL but
| on reissue decides to use Let's Encrypt and fails because one
| requires an email and the other doesn't. Or the verification
| method gets deprecated and requires manual software updates. Or
| the tool decided to rename it's environment variables and auto
| updated so now it can't find it's DNS auth keys.
| akerl_ wrote:
| Interesting. I was using acme.sh and switched to lego, largely
| because I manage my servers with Puppet and acme.sh's config
| format didn't lend itself well to config management.
|
| I renew certs for several boring things like nginx instances,
| but also things like my printer's web portal, a Synology NAS,
| and a Unifi console that require bespoke commands to reload the
| cert.
|
| I've got monitors that fire if any of my certs drop below 27
| days before expiry, and over the past ~2 years, 100% of the
| problems have been with the custom hooks. I'll log into the
| server and find that it _did_ reissue the cert, but the command
| to make the Synology ingest the new cert failed for some arcane
| reason. The servers where it's just running systemctl restart
| nginx or similar don't fail.
| Macha wrote:
| Huh, the environment variable thing was specifically aimed at
| acme.sh which rather arbitrarily changed the config value
| from ACMEDNS_UPDATE_URL to ACMEDNS_BASE_URL, never
| acknowledged this in a changelog and then silently failed
| after an automatic upgrade as recommended by the default
| install:
|
| https://github.com/acmesh-
| official/acme.sh/commit/2ce145f359...
|
| It's also cleared out my .account.conf files when run on the
| suggested cron.
|
| I've started using updown which also monitors my TLS certs
| simply because I no longer trust the process to work as
| documented.
| akerl_ wrote:
| Ah, I plausibly didn't hit this because I was permanently
| on a fork of the acme.sh codebase, after I opened a PR to
| support Route53 subzones and it was ignored.
| Aissen wrote:
| I'm using certbot and acme/autocert, and it has been a few
| years since I last fiddled with this (even though I have CD
| happening !).
|
| I was a bit mad at certbot for having chosen snap as its
| recommended installation platform, but my solution to use pypi
| + CD seems to be quite stable (famous last words).
|
| The go services I run with autocert have had no manual
| intervention ever since the ACMEv2 move.
| crotchfire wrote:
| This is why I use .onion urls instead of WebPKI.
| memset wrote:
| Shameless plug: I've built a tool that automatically generates
| certs and uploads to destinations.
| https://github.com/poundifdef/certmaster
|
| It uses Lego under the hood to issue certs, and then has custom
| connectors to upload to destinations. Right now those are email
| (indeed, more humans handling certificates), sftp, and hetzner
| load balancers. I'm working on adding the ability for it to
| automatically renew and re-upload when certs are 30 days from
| expiration.
| throw0101d wrote:
| A 'competitor' to this would be GetSSL which is a pure-shell
| ACME client (plus OpenSSL and cURL) and can be executed on one
| host, but send verification tokens to remote systems (where you
| may not have _cron_ access):
|
| > _Get certificates for remote servers - The tokens used to
| provide validation of domain ownership, and the certificates
| themselves can be automatically copied to remote servers (via
| ssh, sftp or ftp for tokens). The script doesn 't need to run
| on the server itself. This can be useful if you don't have
| access to run such scripts on the server itself, as it's a
| shared server for example._
|
| * https://github.com/srvrco/getssl
| at_a_remove wrote:
| I _get_ why it is important, but certs can be a tremendous pain
| to deal with when you 're in a niche industry and reliant on the
| only vendor in town, and the consequences of a broken cert were
| minimal.
| 0xbadcafebee wrote:
| The whole cert regime is wrongheaded. Issuing certs is
| vulnerable to exploits, the methods are clunky and problematic,
| certs expire so quickly that stuff is breaking more often than
| security issues could ever come up, there's no consideration to
| private networks, we still don't have content integrity without
| encryption, content proxy caching is dead, CAs can do what they
| want (there's too many CAs, cert logs aren't required to be
| validated in real time, you can't force a CA _not_ to do
| something shady under a government gag order, etc), and there
| 's a fundamental conflict between the owner of a domain and the
| issuer of a cert and the owner of a cert and the owner of some
| IP space or DNS server records.
|
| But because it's this giant behemoth involving multiple
| industries, technological components, and vested interests, it
| will never change, save to implement bandaid after bandaid
| after hack after kludge. Literally the only thing that changes
| it is when someone with enough power (a market-dominant
| browser) forces it and everyone has to comply, or face creating
| a total shit-show due to lack of conformance. And we're all
| hopelessly dependent on it so it's not like we can do anything
| about it. The security of the web is a Kafkaesque nightmare.
| XorNot wrote:
| Here's the thing though: the alternatives are also god-awful,
| and in part it is because the people who make them wind up
| being very stupid about their choices - i.e. if you want a
| distributed trust environment fine, but your first, primary
| and most important use-cases are _still_ going to be
| government, and corporate, and specifically financial.
|
| If your "libre" plan does not involve a way to bring those
| entities onboard and direct priority to servicing their
| needs, then _no one_ is going to use it because we all
| actually live in the real world where those are the things we
| _need_ security in communicating with.
| gur48 wrote:
| More revocations != more customers losing keys.
|
| It may just as well mean more capable audits to detect
| lost/compromised keys.
|
| The article also says nothing of certificate validity periods,
| which are critical to their lifecycle and analysis.
|
| The conclusion that humans degrade information security is true
| but ultimately removed from the reality of operating a business.
| throw0101d wrote:
| A reminder that if you an internal-only server where the typical
| _http-01_ verification connection method will not work, one can
| use _dns-01_ by using DNS aliasing /CNAME, especially if you
| cannot easily/dynamically update DNS records:
|
| * https://dan.langille.org/2019/02/01/acme-domain-alias-mode/
|
| * https://www.eff.org/deeplinks/2018/02/technical-deep-dive-se...
|
| So if you want a cert for _www.internal.example.com_ , you will
| first have do a one-time change to have a __acme-
| challenge.www.internal..._ CNAME created to point to any other
| (sub-)domain where you can easily update things dynamically,
| e.g., _www-internal.example-dnsapi.com_.
|
| When request the cert for "www.internal...", LE/ACME will look up
| the corresponding _acme-challenge record, and go to " __acme-
| challenge.www-internal.example-dnsapi.com_. The nonce token will
| be there in the 'final' destination following the CNAME in a TXT
| record, which shows LE/ACME that you control the DNS chain.
|
| To do the DNS updating, you can use a CLI/Python library like
| Lexicon, which supports dozens of APIs:
|
| * https://github.com/AnalogJ/lexicon
| ryandv wrote:
| One advantage of the DNS01 approach is that you can also order
| certificates for standby instances behind a load balancer, all
| fronted by the same domain name. Since the standbys are not
| serving traffic, you can't get an HTTP01 challenge/response
| through, but you can still do so with DNS01 and get a
| certificate delivered to that machine, so that it's always
| ready to serve TLS in the event of a failover.
| JoshTriplett wrote:
| tls-alpn-01 woks really well as well. You don't have to mess
| with DNS at all, or have an HTTP port open; you just listen on
| HTTPS.
| throw0101d wrote:
| Yeah, except for the the _internal-only server_ part where
| the hostname may not even be accessible to the public
| Internet (certainly not as an A(AAA) record).
|
| And given the number of internal applications many large/r
| organizations have, giving all those internal dev teams
| update access to the main corporate domain is not something
| I'd really feel comfortable doing.
|
| But being able to give API access to (e.g.)
| foo.dnsauth.example.com is something I have done: main domain
| needs to be touched once for the CNAME, and the sub-domain
| isn't critical for day-to-day operations of a company.
| JoshTriplett wrote:
| The post I was replying to used www.internal.example.com ,
| which looked like a public name. Internal-only names should
| not look like subdomains of public DNS names.
|
| But yes, if you want a certificate for a name that isn't
| accessible externally and _can 't_ be made accessible
| externally, dns-01 works.
| throw0101d wrote:
| > _Internal-only names should not look like subdomains of
| public DNS names._
|
| Since when? I, and many other people I know, have been
| doing that for decades.
|
| If I (the company) own _example.com_ , I can choose to
| have some sub-domains internal-only, some external-only,
| some DNS records resolve to the same IP value regardless
| of in/outside, and other DNS records resolve to different
| IP values depending on in/outside.
|
| * https://en.wikipedia.org/wiki/Split-horizon_DNS
| p_l wrote:
| However, internal domains should have their top domain
| externally registered and controlled... so ultimately
| there's no difference because you're going to need at
| least a dummy zone file to satisfy registrar.
| diarrhea wrote:
| I own example.com and actually created NS records for
| internal.example.com, to manage that entire zone at an entirely
| different provider. I did that because the original provider
| had terrible API access for Caddy to use. The new one has
| better support, and a breach there does not pwn the root
| domain. I was happy with that setup, but wasn't aware one could
| delegate to an entirely different domain using ACME primitives.
| Makes sense though, it's like the NS approach but simpler, and
| allows using some throwaway domain. Thanks!
| yourapostasy wrote:
| Can anyone with good experiences with a commercially-supported
| ACME stack recommend a vendor they're happy with, where the stack
| was integrated into their own in-house certificate authority? I'm
| especially looking for vendors that have very good documentation
| and tracing introspection to let our own engineers who read the
| documentation rapidly troubleshoot the stack, and our own
| developers build robust integrations to the client to rotate the
| certificates.
| tlyman27 wrote:
| You should look at Venafi and their offerings. They have
| integrations that allow you to connect to inhouse CA for
| certificate issuing. You can also add your own integrations
| when provisioning certificates to machines or choose from some
| of the many existing integrations.
|
| Disclosure: I work for Venafi
| schoen wrote:
| I haven't used it (so I might be excluded from answering your
| question), but I think SmallStep may be relevant to you.
|
| https://smallstep.com/docs/certificate-manager/acme/
|
| At least they are doing a lot of "your own organizational PKI"
| tools and they care about ACME integration.
___________________________________________________________________
(page generated 2024-01-30 23:01 UTC)