[HN Gopher] SSH certificates: the better SSH experience
___________________________________________________________________
SSH certificates: the better SSH experience
Author : jandeboevrie
Score : 199 points
Date : 2026-04-03 09:52 UTC (13 hours ago)
(HTM) web link (jpmens.net)
(TXT) w3m dump (jpmens.net)
| thomashabets2 wrote:
| Every couple of months someone re-discovers SSH certificates, and
| blogs about them.
|
| I'm guilty of it too. My blog post from 15 years ago is nowhere
| near as good as OP's post, but if I though me of 15 years ago
| lived up to my standards of today, I'd be really disappointed:
| https://blog.habets.se/2011/07/OpenSSH-certificates.html
| kaoD wrote:
| I've known SSH certs for a while but never went through the
| effort of migrating away from keys. I'm very frustrated about
| manually managing my SSH keys across my different servers and
| devices though.
|
| I assume you gathered a lot of thoughts over these 15 years.
|
| Should I invest in making the switch?
| ibotty wrote:
| Yes. Caveat: It might not really be worth it if all your
| infrastructure is managed by these newfangled infrastructure-
| as-code-things that are quick to roll out (OpenShift/OKD,
| Talos, etc.) and you have only one repo to change SSH keys
| (single cluster or single repo for all clusters).
|
| There are some serious security benefits for larger
| organizations but it does not sound as if you are part of
| one.
| thomashabets2 wrote:
| If your use case is such that you are frustrated about
| managing keys, host or user keys, then yes it does sound like
| SSH certs would help you. E.g. when you have many users,
| servers, or high enough cartesian product of the two.
|
| In environment where they don't cause frustration they're not
| worth it.
|
| Not really more to it than that, from my point of view.
| otabdeveloper4 wrote:
| You will have to manage your SSH CA certificates instead of
| your keys.
|
| The workflows SSH CA's are extremely janky and insecure.
|
| With some creative use of `AuthorizedKeysCommand` you can
| make SSH key rotation painless and secure.
|
| With SSH certificates you have to go back to the "keys to the
| kingdom" antipattern and just hope for the best.
| jamiesonbecker wrote:
| Exactly. We'd had discussions about building
| https://Userify.com (plug!) around SSH certificates, but
| elected to go with keys instead, because Userify delivers
| most of the good things around certificates without the
| jank and insecurity.
|
| It's not that certificates themselves are insecure
| themselves, it's that the workflows (as the parent points
| out) are awful. We might still add some automation around
| that (and I think I saw some competitor tooling out there
| if you're committed to that path) but I personally feel
| like it's an answer to the wrong question.
| cyberax wrote:
| > With SSH certificates you have to go back to the "keys to
| the kingdom" antipattern and just hope for the best.
|
| Whut? This is literally the opposite.
|
| With CA certs you can create short-lived certificates, so
| you can easily grant access to a system for a short time.
| namibj wrote:
| And what about the CA?
| cyberax wrote:
| It's no different compared to regular SSH private keys.
| You need to protect it from compromise.
|
| However, it provides you an additional layer of
| protection, because it does not need to be on the
| critical path for every SSH connection. My CA is a
| Nitrokey HSM, for example. I issue myself temporary certs
| that are valid only for 6 hours for ephemeral private
| keys.
| anyfoo wrote:
| A big problem I have with ssh carts is that they are not
| universally supported. For me, there is always some device or
| daemon (for example tinyssh in the initramfs of my gaming pc
| so that I can unlock it remotely) that only works with "plain
| old ssh keys". And if I have to distribute and sync my keys
| onto a few hosts anyway, it takes away the benefits.
| TZubiri wrote:
| Might actually be a positive instead of a negative. Gaming
| use-cases should have not any effect on security policies,
| these should be as separate as possible, different auth
| mechanisms for your gaming stuff and your professional
| stuff ensures nothing gets mixed.
| namibj wrote:
| Upgrade to a better one in initramfs?
| AceJohnny2 wrote:
| Adding to this: while certs are indeed well-supported by
| OpenSSH, it's not always the SSH daemon used on alternate
| or embedded platforms.
|
| For example, OpenWRT used Dropbear [1] instead, which does
| not support certs. Also, Java programs that implement SSH
| stuff, like Jenkins, may be doing so using Apache Mina [2]
| which, though the underlying library supports certs, it is
| buggy [3] and requires the application to add the UX to
| also support it.
|
| [1] https://matt.ucc.asn.au/dropbear/dropbear.html
|
| [2] https://mina.apache.org/sshd-project/
|
| [3] I've been dealing for years with NullPointerExceptions
| causing the connection to crash when presented with certain
| ed25519 certificates.
| dizhn wrote:
| I am keeping an eye on the new (and alpha) Authentik agent
| which will allow idp based ssh logins. There's also SSSD
| already supported but it requires glibc (due to needing NSS)
| meaning it's not available on Alpine.
| gnufx wrote:
| If you mean using OIDC, in that space there's at least
| https://github.com/EOSC-synergy/ssh-oidc,
| https://dianagudu.github.io/mccli/ and OpenPubkey-ssh
| discussed in https://news.ycombinator.com/item?id=43470906
| (which might mention more).
|
| How does SSSD support help with SSH authN? I know you can
| now get Kerberos tickets from FreeIPA using OIDC(?), but I
| forget if SSSD is involved.
| cyberax wrote:
| It depends on what you want to do. CA certs are easy to
| manage, you just put the CA key instead of the SSH public key
| in authorized_keys.
|
| They also provide a way to get hardware-backed security
| without messing with SSH agent forwarding and crappy USB
| security devices. You can use an HSM to issue a temporary
| certificate for your (possibly temporary) public key and use
| it as normal. The certificate can be valid for just 1 hour,
| enough to not worry about it leaking.
| papyDoctor wrote:
| Another useful feature of SSH certificates is that you can sign
| a user's public key to grant them access to a remote machine
| for a limited time and as a specific remote user.
| TZubiri wrote:
| The capacity to grant access as a specific remote user is
| present without certs as well right? The typical
| authorized_keys file lives under a user directory and grants
| access only to that user.
| blueflow wrote:
| The main advantage of certificates is that you are able to
| do that from the CA without touching the target machine.
| lokar wrote:
| Exactly. This is really useful in larger organizations
| where you may want more complex rules on access. For
| example, you can easily build "break glass" or 2nd party
| approved access on demand. You can put whatever logic you
| need in a CA front-end.
|
| You can also make all the certs short-lived (and only
| store them in ram).
| pphysch wrote:
| Certs may still be the right approach, but OpenSSH also
| supports an AuthorizedKeysCommand which could be a secure
| HTTPS request to a central server to pull down a
| dynamically generated authorized_keys file content for
| the particular user and host.
|
| If your endpoints _can_ securely and reliably reach a
| central server, this gives you maximum control (your
| authorized_keys HTTPS server can have any custom business
| logic you want) without having to deal with certs /CAs.
| Stefan-H wrote:
| I think the scary reality is most people conflate "keys" and
| "certificates". I have worked with security engineers that I
| need to remind that we do not use SSH certs, but rather key
| auth, and they have to think it through to make it click.
| tracker1 wrote:
| I'm consistently amazed how many developers and security
| professionals don't have a clear understanding how PPK even
| works conceptually.
|
| Things like deploying dev keys to various production
| environments, instead of generating/registering them within
| said environment.
|
| One of the worst recent security examples... You can't get
| this data over HTTPS from $OtherAgency, it's "not secure" ...
| then their suggestion is a "secure" read-only account to the
| other agency's SQL server (which uses the same TLS 1.3 as
| HTTPS). This is from person in charge of digital security for
| a government org.
| tolciho wrote:
| Or when the security team at some other company emails you
| their private key.
| tialaramex wrote:
| One key technological cause is that PKCS#12 standardizes a
| format (you've most likely seen it as .PFX files) in which a
| certificate and its associated private key are bundled. This
| is in an effort to simplify the software...
|
| So you get a situation where the lay person is given a
| "certificate" but it's not really just the certificate it's a
| PFX file and so e.g. no they mustn't show it you, it has
| their private key inside it and so you will learn that key
| and if you're honest you've just ruined their day because
| they need to start over...
|
| I would say in my career I've had at least two occasions
| where I did that and I felt awful for the person, because I
| had set out to help them but now things are worse, and I've
| had a good number of later occasions where I spent a lot more
| of their time and mine because I knew I need to be very sure
| whether their "certificate" is actually a certificate (which
| they can show me, e.g. Teams message me the file) or a PFX
| file (thus it has their private key) and I must caution them
| to show nobody the file yet also try to assist them.
| V-eHGsd_ wrote:
| oh man, I referred back to your blog post when I wrote the ssh
| certificate authority for $job ... ~10 years ago.
|
| Thank for writing it!
| Thom2000 wrote:
| Sadly services such as Github don't support these so it's mostly
| good for internal infrastructure.
| lights0123 wrote:
| They do, for Enterprise customers only:
| https://docs.github.com/en/enterprise-cloud@latest/organizat...
|
| They've rolled their host key one time, so there's little
| reason for them to use it on the host side.
| linsomniac wrote:
| In our dev/stg environment we reinstall half our machines every
| morning (largely to test our machine setup automation), and SSH
| host certificates make that so much nicer than having to persist
| host keys or remove/replace them in known_hosts. Highly
| recommended.
| Tepix wrote:
| The author lists all the advantes of CA certificates, yet doesn't
| list the disadvantages. OTOH, all the many steps required to set
| it up make the disadvantages rather obvious.
|
| Also, I've never had a security issue due to TOFU, have you?
| adrian_b wrote:
| TOFU is convenient, but not necessary.
|
| Choosing to use TOFU is a distinct choice from the choice of
| using the keys generated by SSH, instead of using certificates.
|
| If you do not want to use TOFU, for extra security, you just
| have to pair the computers by copying between them the
| corresponding public keys through a secure channel, e.g. by
| using a USB memory.
|
| Using certificates does not add any simplification or any extra
| security.
|
| For real security, you still must pair the communicating
| computers by copying between them the corresponding
| certificates, through a secure channel, e.g. a USB memory.
|
| When you use for HTTPS the certificates that have come with
| your Internet browser, you trust that the installer package for
| the browser has come to that computer through a secure channel
| from the authority that has created the certificates. This is
| usually an assumption much more far fetched than the assumption
| that you can trust TOFU between computers under your control.
|
| Certificates may be useful in big organizations, if other
| functionality is needed beyond just establishing secure
| communication channels, e.g. if you want to use certificate
| revocation.
|
| In the list of "advantages" enumerated in the parent article,
| more than half of them are false, because if certificates are
| implemented correctly, completely equivalent actions must be
| executed when SSH keys without TOFU are used and when
| certificates are used.
|
| Perhaps the author meant by writing some of the "advantages"
| that the actions that supposedly are no longer needed with
| certificates are done by an administrator, not by the user.
| However that is also applicable with SSH. An administrator
| could install the certificates, so that no action is required
| from the user, but an administrator can also install the SSH
| public keys, so that no TOFU is ever needed from the user.
|
| Using certificates requires exactly the same steps like using
| keys generated by SSH (i.e. generating certificates and copying
| them between computers through secure channels, to pair the
| servers and the authorized users), but it may need additional
| steps, caused by the fact that certificates provide additional
| functionality.
| gkoz wrote:
| Are you pairing computers by copying certificates to visit
| this site?
| _hyn3 wrote:
| Touche.. actually a good point, but actually those are two
| different situations. With one, I'm accessing a website and
| trusting that the certificate is signed by someone I trust;
| so the trust in my browser certificates (which include
| certificates from hundreds of certificate authorities all
| over the world, any one of which could be compromised,
| robbed, or controlled by an adversarial person or even
| government) is extended to the site that I'm visiting. To
| say this is weak sauce rather understates how bad this
| actually is. (To paraphrase Churchill, this is the worst
| possible design, except for all the rest.)
|
| With the other, I'm logging into a server for the first
| time (and I could simply deploy the same trusted host key
| to all my ssh servers via an autoscaling configuration or
| whatever). I think it's debatable if TOFU is worse or
| better than your (granted clever) metaphor.
|
| (to those who'd recommend userify, yes - great for the
| client login issue and definitely increases security, but
| to parent's point, TOFU is still needed unless you want to
| distribute _host_ pubkeys)
| adrian_b wrote:
| Pairing is absolutely necessary for bidirectional
| authentication, where each party must verify the identity
| of the other end.
|
| To visit this site, there is no pairing, because the site
| does not know who I am.
|
| In order to verify the identity of the HN site, I must
| trust that the maintainers of the installation packages of
| the browsers that I use (Firefox, Vivaldi, Chromium) have
| ensured that the built-in certificates have reached me
| through a secure path. This actually requires much more
| trust than when someone answers "yes" to the SSH unknown
| host message.
|
| If I use certificates for accessing e.g. the network of my
| employer, then my work computer must be paired with some
| corporate server, i.e. a unique certificate has been
| generated for myself and it has been copied to some
| certificate authority server for signing and then to my
| computer, and also a certificate of the local certificate
| authority has been copied to my personal computer.
|
| While pairing is unavoidable for bidirectional
| authentication, it is not necessarily direct between the
| end points. Both end points must have been paired with at
| least one other computer but they need not have been paired
| between themselves previously if there exists some path
| through secure connections that have been originally
| created by pairing.
|
| When certificates are used, usually the pairings are not
| done directly between end points, but each computer must be
| paired with the server hosting the certificate authority.
|
| The term "pairing" is not used frequently, but it should
| have been preferred, because frequently the users do not
| understand which are the exact actions on which the
| security of their communications depend, which leads to
| various exploits. The critical security actions are those
| that perform the pairing.
|
| "Pairing" of 2 systems, e.g. A and B, means that some
| information must be transmitted through a secure channel
| from A to B and some other information must be transmitted
| through a secure channel from B to A. An alternative
| pairing method is to generate both pieces of information on
| one of the 2 systems and transmit both of them through a
| secure channel to the other. The information exchange
| channels must already be secure, because before pairing
| authentication is impossible.
|
| The pairing between a PC and the server hosting the
| certificate authority can be done in various ways,
| depending on where the PC certificate is generated. If the
| certificate is generated at the certificate authority than
| both it and the root certificate must be copied through a
| secure channel to the PC. If the certificate is generated
| on the PC, it must be sent through a secure channel to the
| CA for signing, then it must be sent back also through a
| secure channel.
|
| In practice, administrators are not always careful enough
| for the channels through which certificates are copied to
| be really secure. For instance they may be copied through
| network links that are not yet authenticated, which is
| equivalent with the TOFU method optionally used by SSH.
| akerl_ wrote:
| > Also, I've never had a security issue due to TOFU, have you?
|
| This is a bit like suggesting you've never been in a car crash,
| so seat belts must not be worth considering.
|
| Do you feel that beyond the obvious and documented work in
| setting them up, there are disadvantages to using SSH
| certificates?
| otabdeveloper4 wrote:
| Your ISP or telecom has to be compromised for TOFU to be
| relevant to anything. In practice that never happens.
| fc417fc802 wrote:
| Not just your ISP. If an attacker slipped a device onto
| your LAN _and also_ you happened to be sshing to a new box
| for the first time then TOFU poses a problem. But that 's
| an awfully limited attack surface. It's similar to the
| difference between leaking a fax while it's sent versus
| leaking years old emails that are just sitting there on an
| internet accessible server.
|
| As for your ISP I think you should never rely on TOFU over
| the public internet. If you really don't want to do ssh
| certs it's easy enough to make the host key available
| securely via https.
| adrian_b wrote:
| Certificates provide extra features, like revocation.
|
| However, if you do not need the extra features provided by
| certificates, using SSH-generated keys is strictly equivalent
| with using certificates and it requires less work.
|
| TOFU is neither necessary nor recommended, it is just a
| convenience feature, to be used when security may be lax.
|
| The secure way to use SSH is to never use TOFU but to pair
| the user and the server by copying the public keys between
| the 2 computers through a secure channel, e.g. either by
| using a USB memory or by sending the public keys through
| already existing authenticated encrypted links that pass
| through other computers. (Such a link may be a HTTPS download
| link.)
|
| When using certificates, a completely identical procedure
| must be used. After certificates are generated, like also
| after SSH keys are generated, the certificates must be copied
| to the client computer and the server computer through secure
| channels.
| palata wrote:
| > TOFU is neither necessary nor recommended
|
| Just to make it clear: this does not mean that it is fine
| to blindly accept the message on first use.
|
| The "secure way" implies copying the server's public key as
| well, which people generally don't do, right? Which is
| equivalent to verifying the fingerprint shown with the TOFU
| message, correct?
| adrian_b wrote:
| Like I have said the secure way requires the secure
| copying of both keys before the first connection attempt
|
| The server public key must be copied into "known_hosts"
| on the client, while the client public key must be copied
| into "authorized_keys" on the server.
|
| When this is the procedure that is always followed, any
| message shown by SSH about an unknown host means that the
| connection must be aborted, because the identity of the
| server is unknown.
|
| You cannot truly verify the "fingerprint" displayed by
| SSH, unless you simultaneously have access to another
| computer, where you have a copy of the fingerprint. What
| is usually meant by "verifying" is that you remember a
| few digits of the fingerprint, and those match.
|
| You could have copied the fingerprint from the server, to
| be able to truly verify it, but that does not make sense,
| because in that case you should have copied the entire
| key, not just the fingerprint, and you should have
| installed it in the client.
|
| When you use only authentication with digital signatures,
| it does not make sense to use any other procedure,
| because you must make at least one of the two copies
| anyway, so when copying the client key to the server you
| can take the server key, to carry it back to the client.
|
| The TOFU method is meant to be used together with
| password-based authentication, in less secure
| applications, where no physical access to the server is
| required for setting up SSH on the client.
|
| By "less secure" I mean for example applications
| equivalent to HTTPS, where the client is not really
| authenticated, e.g. when providing a public password
| allowing read-only access to an SSH server through
| Internet.
| palata wrote:
| > Like I have said
|
| Sure, I just wanted to make sure that nobody would
| understand "TOFU is neither necessary nor recommended,
| just ignore that message and say "yes" when it appears".
|
| Both ends need to be sure of _who_ is on the other end,
| and there are different ways to achieve that. The way SSH
| works is that if you haven 't copied the server public
| key locally, it will explicitly ask you to verify it the
| first time.
|
| I am not sure that "SSH does TOFU". SSH asks you to
| verify. The human who YOLOs it and approves it without
| checking is the one doing TOFU, and this is not really
| secure.
| akerl_ wrote:
| > When using certificates, a completely identical procedure
| must be used. After certificates are generated, like also
| after SSH keys are generated, the certificates must be
| copied to the client computer and the server computer
| through secure channels.
|
| That is not the case, and is a major advantage of
| certificates.
| zamadatix wrote:
| If you have some form of access to set up the CA config on the
| box before connecting then you can use the same access channel
| to avoid needing to rely on TOFU for setting up the key access
| all the same.
|
| This can be anything from being part of the install script to
| customized deployment image to physical access to access via a
| host in virtualized scenarios.
|
| TOFU only really comes into play when the box is already set up
| and you have no other way to load things onto the box other
| than connecting via SSH to do so. But, again, that would be the
| same story if you were intending to go the certificate approach
| too.
| znpy wrote:
| > Also, I've never had a security issue due to TOFU, have you?
|
| You haven't _yet_ had a security issue due to TOFU - FTFY.
| jcalvinowens wrote:
| You can also address TOFU to some extent using SSHFP DNS records.
|
| Openssh supports checking the DNSSEC signature in the client, in
| theory, but it's a configure option and I'm not sure if distros
| build with it.
| jsiepkes wrote:
| On top of that you would need something to secure DNS. Like
| DNSSEC or at the very least use DNS with TLS or DNS over HTTP.
| None of these are typically enabled by default.
| jcalvinowens wrote:
| Anything that uses system-resolved is probably doing DNSSEC
| validation by default. It's becoming much more common.
|
| Additionally, as I mentioned, openssh itself has support for
| validating the DNSSEC signature even if your local resolver
| doesn't. I actually don't think it can use the standard
| resolver for SSHFP records at all, but I'm not sure.
| fc417fc802 wrote:
| Any idea if there's a standardized location, something like
| /.well-known/ssh?
| bobo56539 wrote:
| With the recent wave of npm hacks stealing private keys, I wanted
| to limit key's lifetimes.
|
| I've set up a couple of yubikeys as SSH CAs on hosts I manage. I
| use them to create short lived certs (say 24h) at the start of
| the day. This way i only have to enter the yubikey pin once a
| day.
|
| I could not find an easy way to limit maximum certificate
| lifetime in openssh, except for using the
| AuthorizedPrincipalCommand, which feels very fragile.
|
| Does anyone else have any experience with a similar setup? How do
| you limit cert max lifetime?
| agwa wrote:
| Instead of using a CA, why not set the key's PIN policy to
| "once" and use an agent (e.g.
| https://github.com/FiloSottile/yubikey-agent/) that holds an
| active session to the yubikey? You start the agent at the
| beginning of the day, enter the PIN once, and then stop the
| agent at the end of the day.
| bobo56539 wrote:
| Wanted to avoid having the key in USB slot all the time. I
| have the version that sticks out and i carry it on my key
| chain. So it's easy to break in a laptop.
| moviuro wrote:
| All those articles about SSH certificates fall short of
| explaining how the revocation list can/should be published.
|
| Is that yet another problem that I need to solve with syncthing?
|
| https://man.openbsd.org/ssh-keygen.1#KEY_REVOCATION_LISTS
| blipvert wrote:
| If you generate short lived certificates via an automated
| process/service then you don't really need to manage a
| revocation list as they will have expired in short order.
| jamiesonbecker wrote:
| But then you can't log in if your box goes offline for any
| reason.
| blipvert wrote:
| Hmm. For user certs you can have the service sign them for,
| say an hour, so long as you can ssh to your server in that
| time then there's no need for any other interaction.
|
| Sure you need your signing service to be reasonably
| available, but that's easily accomplished.
|
| Maybe I misunderstand?
| jamiesonbecker wrote:
| That works for authn in the happy path: short-lived cert,
| grab it, connect, done.
|
| Except for everything around that:
|
| * user lifecycle (create/remove/rename accounts)
|
| * authz (who gets sudo, what groups, per-host
| differences)
|
| * cleanup (what happens when someone leaves)
|
| * visibility (what state is this box actually in right
| now?)
|
| SSH certs don't really touch any of that. They answer
| _can this key log in right now,_ not _what should exist
| on this machine._
|
| So in practice, something else ends up managing users,
| groups, sudoers, home dirs, etc. Now there are two
| systems that both have to be correct.
|
| On the availability point: "reasonably available" is
| doing a lot of work ;)
|
| Even with 1-hour certs:
|
| * new sessions depend on the signer
|
| * fleet-wide issues hit everything at once
|
| * incident response gets awkward if the signer is part of
| the blast radius
|
| The failure mode shifts from _a few boxes don 't work_ to
| _nobody_ can get in _anywhere_
|
| The pull model just leans the other way:
|
| * nodes converge to desired state
|
| * access continues even if control plane hiccups
|
| * authn and authz live together on the box
|
| Both models can work - it's more about which failure mode
| is tolerable to you.
| blipvert wrote:
| Well, yes, pick your poison.
|
| But for just getting access to role accounts then I find
| it a lot nicer than distributing public keys around.
|
| And for everything else, a periodic Ansible :-)
| gnufx wrote:
| Public keys (for OpenSSH) can be in DNS
| (VerifyHostKeyDNS) or in, say, LDAP via KnownHostsCommand
| and AuthorizedKeysCommand.
| moviuro wrote:
| That sounds like a lot of extra steps. How do I validate
| the authenticity of a signing request? Should my signing
| machine be able to challenge the requester? (This means
| that the CA key is on a machine with network access!!)
|
| Replacing the distribution of a revocation list with
| short-lived certificates just creates other problems that
| are not easier to solve. (Also, 1h is bonkers, even
| letsencrypt doesn't do it)
| toast0 wrote:
| 1h is bonkers for certs in https, but it's not
| unreasonable for authorized user certs, if your issuance
| path is available enough.
|
| IMHO, if you're pushing revocation lists at low latency,
| you could also push authorized keys updates at low
| latency.
| gunapologist99 wrote:
| Anyone tried out Userify? It creates/removes ssh pubkeys locally
| so (like a CA) no authn server needs to be online. But unlike
| certs, active sessions and processes are terminated when the user
| access is revoked.
| jamiesonbecker wrote:
| We're in the process of updating the experience to _this
| century_! ;)
|
| We've always taken the stance that crusty is better than
| vulnerable, but it turns out that not having a modern
| experience after 15 years is starting to feel like maybe we
| need to step up the features and shininess :)
| sqbic wrote:
| I've had very good experiences with SSH Communication Security
| company's (the guys who invented SSH) PrivX product to manage
| secure remote access, including SSH certificates and also cert
| based Windows authentication. It supports other kinds of remote
| targets too, via webui or with native clients. Great product.
| jamiesonbecker wrote:
| SSH certs quietly hurt in prod. Short-lived creds + centralized
| CA just moves complexity upward without solving the core problem:
| user management.
|
| The system shifts from many small local states to one highly
| coupled control point. That control point has to be correct and
| reachable all the time. When it isn't, failures go wide instead
| of narrow.
|
| Example: a few boxes get popped and start hammering the CA. Now
| what? Access is broken everywhere at once.
|
| Common friction points: 1. your signer that
| has to be up and correct all the time 2. trust roots
| everywhere (and drifting) 3. TTL tuning nonsense (too
| short = random lockouts, too long = what was the point)
| 4. limited on-box state makes debugging harder than it should be
| 5. failures tend to fan out instead of staying contained
|
| Revocation is also kind of a lie. Just waiting for expiry and
| hoping that's good enough.
|
| What actually happens is people reintroduce state anyway:
| sidecars, caches, agents... because you need it.
|
| We went the opposite direction: 1. nodes pull
| over outbound HTTPS 2. local authorized_keys is the
| source of truth locally 3. users/roles are visible on
| the box 4. drift fixes itself quickly 5. no
| inbound ports, no CA signatures (WELL, not strictly true*!)
|
| You still get central control, but operation and failure modes
| are local instead of "everyone is locked out right now."
|
| That's basically what we do at Userify (https://userify.com).
| Less elegant than certs, more survivable at 2am. Also actually
| handles authz, not just part of authn.
|
| And the part that usually gets hand-waved with SSH CAs:
| 1. creating the user account 2. managing sudo roles
| 3. deciding what happens to home directories on removal
| 4. cleanup vs retention for compliance/forensics
|
| Those don't go away - they're just not part of the certificate
| solution.
|
| * (TLS still exists here, just at the transport layer using the
| system trust store. That channel delivers users, keys, and roles.
| The rest is handled explicitly instead of implied.)
| ngrilly wrote:
| How do you solve TOFU?
| jamiesonbecker wrote:
| Well, TOFU is really just the model for how the chain of
| trust is established.
|
| In practice there isn't really _trust on first use_ : there's
| _verify the key matches what's expected_ , or _distribute
| keys out-of-band_ (including certs).
|
| If that verification step isn't happening, then it's not
| TOFU, it's just blind trust.
|
| From an automation/autoscaling angle, the same thing shows up
| again:
|
| 1. either keys are pre-baked / distributed
|
| 2. or, something signs them at boot
|
| Signing an instance key is just another way of distributing
| trust. It doesn't remove the need for a root of trust, it
| moves it.
|
| Certificates just add extra steps around the same underlying
| task.
| ngrilly wrote:
| I agree. I was just wondering if Userify had a solution for
| distribution the server signatures to the users.
| longislandguido wrote:
| This discussion is full of schizo solutions to "secure" SSH, most
| of which make no practical sense or have no technical basis.
|
| There really needs to be a definitive best practices guide
| published by a trusted authority.
| kackerlacker wrote:
| In my view it is more important to stop using software keys so
| probably use sk (fido) for both host and user.. From there CAs
| would be a next step.. The level of documentation and example
| setups is astoundingly poor if you even look at step 2 for any
| feature. I.e. SK keys are reasonably understood for user keys
| but the setup as host keys is vague and needs testing to see if
| it really works.
| TZubiri wrote:
| >then I don't need to type the target user's password; instead I
| enter the key's passphrase, a hopefully much more complicated
| combination of words, to unlock the private key.
|
| This sentence is a bit of a red flag, it looks like the author is
| making a (subtle) mistake in the category of too much security,
| or at least misjudging the amount of security (objectively
| measurable entropy) needed. This is of course a less
| consequential error than too little entropy/security measures,
| but still if one wants to be a cybersecurity professional,
| especially one with influence, they must know exactly the right
| amount needed, because our resources are limited, and each
| additional bit of entropy and security step not only costs time
| of the admin that implements it, but of the users that have to
| follow it, and this can even impact security itself by fatiguing
| the user and causing them to circumvent measures or ignore
| alerts.
|
| On to specifically what's wrong:
|
| Either a key file or a password can be used to log in to a server
| or authenticate to any service in general. Besides the technical
| implementation, the main difference is whether the secret is
| stored on the device, or in the user's brain. One is not more
| correct than the other, there's a lot of tradeoffs, one can
| ensure more bits and is more ergonomic, the other is not stored
| on device so it cannot be compromised that way.
|
| That said a 2FA approach, in whatever format, is (generally
| speaking) safer than any individual method, in that the two
| secrets are necessary to be granted access. In this scenario one
| needs both the file and the password to authenticate, even if the
| password is 4 digits long, that increases the security of the
| system when compared to no password. An attacker would have to
| setup a brute force attempt along with a way to verify the
| decryption was successful. If local decryption confirmation is
| not possible, then such a brute force attack would require to
| submit erroneous logins to the server potentially activating
| tripwires or alerting a monitoring admin.
|
| There's nothing special about the second factor authorization
| being equal or equivalent in entropy to the first, and there's
| especially no requirement that a password have more entropy when
| it's a second authorization, in fact it's the other way around.
|
| tl;dr You can consider each security mechanism in the wider
| context rather than in isolation and you will see security
| fatigue go down without compromising security.
| aquafox wrote:
| I work in a corporate setting and the money and time we wasted
| because of Zscaler and its SSL inspection [1] is beyond your
| wildest imagination. Whenever I see a "SSL certificate problem:
| self-signed certificate in certificate chain" error, I know I'm
| in trouble.
|
| [1] https://www.zscaler.com/resources/security-terms-
| glossary/wh...
| nayuki wrote:
| Zscaler was deployed on a Windows 11 machine at a place that a
| friend worked at. When I assessed the software, its behavior
| was downright evil like malware. As we all know, it injects its
| own root certificate into the operating system in order to
| conduct man-in-the-middle attacks on TLS/HTTPS connections to
| monitor the user's web activity.
|
| Furthermore, it locks down the web browser's settings so that
| you cannot use a proxy server to bypass Zscaler's MITM. I saw
| this behavior in Mozilla Firefox, where the proxy option is set
| to "No proxy" and all other options are disabled and grayed
| out; I imagine that it does the same to Google Chrome. If you
| try to modify the browser's .ini(?) file for proxy settings,
| Zscaler immediately overwrites it against your will. Zscaler
| worked very hard to enforce its settings in order to spy on the
| computer user.
|
| And as you'd expect, if you open up the Zscaler GUI in the
| system tray, you are presented with the option to disable the
| software if you have the IT password. Which of course, you
| don't have. Then again, that might be an epsilon better than
| the Cybereason antivirus software, which just has a system tray
| icon with no exit option, and cannot be killed in Task Manager
| even if you are a local administrator, and imposes a heavy
| slowdown if you're open hundreds of small text files per
| second.
| pxc wrote:
| I feel the same way about Cisco Umbrella where I work.
|
| The worst breakage by far is protocol breakage; basically
| anything that uses HTTP as a basis for building some other
| protocol gets broken all the time. None of the people
| implementing it seem aware. They buy the vendor's claim that
| it's "transparent", when in fact even "inspect/trace-only"
| modes often break all kinds of shit.
|
| I've seen Umbrella break: - Git -
| RubyGems - `go mod` - OrbStack - Matrix
| - Cargo - all JDKs - Nix - Pkgsrc - all
| VMs
|
| and probably some other things I'm forgetting. When this
| breakage is reported, the first round of replies is typically
| "I visited that domain in my enterprise-managed browser and
| it's not blocked". That is, of course, a useless and irrelevant
| test.
|
| Often it takes hours to even fully diagnose the breakage with
| enough confidence to point the finger at that tool and not some
| other endpoint security tool.
|
| I'm not sure if the people buying and deploying tools in this
| category don't know how much stuff it breaks or just don't
| care. But the breakage is everywhere and nobody seems prepared
| for it.
| tptacek wrote:
| SSH certificates aren't X.509 certificates.
| yason wrote:
| Despite the drawbacks of its grassroot nature TOFU goes a
| looooong way.
|
| With my own machines I can just physically check that the server
| host key matches what the ssh client sees. Once TOFU looks good
| I'm all set with that host because I don't change any of the keys
| ever.
|
| In a no-frills corporate unix environment it's enough to have a
| list of the internal servers' public keys listed on an internal
| website, accessible via SSL so it's effectively signed by a known
| corporate identity. You only need to check this list once to
| validate carrying out TOFU after which you can trust future
| connections.
|
| In settings with huge fleet of machines or in a very dynamic
| environment where new machines are rolled out all the time it
| probably makes things easier to use certificates. Of course,
| certificates come with some extra work and some extra features so
| the amount of benefit depend on the case. But at this scale TOFU
| is breaking down bad on multiple levels so you can't afford a
| strong opinion against certificates, really.
|
| I wish web browsers could remember server TLS host keys easily
| too and at least notify me whenever they change even if they'd
| still accept the new keys via ~trusted CAs.
| gnufx wrote:
| Life is easier if you can use Kerberos SSO, i.e.
| GSSAPIAuthentication in OpenSSH. (If we're talking certificates,
| presumably it is OpenSSH, or does anything else implement them?)
| tptacek wrote:
| Why would you do that rather than just hooking SSH up to a real
| IdP with certificates?
| tacostakohashi wrote:
| One constant source of amazement for me is people not using ssh
| keys / using passwords with ssh.
|
| Especially at a BigCo, where there are different environments,
| with different passwords, and password expiry/rotation/complexity
| rules.
|
| Like, when asking for help, or working together... you say to
| them "ok, lets ssh to devfoo1234", and they do it, and then type
| in their password, and maybe get it wrong, then need to reset it,
| or whatever... and it takes half a minute or more for them to
| just ssh to some host. Maybe there are several hosts involved,
| and it all multiplies out.
|
| I mention to them "you know... i never use ssh passwords, i don't
| actually know my devfoo1234 password... maybe you should google
| for ssh-keygen, set it up, let me know if you have any problems?"
| and they're like "oh yeah, thats cool. i should do that sometime
| later!".... and then they never do, and they are forever messing
| with passwords.
|
| I just don't get it.
| pphysch wrote:
| Unregulated/decentralized SSH key usage (i.e. allowing ssh-
| copy-id) is a dream for hackers to move laterally through a
| network. That's why many orgs disable it, and otherwise haven't
| invested resources in getting a proper centralized CA/authz
| server set up.
| bombcar wrote:
| Keys is great for individual use, or for company use if you
| have centralized key control (and issue one or more keys per
| user).
|
| Often you either end up with one "dev ssh key" for all machines
| (which is bad) or you end up with people sharing around keys
| and unidentified keys on machines.
|
| Passwords at least are "simple" for people to work with.
| TacticalCoder wrote:
| Since a few years now I only ever use SSH private keys safely
| hidden behind a HSM with a tinier than tiny attack surface:
| Yubikeys do it for me (but other vendors would work too). My
| SSH keys do _not_ have a password but when I log in using SSH,
| it requires me to physically touch my Yubikey (well one of my
| Yubikeys).
| znpy wrote:
| It's very interesting that you can force the execution of a
| specific command.
|
| I wonder if that can be somehow employed to somehow get kubectl
| to invoke ssh in order to get an authentication tokens.
|
| I would be cool to authenticate to kubernetes via ssh.
___________________________________________________________________
(page generated 2026-04-03 23:00 UTC)