[HN Gopher] TOFU: Do You Check?
___________________________________________________________________
TOFU: Do You Check?
Author : c7DJTLrn
Score : 58 points
Date : 2023-09-22 16:36 UTC (6 hours ago)
(HTM) web link (cedwards.xyz)
(TXT) w3m dump (cedwards.xyz)
| thealistra wrote:
| Isn't this mechanism by default to check if the prompt shows on a
| used machine to detect cert rotation? And then you can ask the
| admin.
|
| Seems like too much work for checking the fingerprints if not in
| an obvious security first context.
| tedunangst wrote:
| Nobody ever mentions that TOFU requires the attacker to commit to
| the attack before they know whether you know the fingerprint.
| Somebody _could_ set up on that public wifi and intercept ssh,
| but they 'd be detected as soon as anybody with a previously
| known key tries to connect. They can't wait for you to connect to
| an unknown server, because they don't know what you don't know.
| If you're not sure about the network, you can connect to a known
| host first to check for funny business.
|
| Or turn on the option for ascii art keys.
| SoftTalker wrote:
| > they'd be detected as soon as anybody with a previously known
| key tries to connect
|
| Would they? Or would the person just delete the old key from
| their ~/.ssh/known_hosts and accept the new one?
| tedunangst wrote:
| If you're deleting keys from known_hosts, you're not trusting
| on first use.
| mjochim wrote:
| That is correct. And still, many, many devs and admins
| would do exactly that. Which is why in many, many cases,
| Tofu really isn't Tofu but "trust whatever."
|
| When designing or evaluating security, one should not
| ignore that this is a part of reality.
| foobiekr wrote:
| This. Hell, in almost every org I've worked there's a guy who
| has helper scripts to do this automatically.
| c7DJTLrn wrote:
| Good point. If we're being imaginative, there's various side
| channels an attacker could use to carefully target their
| attack, such as seeing a new MAC or DHCP lease on the local
| network.
| [deleted]
| xg15 wrote:
| True. Also depends on what the target is. I.e., if the
| attacker wants to impersonate machines that are only
| avaliable on the local network, then it's probably reasonable
| to conclude that a machine which was never seen on that LAN
| before also hasn't yet connected to the impersonated host.
|
| However, that isn't true for "public" SSH endpoints, such as
| github, gitlab etc - e.g. if the attacker is impersonating
| the wifi if the nearest Starbucks to snoop on all the hipster
| solo-devs there, he'll probably be unable to impersonate
| github, because even people who use that wifi for the first
| time have probably connected to github before, on a different
| network.
| NoZebra120vClip wrote:
| I was using a system at work, and so eventually I needed to ssh
| into it, and I was confronted with the usual TOFU prompt. So I
| contacted Support, to ask how I could ascertain the Host Keys in
| use and check their fingerprints. And they replied, "Mmm, we
| don't do that."
| Am4TIfIsER0ppos wrote:
| Only once. ssh needs to make the fingerprint shown in the
| terminal match what is stored in known_hosts. If I already have
| public keys why can't I match them against the new connection? I
| could preload the file but why not copy it from my desktop's file
| into the terminal.
| cesarb wrote:
| It's implied in the examples on this post, but when using public
| key authentication, an attacker with a stolen SSH host key can
| pretend to be the server to the client, but cannot pretend to be
| the _client_ to the _server_. Which means that the attacker has
| to pretend to be the server, without being able to see how the
| server would answer. For instance, if the ssh connection was a
| "git pull", the attacker has to know enough about the repository
| state (and about how much of that state the git client has
| already seen) to avoid complaints from git; if the ssh connection
| is a shell connection, the attacker has to know enough about the
| target machine to make the command outputs seem realistic enough
| for that particular machine; and so on.
|
| And as for "SSH has no forward secrecy", I thought SSH used
| Diffie-Hellman or similar to derive the session key? The SSH host
| key (and the client's public key) is for authentication only,
| stealing them does not reveal anything about past sessions.
| progmetaldev wrote:
| SSH does have forward secrecy. The OP stated in a previous
| comment that they have fixed this in the article.
| mindslight wrote:
| > _[the attacker] cannot pretend to be the client to the
| server_
|
| Isn't the most common way of identifying ssh clients still just
| public key authorization? Meaning the attacker can MITM the
| real client connecting to the server, satisfy the real client's
| request, and then likely carry on doing whatever it likes.
|
| edit: actually no - I can see where my intuition is wrong (out
| of date from the time of widespread password auth). If the ssh
| server only accepts input that's been effectively signed by the
| authentication key, then there's no way for the MITM to send
| its own requests. Yet another reason to not use password auth.
| m_x wrote:
| Aren't SSH host certificates the answer? Why are they not more
| widely used?
| fragmede wrote:
| Especially since they're distributable via DNS. If your server
| has an entry in DNS, you shouldn't need to confirm host key.
| c7DJTLrn wrote:
| According to the guy who invented them: "I'm a shitty shitty
| publicist" (Damien Miller)
| surfer7837 wrote:
| Wouldn't that require a responsible body (ie a CA) to verify
| who they say are. Not sure that's needed for SSH
| cpach wrote:
| Yes, but it can be an internal CA. There are even FOSS
| products for that: https://goteleport.com/
| tetha wrote:
| We've switched to them based on Hashicorp Vault and I think
| they are an improvement in security and convenience.
|
| Before SSH host certificates, using tooling like ansible was
| seriously annoying, especially to the monthly ansible users,
| compared to the daily ansible users. If you tried using ansible
| once a month without our host key certificates, you'd have
| ansible barf about a dozen unknown host keys due to them being
| rebuilt and other things. Then you fix that, then they do that
| one ansible run, then it lies around for a month and back you
| go to square one.
|
| I'm not sure HOW you'd stage an actual attack there, but people
| got used to just accept SSH host keys whatsoever, so the
| vulnerability was there.
|
| Now we have host key signing based on Vault. This is a huge
| improvement in my book. Base infra guys know when they've reset
| a VM to a base image (which resets SSH host keys), so they know
| when to expect a TOFU request from their SSH upon first connect
| to the just rebuilt system, so they accept those. Afterwards,
| one of the first things the config management does is to sign
| those host keys and then the accepted host keys are usually
| deleted again.
|
| There probably is a window of vulnerability in there, but
| that's getting pretty hard to attack and I'm sure there are
| easier ones in the overall infrastructure.
|
| The main problem at this however is that you need some safe and
| secure place for the CA, and ideally automation to sign those
| host keys. And it needs to be enough of a problem to do all of
| that.
| morelisp wrote:
| I tried to check but 99% of our clients rotate host keys whenever
| they restart the container or the LB sends you to a different
| node or whatever. And if you email them to verify they take 3
| weeks to respond, which is 20 daily reports they now don't get
| and so don't want to pay for, so... nope.
| SoftTalker wrote:
| When I set up a system for a user, I pre-populate
| /etc/ssh/ssh_known_hosts for all our internal systems.
|
| But I've certainly been guilty of just hitting "yes" on first
| connection to a machine I'm unfamiliar with.
| MenhirMike wrote:
| If you don't check and just allow it, does that count as
| OpenTOFU?
| pxx wrote:
| > Not to mention that SSH has no forward secrecy.
|
| What? This is not right, not even for the obsolete v1 ssh:
| https://utcc.utoronto.ca/~cks/space/blog/tech/SshForwardSecr...
| c7DJTLrn wrote:
| Fixed, thanks.
| foobiekr wrote:
| In the real world, "trust on first use" is literally true - it's
| never "check on first use." SSHFP DNS records would help a lot,
| or some other scheme. For all of the positive image ssh has,
| mostly because telnet and ftp were so ridiculously terrible, ssh-
| as-used is pretty mediocre.
|
| And the reality is: "No."
|
| https://www.usenix.net/system/files/login/articles/105484-Gu...
| NovemberWhiskey wrote:
| Ideally, the DNS SSHFP record would help solve this problem from
| Internet-exposed services, in combination with DNSSEC of course.
| upofadown wrote:
| SSH could really use a better default key fingerprint. It's 42
| characters of base64. So a hodgepodge of upper and lower case
| Latin characters with some numbers and special characters mixed
| in. Not even any spaces to give it some structure. Basically just
| the output of of a 256 bit hash dumped directly to the terminal.
|
| Perhaps more people would check it if the usability was
| improved...
| bentcorner wrote:
| Usability would be improved if there was a one-click way to
| verify it. Off the top of my head a QR code you could scan so
| it'd be enough "out of band" that the odds of a compromise
| affecting both paths would be unlikely. Ideally someone smarter
| than me would come up with a way that it wouldn't require TOFU
| at all.
| m-p-3 wrote:
| A well-known URI that provides SSH keys would be nice.
| dinosaurdynasty wrote:
| Or you know SSHFP records + DNSSEC to verify hostkeys.
|
| Which is something OpenSSH already supports.
| tedunangst wrote:
| Turn on ascii art.
| susam wrote:
| > SSH could really use a better default key fingerprint. It's
| 42 characters of base64.
|
| You can get an ASCII graphic for the host key with the
| VisualHostKey option. For example: ssh -o
| VisualHostKey=yes foo.local
|
| For example, $ ssh -o VisualHostKey=yes
| tty.sdf.org Host key fingerprint is
| SHA256:ZjwbO7AU8rHJExYrmZS2LqGZ7WfdoELfMrF54W92PYA
| +--[ED25519 256]--+ | ... | | .oo o
| | | .=.* | | . .* B | | =
| o O S. | |+ + o.o*E=. | | o o O.++ o
| | | o X = +.. o | | + + +.. . |
| +----[SHA256]-----+ user@tty.sdf.org's password:
|
| This graphic looks like a pigeon to me.
| jjav wrote:
| Yes, highly recommended. Easy to get used to the visual
| pattern of your important servers and notice if anything is
| off.
| dheera wrote:
| No, often I don't check, because my managers rush me to produce
| results on tight timelines, so I don't have time to "do things
| right".
|
| Slack message at 3pm asking for updates to be in a doc by 6pm.
| What do you expect?
|
| I guess the problem is less serious on a corporate VPN, but
| still, the principle applies everywhere. If you want security,
| give your employees time to do things right. Usually that means
| at least double what you think it will take.
| shadowgovt wrote:
| It seriously shoots any Zero Trust claims a corporation makes
| in the foot if people are "yes"ing through that dialog.
|
| Of course, to really have Zero Trust you have to make it
| impractical for users to "yes" through the dialog.
| dinosaurdynasty wrote:
| For SSH: setup an SSH CA and then disable asking in the
| client config.
| Apreche wrote:
| I don't check. That's because I'm one of those weirdos who uses
| certificates, so the prompt never appears.
| tjk_ wrote:
| Bitbucket dev here. Firstly, thanks for the post, this topic is
| too often skimmed and as someone who worked directly on the host
| key rotation here, it was shocking to find how many devs had zero
| idea what "that prompt from git on a new computer" was.
|
| Secondly, I just wanted to say that there is an interesting side-
| effect to everything covered in that blog that effectively leads
| to a gap in most people's mental models (or maybe just mine).
| Because of the way these systems were defined/the way people
| interact with them, there are several conundrums when a large
| centralized host needs to deprecate an old key.
|
| If a key were suspected of being compromised, it seems obvious
| that you could just stop serving the old key and tell all of your
| users to be prepared to re-TOFU. However, since _most_ devs
| blindly type "yes" when interacting with a large git host, this
| effectively primes the pumps for any bad actor that have MITM
| control but have not actually stolen the keys. This gives them a
| relatively long (if not infinitely long) window of time where a
| user will not be surprised to see "the prompt" and blindly accept
| trust of a host that could be controlled by the bad actor. If a
| key was completely successfully compromised by a bad actor, and
| said bad actor had MITM control of the victim network, then
| requests with the _old_ key would never actually reach the
| correct host and just quietly continue working (assuming the bad
| actor was savy enough to setup a remote system to properly behave
| as a git host without a prior copy of the target repository).
|
| Damned if you do, damned if you don't, so always carefully TOFU.
|
| Also,not sure why the author linked off to a Bitbucket Cloud blog
| post for the SSH keys, they are documented here[1] along with our
| recommended best practices WRT TOFU.
|
| 1 - https://support.atlassian.com/bitbucket-
| cloud/docs/configure...
| c7DJTLrn wrote:
| Thanks. I linked to the blog post because it was one of the
| first results in the search engine.
|
| How are you guys tackling host key rotation? Do you do it
| periodically, or on compromise? How do you protect such an
| important set of keys?
| tedunangst wrote:
| sshd also supports online key rotation where it tells clients
| about new keys, but I think it requires the new key be
| installed on the server, so it ends up getting compromised at
| the same time as the existing key.
| eichin wrote:
| We actually ship a bunch of vetted pubkeys in our developer
| tools package (in ssh_known_hosts2) . We tried to get a little
| bit out of the loop and tell our devs that they could use the
| instructions at https://bitbucket.org/blog/ssh-host-key-changes
| (confirm it doesn't cert fail, then read the curl
| https://bitbucket.org/site/ssh bit and see that it also checks
| certs, so we can leverage "trusted https cert" to "trusted ssh
| key"... much better than TOFU...
|
| Didn't work, because there's no trailing newline on the output
| of site/ssh. So even if it works, it corrupts the next
| addition.
| myself248 wrote:
| Yeah I've done it. Not for Git but for SSH. I'll call the admin
| out-of-band and read him part of the fingerprint, and ask him to
| read another part back. Only done it a few times when I'm setting
| up a new machine and don't have an already-authenticated one to
| check against.
|
| If I've got suspicions about the network connection I'm on, I'll
| first ask him some questions about our adventures years ago, that
| nobody else would likely know about.
|
| And one time when the host key changed, and I wasn't expecting
| it, he said my call was how he knew he replaced the right box.
| Tongue-in-cheek, but only barely.
|
| The only time I immediately dismiss the key-changed message is
| when I'm cycling yet another raspberry pi image into the same IP
| as an old one or whatever, and I'm on the local network so I know
| exactly what's going on.
| neilv wrote:
| The last time I was jump-starting an internal wiki, for a place
| that had no docs... when I was documenting how to set up a
| workstation for checking out and running the code from Git SSH...
| it turned out that no one seemed to document or know what the
| fingerprint should be. Not even any documentation I could find
| from the major Git provider.
|
| I can understand not expecting every user to do a one-time
| 10-second diligence process. But the "enterprise" provider
| seemingly not expecting any of its customers' IT people to even
| do a one-time 10-second check when integrating mission-critical
| infrastructure... is a concern.
| ghusto wrote:
| I think the attack in all cases would require pointing DNS of
| gitprovider.net to their own server. This means they control DNS
| on that network, which means they're running it.
|
| What am I doing on a network I don't trust with a machine I've
| never used? The example given is that I've started a new job.
| Well okay, their machine, their network, and I guess their
| corporate gitprovider.net account they're getting me to use, so
| to heck with it.
|
| If we're talking about using _my_ gitprovider.net account
| however, that's a different story. I simply wouldn't use my own
| account on a laptop that's not mine.
| toxik wrote:
| Another good question is, why not post the fingerprint in a TXT
| record?
| dmurray wrote:
| Trust On First Use is exactly what I do.
|
| If I have to connect to a new machine, I identify it by its
| hostname, and send it my credentials (whatever credentials the
| owner just sent me in plaintext).
|
| If I get the warning and I'm pretty sure I connected to this
| machine before, I ask whoever owns the machine "hey did you do
| something weird with this machine? Like reimage it or rotate SSL
| certs or change the DNS entry so it's actually a different
| machine?"
|
| 99% of the time I'm not connecting to a machine for the first
| time, so (if I may be allowed some bad maths) this reduces my
| attack surface by a factor of 100.
|
| (If the owner didn't send me plaintext credentials but instead
| added my SSH key, I don't expose anything by using that, but I
| immediately follow up by doing other sensitive things like
| scp'ing code or credentials there).
| shadowgovt wrote:
| "hey did you do something weird with this machine? Like reimage
| it or rotate SSL certs or change the DNS entry so it's actually
| a different machine?"
|
| "I dunno."
| Muromec wrote:
| "Maybe the other department reprovisioned the bastion about 2
| months ago, but maybe it was another fiscal year"
| MarkSweep wrote:
| Maybe there should be an "expected key" argument to "git clone".
| GitHub could include it in the copy-pasted clone command. Problem
| solved.
| 3pt14159 wrote:
| I had this phase where I tried to do everything the right way. I
| used to check whenever I thought I could, including to my own
| servers. And you know what? 100% of the time the check passed.
|
| After a while I sat back and thought to myself "is this really
| worth the cost in time?" And I concluded that no it wasn't.
| Plenty of people run scanners looking for bad stuff online,
| ranging from poisoned SSH to supply chain attacks on OSS. These
| are the people that find the baddies and the baddies eventually
| get targeted. Me as a lowly developer is just not spending time
| fruitfully by checking github.coms SSH fingerprint. It would be
| better if I scanned the web app that I'm working on for, say, SQL
| injection bugs or misapplied CORS headers or whatever else.
|
| Ultimately security checks are an economic activity. The
| likelihood you are targeted via a given threat, the payoff to the
| attacker if you are targeted, the downside to the attacker if
| they are caught, etc. I think being realistic is prudent.
| Muromec wrote:
| >Me as a lowly developer is just not spending time fruitfully
| by checking github.coms SSH fingerprint
|
| ... and when you don't you drive up the probability of enabling
| the 4-stage attack that leaks private keys to Chines govt. Of
| course the probability is only 0.01% and it's worth to spend
| time on something better. And when you do check and other 1000
| devs in the organization don't, what's the point, right?
|
| This is how this stuff happens and we all read a nice post
| morten here somewhat 2 years later.
| seabass-labrax wrote:
| MITM for GitHub or another code forge is one of the things I'm
| least concerned about, because GitHub to me is a place where
| things go to be made public. One of the security principles I
| live by is not to upload anything to the Internet that I wouldn't
| mind being shared completely in public.
|
| The only exceptions I make are video conferencing and email.
| Jitsi provides great end-to-end encryption for the former
| (although the 8x8 server's recent prohibition of anonymous use is
| disappointing).
| ghusto wrote:
| Same. I presume every company I have an account on will be
| compromised at some point, and all my account data will be out
| in the wild. This presumption hasn't disappointed so far ;)
___________________________________________________________________
(page generated 2023-09-22 23:00 UTC)