SSH suddenly refused across multiple machines? reddit / selfhosted ---------------------------------------------------------------------- CrispyBegs · 2023-10-03 15:46 please don't shout at me, I did post this on linux4noobs but got no response, so I'm trying my luck here I have 5 HP elitedesk headless machines all with ubuntu installed on them, all running a variety of docker services. I noticed this morning that my usual watchtower updates had failed on 4 of them, and it reminded me that I hadn't updated the machines in a while. I SSH'd into each one and updated then upgraded them, and then rebooted. When I tried to SSH in again my connection was refused on 4 of them, with the message `ssh: connect to host` `192.168.1.XXX port 22: Connection refused` One of them still lets me connect just fine. The machines are definitely up and running as I can connect to their samba shares just fine and their docker services are all working, it's just SSH that has suddenly become an issue. Any idea why this has happened, and how I can fix it without dismantling everything and carrying them back to a monitor and keyboard? ---------------------------------------------------------------------- MisterBazz · 2023-10-03 15:53 ufw was enabled, or iptables modified? ---------------------------------------------------------------------- CrispyBegs · 2023-10-03 15:54 nothing was changed by me in those regards, unless it happened during the update process perhaps? ---------------------------------------------------------------------- justinhunt1223 · 2023-10-03 16:06 Do you access the services on them via IP address? I would reboot one of them and see if it fixes it. If it does, check the logs, sshd might have stopped ---------------------------------------------------------------------- CrispyBegs · 2023-10-03 16:08 i do yes. i did reboot all of them after updating, which is when the refusals started happening. guess i'm actually going to have to go and get them and manually restart them aren't i ---------------------------------------------------------------------- proc-sysrq · 2023-10-03 16:26 I see three scenarios: 1. The SSH service isn't running 2. Firewall rules are blocking port 22 3. The machines failed to fully boot but were able to bring up their network Your best bet is checking remote logs if you're shipping the system logs to a central server, using a config management tool to peek in if you're running something like Puppet, or attaching a monitor and keyboard. ---------------------------------------------------------------------- justinhunt1223 · 2023-10-03 17:15 Just plug a keyboard into them and press control alt delete. No need for a monitor or anything ---------------------------------------------------------------------- Marvin_the_Minsky · 2023-10-03 18:13 CAPS LOCK? ---------------------------------------------------------------------- mirokra · 2023-10-03 18:43 > The machines failed to fully boot but were able to bring up their network Or, a supply chain attack during upgrade, and are part of botnet now. ---------------------------------------------------------------------- coolkathir · 2023-10-03 19:57 I had faced this issue out of nowhere with my AWS instance. I uninstalled and reinstalled a new version of putty which resolved the issue. ---------------------------------------------------------------------- the-noidea · 2023-10-03 20:47 Maybe you are running into this when using certificate based authentication: https://askubuntu.com/questions/1409105/ubuntu-22-04-ssh-the-rsa-key- isnt-working-since-upgrading-from-20-04 ---------------------------------------------------------------------- a-nerd-has-no-name · 2023-10-03 20:55 I recently experienced a similar issue. While I still don’t know exactly what the issue actually is, it seemed related to Windows 11, zero-tier, Cloudflare WARP agent, and my Wi-Fi router. Long story short, I had to create a static lease for my laptop. Once I released and renewed IP for the Wi-Fi adapter, I was suddenly able to SSH into my Linux servers. Weird, but hope this helps. ---------------------------------------------------------------------- CrispyBegs · 2023-10-03 21:01 thanks. i didn't understand almost any of that thread, but i think i'm just going to have to physically bring them in and take a closer look unfortunately ---------------------------------------------------------------------- the-noidea · 2023-10-03 21:06 Maybe try another/newer ssh client first if you haven't, could also be yours is too old. Good luck! ---------------------------------------------------------------------- CrispyBegs · 2023-10-03 21:07 thanks! i use [https://warp.dev](https://warp.dev) which is always up to date with the latest version, and two of the five machines let me SSH in just fine, so I don't think it's the client? ---------------------------------------------------------------------- adamshand · 2023-10-03 21:14 The error you are getting means that ssh isn't available on port 22 on the server you are trying to connect to. There are a few things that could cause this: - something has gone wrong with the boot process and it's not starting the SSH service. - something is blocking your access to port 22 like a change in firewall rules - ssh is running on a different port I'd try nmaping your hosts and seeing if ssh is running on a non- standard port. Then I'd try rebooting them again. If neither work than it's keyboard + monitor time. Side notes: - This is an example of why running a centralised syslog host can be helpful. You'd have the boot logs and could potentially see what's going wrong. - It's also a good example of why it's usually a good idea to reboot servers one at a time and check that each is working. - And finally having a web based ssh client on each server can sometimes provide an emergency way to connect. Obviously won't help if ssh isn't working, but can help get around firewall issues. ---------------------------------------------------------------------- CrispyBegs · 2023-10-03 21:16 all good advice, thank you. i'm going to take it. i'm not sure what this is or how to do it: >*I'd try nmaping your hosts and seeing if ssh is running on a non- standard port* do you have any pointers by any chance? ---------------------------------------------------------------------- the-noidea · 2023-10-03 21:17 Never heard of warp, looks nice though. We had a similar situation with sftp at work and a newer client did the trick after upgrading the server to Ubuntu 22.03. Seems to be not that easy in your case unfortunately. ---------------------------------------------------------------------- CrispyBegs · 2023-10-03 21:37 oh it's so good. mac-only annoyingly, but i believe they're working on a linux version ---------------------------------------------------------------------- jorissels · 2023-10-03 22:00 Have you set your IP’s static? If not it is possible they requested a new ip from dhcp and that the “server ip’s” now belong to something else that doesn’t allow ssh. Try an ip scanner to check! :) ---------------------------------------------------------------------- CrispyBegs · 2023-10-03 22:01 yes everything has its own static IP. As I mentoned, all the services on each machine and smb serving are all working fine, so it's def just the ssh that's stopped working, nothing else. ---------------------------------------------------------------------- adamshand · 2023-10-03 22:04 Ah, sorry. nmap is a command line tool which will scan hosts or networks for open ports. If you scan one of your hosts it will show you all the open ports. Lots of info on Google, eg. https://linuxhandbook.com/nmap-scan-ports/ You then need to figure out what each open port maps too. If there are ports you don't recognise you can sometimes figure out what they are by using telnet to connect to the port, eg. ``` ❯ telnet mahoe.xxx.nz 22 Trying 139.180.171.61... Connected to mahoe.xxx.nz. Escape character is '^]'. SSH-2.0-OpenSSH_9.2p1 Debian-2 ^] telnet> q Connection closed. ``` ---------------------------------------------------------------------- CrispyBegs · 2023-10-03 22:05 thank you brother ---------------------------------------------------------------------- CrispyBegs · 2023-10-03 22:08 ah intersting, trying it on the first machine says this: `nmap -sT 192.168.1.35` `Starting Nmap 7.92 ( https://nmap.org ) at 2023-10-03 23:06 BST` `Nmap scan report for 192.168.1.35` `Host is up (0.018s latency).` `Not shown: 985 closed tcp ports (conn-refused)` `PORT STATE SERVICE` `139/tcp open netbios-ssn` `445/tcp open microsoft-ds` `548/tcp open afp` `3000/tcp open ppp` `3001/tcp open nessus` `3389/tcp open ms-wbt-server` `8000/tcp open http-alt` `8080/tcp open http-proxy` `8083/tcp open us-srv` `8088/tcp open radan-http` `8181/tcp open intermapper` `8443/tcp open https-alt` `8888/tcp open sun-answerbook` `9001/tcp open tor-orport` `55555/tcp open unknown` `Nmap done: 1 IP address (1 host up) scanned in 0.36 seconds` so somehow I guess port 22 has been closed, no idea how though ---------------------------------------------------------------------- 4LAc · 2023-10-03 22:31 Another reason for this can be the permission of the home directory of the machines you're trying to ssh into. If ```/home/yourusername``` isn't owned by ```yourusername``` then you won't be able to ssh into it with keys. Passwords should still work (if you can enable passwords). ```ls -lah``` from the /home directory will show you if that's the cause [oops, you won't be able to run this without logging in - sorry] After that there's permissions on the .ssh files & folders too: This is good guide for that - https://www.tecmint.com/set-ssh- directory-permissions-in-linux/ ---------------------------------------------------------------------- lannistersstark · 2023-10-04 00:00 it has happened to me before and it required manual reboot for some weird reason. ---------------------------------------------------------------------- OldManinTights · 2023-10-04 01:05 Fail2ban has banned the ip address. Happened on mine several times. ---------------------------------------------------------------------- adamshand · 2023-10-04 03:36 It's unlikely that ssh moved to another port without you knowing, but worth a look! ---------------------------------------------------------------------- EpicRisc · 2023-10-04 05:47 I had a similar issue a few days ago after upgrading my Laptop from Ubuntu 20 to 22. To fix this I had to write this into my /etc/ssh/ssh_config: ``` # Fallback for legacy keys HostKeyAlgorithms +ssh-rsa PubkeyAcceptedKeyTypes +ssh-rsa ``` Long term I obviously need to Upgrade my ssh keys to a better encryption. You can read about this issue here: https://askubuntu.com/questions/1409105/ubuntu-22-04-ssh-the-rsa-key- isnt-working-since-upgrading-from-20-04 Maybe this is it? Also what does ssh say if you increase verbosity? ---------------------------------------------------------------------- CrispyBegs · 2023-10-04 08:06 if i understand that report properly, it hasnt' moved to another port, it's not available on any port ---------------------------------------------------------------------- CrispyBegs · 2023-10-04 08:06 hmm i don't have F2B installed ---------------------------------------------------------------------- adamshand · 2023-10-04 09:16 Nmap can't show that for sure, the name it gives each port is based on what's defined in `/etc/services`, not what's actually running on the port. So in theory, ssh could be running on any of the tcp ports shown as open by nmap. If there are any port numbers you don't recognise and know what's running on them, you can check by telnet'ing as I showed you. But in practice, it's pretty unlikely that ssh moved to another port. More likely that it just didn't start for some reason or that there's a firewall in the way. ---------------------------------------------------------------------- CrispyBegs · 2023-10-04 09:59 got it, thank you mr hand ---------------------------------------------------------------------- arcadianarcadian · 2023-10-10 09:53 increase verbosity with ssh -vvv, then ask again. ---------------------------------------------------------------------- CrispyBegs · 2023-10-10 09:54 `OpenSSH_8.1p1, LibreSSL 2.7.3` `debug1: Reading configuration data /etc/ssh/ssh_config` `debug1: /etc/ssh/ssh_config line 47: Applying options for *` `debug2: resolve_canonicalize: hostname 192.168.1.35 is address` `debug2: ssh_connect_direct` `debug1: Connecting to 192.168.1.35 [192.168.1.35] port 22.` `debug1: connect to address 192.168.1.35 port 22: Connection refused` `ssh: connect to host 192.168.1.35 port 22: Connection refused` ---------------------------------------------------------------------- arcadianarcadian · 2023-10-10 09:59 Your SSH port (default 22) definitely not open. You should check it. ---------------------------------------------------------------------- CrispyBegs · 2023-10-10 10:00 yep thanks, i need to go get the machines and see what's up appreciate the `-vvv` tip btw, not seen that before, thank you