Remote Server won't come back online after "sudo reboot" reddit / selfhosted ---------------------------------------------------------------------- Ieris19 · 2023-11-21 22:58 I know I probably fucked something up, but still want some advice. I have two houses, halfway across the world from each other. Whenever I am on holidays in my second home, I would like to still access my home network and vice versa. I have a Tailscale VPN setup for this and I regularly SSH into my server from other devices to configure it, rather than use the physical device. I tend to only access it whenever I need to turn it on or off. TIFU by trying to reboot it. I was configuring some network stuff for my brand-new project with installing PiHole, and after debugging a little issue, having changed many configurations and being unsure about how to restart everything needed for the configs to take effect, the answer that I was following suggested rebooting to sort all issues in one go. Having tried \`sudo reboot\` on my local VM earlier today, I thought it couldn't hurt, ran the command on my remote, and it hasn't come back online yet. It should be automatically connecting to tailscale on startup, it has worked like that in the past, but it hasn't this time. It has been an hour since tailscale last connected with the device. What did I do wrong, and how do y'all handle rebooting your bare metal when you don't have access to the physical server atm? ---------------------------------------------------------------------- joecool42069 · 2023-11-21 23:09 Without the exact configurations/commands you entered and inability to remotely troubleshoot, there's absolutly no way to tell what you did wrong. If you were playing with the network configurations, it would seem to be that? How do we handle bare metal remotely? IP KVM, be it an internal IPMI, iDrac, ILO, whaterver. Or by an external IP KVM, such as PiKVM or by older, yet still pretty expensive enterprise IP KVMs. ---------------------------------------------------------------------- Simplixt · 2023-11-21 23:13 Makes no sense guessing what went wrong, just look it up as soon as you are home. If it's critical to debug your baremetal remotely, build a IP KVM with PiKVM [https://pikvm.org/](https://pikvm.org/) I'm just using a WiFi plug so that I can force as reboot (as I set my server to start after powerloss) and I have an additional VPN configured directly on my router. It's not a good idea to have it just on the application server that can break easily. ---------------------------------------------------------------------- Ieris19 · 2023-11-21 23:16 I have a feeling it's one of two things, either I fucked up the DNS settings which is what I was doing, and the computer can no longer connect to the internet, in which case I think I can access it if a friend goes home and sends me the IP address of my router (it should still listen to SSH if I ask my friend to port forward my SSH port temporarily right? I'm comfortable with sharing my router password since I can change that easily remotely) More likely, the server just doesn't properly shutdown, last time I shut it down after use it was still trying to shut down a week later when I noticed the power wasn't fully out (I wasn't using it for a long period). I did "test" that it properly turned off after and it seemed fine, but maybe there's something wrong and it won't shut down, since it is stuck somewhere it can't reboot and thus come back online. Can I ssh into a server that's currently not running any sessions itself? I tend to have a local session normally so not sure if that matters ---------------------------------------------------------------------- Ieris19 · 2023-11-21 23:17 IP KVM sounds so aliens to me, never heard of it ---------------------------------------------------------------------- joecool42069 · 2023-11-21 23:20 # ¯\_(ツ)_/¯¯\_(ツ)_/¯ ---------------------------------------------------------------------- joecool42069 · 2023-11-21 23:21 I like my PiKVM. ---------------------------------------------------------------------- Ieris19 · 2023-11-21 23:21 I am a student and this is a hobby, so I am using a router I barely understand the capabilities of, and learning along the way. I won't be home for another month, so I guess I'm screwed. What would you recommend if I can't afford any additional hardware beyond the one laptop I use as a server? The VPN is Tailscale, so it still runs since it's a mesh, just the server in question never connected to it to begin with, meaning it is not powered on and/or has no internet connection. ---------------------------------------------------------------------- Ieris19 · 2023-11-21 23:22 Is that a shrug emoji? ---------------------------------------------------------------------- Ieris19 · 2023-11-21 23:24 Do I understand correctly that the KVM is a VM running on the kernel and thus, allows me to restart my "server" without powering off the kernel, thus giving me a "fallback" in case my server fails to restart? ---------------------------------------------------------------------- joecool42069 · 2023-11-21 23:24 yeah, i don't think it pasted well. maybe this one. ¯\\\_(ツ)\_/¯ ---------------------------------------------------------------------- Ieris19 · 2023-11-21 23:25 Haha, well, at least it's basically what I have left to do now haha ---------------------------------------------------------------------- joecool42069 · 2023-11-21 23:25 no. it's literally a separate computer. it has zero dependenies on the host OS. You can even enter the bios. It can be a little computer in the the server or external. ---------------------------------------------------------------------- Simplixt · 2023-11-21 23:28 It's a hobby for most of us ;) Possible solutions: \- WiFi plug to force shut down and restart your server remotely \- Look up if your router allows to configure a own VPN connection. Many consumer routers are allowing it nowadays easily. Otherwise, get a dedicated device for creating a second VPN connection to your home \- Get a RaspberryPi to build an IP KVM Otherwise, you must life with the risk something can always break with an update and you can't fix it remotely ;) ---------------------------------------------------------------------- Ieris19 · 2023-11-21 23:30 Then I am confused, I have no additional hardware aside from my old laptop serving as a server. Can I have a KVM setup or would I need something else? ---------------------------------------------------------------------- joecool42069 · 2023-11-21 23:31 something else. ---------------------------------------------------------------------- Ieris19 · 2023-11-21 23:33 I can't afford another computer; I barely make it to the end of the month as it is. WiFi plug to force shutdown/restart sounds doable, but it won't help if I have a dynIP that I don't know unless I'm connected through VPN. And sadly, Tailscale doesn't support a router, so I'd need a second server acting as a subnet server to have remote access to that network which I can't afford. So I guess I gotta live with it unless I'm missing something, sucks ---------------------------------------------------------------------- Simplixt · 2023-11-21 23:44 Just check if your router can do DynDNS ---------------------------------------------------------------------- Ieris19 · 2023-11-22 00:02 It can but I need a domain for it, which was exactly what I planned to do with my VPN+DNS setup ---------------------------------------------------------------------- speculatrix · 2023-11-22 00:14 What can sometimes happen is a whole new kernel changes the network device names. This can cause the network configuration to go wrong, particularly if you need a static up, so it fails to set up the right interface and routes. To fix that requires console access, either serial or keyboard/video. For serial console, a raspberry pi or a WOOBM might work. https://mikrotik.com/product/woobm But this is a total guess. ---------------------------------------------------------------------- f0xsky · 2023-11-22 00:29 and thats why you run servers with IPMI/ILO/iDRAC :) ---------------------------------------------------------------------- -eschguy- · 2023-11-22 00:32 In general I don't reboot bare metal without being on-site. ---------------------------------------------------------------------- -my_dude · 2023-11-22 01:02 Do you have anything else on the network with tailscale? You might be able to ssh into it from another device at home if it powered back on. ---------------------------------------------------------------------- Ieris19 · 2023-11-22 02:00 Yeah, I have a few devices more on the net, my laptop with me now and my phone included. But I just gotta hope it powers back on. I have a friend checking in my apartment in a few weeks. Hopefully they can help me haha ---------------------------------------------------------------------- Ieris19 · 2023-11-22 02:51 What’s that? ---------------------------------------------------------------------- Ieris19 · 2023-11-22 02:53 WOOBM wouldn’t help since the computer is off right? If it had turned on at all, the ethernet cable plugged in should’ve kicked in without issues no mater the network device names. Then tailscale would come online ---------------------------------------------------------------------- Ieris19 · 2023-11-22 02:53 And this, is probably the lesson I gotta learn. No matter what happens, don’t reboot unless physically there ---------------------------------------------------------------------- f0xsky · 2023-11-22 03:33 its a way to connect to to the server even when OS is not running with console access you can do anything as if you had a keyboard and mouse connected but remotely. but you do have to pay a premium for these kind of motherboards. ---------------------------------------------------------------------- Ieris19 · 2023-11-22 04:28 I use an old laptop as a server. And it wouldn’t help unless I had access to both a VPN and the power button. But thanks for the suggestion! ---------------------------------------------------------------------- Shulya · 2023-11-22 04:37 I dont think there is anything to do now, except manually interact with your server. For next time, I suggest buying a smart plug, smart outlet, whatever you call it, and configure your server bios to always come back on after power comes back to it. Though, if you fucked the network settings of your machine, that wouldn't fix anything about it. What I have is a Raspberry Pi with a wireguard vpn on it, so in the worst case, I can ssh into it, then SSH into a server in the same local network. But again, if the network settings are so fucked that it doesn't even connect to the local network anymore, there is nothing much to do except going to the server and fixing it there. Unless you have a PiKVM (Its a raspberry pi with an HDMI In/Out that can interact with your computer even if its off), an IMPI/ILO/iDRAC (its a chip on server motherboards that can power on/power off/show screen of the server even if it's off). Good luck for your adventures ---------------------------------------------------------------------- Ieris19 · 2023-11-22 05:05 At most, I fucked up my DNS which is what I was attempting to modify. But yeah, guess I gotta get my ass there and soon hopefully have a second vpn device that allows me to access the private lan there ---------------------------------------------------------------------- hannsr · 2023-11-22 06:22 Ipmi/ikvm/ilo/pikvm can control the power power, that's what makes it so great. But yeah, without VPN that wouldn't help much. Maybe your router supports VPN as well and you can set that up as a backup solution if something goes wrong? ---------------------------------------------------------------------- qfla · 2023-11-22 06:49 KVM VMs in Linux kernel just happen to have the same name. What we are talking about here is often called IP KVM which means i believie Keyboard Video Mouse over IP. Its a separate device (with enterprice servers its often integrated into motherboard) that you plug into network and into your PC thats lets you control it remotely ---------------------------------------------------------------------- servergeek82 · 2023-11-22 06:53 iLO even has an app lol Love having a server with a console built in. ---------------------------------------------------------------------- jkirkcaldy · 2023-11-22 08:58 You should put a smart plug on your remote server with the bios setting to always power on after power loss. This won’t help you if you’ve fucked up network settings, but it’s a life saver if your computer locks up or doesn’t come back. Turn off the smart switch, wait a few seconds and turn it back on again, your computer should start back up. ---------------------------------------------------------------------- Ieris19 · 2023-11-22 13:07 My router supports VPN, but my VPN (Tailscale) doesn’t support my router. I could have a second Wireshark VPN just in case, connecting my router and my laptop or smth. But don’t know how much that could help ---------------------------------------------------------------------- speculatrix · 2023-11-22 13:48 I'm suggesting the computer might be on but the network settings are now wrong ---------------------------------------------------------------------- Ieris19 · 2023-11-22 14:37 Would the router resolve DNS if the computer’s DNS is fucked? Or would the computer need its internal DNS to even request the router’s DNS? Because if it’s the later, the computer might be on, if it’s the former, then the network settings should be okay ---------------------------------------------------------------------- mazowo · 2023-11-22 15:02 It can have many reasons as to why the server won't connect again. Most likely one of the configurations changes caused it to loose internet connectivity. Don't worry, it happens to everyone at some point. I can't tell you what exactly you did wrong, but I can give you tips on how to prevent it in the future: Firstly, take the time to research about whichever service you want to change. Typically, there is a way to reboot any service, without rebooting the entire server. Once you figured that out, you can write a little script that backd up your original config, applies the new one and then reverts the old one back unless you manually cancel within 30 or more seconds. This ensures that, if your server does loose internet connectivity, your changes are reverted automatically. If everything works as exoected, you just CTRL + C to cancel the rest of the script. Also make sure to spawn the script in an independant tool like tmux, so it doesn't end with your session. Only once you are sure that all your configs 100% work, you may or may not fully restart the server. ---------------------------------------------------------------------- Ieris19 · 2023-11-22 15:06 Lesson learnt I guess! Taking notes from all of your valuable advice ---------------------------------------------------------------------- -eschguy- · 2023-11-22 15:23 9/10 times you're probably fine. But that tenth time is going to be a *biiiiiiiiiitch* ---------------------------------------------------------------------- Ieris19 · 2023-11-22 15:27 Haha, I can see lol. I am too used to Windows. Changed my DNS and assumed after being prompted that I needed to reboot. Probably only had to restart systemd-resolv somehow ---------------------------------------------------------------------- qwertyvonkb · 2023-11-22 15:38 If you had ilo or idrac for instance, it would help alot. ---------------------------------------------------------------------- Ieris19 · 2023-11-22 15:39 I keep hearing about those but don’t fully understand what it means ---------------------------------------------------------------------- hannsr · 2023-11-22 17:43 There are multiple names (idrac, ilo, ipmi, pikvm) for the same thing. Those are remote management systems that let you control the host system via Ethernet. So if you could VPN to your remote network, you could then use whatever you have available of those (most likely pikvm since you're using an old laptop) to see the display output of your house remotely. It's like you're sitting in front of it, with all the options like pressing the power button as you'd have being physically there. My strategy for your setup would be to get some sort of dyndns, then set up a VPN on the router. That way if, for whatever reason, your tailscale tunnel doesn't come up after a reboot, you can connect to your routers VPN, pull up the kvm, and see what's happening on the screen and for example use the local shell. But pikvm sadly aren't cheap and used IP kvm are mostly junk that runs on java or are still expensive. But if you can get your hands on either of those for cheap, and can afford to, I'd get one in a heartbeat. In your current situation though... As long as you don't have anyone to assist you on site, I don't see how you'd solve it.