Warning the the noobs reddit / selfhosted ---------------------------------------------------------------------- Nikto_90 · 2023-10-15 12:02 About 2 years ago i started a new journey of learning more about tech, self-hosting, and coding (python). Since then I have implemented home assistant, various self-hosted tools, and a cloud server that runs various things i want access to no matter whats going on at home. One of the things i finally go to this weekend was security for my cloud server. I still prefer GUIs over CLI and was looking for firewall solutions i could run on my cloud server with easy to change rules. While i use a Tailscale VPN to access it from my devices i realised that having minimal protection is still probably a bad idea. I came across Copilot (with firewalld) and decided to give it a test. I was not prepared for what i would find the the logs section. I was fiddling/tinkering around with various menus (as a usually do as my primary way of learning) i located a sshd section of the logs. In this section i identified that there was anywhere between 5-10 attempts at SHHing into my server both on port 22 as well as other random ports. I checked the IP addresses and most were from China/Hong Kong/Singapore. I have no idea how anyone could identify my server's IP and begin attacks on it (still a noob in that respect), I have not really done much with the server at all oter than host a few things via docker (pihole, nginx, gokapi, a few other things) ; but this certainly appears like a problem. I spent the last 24 hours figuring out how to setup FirewallD in such a way that i can continue to access my server via Tailscale and the various APIs that i have sitting on it, without allowing any other direct connections to the server, and also ensuring that with any reboot of the server the whole thing reinstates in such a way that i get tailscale connection back without having to do anything. The warning to all noobs like me is this: just because it feels like its safe, and no one should know about the existence of your server and/or its IP address doesnt mean that you are safe. **You are not safe!** Would any veterans suggest tools/monitors/methods/alerts etc for us noobs to monitor and increase our security? Is there a reason or something I did that has resulted in someone identifying my server and attacking it? ​ ​ ​ ​ ​ ​ ---------------------------------------------------------------------- ElevenNotes · 2023-10-15 12:13 Security through obscurity is no security. If you have open ports, scripts will try to login. ---------------------------------------------------------------------- p_235615 · 2023-10-15 12:18 > I have no idea how anyone could identify my server's IP and begin attacks on it Most attacks are just random script driven stuff, which goes through the whole IPv4 range (there is really not that much IPv4 adresses, if you trying like 50 in parallel every few seconds), and just randomly looking for vulnerable versions of services and trying various easy access tricks. For any reasonable up to date server these are no real threats beside annoying log fill. You can use fail2ban to block IPs after few failed attempts or just straight up block IP ranges based on geolocation. ---------------------------------------------------------------------- Nikto_90 · 2023-10-15 12:18 I guess you never really think this could be an issue until you see it. I assumed these login attempts were bots, but still no idea how my server got on their radar in the first place. There is something odd that i outline [here](https://www.reddit.com/r/ pihole/comments/178e481/weird_client_issue_tailscale_ips_not_showing/) that could be connected? ---------------------------------------------------------------------- sk1nT7 · 2023-10-15 12:31 > Masscan – Worlds fastest scanner can scan the entire Internet in under 6 minutes, transmitting 10 million packets per second.  So it is basically very easy to detect open network ports. Especially those default ones like 80, 443, 22, 21 etc. As soon as you configure port forwarding and expose stuff to the Internet, automated bots will hit it. Also the more benign things like Censys, Shodan and so on, which also constantly crawl and index the IPv4 space. Harden your SSH service accordingly: - Disable password auth and use public key authentication only - Disable the possibility to login as root directly - Harden your SSH ciphers and algos. May use this audit python script https://github.com/jtesta/ssh-audit - May define a non-standard port for your SSH service. This is security through obscurity but most automated bots only look for TCP/22. So if you use TCP/65432 as an example, you won't get that many hits, if any at all. - Install fail2ban so that multiple failed logins lead to an IP ban of the attacker. If you have pubkey auth only, this would be rather optional but just install it. Better safe than sorry. ---------------------------------------------------------------------- SceneAcceptable9176 · 2023-10-15 12:37 If it’s public, it’s on the radar. ---------------------------------------------------------------------- Plane_Resolution7133 · 2023-10-15 12:51 Add [CrowdSec](https://www.crowdsec.net) to that list. ---------------------------------------------------------------------- Sphyix · 2023-10-15 12:55 I enjoy leaving port 22 open, then putting endlessh behind it. ---------------------------------------------------------------------- protocolnebula · 2023-10-15 12:59 I’ve just open the important ports that have self security and the others are behind traefik + Authy with 2fa (except in local that is just username + password) I have my domain with dyndns, so I can access anywhere with a bit of security Of course, I’m not blinded at 100%, for sure some hacker can enter into my server but, good luck haha ---------------------------------------------------------------------- ju1ce1ess · 2023-10-15 13:03 You mean cockpit? Not Copilot right. ---------------------------------------------------------------------- phein4242 · 2023-10-15 13:06 If you dont want something to be accessible by others, dont connect it to the internet at all, and dont rely on 3rd parties to provide you with security (if its free, guess who is the product). ---------------------------------------------------------------------- Nikto_90 · 2023-10-15 13:12 Yes. I keep saying copilot for some reason 🤦‍♂️ ---------------------------------------------------------------------- DaegurthMiddnight · 2023-10-15 13:13 Lol! It's hilarious actually. ---------------------------------------------------------------------- WiseCookie69 · 2023-10-15 13:13 Normal background noise, which you'll have with literally every service that's listening on default ports and accessible from the internet 🤷‍♂️ ---------------------------------------------------------------------- DaegurthMiddnight · 2023-10-15 13:15 Ports don't have self security, what do you mean by that? ---------------------------------------------------------------------- Evelen1 · 2023-10-15 13:19 I see no harm in attempts as long as they are unsuccessful. I have the same here, they try most default stuff admin:admin ect. ​ Good thing to have a long and secure password ---------------------------------------------------------------------- apcyberax · 2023-10-15 13:46 this is the danger of not knwoing how to read a firewall rule. If you have anything on the internet it will get found ---------------------------------------------------------------------- apcyberax · 2023-10-15 13:47 every open SSH server on the intenet in the IPv4 address space nmap [0.0.0.0/0](https://0.0.0.0/0) \-p22 ---------------------------------------------------------------------- LastTreestar · 2023-10-15 13:47 Why have a "cloud server" at all?? Use a Tailscale or another VPN. ---------------------------------------------------------------------- Nikto_90 · 2023-10-15 13:48 I do use Tailscale connect to the server. I’m not sure I understand your question. ---------------------------------------------------------------------- autisticit · 2023-10-15 13:49 I recently learned about masscan. My dynamic IP was being scanned so I just changed it three times, and to my surprise the scanning was still going on... ---------------------------------------------------------------------- LastTreestar · 2023-10-15 13:51 I fixed the question because I knew it would confuse you. ---------------------------------------------------------------------- lucassou · 2023-10-15 13:51 Having multiple ssh connexion attemps per day (or hours, minutes, seconds... ?) is very standard for anything directly connected accessible from the internet. It's most likely not someone targeting you but just some random bot that tried to ssh with commonly used credidential. ---------------------------------------------------------------------- fab_space · 2023-10-15 13:55 - wazuh (and fix where needed) - crowdsec (and check decisions list) - netdata to monitor everything and be notified is a good start ---------------------------------------------------------------------- Sansui350A · 2023-10-15 14:18 Have hole, will be fucked. ---------------------------------------------------------------------- No_Dragonfruit_5882 · 2023-10-15 14:25 How is this still unknown? Getting 4000 Login attempts a Day on ssh on my Research cluster. Either Privatekey auth or dont open your Ports. Not even a Minute, if they bruteforced your pw and the access was enabled your whole network could be backdoored. ---------------------------------------------------------------------- AnApexBread · 2023-10-15 14:26 Security through obscurity IS security but it should not be the ONLY form of security. Changing the SSH port doesn't stop an actual hacker but it does stop scripts aimed at port 22. Additionally, it decreases the number of logs you must go through, allowing you to find more important stuff quicker. It should not be the only way you protect SSH but from a cost benefit comparison it costs almost nothing to change the port and it gives you more than zero benefit. Therefor it's worth doing. ---------------------------------------------------------------------- dazchad · 2023-10-15 15:00 People are downvoting you, but the reason we win the fight against spam is, among others, security through obscurity… led by dozens of super smart grey beards ---------------------------------------------------------------------- dazchad · 2023-10-15 15:02 I’m not sure either. Maybe GP meant you can close all incoming ports, and access through Tailscale VPN ---------------------------------------------------------------------- Nikto_90 · 2023-10-15 15:07 I use Tailscale to make the connection secure. I’ve also made it so that the containers cannot be accessed from outside (setting the container ports to 127.0.0.1:5678:5678, for example). ---------------------------------------------------------------------- Scimir · 2023-10-15 15:09 Giving out your ip in most cases is not any more or less dangerous than not doing it. As long as you have exposed services you need to consider security. In most cases it is as easy as having strong passwords and 2fa where ever you can. Once you get more comfortable with the technology you should consider implementing remote access options like a vpn, put exposed services behind a proxy and ditch password auth in favour of keys. To be honest, your data rarely is from any interest to most people with malicious intent but the chance always exist. Especially when nobody has to pick your services out of the worldwide pool because you use leaked passwords or outdated services. Most attacks will be automated and therefor you are a lot safer as soon as you put just a little more effort into it than most. If you want to monitor your public services you can use services like Shodan (lifetime membership is on sale for 5$ quite often). For internal monitoring you can start with something easy like UptimeKuma and later switch to full blown self hosted monitoring such as CheckMK, Zabbix etc. ---------------------------------------------------------------------- Regfdip · 2023-10-15 15:11 Could you not have just blocked port 22 at the firewall and used tailscale to SSH into your machine, that's what I have been doing for over a year now and I never see SSH attempts. ---------------------------------------------------------------------- pigers1986 · 2023-10-15 15:12 do not expose, unless you reallllllyyyyyyyy have to (f4cking TVs and Emby) I keep all my stuff (except Emby) under Tailscale - so far no issues most of my hosts are internet facing and also have CrowdSec installed with bouncer. ---------------------------------------------------------------------- Nikto_90 · 2023-10-15 15:12 Yup that’s what I’ve done now. ---------------------------------------------------------------------- Big-Finding2976 · 2023-10-15 15:14 Sorry, we won the fight against spam? Can someone tell my inbox please! ---------------------------------------------------------------------- LastTreestar · 2023-10-15 15:18 You're not answering the question. Why do you need a "cloud" server??? ---------------------------------------------------------------------- Nikto_90 · 2023-10-15 15:20 Because I don’t really have a home server to host/run things on at the moment. Also want to the separated from my home internet/power in case something drops. ---------------------------------------------------------------------- ElevenNotes · 2023-10-15 15:20 It takes me a few seconds to scan all 65k ports, then another few seconds to get the protocol behind the port. Are you aware of that? ---------------------------------------------------------------------- nerdandproud · 2023-10-15 15:27 I know it looks scary but as long as your sshd only accepts login with keys only and is kept up to date those login attempts do not matter! Cryptography works and we've been betting our world's economy on that for a while. In that configuration sshd is as secure if not more secure (extremely good track record) as any VPN. ---------------------------------------------------------------------- ElevenNotes · 2023-10-15 15:30 No, SPF, DKIM and more helps better, unless you talk about NDR and rcpt-to, sure there you should never send an NDR and always accept any rcpt-to. ---------------------------------------------------------------------- basicallybasshead · 2023-10-15 15:30 Thanks. It is important to share information like this to enhance overall security literacy. ---------------------------------------------------------------------- dummptyhummpty · 2023-10-15 15:41 Yeah, I feel like this is IT101. But I guess not everyone knows. ---------------------------------------------------------------------- j0hnp0s · 2023-10-15 15:41 I replace fail2ban with crowdsec, and leave ssh port at 22. As you said, ports are easy to scan. And 22 is a fast honeypot into the community's blocklist ---------------------------------------------------------------------- dazchad · 2023-10-15 15:48 You may be new to the internet. 20 years ago it was pure chaos. Of course. I’m talking about the big providers (google, Microsoft, etc). If you self host your email then it depends on your implementation ---------------------------------------------------------------------- weeklygamingrecap · 2023-10-15 15:59 With the rise of YouTubers telling everyone how easy self hosting is anyone can jump in anywhere and follow a guide. This is great to help people learn but it also bad because it assumes that people are also following best practices. I'm not sure how to solve this without having everyone start a video like "remember to see episodes 1-50 first before you start this!" Even a seasoned IT person can make mistakes while trying out new things with the amount of open source projects that get mentioned around here. ---------------------------------------------------------------------- AnApexBread · 2023-10-15 16:19 Did you bother reading what I said? I said specifically that it won't stop a hacker but it will stop a bot. ---------------------------------------------------------------------- ElevenNotes · 2023-10-15 16:19 Um, you know that "bots" scan all 65k ports too? ---------------------------------------------------------------------- AnApexBread · 2023-10-15 16:22 Yes, I'm aware But again. It costs almost nothing to change a port and it gives more than zero benefit. I'm sorry that you're still stuck in the archaic mentality that if something doesn't provide 100% security then it's not worth doing. ---------------------------------------------------------------------- ElevenNotes · 2023-10-15 16:23 It does not stop anything, so why do it at all? ---------------------------------------------------------------------- JoeB- · 2023-10-15 16:30 >The warning to all noobs like me is this: just because it feels like its safe, and no one should know about the existence of your server and/or its IP address doesnt mean that you are safe. **You are not safe!** What on earth would make you think that you are safe? The internet is a scary place. ​ >Would any veterans suggest tools/monitors/methods/alerts etc for us noobs to monitor and increase our security? ***For monitoring***... Use a good firewall and export firewall events to an external time-series database, e.g. InfluxDB, Elasticsearch, OpenSearch, etc. I export all firewall events from a pfSense firewall to an Elasticsearch/Logstash/Kibana (ELK) server and keep data on for a rolling 12-month period. Here is a screenshot of my [**Firewall Blocked Events**](https://i.imgur.com/IzF6Fjw.png) Kibana dashboard for the last year. I have a lowly residential Internet service with dynamic IP, and open ports on my firewall for an IPsec VPN server only when travelling. Regardless, the pfSense firewall dropped 3,609,328 connection attempts over the last 12 months. That is an average of almost 7 attempts per minute, or one every 9 seconds. Some of those are benign web crawlers, but most are nefarious. The top 15 ports scanned over the last 12 months were... 1. **23** telnet 2. **6379** redis db cache 3. **22** ssh 4. **8080** http-alt 5. **80** http 6. **8443** pcsync-ssl 7. **443** https 8. **3389** ms-term-services 9. **5555** \[*ServeMe*\] 10. **1433** ms-sql-s 11. **81** \[*RemoConChubo*\] 12. **2375** docker 13. **8081** blackice 14. **2376** docker rest api 15. **2323** 3d-nfsd **To increase security when exposing a system to the Internet**... * [**Network Isolation**](https://www.sciencedirect.com/topics/computer- science/network-isolation) \- Use subnets and/or VLANs to limit communication from the system exposed to the Internet from the LAN. This also may be referred to a DMZ. * **Server Hardening** \-Lock down the exposed system to minimize the attack surface. It can be daunting as in [Linux Hardening Guide](https://madaidans-insecurities.github.io/guides/linux- hardening.html), or more simplified. Google *linux hardening* or *bastille linux* for more info. These guidelines are specifically for Linux. Hardening a Windows server exposed to the Internet will be different of course, but the goal is the same, to minimize the attack surface. * **Active Measures** \- Use an intrusion detection system / intrusion prevention systems (IDS/IPS) such as open-source [Suricata](https://suricata.io/) on the firewall, and installing file system integrity monitoring, such as the open-source [Wazuh](https://wazuh.com/) on the exposed server. These are combined in one open-source solution, [Security Onion](https://securityonionsolutions.com/) ---------------------------------------------------------------------- AnApexBread · 2023-10-15 16:36 >It does not stop anything, so why do it at all? Man, reading is hard for you, isn't it? If the bot is targeting port 22 and you put the SSH port on 2222 then it stops the bots targeting port 22. This also means less logs I have to look through because now I don't have to sift through all the stupid bots. ---------------------------------------------------------------------- Serge-Rodnunsky · 2023-10-15 16:36 5-6 attempts is really nothing. Perfectly normal random bots simply trying every single ip on standard ports like 22/80/8080 etc. You’re not under an attack, to mention that if you’re not using a default login/pw like “admin/admin” it would take hundreds or thousands of attempts to brute force attack the ssh credentials. Not 5-6. These bots, which are constantly scouring the internet are just looking for ports that are either open (no pw) or have default pw. ---------------------------------------------------------------------- ElevenNotes · 2023-10-15 16:43 The bot does not target a single port. The bit scans all ports and then scans the protocols behind and informs the other workers to start adding that IP and port to the list of server to run login attacks against. This is not very difficult to understand. Painting your front door black does not prevent burglary. ---------------------------------------------------------------------- ellipticalporcupine · 2023-10-15 17:09 This is always a good starting point: https://www.cisecurity.org/benchmark/debian_linux Automated version: https://github.com/ovh/debian-cis ---------------------------------------------------------------------- AnApexBread · 2023-10-15 17:25 >The bot does not target a single port. The bit scans all ports and then scans the protocols behind and informs the other workers to start adding that IP and port to the list of server to run login attacks against. This is not very difficult to understand. Painting your front door black does not prevent burglary. Some bots do, others don't. I'm not going to keep discussing this with you. I've done cybersecurity for over a decade. Deception is not only a good good practice, it's the next generation of defensive cybersecurity. ---------------------------------------------------------------------- dal8moc · 2023-10-15 17:38 Just a note. How do you think your server connects to the internet? The moment you establish a connection your ip needs to be known to the other side. It’s just an address other systems can use to reach your “door”. There is no real way to “hide” it. You can make your ports go dark by not averting icmp messages. So any machine trying to reach your service won’t get any message back. But once you reach out to other services (email, web, etc) you’ll hand out your ip. That is perfectly normal. ---------------------------------------------------------------------- gold76 · 2023-10-15 17:39 5-10 is nothing, I have 188 blocked ip’s currently (fail2ban) and that’s low. ---------------------------------------------------------------------- ElevenNotes · 2023-10-15 17:40 Just because someone is doing something since a while does not mean they are good at what they are doing. ---------------------------------------------------------------------- apbt-dad · 2023-10-15 17:40 Thanks for chuckle. Installing it. ---------------------------------------------------------------------- ItalyPaleAle · 2023-10-15 17:50 Just know you’re wasting a lot of resources yourself, including precious TCP sockets and TCP ports on your NAT :) ---------------------------------------------------------------------- skmcgowan77 · 2023-10-15 17:53 >I assumed these login attempts were bots, but still no idea how my server got on their radar in the first place They didn't get specifically your IP and didn't specifically target you or your server, there's just countless bots out there running that scan the entire public IP space over and over looking for open ports. They then determine the protocol utilized on that port, and typically then fire off worker scripts to attempt various attack vectors known to exploit vulnerabilities in that protocol. Take a deep breath, change at least the SSH port to something non 22, setup crowdsec (the "punkbuster" of ip blocking, i.e. a blacklisting service) or fail2ban, maybe a Honeypot on port 22, a WAF and/or an IDS. That should get you into a better position AND give you a kickstart into learning more about security solutions. Sláinté ---------------------------------------------------------------------- Shitpid · 2023-10-15 18:04 This is the best thing I've seen in the thread. Thanks for sharing. ---------------------------------------------------------------------- Sphyix · 2023-10-15 18:07 For the number of connections that I get, around 2MB of ram and close to no CPU at all ---------------------------------------------------------------------- ball_soup · 2023-10-15 18:12 This reminds me of the user on here a few weeks ago that told me that security updates aren’t that important because who cares about our tiny little home networks? We’re not the big fish so it shouldn’t matter. Scripts don’t care. They’ll look for any open ports on a range of IPs. ---------------------------------------------------------------------- ItalyPaleAle · 2023-10-15 18:14 How many concurrent connections do you have however? If your server is behind a NAT like I assume it is, what I am mostly concerned with is port exhaustion on the NAT. ---------------------------------------------------------------------- ball_soup · 2023-10-15 18:15 Being condescending to someone else doesn’t make your words less untrue. ---------------------------------------------------------------------- Sphyix · 2023-10-15 18:23 I'm getting at most 20 connections simultaneously, no way of reaching port exhaustion with something like this ---------------------------------------------------------------------- maddsua · 2023-10-15 18:25 I'd try this in production sometime, just the idea gives me a blast! ---------------------------------------------------------------------- maddsua · 2023-10-15 18:30 I remember chinese IP trying to log into my mikrotik routerboard around 2 years ago, yeah, I forgot that it's all exposed after the reset. There was a ton of logs trying to login as admin, poor bastards =) ---------------------------------------------------------------------- rafipiccolo · 2023-10-15 18:32 Dick jokes are the best :) ---------------------------------------------------------------------- BarServer · 2023-10-15 18:34 > I have no idea how anyone could identify my server's IP and begin attacks on it They didn't identify your server. They just scan and probe the entire internet address space. The same as when you receive a phishing mail (for example for Amazon). They, most likely, don't know you have an account associated with that address. They just send their phishing mails to every address they have. ---------------------------------------------------------------------- mpcabete · 2023-10-15 19:06 Why disabling ssh with password is a recommendation, is password login insecure even with a robust password? ---------------------------------------------------------------------- sk1nT7 · 2023-10-15 19:11 Password authentication itself is secure and robust. However, if supported, it allows for brute-forcing attacks where an attacker just tries unlimited guesses of username:password tuples. If you support pubkey auth only, an attacker would have to have your private key file; which cannot be guessed or brute-forced (assumed a random keyfile was created Witz >=2048-bit). So it can be seen as hardening in depth. ---------------------------------------------------------------------- Bytepond · 2023-10-15 19:27 Because it’s scanning every single ip all the time? ---------------------------------------------------------------------- DarligUlvRP · 2023-10-15 19:31 5-10 ssh login attempts in how long a period of time? I use to get that in about a minute until I figured it out. Started running endlessh and they eventually stopped. One thing to do is use port knocking. ---------------------------------------------------------------------- trisanachandler · 2023-10-15 19:37 >May define a non-standard port for your SSH service. I somewhat have to argue on this one, but it is somewhat true. I might also recommend locking down what IP's you can SSH from, including either just your home and a VPN. Depending on the cloud provider, use their API to update your home IP. ---------------------------------------------------------------------- Charming-Molasses-22 · 2023-10-15 20:04 You shouldn't be less worried about how many different IPs are attempting to login via ssh and more worried about restricting access to ssh. E.g I use this simple script to get alerts on my Telegram every time there's a successful ssh login. That way I know if my ssh keys are safe or not. https://github.com/64mb/SLAT-ssh-login-alert-telegram ---------------------------------------------------------------------- Ion_GPT · 2023-10-15 20:17 You can do an experiment. Install a VNC server with “password” as password on a computer directly connected to the internet. It will take less than one minute until the mouse will start moving on the screen showing someone connected to your VNC server. ---------------------------------------------------------------------- zer04ll · 2023-10-15 20:21 use cloudfare ---------------------------------------------------------------------- fjortisar · 2023-10-15 20:41 Shodan doesn't scan every port (it scans a list of about 1300), it's also not that common for bots to do it either. Set up an SSH server on 22 and one on another port and compare how many times they get brute forced. ---------------------------------------------------------------------- idreamofjeanshorts · 2023-10-15 21:03 Someone can intercept a password, keys do a diffie Hellman key exchange to prevent sending the password as plain text, ensuring non- repudiation. ---------------------------------------------------------------------- Nikto_90 · 2023-10-15 21:06 yup, i do ---------------------------------------------------------------------- dazchad · 2023-10-15 21:13 Sorry. Didn't mean to. Reworded my comment. ---------------------------------------------------------------------- hmoff · 2023-10-15 21:14 That’s an issue on outbound connections, not inbound. ---------------------------------------------------------------------- felansky · 2023-10-15 21:30 If you're being specifically targeted, obscurity gives you nothing at all. If you're being scanned by 7 thousand scripts, 90% of which are targeting 22 specifically, it does decrease the chance of a successful attack. As long as you understand it's a decrease and not elimination, you can get benefits from it. Making obscurity the only defense angle is ignorance and stupidity. Making it one of the angles however, definitely reduces the attack surface, and - possibly even more importantly - improves observability for your defenses in other areas. ---------------------------------------------------------------------- dinosaurdynasty · 2023-10-15 21:33 Public key auth is just _better_ in a lot of ways (for instance, if you accidentally log into the wrong server with a password (or just reuse it), welp... with a key it's not _as_ big of a deal (client security issues notwithstanding)) Also you can automatically sign into servers with keys, you can't with passwords, so it's better ergonomics ---------------------------------------------------------------------- lilolalu · 2023-10-15 21:37 I enjoyed the detailed description on "masscans" GitHub page on how to scan the entire internet: _masscan 0.0.0.0/0 -p0-65535 --max-rate 100000_ This increases the rate to 100,000 packets/second, which will scan the entire Internet (minus excludes) in about 10 hours per port (or 655,360 hours if scanning all ports). 655,360 hours is approximately 74.8 years. ---------------------------------------------------------------------- ElevenNotes · 2023-10-15 21:37 I have SSH available via :22 and :443 and there is almost no difference. ---------------------------------------------------------------------- skibare87 · 2023-10-15 21:42 Right? He wasn't lying about the noob part. If it's on the internet it will get scanned or someone will try to access it. What matters is the security and procedures in place to handle it. Be worried when someone breaks in, not just knocks. ---------------------------------------------------------------------- MrSliff84 · 2023-10-15 21:58 At hetzner it's possible run a pfsense instance and run your other VPS behind it in a private network. So you can benefit of a gui driven firewall and also hide your ssh behind the firewall. The smallest cloud VPS for 4 bucks a month is good enough for pfsense. ---------------------------------------------------------------------- lonlygamerx · 2023-10-15 22:26 Recommend putting your server on zerotier and creating a private network on there and not port forwarding the port 22 as doing it this way, you can make sure only people who have zerotier and are connected to your private network on zerotier can access your server. This can help increase your security as no one else can connect to your server plus if you ever forget what IP to ssh to you can always go on zerotier to check or even change it. ---------------------------------------------------------------------- Nikto_90 · 2023-10-15 22:32 Yup, I use Tailscale for this ---------------------------------------------------------------------- surgeimports · 2023-10-15 22:41 I use Cloudflared (cloudflare tunnel) to connect my selfhosting to my domain via only that port ---------------------------------------------------------------------- primalbluewolf · 2023-10-15 22:52 Those are both well-known ports, so that's not any surprise. ---------------------------------------------------------------------- primalbluewolf · 2023-10-15 22:56 > still no idea how my server got on their radar in the first place. It's because you have a public IP address. This is the internet equivalent of calling every number in the phone book, one at a time. Computers can do that very fast. If you check out the page on masscan, you may note that scanning the entire ipv4 address space can be done in a matter of minutes. The shotgun approach is easy: doorknock the internet on x port, try find poorly secured ones. ---------------------------------------------------------------------- npsimons · 2023-10-15 23:00 > In this section i identified that there was anywhere between 5-10 attempts at SHHing into my server both on port 22 as well as other random ports. I checked the IP addresses and most were from China/Hong Kong/Singapore. > > I have no idea how anyone could identify my server's IP and begin attacks on it It's not you personally - this is pretty typical, and they are basically just trying every IP address out there. It's the sad flipside of the double-edged sword that is progress - computing cycles, RAM and bandwidth are all dirt cheap, even for China, so they are just throwing everything at the (fire)wall and seeing what sticks. And this has been going on for a **long** time - I can't remember when I first saw logs like these (early aughts?), but I've since relegated them to fail2ban and the like so I don't lose sleep or clog my tailing window with them. ---------------------------------------------------------------------- primalbluewolf · 2023-10-15 23:00 100,000 packets/second is about one 25th of its capability, provided a fast enough internet connection and server. ---------------------------------------------------------------------- primalbluewolf · 2023-10-15 23:01 Presumably that the service running on that port implements its own for of authentication. ---------------------------------------------------------------------- CJtheDev · 2023-10-15 23:25 Even if the attacker get unlimited tries do you know how hard it is to brute force randomly generated 32 character long password with capital, small letters, numbers and symbols. I am a person that believes that leaving password authentication on with strong password best idea. In case you mess up and can't access using pubkey and you not at home ( happened to me lol ) ---------------------------------------------------------------------- DaegurthMiddnight · 2023-10-15 23:27 Ah makes sense ---------------------------------------------------------------------- lannistersstark · 2023-10-16 01:04 > In this section i identified that there was anywhere between 5-10 attempts at SHHing into my server both on port 22 as well as other random ports. Eh, this isn't really an issue. People can't just magically login to your ssh if you harden it(Key auth and disabling passwords is sufficient for most people). ---------------------------------------------------------------------- lannistersstark · 2023-10-16 01:06 >Would any veterans suggest tools/monitors/methods/alerts etc for us noobs to monitor and increase our security? I mean, I don't really see an issue there. Granted you have performed some due security updates : Disabling password login, using key-pair auth for ssh, etc, then even leaving the default port 22 out there wouldn't really have any issues. People/bots can't just magically login to 22. ---------------------------------------------------------------------- Weird_Divide_8799 · 2023-10-16 01:20 Attempts are kind of normal. Few tips: -Dont ever enable root user for direct login/access. -Ideally use long complex passwords with combination of ssh certificates. -Works fine for me: https://www.digitalocean.com/community/tutorials/how-fail2ban-works- to-protect-services-on-a-linux-server ---------------------------------------------------------------------- Brave-Film9475 · 2023-10-16 01:26 if you have tailscale already why not bind your service and ssh to it? ---------------------------------------------------------------------- protocolnebula · 2023-10-16 02:40 You are not doing auth always, and less for an specific file, maybe if you change between files. The worst scenario is the encryption for the file (ssl), but you may just run ftp or guest ftp and solved. ---------------------------------------------------------------------- protocolnebula · 2023-10-16 02:42 As in another example, I have one or two service that has their own implementation, of course the ports have not Also these services are Jellyfin (for faster streaming) and plex (same case). Additionally they are also proxied by traefik ---------------------------------------------------------------------- da_frakkinpope · 2023-10-16 03:00 I love it. ---------------------------------------------------------------------- JaKami99 · 2023-10-16 03:43 Changing your SSH Port does not increase security. fail2ban is only needed if you have bad passwords. Just don't use passwords and if you do, make em strong. Then you are done with hardening :D ---------------------------------------------------------------------- lechauve911 · 2023-10-16 04:08 I usually use rh digital ocean firewall and block all ip acess except from my tailscale ---------------------------------------------------------------------- autisticit · 2023-10-16 05:03 It was scanning a whole subnet. Don't know why the downvotes smh. ---------------------------------------------------------------------- SignificantTrack · 2023-10-16 05:07 You're assuming folks don't reuse keys :) ---------------------------------------------------------------------- benn447 · 2023-10-16 05:10 Setup a twingate connector in the cloud, and only allow the twingate connector to access secure ports such as 22. If you can get away with it, dont expose any of the listening ports to the internet and only access them through twingate. Twingate is free for teams up to 5 users. ---------------------------------------------------------------------- markbeelen · 2023-10-16 05:20 With Tailscale ssh you can leave port 22 closed on your router. ---------------------------------------------------------------------- Disastrous_Elk_6375 · 2023-10-16 05:40 > Sorry, we won the fight against spam? I haven't seen a viagra, pills, hair product etc. spam in my gmail in years. ---------------------------------------------------------------------- am_kobold · 2023-10-16 07:13 SSH scanning has been a fact of life for a long while now. The IPv4 address space is small enough for all possible IPs to be scanned fairly easily. Using a non-standard port helps, but disabling password auth and root login is basically required. ---------------------------------------------------------------------- Freeman371 · 2023-10-16 07:17 Why open SSH port to the internet ? My server only have a port redirection to 443, is it safer ? ---------------------------------------------------------------------- teambob · 2023-10-16 08:11 The attacks are automated against every IP on the internet ---------------------------------------------------------------------- Nikto_90 · 2023-10-16 08:14 I use Tailscale for this. ---------------------------------------------------------------------- ProffesionalAds · 2023-10-16 08:24 Netmaker is another alternative that is great for this. Offers upto 30 free devices. ---------------------------------------------------------------------- Empyrials · 2023-10-16 08:41 You use Tailscale you said? Just block ALL inbound traffic. That’s what I do with my gcloud Bitwarden server. It’s accessible via Tailscale and that’s it ---------------------------------------------------------------------- Krieg · 2023-10-16 09:14 My "solution" is not to open any port. My webserver is tunneled via Cloudflare. And Tailscale for the rest. ---------------------------------------------------------------------- Bradyns · 2023-10-16 10:42 At a billion guesses a second, it would take \~10^(39) years. ---------------------------------------------------------------------- ReferenceLive · 2023-10-16 10:55 Even if you reuse the key, the servers all only have the public part so if you compromise server A you still cannot access server B with it. While in case of a password if you compromise server A, wait for the user to login, you now have the password for server B. ---------------------------------------------------------------------- SignificantTrack · 2023-10-16 11:22 I don't disagree with what you are saying about keys. What I was trying to aim at is that whether the password gets compromised, or the key (private) gets compromised, the impact would be the same. ---------------------------------------------------------------------- dinosaurdynasty · 2023-10-16 14:46 You can reuse keys because logging into a server doesn't give it the ability to _do_ anything with it (public key cryptography ftw). The only reason _not_ to reuse keys is privacy/identity separation. ---------------------------------------------------------------------- dinosaurdynasty · 2023-10-16 14:46 Keys are compromised _much less_ than passwords are. And if you are paranoid, you can put your private key on a hardware token. ---------------------------------------------------------------------- Low-Chapter5294 · 2023-10-16 14:59 Fail2ban guarantees a smaller attack surface - no unlimited password retrys. ---------------------------------------------------------------------- Sekhen · 2023-10-16 15:20 That's why you limit port access to select few IPs. I have rdp open toward the internet. But only to my phone on VPN, and office network. ---------------------------------------------------------------------- Nikto_90 · 2023-10-16 15:27 What I found is that I still have to allow port 22 traffic for the Tailscale interface, otherwise nothing works, and I cannot access anything through Tailscale or otherwise. ---------------------------------------------------------------------- SignificantTrack · 2023-10-16 15:32 Again, I am not arguing with you. Though I believe the "Keys are compromised much less than passwords" (we can just look at the amount of compromised accounts over the years) is true, I think compromised keys have a much higher severity and impact to organizations as those are not only access keys that it might be used about. Myself, I would recommend folks to look at step-ca or the SSH commercial solution for short lived access keys and certificates. I use a Yubikey and bought one for every employee I have when cloudflare had a $10 deal. There's really no argument here with you, just pointing out that folks do not rotate keys, do not secure them properly, and think like keys are panacea for security. ---------------------------------------------------------------------- SignificantTrack · 2023-10-16 15:34 You can, but you really shouldn't. As with a private key compromise, the attacker gets unlimited access in most cases to the server, especially if there is no password utilized there. ---------------------------------------------------------------------- dinosaurdynasty · 2023-10-16 15:46 Pretty much any case you have a private key compromise, your entire `.ssh` directory was read, which means _all_ your keys are compromised anyway ---------------------------------------------------------------------- SignificantTrack · 2023-10-16 16:12 Keys are stored in many other places than just the .ssh directory. You can store your private keys anywhere, your dropbox, onedrive, github access keys, local storage or others. .ssh directory is probably harder to get to as it is local rather than any of the other services I mentioned above. ---------------------------------------------------------------------- dinosaurdynasty · 2023-10-16 16:13 Well uh, maybe don't store private keys on dropbox ---------------------------------------------------------------------- Empyrials · 2023-10-16 16:50 Inbound??? I have no open ports and no public ip attached to my server. Same at home and I can access everything ---------------------------------------------------------------------- Nikto_90 · 2023-10-16 17:00 I’ve just gone back to test it now and for whatever reason when I was fiddling with the setup over the weekend I couldn’t get the connection via Tailscale to work without the ports being enabled on the Tailscale interface, but now that’s not the case. Thinking about it now it’s possible it’s just the interface that needed to be enabled but I did that and the ports together and thought maybe it was the ports that enabled it. Or maybe I was doing something else wrong? ---------------------------------------------------------------------- Nikto_90 · 2023-10-16 17:15 I need some ports open for https as I have a few things that needs to be accessed by others who aren’t on Tailscale + some webhooks. I use Nginx on the incoming 443 to route to various containers. ---------------------------------------------------------------------- Nikto_90 · 2023-10-16 17:32 I’ve now closed (I think) all access to everything from the internet except ports 80 and 443 as I need those open for various things to be accessible outside of Tailscale. ---------------------------------------------------------------------- Krieg · 2023-10-16 17:34 Cloudflare will tunnel everything with no ports open. ---------------------------------------------------------------------- Nikto_90 · 2023-10-16 18:45 I’ve just discovered something I wasn’t expecting. Currently have no ports open. Nginx is running and is still managing to route to the containers via Nginx. I have a domain sat on cloudflare that points wildcard to my server IP. ---------------------------------------------------------------------- indykoning · 2023-10-16 19:10 Yep, they're not specifically attacking your server. They're attacking every single IP they can get their grubby little hands on. This is why people always suggest opening as little ports as you need. Also why people suggest not using common names for your subdomains as they will try and go down the list. Anything publicly accessible will be indexed and will be attacked at some point. I think honestly the simplest solution is using services like Cloudflare where you can, they have dashboards already and built in rules. It's also easy to add custom rules e.g. "show a captcha to anyone not from my country" If you do not want to/can not use Cloudflare look into crowdsec. Takes a bit more setup but it catches a lot! ---------------------------------------------------------------------- Nikto_90 · 2023-10-16 19:12 I have a domain that points to my server and runs via cloudflare. Which settings should I dig into? ---------------------------------------------------------------------- atheken · 2023-10-16 19:51 Someone having knowledge of your _public_ IP shouldn't matter. You could possibly be a target of a directed attack, but most of the time, it's bots that want to try to exploit extremely weak or broken configurations. I basically have four ports open: - 80+443 to Caddy (an HTTPS Reverse Proxy) - 22 for ssh. - 5xxxx to Wireguard (I don't actually remember the port). Some tips: - Just because an app is behind TLS on port 443 doesn't mean it's secure. If it's not behind an auth later like Authelia, a random crawler could still make requests to it. - Keep hosted apps up to date, and add an Auth layer if you can. Don't serve on port 80, that's only for Let's Encrypt to validate requests. - Harden SSH: disable root login, disable password login, and follow the recommendations from the scan you'd run here: https://www.ssh- audit.com Anyway, bots are _constantly_ scanning public IPs for open ports. It's now so ridiculously fast to do that moving SSH to some other port will really only cut the absolute dumbest bots (maybe a few, but barely worth the effort, IMO). OTOH, moving to some other port might remove some of them, and you can set the port in your client ssh_config so you never have to think about it. ---------------------------------------------------------------------- Nikto_90 · 2023-10-16 20:14 Great summary, thank you. I have NGINX running which I’ve just learned forces open ports 80 and 443. I couldn’t understand why even when I removed those ports from the allow list I could still connect, but this seems like the answer there. Everything I have running and publicly accessible (via cloudflare) atm has its own auth. I’ve come across Authelia before, thanks for the tip. I’ll look into it a bit more. Does it work before the endpoint? Almost everything I’m running is in a docker containers (for the moment) and I made those only available via localhost, so no direct connection are possible, only Tailscale and via Nginx. I’ve now locked down all other ports. My current issue is that when the server reboots the firewall startup faster than the Tailscale container and that results in blocked SSH connections via Tailscale. Oddly other connections via Tailscale are possible still. I have to reload the firewall for the the Tailscale interface to “register” and allow Tailscale ssh connection. I have no idea why it’s just the ssh that gets blocked. My current workaround is I’ve created a workflow that runs the reload command and I run that as I’m still able to access the container 🤷‍♂️ ---------------------------------------------------------------------- indykoning · 2023-10-16 20:30 A couple interesting places to look at if Cloudflare is already proxying your network is the Analytics > Security tab and even just the Analytics > Traffic tab to see just how many requests come from random countries. You can view more on blocked clients under Security > Events Most useful place to harden your security is the Security > WAF This is where you can set up rules to block. You can click together your rules using the fields like "country" or "region" Note that known bots in this context means "good" bots like Google crawlers. on public websites you'd "use a challenge for anyone outside the region of europe and is not a known bot" for instance Then as an action you can choose Block or Managed Challenge. Block means there's no way to get in, Managed Challenge means Cloudflare will try to do an automated check if it fails you need to do a captcha. I recently enabled this for a customer receiving lots of spam on their site, of 37k requests only 7 from the spam country got through the challenge. And the spam was gone without interrupting real customers. ---------------------------------------------------------------------- indykoning · 2023-10-16 20:36 Though do note that Cloudflare only secures http(s) traffic, and only does so if you let them proxy your requests instead of only being DNS. So your SSH is not protected! Simplest solution which I use for my homeserver is not to open the SSH server to the internet. But for the most part disabling password authentication and only allowing public/private keys should be good enough ---------------------------------------------------------------------- atheken · 2023-10-16 21:11 Even if and app has it's own auth, you may still wish to have another layer for external requests - this is making it so that bugs in the app layer aren't exposed without going through Authelia (or Authentik, for example). Not all bugs that could be dangerous require auth, many do not. Inside your network, you will usually be able to access ports on your servers, but those systems rarely/can't modify your NAT Gateway (router) to allow traffic in. If you don't open those ports on your router, then public traffic shouldn't flow to them. I don't know specifically what your tailscale issue is, but I'd make sure you're routing to an internal IP, rather than a hostname. ---------------------------------------------------------------------- Empyrials · 2023-10-16 21:11 But you’re running Tailscale right? If so then it works as intended. Just to verify, get your console access, turn off Tailscale, and see what happens. I also have a cloud flare domain wild carded, certbot, nginx, & Bitwarden running with Tailscale on the host and it functions great. In turn I have PFsense at home allowing the rest of the network access so my family doesn’t have to have Tailscale on at home, they can hit the cloud flare domain for Bitwarden access. ---------------------------------------------------------------------- Nikto_90 · 2023-10-16 21:21 That example is without Tailscale. I found after some port scans that even though I don’t have 80 and 443 explicitly allowed in firewalld they are still open and allowing the connection via the cloudflare domain. I’m not wrapping my head around why the cloudflare domain in your example is needed when you use Tailscale? Or are you saying that the pfsense at home is connecting to your server without Tailascale at all? I have Tailscale in a docker container. I think that’s creating an issue where the Tailscale interface isn’t up when the firewall boots up, resulting in SSH connections via Tailscale being rejected until i reload firewalld. Oddly the containers are all still assessable. ---------------------------------------------------------------------- TheSteevLP · 2023-10-16 23:24 A very good practice is having root ssh completely turned Off and only having keyfile authentication enabled. ---------------------------------------------------------------------- Lazydayz137 · 2023-10-17 00:04 I think it is Cent that at least used to tell you how many log on attempts since your last successful. Literally thousands on a public ip every time I logged in. Wild world. ---------------------------------------------------------------------- Empyrials · 2023-10-17 13:17 i'm not sure what's with your firewall but for sure look ito it. The cloudflare domain isn't necessary for you, however it is for how i have mine configured. I allow devices behind my pfsense to access my tailscale domain, because of that, i added the domain name so other devices that don't have the tailscale app installed, to know what IP to goto so pfsense will route it correctly. I don't know why but the Tailscale plugin for pfsense wont allow other clients to do DNS lookup via the tailscale network, so i needed a way that all my clients would always know what IP to goto, hints my cloudflare domain which is pointed to a private IP address. You CAN run tailscale in a docker, however i don't. If you run tailscale in a docker, it has it's own IP via docker, so you'd need to add an Advertised Route for whatever ip your bitwarden is, then go accept that route in your admin console for tailscale. ---------------------------------------------------------------------- Nikto_90 · 2023-10-17 13:22 I don’t have Bitwarden. Ah so your pfsense firewall is a node on the Tailscale VPN and it routes a connection from your devices on your home network to that Tailscale IP even though the device doesn’t have Tailscale? Makes sense I think. Yes I advertise the various networks that exist inside my server via Tailscale and they are all accessible no matter what. Oddly it’s the SSH via Tailscale that doesn’t work post reboot until I reload the firewall. A bit annoying but ultimately I won’t use it back day to day and I’ve build a solution to quickly reload the firewall when needed. ---------------------------------------------------------------------- FoxxMD · 2023-10-17 13:50 [You should try out the go implementation of endlessh](https://github.com/shizunge/endlessh-go). It has prometheus stat exports to make fun dashboards. I've been running it for years at this point. Never fails to amuse ---------------------------------------------------------------------- Live_Shallot1353 · 2023-10-17 17:36 Offtopic: Are these 3 tools needed if I only access my network through wireguard vpn? ---------------------------------------------------------------------- fab_space · 2023-10-17 17:37 yes of course: security is not just obscurity. ---------------------------------------------------------------------- Sphyix · 2023-10-17 18:02 I like it, ty, didn't know it existed ---------------------------------------------------------------------- MonkAndCanatella · 2023-10-19 02:00 I was under the impression you just use one key per client and add your public to whichever hosts you plan to SSH to. ---------------------------------------------------------------------- NegativeK · 2023-10-21 22:13 > leaving password authentication on with strong password best idea The two reasons it isn't recommended: * People are typically bad at strong passwords. * If your password is leaked (whether due to reuse, a breach, or stealer malware,) your login is compromised. ssh with keys is effectively MFA against those concerns. Concerns about remote access reliability need to balance how important uptime really is versus security, while acknowledging that to be human is to make mistakes. ---------------------------------------------------------------------- NegativeK · 2023-10-21 22:17 No. SIEM and other cybersecurity tools aren't appropriate for the vast majority of people. You should think through the actual risks of systems being compromised and consider if you can spend the resources learning tooling to protect the systems and their data. But no matter what you do, you should check your system configs for secure settings, patch and update, and lock down services to what's necessary. ---------------------------------------------------------------------- NegativeK · 2023-10-21 22:21 > It does not stop anything Yes it does. There are plenty of half-ass bots out there that aren't checking for every form of obscurity. Obscurity isn't security, but it sure helps waste the time of the attackers -- and they don't have infinite resources. ---------------------------------------------------------------------- NegativeK · 2023-10-21 22:23 > What on earth would make you think that you are safe? Everyone has to learn some time. OP learned and shared their knowledge so that others can learn sooner. ---------------------------------------------------------------------- ElevenNotes · 2023-10-21 22:30 and you feel that this half-assed bot like you call it is dangerouse to you, how exactly? ---------------------------------------------------------------------- NegativeK · 2023-10-21 22:37 You've moved the goal post. ---------------------------------------------------------------------- ElevenNotes · 2023-10-21 22:39 Don't be so negative NegativeK. ---------------------------------------------------------------------- NegativeK · 2023-10-21 22:43 Here are further resources on the use of deception in cybersecurity, if you're interested. https://www.blackhillsinfosec.com/active-defense-cyber-deception- intro/ ---------------------------------------------------------------------- ElevenNotes · 2023-10-21 22:54 No thanks, I'm able to leave port 443 open as a comercial data centre provider. If you need some tipps, I can give you some. ---------------------------------------------------------------------- omgredditgotme · 2023-10-28 17:00 Haha, dude... if I provision a VPS /w key-based SSH access out of the box, which I pretty much always do, there's usually pages upon pages of failed/temporarily banned IP's. Key-based authentication is my recommendation. Bind all services to the tailscale wireguard network interface only, since you're already using it anyway, you might as well.