.env crawler on my self-hosted Shlink (URL Shortener) instance. Should I be worried? reddit / selfhosted ---------------------------------------------------------------------- dejovm · 2023-11-04 18:15 Hello, i recently discovered, that some crawler is trying random paths on my domain to find unsecured .env files. My router has port 80 and 443 exposed to the internet. The ports are facing to my Nginx Proxy Manager and I use Docker, where I host Shlink and my Nginx Proxy Manager. Should I be worried and secure or even close my service? Or is there a great way to block these requests. Or is there even nothing to worry about? I am using my domain with Cloudflare. It did surprise me, the requests weren't blocked automatically. ​ Thanks in advance! https://preview.redd.it/l0rxnpnsgdyb1.png?width=1571&format=png&auto=w ebp&s=01aab72dcf7729656d93e486599977f10c92a446 ---------------------------------------------------------------------- VsevolodLNM · 2023-11-04 18:51 Just ban his ip, nothing more you can do really, except for securing you envy files ---------------------------------------------------------------------- Disastrous_Elk_6375 · 2023-11-04 18:51 Welcome to the internet. (that's mild as far as access logs go) ---------------------------------------------------------------------- root_switch · 2023-11-04 19:18 If it’s just you and maybe family or friends that access your resources then why even poke holes in your router, you should use tailscale or WireGuard or something. ---------------------------------------------------------------------- WiseCookie69 · 2023-11-04 19:32 Normal background noise. You expose stuff to the public and in return you make friends with a bunch of bots. ---------------------------------------------------------------------- keshavdaboss · 2023-11-04 19:37 What took are you using in the picture? Looks pretty cool! ---------------------------------------------------------------------- JL_678 · 2023-11-04 19:53 I am currently using [Crowdsec](https://www.crowdsec.net/) to try and remediate things like this. It might be worth a look for you. ---------------------------------------------------------------------- SillyLilBear · 2023-11-04 20:01 Just another tuesday. ---------------------------------------------------------------------- dejovm · 2023-11-04 20:03 You mean tool? The tool is [Shink](https://shlink.io/) itself self hosted . And the web interface you see in my screenshot is the [Shlink Client](https://app.shlink.io/). ---------------------------------------------------------------------- dejovm · 2023-11-04 20:06 That would probably the best solution for me, but as my operating system I use unRAID and for me, it is a complicated to set Crowdsec there up. ---------------------------------------------------------------------- tech2but1 · 2023-11-04 20:27 What are you using as a router? I've just switched to using OPNsense specifically to get IDS/CrowdSec on the network. ---------------------------------------------------------------------- ok9021 · 2023-11-04 20:30 Cloudflare -> Select your domain -> Security -> WAF -> Create rule If incoming requests match… -> Edit expression ``` (http.request.uri.path contains "/.env") or (http.request.uri.path contains "/.git") or (http.request.uri.path contains "/.DS_Store") or (http.request.uri.path contains "config") or (http.request.uri.path contains "shell") or (http.request.uri.path contains "/wp-admin") or (http.request.uri.path contains "/wp-class") or (http.request.uri.path contains "/wp-login") or (http.request.uri.path contains "/wlwmanifest.xml") or (http.request.uri.path contains "/xmlrpc.php") or (http.request.uri.path contains "/autodiscover.xml") or (http.request.uri.path contains "/autodiscover.src") ``` Then take action… -> Block ---------------------------------------------------------------------- sowhatidoit · 2023-11-04 21:17 Bold of you to assume the gender of a bot. :) ---------------------------------------------------------------------- dejovm · 2023-11-04 21:17 thanks!, I just adjusted my settings in Cloudflare. This is a great solution! ---------------------------------------------------------------------- dejovm · 2023-11-04 21:22 Sadly, I am using a standard home router: FRITZ!Box 7590. I would like to upgrade to OPNsense, but the whole mesh system wouldn't work anymore. ---------------------------------------------------------------------- HugoDos · 2023-11-04 21:26 Let me know why it is complicated, and I can help Laurence CrowdSec Support ---------------------------------------------------------------------- ddpbsd · 2023-11-04 21:31 Only worry if they get a 200 ---------------------------------------------------------------------- rad2018 · 2023-11-04 21:52 First 15 minutes connecting another new web server, varied bot attacks somewhere between 3500 and 4000 hits. If it weren't for the 3 layers of firewalls, 4 if you include the server's based one, the potential attacks would be much higher. Majority of the attacks were XSS and CSRF. ---------------------------------------------------------------------- tech2but1 · 2023-11-04 21:58 I suppose you could still use OPNsense behind the FRITZ!Box. Bit more of a faff to set up but would still give you the protection it offers. ---------------------------------------------------------------------- aaron__walker · 2023-11-04 22:13 They is already a prebuilt WAF rule for this and more in CF too ---------------------------------------------------------------------- dejovm · 2023-11-04 22:40 My problem is the implementation of the Crowdsec Bouncer. I would like to protect my Nginx Proxy Manager (Docker) with Crowdsec. Installing a Security Engine isn't too hard. It works fine, if i just install the official crowdsecurity/crowdsec image. But generating alerts and actually getting the bouncer working is too much for me as a newbie. I just dont know where and how to start. ---------------------------------------------------------------------- mxrider108 · 2023-11-04 23:26 oh interesting! where do I find that? ---------------------------------------------------------------------- timberhilly · 2023-11-05 00:09 Have a look around ---------------------------------------------------------------------- d_maes · 2023-11-05 00:13 Was playing with the firewall rules on my mikrotik the other day. The moment I deleted the rule to drop input from wan, I saw the failed ssh attempt logs roll over my screen. ---------------------------------------------------------------------- DimasDSF · 2023-11-05 00:18 For cloudflare solution to work you still need to make sure you only allow traffic to reach your 80 and 443 if it comes from cloudflare to your domain name and not just from anywhere targetting your IP otherwise they just bypass cloudflare. In CF documentation there should be a list of cloudflare proxy ip ranges that you need to allow in your firewall while denying all others ---------------------------------------------------------------------- matjako1 · 2023-11-05 00:59 There was a forked version of NPM by Baudneo but it was quietly deleted from github. I am running this together with Crowdsec and it works great. Anyone knows what happened with this fork? ---------------------------------------------------------------------- rad2018 · 2023-11-05 01:03 🤣 ---------------------------------------------------------------------- RB5Network · 2023-11-05 01:28 Hey, curious to exactly what this means. Are you referring to only keeping port 443 and 80 open for a reverse proxy service like NPM? ---------------------------------------------------------------------- SEND_NUKES_PLS · 2023-11-05 01:33 They could use Cloudflare tunnels as well, that way no ports would need to stay open. ---------------------------------------------------------------------- DimasDSF · 2023-11-05 01:40 Nope, basically when you "open" your ports on the firewall you can allow specific hosts to access it or every host, now lets say I have a domain name "example.com" its proxied via cloudflare to my ip eg. 111.111.111.111 Now if I try accessing 111.111.111.111:80 I'll hit the service hosted by my server, same as if I had used example.com but the IP request does not use cloudflare at all it goes directly to the server so any web app firewall setting on cf is ignored. to prevent it from happening only allow hosts from the cf proxy ip range to access the open ports. Now if I try 111.111.111.111:80 I'll hit the firewall as my residential IP is not allowed access to this port, but using example.com I'll first hit cloudflares proxy and that then reaches out to the server on 111.111.111.111:80 and as cf proxies ips are whitelisted it can access the service and let you in ---------------------------------------------------------------------- JL_678 · 2023-11-05 02:05 I tried both docker NPM/Crowdsec forks and could not get either working. :-( In the end, I installed NPM in an LXC container in Proxmox and then installed Crowdsec in that. ---------------------------------------------------------------------- europacafe · 2023-11-05 02:41 Could you explain what the expressions actually do? ---------------------------------------------------------------------- No-Entrepreneur-6027 · 2023-11-05 03:03 Create a ".env" slug that forwards them to FBI cybercrime site. ---------------------------------------------------------------------- themedleb · 2023-11-05 03:27 I use Tailscale and it works, but If I want to use only Wireguard (without relying on Tailscale's Infrastructure), I don't have to open ports in my router and still reach my home network remotely? If it's possible, then how? if not, then why? ---------------------------------------------------------------------- nik282000 · 2023-11-05 04:12 My ssh logs gave me a cool list of 5k user names to never use! ---------------------------------------------------------------------- Readmymind · 2023-11-05 04:40 You still open a port to access the wireguard VPN from the internet, and once you're connected, it's as if you're connected to the home network. Then you don't need to open any other ports for your other services to the internet. I assume tailscale works the same way, no? ---------------------------------------------------------------------- coldblade2000 · 2023-11-05 05:34 Basically filters out requests that are searching for certain files there. Among them are .env or WordPress files that could store credentials, .git folders that could potentially leak source code or credentials or other sensitive files ---------------------------------------------------------------------- europacafe · 2023-11-05 05:46 thanks ---------------------------------------------------------------------- Genericsky · 2023-11-05 05:50 Another way that doesn't end up in exposing your home network is hosting a VPN coordinator server in a cheap VPS for like $5 a month. You may experience some latency, but it's probably worth it over the security risks that come from opening your ports. Look into [headscale](https://github.com/juanfont/headscale), which is basically an open-source implementation of the Tailscale control server you can actually self-host. It works out of the box with your Tailscale clients and in the same magic manner, with the added benefit of you being the actual owner of the VPN ---------------------------------------------------------------------- root_switch · 2023-11-05 06:08 Nah tailscale is a bit different, no open ports what so ever which is the major benefit to using it. They rely on relays, which yes you can self host (most likely on a VPS though). Have a read if you’re interested: https://tailscale.com/blog/how-tailscale-works/ ---------------------------------------------------------------------- silentassassin22 · 2023-11-05 07:05 can I get that list ---------------------------------------------------------------------- mitchellcrazyeye · 2023-11-05 07:27 Anything that bots and crawlers can hunt for can be found ---------------------------------------------------------------------- predmijat · 2023-11-05 08:00 Move SSH port to a higher one and bot noise drops to 0. Use SSH keys and don't worry about usernames :) ---------------------------------------------------------------------- lilolalu · 2023-11-05 08:39 This is not the solution, it's a patch you put on the problem. If your .env file is browsable over the internet, THAT is the problem you should fix. ---------------------------------------------------------------------- lilolalu · 2023-11-05 08:41 This is exactly a display of the lack of understanding what cloudflare is and how it helps you with security. ---------------------------------------------------------------------- Cybasura · 2023-11-05 08:47 What command did you use to get that log? 👀 ---------------------------------------------------------------------- VsevolodLNM · 2023-11-05 09:01 Sorry English isn’t my native language, in my language we assume every gender😈 ---------------------------------------------------------------------- LeeHide · 2023-11-05 09:26 probably journalctl -u sshd -e ---------------------------------------------------------------------- HurricanKai · 2023-11-05 09:31 Using IP ranges isn't the right way to do this. You should use mTLS and enforce it on your reverse proxy. Oh cloudflare the feature is called Authenticated Origin Pulls. Personally I even prefer this, as it doesn't require fiddling with Zero Trust or cloudflared and is extremely straightforward. ---------------------------------------------------------------------- Radsdteve · 2023-11-05 10:01 `cat /var/log/auth.log` ---------------------------------------------------------------------- HugoDos · 2023-11-05 10:19 I understand, we have the https://docs.crowdsec.net and examples via https://github.com/crowdsecurity/example-docker-compose/tree/main/npm However, I sense it's more of a step by step guide you were looking for? If so, we found these recently uploaded videos that may help https://youtu.be/qnviPAMwAuw?si=zgnKgS3Yl9bV1iCr https://youtu.be/8bQh88z3FuY?si=d_RfhhGHn4QN8N_v ---------------------------------------------------------------------- jippen · 2023-11-05 10:47 But don't use 2222. I still get mountains of bot noise there. ---------------------------------------------------------------------- d03j · 2023-11-05 11:39 You can still use crowdsec and have a firewall bouncer on the machine you're hosting shlink. I have a fairly basic router with a single port open, all my services are behind a reverse proxy and ufw implements the decisions from crowdsec's security engine via the firewall bouncer. ---------------------------------------------------------------------- Foo-Bar-Baz-001 · 2023-11-05 11:58 DirectAdmin sits there ---------------------------------------------------------------------- NateY3K · 2023-11-05 12:13 if it's in a docker container, is there anything within the docker container that is sensitive? because if not, then they can try to grab everything they want but they're not going to get anything either way, you could look into having an allow-list of urls in your nginx.conf, making it such that anything outside of that list gets an automatic 404. actually, are you sure that's not already what's happening? i wouldn't expect for cloudflare to do this automatically, and i wouldn't trust them to do it unless this was a specific feature (potential to look into) ---------------------------------------------------------------------- nik282000 · 2023-11-05 12:27 Yup https://github.com/nik282000/DontUseTheseNames/blob/main/UIDList_C ount.csv ---------------------------------------------------------------------- nik282000 · 2023-11-05 12:29 I've actually done both but once or twice a year a persistent bot will try anyway. I could use fail2ban but that's not as fun. ---------------------------------------------------------------------- nik282000 · 2023-11-05 12:30 Followed by some grepping and python to extract the usernames. ---------------------------------------------------------------------- how_now_brown_cow · 2023-11-05 13:47 Use crowdsec, its a joy compared to fail2ban ---------------------------------------------------------------------- mitch8b · 2023-11-05 14:04 Im interested, what do you like about it over f2b? ---------------------------------------------------------------------- Sparrow538 · 2023-11-05 14:34 Just a hacker bot looking for open WordPress files. ---------------------------------------------------------------------- garbast · 2023-11-05 16:26 As long as you dont expose the port there is no need to change the port. You dont get rid of the bots. They are not attacking the ssh but the web ports 80 and 443. SSH has nothing to do with that. ---------------------------------------------------------------------- Leaderbot_X400 · 2023-11-05 21:25 They've got mountains of exploits, some patched, some found. ---------------------------------------------------------------------- cybrian · 2023-11-05 22:30 That’s a much longer tail end than I expected ---------------------------------------------------------------------- Haliphone · 2023-11-05 23:16 How did you get the logs? I'd love to see if I have anything similar ---------------------------------------------------------------------- RuffKez · 2023-11-06 00:37 And fail2ban ---------------------------------------------------------------------- nik282000 · 2023-11-06 01:04 SSH login messages go to your /var/log/auth.log. So I grep all the line with sshd then do some sorting to grab only the user name from each login attempt. ---------------------------------------------------------------------- nik282000 · 2023-11-06 01:10 Yeah, there are a LOT of single attempts. They were all made in alphabetical order within about an hour (it was like 15-20k attempts total) from IPs all over the world, so it was probably one guy's bot net trying lots of common passwords with common names then specific user/pw combos. Also root is omitted from the top of the list because of the way I grep'd the log :/ oops. ---------------------------------------------------------------------- mausterio · 2023-11-06 15:15 This is the way. Also defense in depth. I've used CF WAF for years as a first line defense against these automated scans/attacks. When you have anything webfacing it will be scanned/attacked. Overwhelmingly, these attacks are fully automated typically scanning for vulnerable systems for further hands on keyboard attack after dropping a shell or payload. While these attacks likely won't lead to anything impactful in a secured environment, it is still noise hitting your underlying systems that can lead to legitimate attacks going unnoticed. By itself, though, CF WAF isn't going to be proactive in blocking these scans/attacks before/as they occur. You can leverage free/self-hosted solutions like Suricata IDS/IPS, Unbound DNS, Cloudflare Zero Trust DNS, and Crowdsec to further minimize your threat landscape. No single security solution will 100% cover all threat angles, so it's better to leverage multiple solutions that complement or work in tandem with each other's shortcomings.