Have I been hacked and am now mining crypto? reddit / selfhosted ---------------------------------------------------------------------- pinewoodranger · 2023-12-21 18:04 So im running a few things on a gen 11 NUC, Debian 12 and just noticed systemd-journald is using high CPU. https://preview.redd.it/5o3wklviso7c1.png?width=954&format=png&auto=we bp&s=ee187ca41066834a99415c0321f68e9dcd440781 This obviously isnt normal. Also why is it running under my own user, shouldnt it be root? Running **journalctl -f** looks ok, not too many messages. So I run **ps aux --sort -rss | head -10** https://preview.redd.it/iqncylksso7c1.png?width=1207&format=png&auto=w ebp&s=51b788dbd4e1ae344a4f205477a1a9da8b5fe06b And I check the supposed config **config\_background.json** and I see lines like https://preview.redd.it/g56ypi2hto7c1.png?width=1063&format=png&auto=w ebp&s=d3906613763253b79aa701969717d87aa4b2427f This seems to be a crypto miner. Not sure how this would have happened. Any hints on what to do next, secure myself? ---------------------------------------------------------------------- neuthral · 2023-12-21 18:12 take their crypto with the password, it lookslike a mining pool config with a masked program name ---------------------------------------------------------------------- jdblaich · 2023-12-21 18:21 How are you exposing your services/servers to the internet? ---------------------------------------------------------------------- pinewoodranger · 2023-12-21 18:26 Only nginx reverse proxy to a few services reachable through HTTP. ---------------------------------------------------------------------- DJOCKERr · 2023-12-21 18:31 remindme! 3 days ---------------------------------------------------------------------- RemindMeBot · 2023-12-21 18:32 I will be messaging you in 3 days on [**2023-12-24 18:31:08 UTC**](htt p://www.wolframalpha.com/input/?i=2023-12- 24%2018:31:08%20UTC%20To%20Local%20Time) to remind you of [**this link **](https://www.reddit.com/r/selfhosted/comments/18nt4af/have_i_been_h acked_and_am_now_mining_crypto/kectzx6/?context=3) [**CLICK THIS LINK**](https://www.reddit.com/message/compose/?to=Remin dMeBot&subject=Reminder&message=%5Bhttps%3A%2F%2Fwww.reddit.com%2Fr%2F selfhosted%2Fcomments%2F18nt4af%2Fhave_i_been_hacked_and_am_now_mining _crypto%2Fkectzx6%2F%5D%0A%0ARemindMe%21%202023-12- 24%2018%3A31%3A08%20UTC) to send a PM to also be reminded and to reduce spam. ^(Parent commenter can ) [^(delete this message to hide from others.)] (https://www.reddit.com/message/compose/?to=RemindMeBot&subject=Delete %20Comment&message=Delete%21%2018nt4af) ***** |[^(Info)](https://www.reddit.com/r/RemindMeBot/comments/e1bko7/remind mebot_info_v21/)|[^(Custom)](https://www.reddit.com/message/compose/?t o=RemindMeBot&subject=Reminder&message=%5BLink%20or%20message%20inside %20square%20brackets%5D%0A%0ARemindMe%21%20Time%20period%20here)|[^(Yo ur Reminders)](https://www.reddit.com/message/compose/?to=RemindMeBot& subject=List%20Of%20Reminders&message=MyReminders%21)|[^(Feedback)](ht tps://www.reddit.com/message/compose/?to=Watchful1&subject=RemindMeBot %20Feedback)| |-|-|-|-| ---------------------------------------------------------------------- nathan12581 · 2023-12-21 18:32 How many ports have you exposed on your network? Surely just 443? ---------------------------------------------------------------------- dopey_se · 2023-12-21 18:32 You may have more open than you realize. I've seen folks not realize ssh was reachable over the same fqdn they thought they only had their services exposed on. It certainly looks like you are compromised. I'd start confirming it is only http(s) open and if that is true, something you are running likely has a known exploit. ---------------------------------------------------------------------- pinewoodranger · 2023-12-21 18:36 Yeah. I may have ran a script without looking too closely at what it was doing. Im going to retrace my steps and see if anything turns up. ---------------------------------------------------------------------- nathan12581 · 2023-12-21 18:36 I’d shut down my internet at this point and nuke everything ---------------------------------------------------------------------- nathan12581 · 2023-12-21 18:38 I assume you use docker or virtualise? Maybe turn your entire server off and directly plug that into a computer so it doesn’t have direct access to the internet whilst you have a look ---------------------------------------------------------------------- autogyrophilia · 2023-12-21 18:42 Hot tip. Any executable can have any name, and the systemd-journald executable does not reside there. And it also does not run under your user. At the very least your user denis has been compromised. Most likely by exposing ssh to the internet with a weak password. ---------------------------------------------------------------------- BlockDigest · 2023-12-21 18:55 Yes looks like you have been owned, that’s definitely a crypto miner config file and that’s not actually systemd running as your user. I would disconnect this machine from the internet first and isolate it from the rest of your network. Take a backup of your data and then kill the miner (just in case it triggers encryption of your disk, who knows). Then start reviewing your firewall config in case they are overly permissive. Also, review any application config you have, especially any reverse proxy (they are easy to misconfigure). Lastly (and most likely the culprit) check the version of the software you are exposing over the internet, any outdated software most likely has unpatched vulnerabilities that can be exploited. After you have reviewed all these, build your system again with all the fixes on a brand new OS. Do not redeploy the same system from backup. Also consider using a VPN to access your self hosted services vs exposing them over the internet. It only takes one vulnerable service behind the reverse proxy to get owned. Best of luck. ---------------------------------------------------------------------- pinewoodranger · 2023-12-21 19:03 I'm gonna nuke everything and start anew. But it looks like everything that was in the .local dir was placed there under one specific user group so I can narrow it down to which service might have screwed me here. ---------------------------------------------------------------------- autogyrophilia · 2023-12-21 19:04 Obviously something with rights covering the user densi ---------------------------------------------------------------------- SolidSentence4878 · 2023-12-21 19:05 Why did they downvote this post? Hack the hacker if possible ---------------------------------------------------------------------- Elle221LL · 2023-12-21 19:05 I would recommend formatting the drive, you never know what backdoors etc was installed. ---------------------------------------------------------------------- chmikes · 2023-12-21 19:05 Locate the binary and call the command string on it. It will print all strings it finds in the binary. It may give a hint on what it does. Another track to follow is looking up the expected size and md5 hash of the binary if it exists for real and compare it with the binary you have. Another track is looking at its open file descriptors (lsfd if I remember well). It also give a lot of information like the open files, ip addresses, etc. ---------------------------------------------------------------------- VeryLongHairGuy · 2023-12-21 19:15 >kill the miner I need his identity and his country of residence first. /s ---------------------------------------------------------------------- HosonZes · 2023-12-21 19:15 `nmap -p- ` to see open ports Or `ss -ltpn` from inside this machine ---------------------------------------------------------------------- PuckSR · 2023-12-21 19:20 Im curious about how that happened ---------------------------------------------------------------------- KervyN · 2023-12-21 19:24 Also get a new mainboard, because you cannot trust your HW anymore. (Uefi malware or ring0 malware as search term). ---------------------------------------------------------------------- speculatrix · 2023-12-21 19:25 You might have accidentally installed a typo-squatting package containing malware. https://www.activestate.com/resources/quick-reads/how-open-source- typosquatting-attacks-work/ ---------------------------------------------------------------------- nukacola2022 · 2023-12-21 19:25 Everyone’s focusing on open ports (which isn’t a bad thing to review), but reverse proxies do not necessarily protect vulnerable services. They are not a WAF. Nuke your environment, run the latest code of each of your services when rebuilding, and consider hiding them behind an overlay VPN/mesh network instead of exposing directly over the internet if you can help it. ---------------------------------------------------------------------- speculatrix · 2023-12-21 19:27 Yes, you should be sure to erase the EFI partition so there's no possibility of bootloader malware ---------------------------------------------------------------------- islandsimian · 2023-12-21 19:29 I discovered my QNAP NAS was getting brute-forced because I had UPnP enabled (I'm assuming by default) which bypassed my (at the time) crappy firewall because the ports were higher than what it was paying attention to. Luckily I had turned on 2FA and set passwords. If they had gained access to the QNAP it would have been game over TL;DR: check UPnP settings on your equipment ---------------------------------------------------------------------- Whyd0Iboth3r · 2023-12-21 19:32 Try this. I just used it on my url, and it found the ports I expected to be open. https://hidemy.io/en/port-scanner/ ---------------------------------------------------------------------- adamshand · 2023-12-21 19:45 I'd say it's more likely from something you ran, which is why it's running as your user account, than a hacked service. The good news, if that's true, is that they probably don't have root. ---------------------------------------------------------------------- KervyN · 2023-12-21 19:49 I wouldn't trust anything anymore on said system. Not even the mainboard. If you ever get infected, owned, whatever: the system is going into the trash and I will start over with new hardware. My backup solution knows which files changed, so I can trust the files from the offsite backup, but I will also need to redo this, because it's a push backup, not a pull backup. This is not a joke and you need to cut it out, very hard or you will suffer forever. ---------------------------------------------------------------------- foomatic999 · 2023-12-21 19:52 To narrow down possible suspects you can reach into the forensics toolbox. Reboot from a kali iso and run fls and mactime on the file system. Fls creates a list of everything on the fs and writes it into a file. Mactime reads that file and orders it by timestamp. It also considers all timestamps that are associated with a file. Check what was created around the time the malicious binary was created. This could add some context. Fls and mactime are both part of the sleuthkit forensics suite. ---------------------------------------------------------------------- foomatic999 · 2023-12-21 19:55 Even if it were viable for persistence, as far as we know the attackers gained only normal user privileges. A firmware attack needs root perms. ---------------------------------------------------------------------- atrizzle · 2023-12-21 19:58 There's no password (private key) here, so can't take anything. ---------------------------------------------------------------------- AnderssonPeter · 2023-12-21 20:02 Not that I have been hacked before that I know of, but what are the odds that it survives a full wipe? With that logic you would need to trash everything on the same network, if they have the resources to spread to the hardware surely they can jump to different devices? ---------------------------------------------------------------------- foomatic999 · 2023-12-21 20:04 While you're at it - you can plug the hash of a file into VirusTotal and it will tell you to if it knows the file already. This may be an indicator for how targeted the attack was (probably not very), but also what is already known about the thing. If it's a miner, my guess would be that you have xmrig running. That's popular with attackers that run crypto miners on compromised systems. ---------------------------------------------------------------------- -quakeguy- · 2023-12-21 20:04 You’ve been owned. Disconnect the system from the network (and not just internet). Personally, I would use this as a little exercise in trying to figure out precisely how this happened before completely wiping the system and rebuilding anew. ---------------------------------------------------------------------- Smeeks1126 · 2023-12-21 20:06 I wonder what we can do with his wallet address... ---------------------------------------------------------------------- ChronosDeep · 2023-12-21 20:06 Do you run qbittorrent on it? ---------------------------------------------------------------------- mimikater · 2023-12-21 20:10 Top answer here! ---------------------------------------------------------------------- foomatic999 · 2023-12-21 20:12 Spotted the pentester 😆 ---------------------------------------------------------------------- mimikater · 2023-12-21 20:13 Respekt. Harter Job aktuell! ---------------------------------------------------------------------- foomatic999 · 2023-12-21 20:17 Why's that? Everyone secures their shit properly and domadmins are nowhere to be found? 😉 ---------------------------------------------------------------------- jman6495 · 2023-12-21 20:17 You've been hacked. What OS are you running ? Did you install any packages from unreliable sources or run any random scripts? If not, it's probably SSH. You shouldn't be able to log in with a password on SSH. **You should only ever use keys**. Given the situation, I think your best bet is to nuke everything and restart. You don't know how many backdoors the guy could have set up. ---------------------------------------------------------------------- bityard · 2023-12-21 20:21 > If you ever get infected, owned, whatever: the system is going into the trash and I will start over with new hardware On today's episode of redditors going overboard... ---------------------------------------------------------------------- Necessary-Office3082 · 2023-12-21 20:22 OP please please share those files if you don't plan looking into. My DMs are open 👀 ---------------------------------------------------------------------- Not_your_guy_buddy42 · 2023-12-21 20:31 I kind of really want to know which apps you run and how... ---------------------------------------------------------------------- blackstar2043 · 2023-12-21 20:40 For this type of forensics, I normally run AIDE on all of my machines. And by comparing its snapshots, I can identify when the modification took place and cross-reference it with other logs for further investigation. ---------------------------------------------------------------------- crazedizzled · 2023-12-21 20:42 This is nonsense lol. Don't listen to this guy ---------------------------------------------------------------------- Scoth42 · 2023-12-21 20:47 The idea is that if someone gains full access t the system, they can compromise/install malware into the UEFI firmware that will enable backdoors or other vulnerabilities and allow them to survive even complete reinstalls of the OS itself. Personally I think that's a little bit overboard as it'd require a lot more work and access than a normal user, but it's a possibility. Most likely this was a hit-and- run where someone used scripts or other automated exploits to install miners and hoped they'd last awhile rather than trying to exploit things to that level, but you never know. And yes, once someone is in your network, it's at least feasible they could use other vulnerabilities or lax security since it's "inside the network" to compromise other devices/systems. It's why I keep my internal stuff almost as locked down as my external stuff for most things. ---------------------------------------------------------------------- Scoth42 · 2023-12-21 20:49 See what's in it? Send more to it? Not a lot with just the wallet address. I'm not sure if there's a registry of wallet addresses used in these sort of attacks or not, but maybe that. ---------------------------------------------------------------------- thatguychuck15 · 2023-12-21 21:04 Pretty sure I got hit by this same thing with a debian 12 server I was just testing and messing around with. Did you install qbittorrent-nox or run qbittorrent with the web ui? Until recently the default unpnp setting was “on”. So if you ran qbittorrent-nox and opened the web ui to your lan, you were also exposed directly to the internet. In my case I hadn’t changed the default credentials because I incorrectly assumed it was only open to my lan and some bot added a random torrent and used the “run program on torrent completion” option in qbittorrent-nox to install the crytpminer. Anyway upstream had somewhat recently changed this setting to off, and I added a bug report to debian that followed suit, so it shouldn’t be happening on new installs. If you aren’t using qbittorrent-nox you might want to see if any other services use upnp. But regardless I would wipe and re install. ---------------------------------------------------------------------- TheQuantumPhysicist · 2023-12-21 21:06 From reading the comments, I think everyone is interested in learning more about your system to guess what you f\*\*\*ed up (no offense, we all do one way or another). You obviously did something wrong. Can you share in detail what your system is and how it works? ---------------------------------------------------------------------- _cdk · 2023-12-21 21:08 for sure the guy only renaming his miner as systemd to hide it is installing a boot kit ---------------------------------------------------------------------- TheQuantumPhysicist · 2023-12-21 21:11 Thank you for the information. I'm trying to understand what the dude did wrong, and what you said is interesting. So to understand what you said, you ran a torrent client in your local network and assumed you don't need to secure anything because it's local, and it had upnp enabled by default, which somehow advertised your device to the whole internet, and a bot caught up on that and connected to it, and installed the malware. This upnp thing is getting out of hand! ---------------------------------------------------------------------- znpy · 2023-12-21 21:17 I'd reinstall the whole machine, frankly. ---------------------------------------------------------------------- rokejulianlockhart · 2023-12-21 21:20 Wouldn't that require root access? ---------------------------------------------------------------------- rokejulianlockhart · 2023-12-21 21:21 RAZE THE CITY BLOCK WITH PHOSHPHORUS ---------------------------------------------------------------------- rokejulianlockhart · 2023-12-21 21:22 RAZE THE CITY BLOCK WITH PHOSPHORUS ---------------------------------------------------------------------- thatguychuck15 · 2023-12-21 21:22 That’s pretty much it. This can probably explain it a little better than I can. https://torrentfreak.com/qbittorrent-web-ui-exploited-to- mine-cryptocurrency-heres-how-to-fix-230902/ ---------------------------------------------------------------------- iRedditFromBehind · 2023-12-21 21:28 Has qbittorrent been compromised? ---------------------------------------------------------------------- Theon · 2023-12-21 21:30 Holy hell, thank you both. I've been fortunate to only have had one breach (knock on wood), but all I could do was just nuke the machine, restore the minimum from backups and tighten some "loose spots" I found - I had no idea where to start actually investigating what happened. This is what I've since wanted to know <3 ---------------------------------------------------------------------- WhisperingSh4dows · 2023-12-21 21:31 You have docker exposed? Old versions are easily hackable ---------------------------------------------------------------------- pinewoodranger · 2023-12-21 21:37 Yes, this is exactly what I am running. qbittorrent-nox. ---------------------------------------------------------------------- Corpdecker · 2023-12-21 21:39 Can probably download it from [https://c3pool.com/](https://c3pool.com/) directly (it prompts you do when you visit). Can choose English in top right. The address they list is not that far from me, same as a business called LEDi2 in a shady part of Houston. It looks like, and I may be misunderstanding their model, that you can run miners for them and get paid, so it's likely someone is hacking boxes, uploading miners, and profiting. Were I OP, I'd def contact the c3pool people and tell them the user hash seen in the screenshot, maaaaaaybe it would get them banned. That assumes the c3pool people are legit. Hell it might already be banned but the payload is carried by a worm or something that's running around on its own. ---------------------------------------------------------------------- yasiesolovemos · 2023-12-21 21:46 If you have a miner on your computer, you have its data. Redirect the attacker's account to your own. It's your machine, you have root, you have access to its account. Clone that drive, mount it to another machine and, via VPN, log in to his wallet. ---------------------------------------------------------------------- pinewoodranger · 2023-12-21 21:52 A bit more details: running a media server (plex and jellyfin) with an arr stack, torrent client and nextcloud. No docker. Jellyfin, nextcloud and a second homeassistant machine is routed through nginx which exposes 443. System is debian, everything from debian bookworm repository except jellyfin and plex (which are from their own respective repos). certbot installed through snap. ​ I was messing around a few weeks ago and may have ran something before checking. Im not sure what. ---------------------------------------------------------------------- thcduck · 2023-12-21 21:54 Nginx Proxy Manager or cli? I've heard somewhere that NPM is not that secure, don't have the source tho, my paranoid friend told me this. ---------------------------------------------------------------------- thatguychuck15 · 2023-12-21 21:55 A better rundown than mine can be found here: https://torrentfreak.com/qbittorrent-web-ui-exploited-to-mine- cryptocurrency-heres-how-to-fix-230902/ ---------------------------------------------------------------------- pseudopad · 2023-12-21 21:57 >I would disconnect this machine from the internet first and isolate it from the rest of your network. Take a backup of your data and then kill the miner (just in case it triggers encryption of your disk, who knows). Shutting the system down and rebooting into a live USB distro should prevent the miner from doing anything at all while you're cleaning up. ---------------------------------------------------------------------- pseudopad · 2023-12-21 21:58 It should be sufficient to reflash the bios/uefi and clear the efi partition, shouldn't it? ---------------------------------------------------------------------- BCMM · 2023-12-21 22:01 [Somebody on /r/debian a month ago also found xmrig, mining for c3pool, pretending to be part of systemd](https://www.reddit.com/r/deb ian/comments/17g36ym/journalctld_using_2gb_of_memory/). They also couldn't explain how it got on the system. It had the (nonsensical) name "journalctld" rather than "systemd- journald", though. ---------------------------------------------------------------------- gibberish420 · 2023-12-21 22:03 Docker as in a container? ---------------------------------------------------------------------- donald_trub · 2023-12-21 22:08 > If not, it's probably SSH. You shouldn't be able to log in with a password on SSH. You should only ever use keys. I think this is a pretty large leap to say it's probably SSH if passwords were turned on. With a complex enough password it would still take a very long time to hack in via SSH. The most likely scenario is an exploited web service. ---------------------------------------------------------------------- Lord_emotabb · 2023-12-21 22:08 [https://www.reddit.com/r/utorrent/comments/11c7ipi/is\_utorrent\_a\_b itcoin\_miner\_help/](https://www.reddit.com/r/utorrent/comments/11c7i pi/is_utorrent_a_bitcoin_miner_help/) it has an option on by default to let it use your computer as a miner, you can deactivate it... most people just install things via next>next>complete ---------------------------------------------------------------------- LightShadow · 2023-12-21 22:08 The password looks to have been blacked on in the image, because as-is that's invalid JSON. ---------------------------------------------------------------------- Scoth42 · 2023-12-21 22:11 Potentially, but if we assume it's been compromised then you can't trust any built-in flash utilities, and even something external like a boot USB with an external flashing program could be made to fail (or show success while malwaare stays buried). About the only way to be entirely sure would be to use some kind of Flash/EEPROM reader/burner to wipe the full BIOS flash chip and rewrite a known good bios image. \*That\* would be be pretty much impossible for malware to survive/bypass, but beyond your average user's ability and maybe not practical. Like I said it's vanishingly unlikely something like that would be done to a random compromised computer on the 'net (that kind of stuff is probably saved for state actors infiltrating hardened targets) but not impossible. UEFI rootkits/bootkits have been seen in the wild. ---------------------------------------------------------------------- WhisperingSh4dows · 2023-12-21 22:14 The "password " here is just a way to ID a worker. Not an actual password ---------------------------------------------------------------------- sandmik · 2023-12-21 22:16 I like to use netdata [https://learn.netdata.cloud/docs/installing/docker](https://learn.net data.cloud/docs/installing/docker) It allows me to see quickly how my node and docker containers are doing and it raises alerts for unusual metrics like high CPU. This is all using the free edition. ---------------------------------------------------------------------- WhisperingSh4dows · 2023-12-21 22:17 Docker as in what runs the containers. Dockerd "TCP port 2375 is the default port used by Docker for remote management of containers" ---------------------------------------------------------------------- BriochesBreaker · 2023-12-21 22:20 What's wrong with qbittorrent? ---------------------------------------------------------------------- Corpdecker · 2023-12-21 22:22 This XMrig stuff appears to be all over the place and abused for a long time. I'm going to guess you ran a script or something from github that had this embedded. That said, a search for part of the user ID hash on github turns up no results, so it might be from somewhere else or was scrubbed since. This is one reason I reeeeally don't like those projects that just import dependencies directly from github or other repos, as all it takes is one of those to be compromised before you are too, as opposed to distro vendor packages which are more thoroughly vetted. As others have said, since it's running as your system user, it's far more likely you ran something rather than hacked from the outside, rooted, then it chose your user as the one to do the heavy lifting. If it was from the outside, it was likely targeting something you were running as the user. https://www.trendmicro.com/en\_gb/research/21/l/vulnerabilities- exploited-for-monero-mining-malware-delivered-via-gitHub-netlify.html ---------------------------------------------------------------------- BeiSho · 2023-12-21 22:29 That is different software entirely ---------------------------------------------------------------------- KervyN · 2023-12-21 22:29 correct. "As far as we know". I've seen altered `ps`, `top`, `htop`, `ls` programs to hide stuff. And please, don't listen to me. It is your system and OP will surly be fine. I am just very paranoid nerd when it comes to this, and my private data (like self hosted password manager and stuff like this) is just to valuable for me, to not fuck around :) ---------------------------------------------------------------------- ChronosDeep · 2023-12-21 22:32 It had UPnP ON by default for the webui on macOS, someone connected to my webui, in settings there is a option to run scripts, he set a script from the web to run when a torrent is added, then he added a simple torrent, a python book. That script downloaded the mining software, also tried to alter my authorized_keys by adding his key…. ---------------------------------------------------------------------- KervyN · 2023-12-21 22:36 Is it? google "uefi malware" or "ring0 malware". I have seen very altered systems, so you do not even see a file in the `ls -al` output, because `/bin/ls` was modified. I tend to be paranoid as F when it comes to this. And the fact that there is a random thing running that is mining stuff means to me, that the system got compromised by an automated attack. And when you are vulnerable to people who hide their stuff so badly, then you are surely vilnerable for people who will use your system in the future. Watch some talks from security professional regarding these topics. In the end, this is how I would treat it, and everybody is doing their own due diligence when this happens. I just throw my HW away and buy new HW. I just buy cheap HW, so I don't care as much. ---------------------------------------------------------------------- DeadMansMuse · 2023-12-21 22:37 I don't have a static IP, and I also don't have any forward facing services. But damn if this post hasn't made me want to check all my entry points just in case! ---------------------------------------------------------------------- KervyN · 2023-12-21 22:38 And this leads me to the point, that this will likely happened because an automated attack went through. On a linux system... So if it this open, you might have a lot more bad things running, you will never find without really good forensic work. I've seen systems where `ls -al` did not show files, because the `/bin/ls` was modified. ---------------------------------------------------------------------- KervyN · 2023-12-21 22:41 please, see my other comments here. You might understand me a little better afterwards. Yep, that is totally overkill. I wouldn't trust this system (mainboard + OS) ever again. ---------------------------------------------------------------------- atrizzle · 2023-12-21 22:42 Oh hmm good point ---------------------------------------------------------------------- ChronosDeep · 2023-12-21 22:47 The problem is UPnP enabled by default for the webui, maybe not all releases have it enable by default, or the user does not know what this do, it’s a dangerous setting together with the ability to run scripts set in settings. ---------------------------------------------------------------------- ambiance6462 · 2023-12-21 23:50 >consider hiding them behind an overlay VPN/mesh network instead of exposing directly over the internet is there a way to do this where, like, my mom could still access services by visiting a regular domain without having to configure a VPN or other software? ---------------------------------------------------------------------- RedditSlayer2020 · 2023-12-21 23:57 please elaborate how to login to a wallet ---------------------------------------------------------------------- nukacola2022 · 2023-12-22 00:02 Yes, but you would need to run a “connection” node within her house (could be the crappiest Pi or Celeron you can find) and then each device at her home would split tunnel through it to get to your home lab services. I use TwinGate for these setups. A bit more complicated but the security benefits (imo) are worth it. ---------------------------------------------------------------------- SR_Lut3t1um · 2023-12-22 00:08 You also never know what exploit chaning they are using ---------------------------------------------------------------------- pinewoodranger · 2023-12-22 00:16 I'm 99% sure this was the culprit. UPnP for WebUI on qbittorrent. Router didnt show opened port 8080, just showed the random ports opened for peer discovery. I checked and webui was accessible from my public ip even though it shouldnt have been. ---------------------------------------------------------------------- last__link · 2023-12-22 00:20 I use Cloudflare tunnels with whitelisted email code verification. Protects my external facing apps ---------------------------------------------------------------------- omniterm · 2023-12-22 00:35 Onetime I got a crypto miner installed on my laptop running fedora. About the only thing I found usable was the ftp credentials left in the script. On the ftp side nothing interesting so i deleted everything. I also made the mistake of running the install script as root, took more work to remove everything and still no clue how I got infected in the first place but it was a learning experience. And fun getting to wipe the ftp site that was hosted on 000webhost a free hosting site. ---------------------------------------------------------------------- Budget-Supermarket70 · 2023-12-22 01:06 How does cloudflare prevent your external apps? I know it hides your IP address but if there is a vulnerability in the app how does going through cloudflare protect it? I mean the app is still exposed to the internet just through a different route. Or am I missing something? ---------------------------------------------------------------------- ecitnE · 2023-12-22 01:10 UPnP auto-{opens, closes} ports when needed IIRC (so you probably couldn't see the forwarded port anymore after taking everything down). ---------------------------------------------------------------------- ChronosDeep · 2023-12-22 01:15 Yeah, same thing happend to me last month, on macOS. Seems like only some releases have this setting enabled by default ---------------------------------------------------------------------- Gumbini · 2023-12-22 01:15 I would recommend you disable UPnP in your firewall (most likely your router) settings too. Take full control over the ports you're exposing to the Internet. ---------------------------------------------------------------------- crazedizzled · 2023-12-22 01:17 Instead of throwing it away, just send it to me next time. I'll put it to use instead of wasting it. ---------------------------------------------------------------------- Conroman16 · 2023-12-22 01:37 When using tunnels and sso auth, one can’t get to the application unless they have already bypassed the SSO portal. Thus, if you don’t control one of the emails on the whitelist, it doesn’t matter how vulnerable the application is because Cloudflare won’t allow you to get to it. ---------------------------------------------------------------------- FierceDeity_ · 2023-12-22 01:39 lol also journal, not journald... Nobody caught that detail. ---------------------------------------------------------------------- yasiesolovemos · 2023-12-22 01:52 Well reddit is working its magic and they are downvoting me. The access data to the hacker's wallet will be encrypted, so you will not be able to access them, but you can always do a little research and start with a "man in the middle". From there, you have access to "their IP", which will be behind 1000 firewalls/vpns, of which you can always keep track (difficult or almost impossible). Or, more realistically, you can poison the traffic so that the data it sends is worthless or even corrupts the bulk of other infected computers. But hey, they're just ideas. Here we have come to play. You can always format and throw the computer in the trash, as others have said. ---------------------------------------------------------------------- AviationAtom · 2023-12-22 01:54 Or you could use a router with a firewall that supports proper access lists, restricting inbound access to that port to your mother's IP address ---------------------------------------------------------------------- AviationAtom · 2023-12-22 01:57 For those not in the know: AIDE is roughly close to Tripwire. It's a HIDS that watches key places on your filesystem for changes. ---------------------------------------------------------------------- AviationAtom · 2023-12-22 02:03 That's a pretty interesting attack chain. Thanks for sharing the details with us all! ---------------------------------------------------------------------- RedditSlayer2020 · 2023-12-22 02:06 karma is meaningless why should you care ---------------------------------------------------------------------- yasiesolovemos · 2023-12-22 02:19 Oh no, please... internet points, I'm just saying that since you have some data from the hacker, take advantage of it instead of throwing the entire computer in the trash. Mount the disk in another machine and play. ---------------------------------------------------------------------- natermer · 2023-12-22 02:29 Nuke it and start over. It isn't worth it. The time and effort to "secure" the system is much less then it will take to just re-install everything. And even then it doesn't mean you caught everything. This is common on Windows. They have a root kit to control it remotely and then that root kit remotely installs software as a payload to do mining or spyware or adware or whatever. Malware tools will get the software payload, but miss the rootkit and then the system is "reinfected" in a couple days. Same thing can happen to Linux. I would shut the system down, boot up using USB media, and copy off text configs or important personal files or whatever you think is important. Then start over with a fresh install and only copy back things you can validate. Anything else is just half-measures. A lot of effort for no guarantee. ---------------------------------------------------------------------- Trick_Algae5810 · 2023-12-22 04:19 No clue, but I cant think of anything else that would piss me off as much as someone hacking my system to fucking mine bitcoin. ---------------------------------------------------------------------- SavingsMany4486 · 2023-12-22 04:21 If you are willing to do some configurations for your mom, you could install a client certificate in her web browser(s) so she can authenticate with her certificate to any of your web services. You'd have to add each certificate (or if the certs are signed by a CA, just the CA cert) to a cert store in your web servers and enable mTLS ---------------------------------------------------------------------- ambiance6462 · 2023-12-22 04:26 man. you guys have some dorky moms. ---------------------------------------------------------------------- RedditSlayer2020 · 2023-12-22 04:29 OSINT is kinda fun ---------------------------------------------------------------------- SavingsMany4486 · 2023-12-22 04:33 Thanks ---------------------------------------------------------------------- Not_A_Red_Stapler · 2023-12-22 04:39 Have you tried this yourself? While theoretically it should work, every time I've looked into it, it doesn't work in reality. It might be better to set up a yubikey or something if you can make that work with your services. ---------------------------------------------------------------------- Ashanrath · 2023-12-22 05:23 Whoa, is that really the default? Surely not, that's a massive risk for people who don't know what it means! Disabled in my instances, and I don't remember ever needing to disable it, but it's been a few years. ---------------------------------------------------------------------- Ashanrath · 2023-12-22 05:27 What services does your mum need to access? I've got a handful of services exposed on the public net for family members (Plex, Jellyfin etc) but everything else is only accessible via VPN. I don't want anyone else but me screwing around with anything else. ---------------------------------------------------------------------- alldots · 2023-12-22 05:27 This is exactly why I've never let upnp be enabled on any of my routers. Allowing any software on any of my internal devices to open up holes in my firewall? Hard pass. ---------------------------------------------------------------------- Ashanrath · 2023-12-22 05:29 Do you need external access to all of that? I've got my Plex and Jellyfin externally accessible (via reverse proxy), everything else behind a VPN. ---------------------------------------------------------------------- DaoFerret · 2023-12-22 05:32 One more reason to just vpn into your home network if you need to have remote access. ---------------------------------------------------------------------- DaoFerret · 2023-12-22 05:36 Great for corporate, but how likely is a home user to have a static IP? ---------------------------------------------------------------------- DaoFerret · 2023-12-22 05:38 One more reason people should turn off UPnP. If you need a hole poked in your firewall/NAT, better you should do it manually and know what got done. ---------------------------------------------------------------------- SavingsMany4486 · 2023-12-22 05:50 Yes, I am running this in my homelab. What do you mean by "it doesn't work in reality"? ---------------------------------------------------------------------- RedWyvv · 2023-12-22 06:03 \+1 Last year, my box was hacked because of an improperly configured Docker installation. Had the same crypto miner running on it. ---------------------------------------------------------------------- vrinek · 2023-12-22 06:15 (Assuming you have not nuked the machine yet) Have you tried running `lsof` to grab a list of files that process is running? That might help you identify what it’s actually doing. ---------------------------------------------------------------------- HansFromHell · 2023-12-22 06:16 Try to have a look at the traffic! ---------------------------------------------------------------------- Not_A_Red_Stapler · 2023-12-22 07:00 This will explain it better than I can: [https://security.stackexchange.com/a/1488](https://security.stackexch ange.com/a/1488) It is an old answer though, so maybe things have changed. ---------------------------------------------------------------------- drshajul · 2023-12-22 07:02 How to set this up.. I am running cloudflare tunnels ---------------------------------------------------------------------- Osni01 · 2023-12-22 07:19 Glad to know you found the issue. However, I've read half of the replies in this post and I haven't seen anyone mentioning the obvious here... You got exposed because uPnP opened ports that you didn't want to expose. Rather than relying on whatever software you install to not force uPnP by default, you should disable uPnP altogether in router. Your OP shows you definitely have the knowledge required to set up port forwarding on your own, so disable that crap ASAP because it's nothing but trouble! If you went through ---------------------------------------------------------------------- speculatrix · 2023-12-22 08:30 Turn it off, then turn of pressing the keyboard buttons to go into the boot menu so it doesn't load any efi payloads. then boot off a known good Linux image on a usb drive, and use that to reformat the efi partition and reinstall the os. ---------------------------------------------------------------------- Reasonable-Bear5084 · 2023-12-22 08:44 time to format your system, hacker's already inside ---------------------------------------------------------------------- ftrmyo · 2023-12-22 08:57 Well done, and well learned. Never upnp bro ---------------------------------------------------------------------- ApprehensiveView2003 · 2023-12-22 09:19 Wonder if a firewall with IDS/IPS would have picked this up... ---------------------------------------------------------------------- AviationAtom · 2023-12-22 11:19 It stays reasonably static and if you're nerdy enough it isn't hard to roll out a solution to track any IP changes and automatically update the ACL rule ---------------------------------------------------------------------- neocwbbr_ · 2023-12-22 11:46 Mine[.]c3pool[.]com is exactly what u are looking for. You might have installed a malicious software or got hacked, indeed. Be happy this wasnt a ransomware. ---------------------------------------------------------------------- ChronosDeep · 2023-12-22 11:47 I don’t know, somehow it got enabled for me and many others. There should not be such an option at all. Also you are not forced to change the default user/password which is also a security risk. Plus you even have the ability to run bash/batch commands set in settings. Those 3 things together + uninformed users = big disaster. ---------------------------------------------------------------------- blackstar2043 · 2023-12-22 11:48 My advice for future fresh installs is to deploy PCI-DSS hardening, such as [Debian-CIS](https://github.com/ovh/debian-cis), to prevent further exploitation after the initial intrusion. In your case, [preventing executables from running in /home](https://madaidans- insecurities.github.io/guides/linux-hardening.html#partitioning) would have stopped the cryptominer from starting. An Ansible solution to hardening: https://github.com/konstruktoid/ansible-role-hardening Further down the rabbit-hole is systemd hardening: https://docs.arbitrary.ch/security/systemd.html ---------------------------------------------------------------------- neocwbbr_ · 2023-12-22 11:49 If you do nuke and start it all over, make sure to harden your system security after install. CISecurity Benchmarks are a good start. ---------------------------------------------------------------------- bombero_kmn · 2023-12-22 12:55 Regardless of what an application does by default, you should have UPNP disabled at the router. Never trust a dev to do the right thing ;) ---------------------------------------------------------------------- pinewoodranger · 2023-12-22 13:03 Yup. Looking at the upsides of upnp and there are none if you know what you are doing. Getting rid of it. ---------------------------------------------------------------------- rokejulianlockhart · 2023-12-22 13:46 I think you should tell the OP, not me. ---------------------------------------------------------------------- PovilasID · 2023-12-22 14:25 I have a high sustained utilization alarm on machines. Can be a little annoying but if you leave a fair bit of headroom should be fine. ---------------------------------------------------------------------- Murky-Sector · 2023-12-22 15:09 >EDIT: It seems port 8080 was exposed to the internet even though I didnt see it listed on my routers admin page, uPnP strikes again. 8080 is qbittorrent WebUI and foolishly I used default authentication. This is highly likely the culprit. Thanks for this followup its very useful to know. ---------------------------------------------------------------------- SavingsMany4486 · 2023-12-22 18:27 That user just doesn't seem to know how certificates work and is throwing ignorant concerns as an answer. Client certificate authentication hasn't changed much over the past several years. It is the most secure method of web authentication. It is well proven in many real-world, as I'm sure you're aware pretty much every first- world country uses client certificates in government and military networks, and many banks use it in their own networks. I'll briefly address some of the points in that comment: Point #1 is a bit moot since everyone is using 2FA now. If you can distribute 2 factor keys, you can distribute certs. Point #2 points to a broken page from 2005 that you can no longer access. IDK what it's supposed to be. Point #3, client certs are portable--as you said, a YubiKey is a smart card, so if you can plug it into a computer, you can authenticate with it. Or you can distribute your certificate file if you're ok with 1FA. Point #4 "I don't know if browsers support it." Maybe the person posting the answer shouldn't post one if he doesn't know how the technology works. Every browser that can do TLS can do client certs. Firefox, Chrome, Curl, what have you. If you're using something that doesn't support TLS then I don't see why authentication is a concern for you. I have no idea what point #5 is meant to be. You can use X.509 certs to verify a person's identity. This is how NASA badges, Energy Department IDs and CACs work. You do not have to identify a real person's identity with them, just like how we use them to verify server identity. This doesn't "break" PKI. It depends on your use case and what you want to do. Point #6, you can just choose not to send a client certificate to a website. A site shouldn't ask for client certificates anyway since your client certificate CA wouldn't be something other sites would have. Point #7 is a bit moot if you're using a modern web server made in the past 15 years or so. Point #8, how can a bad guy phish you if he can't access the website he is attempting to mimick? That's the whole beauty of mTLS. Your website cannot be accessed unless the certificate is verified. Separately, your browser will always ask you about sending a cert to a specific server, and only after verifying that server's identity. If you're asked to provide a cert to a site you've never been to before, that's a pretty good phishing indicator. ---------------------------------------------------------------------- Working_Buyer2111 · 2023-12-22 19:07 Install Fail2Ban on any server that is exposed to internet to block unlimited logging attempts. ---------------------------------------------------------------------- Not_A_Red_Stapler · 2023-12-22 21:52 This is good. You should publish it and an updated answer on StackExchange so everyone who goes there can benefit. ---------------------------------------------------------------------- NegativeK · 2023-12-23 00:30 OP should assume that they can't detect compromises and that the systems were insecure in other ways. ---------------------------------------------------------------------- speculatrix · 2023-12-23 00:39 I copied my comment, thanks. ---------------------------------------------------------------------- SavingsMany4486 · 2023-12-23 21:57 Don't think I have enough points. I was thinking of making a blog post but my homelab is so incredibly complicated it would take me weeks to get something like that out. Smallstep has a good post on the topic though: https://smallstep.com/blog/access-your-homelab-anywhere/ ---------------------------------------------------------------------- iamdroppy · 2023-12-24 08:18 dont nuke my crypto bro ---------------------------------------------------------------------- mosaic_hops · 2023-12-24 18:53 It’s a WAF, and on top if this it only passes authenticated traffic through. I.e. nobody can hit it from the open internet unless they have your login. ---------------------------------------------------------------------- UberETH · 2023-12-26 04:00 Not even bitcoin.. it's mining XMR which most server CPU hashrate amounts to about $0.02/day. I found the same exploit on my server downloading a script and installing XMrig under a user account. I looked up the wallet and it had made only 0.40 XMR worth about $70 since November 2022 - not very lucrative of an exploit. I might have mined a nickel's worth of it. Script also tried to get SSH access but was unsuccessful.