My Server Hacked! reddit / selfhosted ---------------------------------------------------------------------- Ashamed-Translator44 路 2023-10-16 08:27 My home server has been hacked and it is mining virtual coins!!! Luckly, the minner is running in a container (immich-machine- learning), i can stop it easily. ## How Did I Find It I found my server is running out of RAM and SWAP, it seems something goes wrong. So i use htop and find this one https://preview.redd.it/ub7bw6rxyiub1.png?width=843&format=png&auto=we bp&s=aa07182f7ea678dad4411f61573d96513d465fc3 than i find this script is running in immich-machine-learning container. when i start the container, it print some logs raise exc requests.exceptions.ConnectTimeout: (MaxRetryError("HTTPSConnectionPool(host='huggingface.co', port=443): Max retries exceeded with url: /api/models/microsoft/resnet-50/revision/main (Caused by ConnectTimeoutError(, 'Connection to huggingface.co timed out. (connect timeout=None)'))")) 0%| | 48/281857 [05:43<781:58:53, 9.99s/KB] [10/16/23 04:02:57] INFO Starting gunicorn 21.2.0 [10/16/23 04:02:57] INFO Listening at: http://0.0.0.0:3003 (9) [10/16/23 04:02:57] INFO Using worker: uvicorn.workers.UvicornWorker [10/16/23 04:02:57] INFO Booting worker with pid: 10 [10/16/23 04:03:12] INFO Created in-memory cache with unloading disabled. [10/16/23 04:03:12] INFO Initialized request thread pool with 4 threads. [10/16/23 04:07:43] INFO Starting gunicorn 21.2.0 [10/16/23 04:07:43] INFO Listening at: http://0.0.0.0:3003 (8) [10/16/23 04:07:43] INFO Using worker: uvicorn.workers.UvicornWorker [10/16/23 04:07:43] INFO Booting worker with pid: 9 [10/16/23 04:07:57] INFO Created in-memory cache with unloading disabled. [10/16/23 04:07:57] INFO Initialized request thread pool with 4 threads. [10/16/23 04:22:57] INFO Starting gunicorn 21.2.0 [10/16/23 04:22:57] INFO Listening at: http://0.0.0.0:3003 (9) [10/16/23 04:22:57] INFO Using worker: uvicorn.workers.UvicornWorker [10/16/23 04:22:57] INFO Booting worker with pid: 10 ## My Home Server Architecture ## System * Debian 12 x64 * docker and docker compose to host my service * one nginx container reverse proxy service ## Network * server has both dynamic ipv6 and dynamic public ipv4 address * the ipv4 address is on the router, I set the Port Forwards to the server * using subdomain * default firewall on openwrt router(Allow-ICMPv6-Forward Allow-IPSec- ESP Allow-ISAKMP) * separate network(has own router and subnet) ## Exposed Port * a port with ssl, nginx use it reverse all my service * a port for v2ray proxy server * 1688 for kms * 5349, 49160-49200 for coturn server i installed wireguard on my router, service like ssh, xrdp, database are avaliable from lan and wireguard. ## Containers This server is running about 30 containers. most of them in one docker-compose file and in one virtual lan. only a few containers like nginx and coturn, mapped the port to host. ## Surmise and Confuse Becuse it just run in the container and it didn't delete database or something, I think this is a hack performed by a script similar to brute force ssh login(Maybe). But I don't know how did hacker do this. * There is no ssh port available from public address * No reverse proxy for this container(But for immich) * This container is not even running properly due to network reasons(logs before) Is my host server still "safe"? Should i reinstall my server? Is there some leak in immich-machine-learning? What should I do? ---------------------------------------------------------------------- Sekhen 路 2023-10-16 08:44 Setting up containers in your sleep? Maybe it's a partner in the household? Check for SSH keys, close all ports from other IPs than your LAN. ---------------------------------------------------------------------- Fun_Fungi_Guy 路 2023-10-16 08:44 What system do you have in place for securing your homelab? Any network traffic analysis such as an IDS/IPS? Personally speaking, I would assume nothing and nuke the whole setup 馃拋 If you don't recall playing with huggingface yourself that is. To me this feels like a pretty bad attempt at cryptojacking if that is what it is. Try to mount the container and extract the config files to see if you find anything malicious if you end up going that way. ---------------------------------------------------------------------- __Hug0__ 路 2023-10-16 08:46 I would start exploring logs /var/log/syslog and /var/log/auth.log. If these logs do not exist (Debian 12 Switched to Journaltl Insthed of Rsyslog), use "Journalctl -f" and "Journalctl -u ssh.Service". Also, at this time "\~10/16/23 04:02:57" look at Nginx logs (/var/log/nginx/\* or Journalctl -u nginx.service). It could be an exploit nginx reverse proxy. ---------------------------------------------------------------------- YankeeLimaVictor 路 2023-10-16 08:50 That's odd. Have you reported it to the immich devs? Seems like the vulnerability could be within the container itself? ---------------------------------------------------------------------- ElevenNotes 路 2023-10-16 08:52 Exposing 1688 for KMS is not a very good idea, it will put you on the radar of Microsoft. By the way, I鈥檓 not seeing any crypto miner in your logs? The only crypto you can mine on a CPU is XMR (Monero). Maybe the source you pulled the container from ships the miner by default? ---------------------------------------------------------------------- TinoOG 路 2023-10-16 08:54 Hard to tell without investigating logs and finding possible traces of an entry point, hardcore security professionals would probably suggest to assume full breach and take everything offline for analysis while rebuilding your system, but this is most probably a miner that came packaged with some other software you ran (that ML thing?). You have to decide what to do but the safest option is to rebuild everything imo ---------------------------------------------------------------------- DarkKnyt 路 2023-10-16 08:55 /u/altran1502 Icymi ---------------------------------------------------------------------- pumpkinsuu 路 2023-10-16 08:58 Machine learning python usually use jupyter which easy get hack. Why do you need to open port? If your server is private just use vpn or something like tailscale. If not show the ip and people will figure out problems. ---------------------------------------------------------------------- B0PE 路 2023-10-16 08:58 Why do you think that this process mines virtual coins? ---------------------------------------------------------------------- Ashamed-Translator44 路 2023-10-16 09:00 Thanks! the time is wrong beacuse of different timezone. i will change to ssh keys to login. the last command output seems ok. ---------------------------------------------------------------------- vixfew 路 2023-10-16 09:01 Always reinstall if you got hacked. There's no 100% sure way to delete malware ---------------------------------------------------------------------- Ashamed-Translator44 路 2023-10-16 09:01 Thanks, i will post this to immich devs ---------------------------------------------------------------------- Ashamed-Translator44 路 2023-10-16 09:03 OK! thank you! i already poweroff the server, and i will backup everything to another disk ---------------------------------------------------------------------- Sekhen 路 2023-10-16 09:34 If you start a command with a space, it's not logged in history. So instead of "docker ps" you'd type " docker ps", you won't be able to know without application logs. ---------------------------------------------------------------------- YankeeLimaVictor 路 2023-10-16 09:57 Why did you delete the post? ---------------------------------------------------------------------- MikeBagdala 路 2023-10-16 16:03 Are you sure you were hacked? The immich-machine-learning container probably just wanted to download a model from huggingface which for some reason was not available. I don't see anything in your log messages that necessarily indicates it being used for malicious purposes. What makes you think that your container was used for mining coins? ---------------------------------------------------------------------- Not_a_Candle 路 2023-10-16 16:50 The ML thing is part of Immich. Unlikely that it has something todo with it. The picture doesn't load for me, but the log posted doesn't show any mining at all. Immich causes really high cpu and memory load over an extended period of time, when pictures are being added/processed, because if the machine learning, face detection and other stuff. CUDA support is barely there, so yeah.. Cpu and ram it is.