Help me figure out what happened here and how to prevent it in the future. All database type docker containers corrupted after host VM's filesystem went into read-only mode. reddit / selfhosted ---------------------------------------------------------------------- Darkextratoasty · 2023-12-11 20:28 I've got Proxmox installed on a mini PC running an Ubuntu 22.04 Server VM that hosts a bunch of Docker containers. I woke up one day to a handful of monitoring alerts telling me that some of my services had gone down. Every docker service that depended on a database container of some kind was getting errors trying to reach their respective databases, and the databases were all freaking out. Any containers based on reddis, postgres, things like influxdb, etc were all having some sort of authentication error. It turned out that the VM's filesystem had converted to read-only, causing the databases to fail, I couldn't even shut down the containers because that requires a write of some kind. Rebooting the VM got the filesystem back to normal and was able to start up all the containers just fine, but the databases still had the authentication errors. The only solution I could find was to delete their volumes and start over, losing all data they had, which wasn't a huge issue, none of them had anything critical. So I have a few questions on this whole ordeal. 1. Is there anything I can do to prevent my databases from flipping out if the filesystem suddenly goes read-only again? 2. Is there any way to see why exactly the VM went read-only in the first place? The rest of Proxmox was fine the whole time. 3. Is there a way to recover data from a corrupted database container (like a reddis container) in case this happens again and there's actually important data? Any ideas, comments, or pointers would be awesome. ---------------------------------------------------------------------- Anejey · 2023-12-11 20:52 Did any of the containers/VMs run out of space? ---------------------------------------------------------------------- Darkextratoasty · 2023-12-11 20:53 The VM has plenty of available space on its drive. I don't think any of the containers ran out of space, their all using volumes, which I don't think have a storage cap by default. ---------------------------------------------------------------------- peekeend · 2023-12-11 21:20 Which version of proxmox are you running ? What type of storage are u using ? Hows the ram, as in did you try a ram test ? ---------------------------------------------------------------------- AlteRedditor · 2023-12-11 21:22 Do you have any backup configuration? Having a RAID should no be that expensive and it'd help with the restoration. There are many services out there that create regular backups for you. ---------------------------------------------------------------------- Darkextratoasty · 2023-12-11 21:38 I do now, just a simple rsync on the volumes folder to my NAS. Unfortunately, the Proxmox machine I'm using doesn't have any way to connect a second storage device for any sort of RAID at the host level, if that's what you mean. ---------------------------------------------------------------------- Darkextratoasty · 2023-12-11 21:40 It's proxmox virtual environment 8.0.9. The storage is a single 2TB M.2 NVME SSD, that's the only storage slot on the mini pc I'm using. I ran a memtest86 overnight when I first set it up and had no issues. It's definitely got plenty of RAM, both on the proxmox host and on the VM. ---------------------------------------------------------------------- thekrautboy · 2023-12-11 22:21 Just to chime in, RAID is not a backup. ---------------------------------------------------------------------- ElevenNotes · 2023-12-11 22:31 If you use XFS they do have a default cap for all not mounted volumes and the image itself. So if the container is producing logfiles you do not mount it will fill up the available space for the container. ---------------------------------------------------------------------- Darkextratoasty · 2023-12-11 22:42 What do you mean by "not mounted volumes"? From my quick google search, it looks like the limit is 10GB, which none of my containers should have been anywhere near. I can't guarantee that none of them hit that, since I didn't check that at the time and I've since deleted the old volumes, but they shouldn't have been very large at all. ---------------------------------------------------------------------- ElevenNotes · 2023-12-11 22:54 Anything you have not mounted via -v is stored in that 10G limit. So like a db dump in a temp folder or huge logs? You can check the size of the containers, if they are bigger than the initial image, there is some storing going on. ---------------------------------------------------------------------- fprof · 2023-12-11 22:58 Next time enter dmesg, it might show you what went wrong. 3 if you are not sure about consistency restore a backup. Redis is only in-memory, so no need to restore data for that. ---------------------------------------------------------------------- Darkextratoasty · 2023-12-11 23:22 They've all got the recommended volumes mounted through the "volumes" section in docker-compose, so unless the image is broken, none of them should be running into that issue. Currently they all look correctly sized, but I can keep an eye on them. ---------------------------------------------------------------------- Darkextratoasty · 2023-12-11 23:25 Yeah I glanced over dmesg when it happened and I didn't see anything obvious, but I don't really know what I'd be looking for. If it happens again I'll try to save if off somewhere to look more closely. ---------------------------------------------------------------------- fprof · 2023-12-11 23:29 If the filesystem is read-only there must be something related to that. ---------------------------------------------------------------------- 100GHz · 2023-12-12 00:15 >NVME Flash disk will go read only on problems. Probably best to backup everything ASAP ---------------------------------------------------------------------- Darkextratoasty · 2023-12-12 00:20 Everything is backed up currently. But why would the VM go read only while the proxmox host is fine? I would think a hardware issue with the storage device would show up on the host as well. ---------------------------------------------------------------------- voidcraftedgaming · 2023-12-12 00:22 Rather than rsync, you'll want to sqldump instead. MySQL and other similar database servers will often leave the disk in an inconsistent state which, if copied or abruptly powered off, will not be functional. You should set up a Cronjob to do something along the lines of ``` 0 0 * * * bash -c "mysqldump -h 127.0.0.1 -u backupuser -pverysecure --all-databases | gzip | ssh user@backupserver "cat - > backups/$(date +%s).sql.gz"' ``` Then a Cronjob on the backup machine / NAS to remove old backups The dump file is guaranteed to be consistent ---------------------------------------------------------------------- thehipstercat · 2023-12-12 01:03 Not sure if same issue but my Proxmox VMs did exactly the same as you describe. Sometimes, seemingly randomly, after daily backup task ran, my VMs ext4 filesystem would go read-only. VMs hard disk are stored on a ZFS RAID 1 with two SSDs. After a few research on Proxmox forums, I found out the culprit was a corruption that can occur when using SATA controller for VMs hard disks when running a backup while the disk is "heavily" used. Switching to SCSI fixed the issue and my VMs never went read-only again. Hopefully that helps you. ---------------------------------------------------------------------- Darkextratoasty · 2023-12-12 01:22 Hmm, I'll have to check if the disk is sata, thanks for the tip ---------------------------------------------------------------------- XTJ7 · 2023-12-12 05:26 nothing is guaranteed until verified. if you don't verify your backups, you don't have backups. ---------------------------------------------------------------------- lilolalu · 2023-12-12 08:59 Debug why your storage has been corrupted. Databases are pretty resilient against errors, mostly should recover fine after the file system is writable again. If not, restore backup. It's actually a good thing what your system did if databases start writing to a corrupted disk the REAL trouble starts. ---------------------------------------------------------------------- lilolalu · 2023-12-12 09:02 Maybe try rsnapshot. Rsync but with retention and incremental versioning. ---------------------------------------------------------------------- iavael · 2023-12-13 08:18 Read logs of your DBMSes and troubleshoot them according to the root cause. ---------------------------------------------------------------------- Darkextratoasty · 2023-12-13 13:57 I've never even heard of the dbms logs, but I'll try to poke around in there, thanks for the suggestion ---------------------------------------------------------------------- iavael · 2023-12-13 14:10 DBMS = database management system: postgres, mysql, redis etc. Usually if they have any issues, it's written in their logs what caused the problem.