๐Ÿ†• Cosmos 0.11.0 - All in one secure Reverse-proxy, container manager with app store and authentication provider, and integrated VPN now has a Docker backup system + Mac and Linux clients available reddit / selfhosted ---------------------------------------------------------------------- azukaar ยท 2023-10-22 13:01 Link: [github.com/azukaar/cosmos- Server/](https://github.com/azukaar/cosmos-Server/) Cosmos 0.11.0 is out with a new backup system to export all your docker containers! The Linux and Mac clients are also out for some early testing, please share your feedback! https://preview.redd.it/htjbg0ze5rvb1.jpg?width=1695&format=pjpg&auto= webp&s=944ae33ae00c67159eddecf4ff923a1a0141e106 The new backup system works by reading the list of containers on your server and exporting a single compose file, with all the setup you need to recreate (in case of crash) or migrate your server. The backup system triggers on every docker change, **including changes you've made outside of Cosmos** (ex. Portainer, etc...). It outputs to a single file in your config folder, which you can backup with various strategy to keep a history of your docker containers state! As a reminder, this exists alongside the existing features: * **App Store** ๐Ÿ“ฆ๐Ÿ“ฑ To easily install and manage your applications, with simple installers, automatic updates and security checks * **Customizable Homepage** ๐Ÿ ๐Ÿ–ผ To access all your applications from a single place, with a beautiful and customizable UI * **Reverse-Proxy** ๐Ÿ”„๐Ÿ”— Targeting containers, other servers, or serving static folders / SPA with **automatic HTTPS**, and a **nice UI** * **VPN** ๐ŸŒ๐Ÿ”’ To securely access your applications from anywhere, without having to open ports on your router. * **Authentication Server** ๐Ÿ‘ฆ๐Ÿ‘ฉ With strong security, **multi-factor authentication** and multiple strategies (**OpenId**, forward headers, HTML) * **Container manager** ๐Ÿ‹๐Ÿ”ง To easily manage your containers and their settings, keep them up to date as well as audit their security. Includes docker-compose support! * **Identity Provider** ๐Ÿ‘ฆ๐Ÿ‘ฉ To easily manage your users, **invite your friends and family** to your applications without awkardly sharing credentials. Let them request a password change with an email rather than having you unlock their account manually! * **SmartShield technology** ๐Ÿง ๐Ÿ›ก Automatically secure your applications without manual adjustments (see below for more details). Includes anti-bot and anti-DDOS strategies. โ€‹ As always, eager to get some feedback on this release, here's the rest of the changelog: \- Docker export feature for backups on every docker event \- Disable support for X-FORWARDED-FOR incoming header (needs further testing) \- Compose Import feature now supports skipping creating existing resources \- Compose Import now overwrite containers if they are differents \- Added support for cosmos-persistent-env, to persist password when overwriting containers (useful for encrypted or password protected volumes, like databases use) \- Fixed bug where import compose would try to revert a previously created volume when errors occurs \- Terminal for import now has colours \- Fix a bug where ARM CPU would not be able to start Constellation โ€‹ happy hosting! ---------------------------------------------------------------------- thekrautboy ยท 2023-10-22 13:52 Always nice to see so many good and frequent updates. Congrats on the release :) ---------------------------------------------------------------------- Moehrenstein ยท 2023-10-22 14:12 Seems like i have to jump in again. Three versions ago i changed back to NPM because it handled 40+ domains faster. ---------------------------------------------------------------------- Paravite ยท 2023-10-22 14:15 I just got into selfhosted using Cosmos and I'd like to cause it makes everything less overwhelming ---------------------------------------------------------------------- azukaar ยท 2023-10-22 14:15 If you are experiencing any performance issue, feel free to document them to me so I can investigate your setup and potentially fix any shortcomings :) ---------------------------------------------------------------------- azukaar ยท 2023-10-22 14:16 Thanks you! That was quite a stressful one... Touched many core services to simplify importing containers ๐Ÿ˜“ ---------------------------------------------------------------------- thekrautboy ยท 2023-10-22 14:20 I am very much not a fan of people using giant compose files with all their services stuffed into it. But as a form of a very simplified export i suppose its fine. Not sure i would call it a actual backup, fingers crossed restores always work without issues ;) ---------------------------------------------------------------------- jasl_ ยท 2023-10-22 16:24 Does it support podman? ideally as non root ---------------------------------------------------------------------- Altair12311 ยท 2023-10-22 16:56 Okay i didnt knew about this and looks amazing ---------------------------------------------------------------------- azukaar ยท 2023-10-22 17:01 No unfortunately it is currently Docker only ---------------------------------------------------------------------- azukaar ยท 2023-10-22 17:03 I guess it's a backup specifically for your docker setup, not including files and volumes basically, On a blank server it should always work as expected to import, as it exports all the data at runtime (ex. if you changed something or the container itself changed. It even export the current container's IP) If you try to merge 2 servers there might be some miss, because of IP range overlap and container name clashes but that's not really the point I did try to migrate a server with \~10 ish app and I was able to do so Worse come to worse you can always edit it manually in case something is not importing \^\^ ---------------------------------------------------------------------- azukaar ยท 2023-10-22 17:04 Glad to hear it :) ---------------------------------------------------------------------- azukaar ยท 2023-10-22 17:04 Thank you!! ---------------------------------------------------------------------- jasl_ ยท 2023-10-22 17:23 Any plans for it? It will help to increase security ---------------------------------------------------------------------- azukaar ยท 2023-10-22 17:33 May be next year once 1.0 is out. The issue is that Cosmos is a OS level service manager, it is meant to run with privilege access because it does a lot of things that requires root access, such as network management, and so on. Supporting non-root for it would be difficult ---------------------------------------------------------------------- thekrautboy ยท 2023-10-22 18:04 Oh i dint mean this as a criticism about your work at all, it was just a general thought about people using giant compose files for their entire host, a discussion that comes up here every now and then and how it is a bad practice. Of course in Cosmosยด case its a different situation entirely. ---------------------------------------------------------------------- azukaar ยท 2023-10-22 18:13 Nothing wrong with a bit of criticism anyway ;p But yes I do agree ---------------------------------------------------------------------- thekrautboy ยท 2023-10-22 18:20 xD ---------------------------------------------------------------------- JustDalek_ ยท 2023-10-22 18:23 Thank you for this! Does this include volume data or do we have to manually back that up ourselves? I'm wondering what the full process of recovery looks like, as if it's a compose file of all containers, I imagine this would restore brand new services unless we handle the volumes right? Or does it also back up volume data as well? ---------------------------------------------------------------------- geo38 ยท 2023-10-22 18:41 The readme is all about security and holes and other bad things that hackers/miscreants/bots can do and how Cosmos can prevent them, but the example run command has this which pretends none of those things actually exist: --privileged -v /:/mnt/host Any sort of security lapse/hole in Cosmos or the software it builds upon gives a miscreant/bot/hacker full read/write access to the entire host system! It's ironic as the readme has many **bolded** statements about how cosmos protects docker containers from one another in case one is breached, but makes the **bold** assumption that cosmos itself is completely secure. Mounting '/' from the host is a **bad** suggestion. Yes, the readme says it's optional, but holy heck don't make it the suggested method as many of the folks using Cosmos are doing so because they don't understand security and they'll use the suggested run command. They don't know this is a **BAD** suggestion. ---------------------------------------------------------------------- EndlessHiway ยท 2023-10-22 18:47 I have been checking out Cosmos and I really like it. A lot more functionality than CasaOS and the fact you have automatic updates for your store's apps is great. Most the apps I tried out from the app store on CasaOS were out of date and hard to update without knowledge of Docker. ---------------------------------------------------------------------- azukaar ยท 2023-10-22 19:45 It does not include volume data, but backing up volumes is a planned feature too, it's just a lot more difficult of a feature to develop :p Basically backing up volumes is easy without Cosmos, it's just a folder, but backing up your container is much harder, that's why I added it in priority ---------------------------------------------------------------------- ItsYaBoyEcto ยท 2023-10-22 19:47 It looks pretty cool ! ---------------------------------------------------------------------- azukaar ยท 2023-10-22 19:51 Cosmos is a fully fledged server management platform, as such it requires those access to the host server in order to operate. "--privileged -v /:/mnt/host" is not as bad of a thing as you would think in that context, in fact it is equivalent to running a daemon like you would with any other alternative (CasaOS, Umbrel, etc..) those are just requirement for Cosmos to run with the same level of exposure as those alternatives who are not docker containers. My only alternative would have been to make Cosmos a daemon and not a container, but then it would make install and maintenance harder I understand your point, and yes ideally it would run as an isolated container, but it's just not possible to have a supervisor software managing your server running in an isolated container with no container, it is contradictory I propose as alternative to run Cosmos with lower privilege, in which case **some features will not work** but the default is to run cosmos with the privileges it requires for all features to work as expected. And the bottom line, the security benefits behind Cosmos for your average home-server outweigh by far this "--privileged -v /:/mnt/host". Slight reminder than a very large portion of people running alternatives like Casa, Umbrel, etc... Also expose those root daemon **without even HTTPS or anything!** ---------------------------------------------------------------------- azukaar ยท 2023-10-22 19:52 Thank you! Yes keeping your containers updated is a huge step toward preserving the server's security ---------------------------------------------------------------------- azukaar ยท 2023-10-22 19:52 Thanks! ---------------------------------------------------------------------- JustDalek_ ยท 2023-10-22 19:52 Ah many thanks :D ---------------------------------------------------------------------- geo38 ยท 2023-10-22 20:14 I think you're missing the point. My issue is sharing / with the container as root. I understand some parts of Cosmos require --privileged. That is not my issue. To be super repetitive, my issue is the combination of --privileged with -v /:/mnt/host The very next statement in your documentation after this says: > in this command, -v /:/mnt/host is optional and allow to manage folders from Cosmos, you can remove it if you don't want it but you will have to create your container's bind folders manually. If it's optional, then don't give it in the sample command that 99% of users will run. You can't tell me that it's not a risk. And saying "other programs have risks, too" is pretty cowardly defense against suggesting users do something as unsafe as mapping their entire host file system to Cosmos when Cosmos doesn't need it. ---------------------------------------------------------------------- azukaar ยท 2023-10-22 20:17 Another way of seeing it is, if Cosmos wasn't a container it would see \`/\` anyway. It's not extra access, it's just a workaround for Docker ---------------------------------------------------------------------- xIndirect ยท 2023-10-22 20:31 I think other poster is more of pointing out why not have -v /CasaFolder:/mnt/host or something similar. I've been in the IT sector for a while and I view that if you have to use a workaround instead of actually solving the problem, there is an issue with your tooling or solution. Secondly, I see your point but I also agree with the other poster. There's tons of security vulnerabilities announced and used on a daily basis and you should never have permissions to things you don't need. Is there a reason why the user couldn't just have a note in docs to say if you want to do X then mount this way? It's a few more lines and doesn't give an insecure by default config right from the get go for users that otherwise wouldn't know better. ---------------------------------------------------------------------- azukaar ยท 2023-10-22 20:43 Don't get me wrong, I am fully aware that you need to reduce as much as possible the amount of access something has but as you said: >you should never have permissions to things you don't need well Cosmos needs to see your files if you want Cosmos to manage your files. It's that simple. By default its on because it is needed for Cosmos to function. You can remove it, but at the expense of some of the functionalities of the server. **By the way** Cosmos, as a Docker management software, **has access to your docker socket**. Which mean, you can remove anything you want from the container, technically, it can add it back itself. Having access to the socket means being able to manage the containers, including itself. In other words, having this mount in the docker run command is just a comfort thing, but in term of privilege, whether it's Cosmos or Portainer or any other docker manager, they have full root access to your system and that's unavoidable. >why not have -v /CasaFolder:/mnt/host or something similar Because it would require users to always update their Cosmos containers to add additional folders all the time, giving a terrible and very error prone user experience. If there is a solution out there, that solves that problem (as in allows Cosmos to continue to work the same without that mount) then I will gladly implement it. But as far as I can see there isn't such solution ---------------------------------------------------------------------- Elle221LL ยท 2023-10-22 21:17 If it weren't a Docker/Portainer replacement and were solely responsible for security, I would use it. Multiple applications are needed to have all the good security measures that you have implemented in one application. ---------------------------------------------------------------------- azukaar ยท 2023-10-22 21:23 Docker is an important ingredient in the mix, to isolate the applications completely, and make things much more streamlined than traditional VM, but I understand if it's not for everyone! ---------------------------------------------------------------------- -SHINSTER007 ยท 2023-10-22 22:20 noob question but would I be able to install this on a free-tier VPS and use it to manage content on a seedbox (without root access) and services on my local network? ---------------------------------------------------------------------- AKAManaging ยท 2023-10-22 23:36 This is a wonderful way of describing something I was looking for. Going to give this a shot. Fingers crossed lol. ---------------------------------------------------------------------- Nnyan ยท 2023-10-22 23:55 Iโ€™ve tried a number of these โ€œstacksโ€ and some have great communities and others not so much. But the peeps with Cosmos are amongst the very best. Extremely helpful. Cosmos and YAMS are the best IMHO. ---------------------------------------------------------------------- azukaar ยท 2023-10-23 03:24 You need root access to manage docker containers that's (almost) unavoidable. Also Cosmos does not support managing remote docker instances. On the other hand, a good (and secure) pattern is to use Constellation (the integrated VPN) on 2 servers with cosmos installed on each. you can connect them together. One of the servers (the seedbox) is the main server running services but it is not exposed on the internet and the only way to access it is to connect to the VPN on the other VPS ---------------------------------------------------------------------- azukaar ยท 2023-10-23 03:25 Thank you so much :) ---------------------------------------------------------------------- umairshariff23 ยท 2023-10-23 05:20 Hey! I've been recently getting into VPN and 2fa and could you give me a eli15 for this? How does this work? I currently have my entire server port forwarded and reverse proxied on port 80 and a duckdns domain, with homepage running on port 80. After implementing this, will port 80 be taken over by cosmos? ---------------------------------------------------------------------- pnlrogue1 ยท 2023-10-23 07:00 Does it support Podman or is it just Docker? If it's just Docker, please consider Podman support as it's open source and most secure by default thanks to running as a user instead of as root yet is almost completely compatible with Docker. ---------------------------------------------------------------------- BadGroundbreaking243 ยท 2023-10-23 08:01 Wow this is interesting. Gotta try this tonight. ---------------------------------------------------------------------- --ddiibb-- ยท 2023-10-23 08:03 looks very cool, will have to give it a good go, thanks for all the effort! :) ---------------------------------------------------------------------- Jasper1224 ยท 2023-10-23 08:04 Wait, just to be clear, is this basically like what Truenas' apps does, but standalone? Cause holy shit I was looking for something like this for a while. Does it support mounting network drives via NFS? ---------------------------------------------------------------------- jasl_ ยท 2023-10-23 08:30 you can keep a privileged container for the things that are necessary and an unprivileged user to run everything else, You even can grant specific network permissions to unprivileged containers ---------------------------------------------------------------------- kapusynyc ยท 2023-10-23 08:44 Dope, it's only for Docker tho. ---------------------------------------------------------------------- azukaar ยท 2023-10-23 09:24 The VPN part is basically a "secret" (encrypted) tunnel between multiple devices/servers. Whenever one device wants to talk to your server, it sends messages via the tunnel, and on the other end, the tunnel dispatch the message to the right port. Using this, you have access to your server without exposing all your ports, so only people connected to your VPN can see it. Keep in mind this is different than a traditional VPN who transfer all your data to the server to hide your IP. Here only the traffic to your server is tunneled. This way your other activities are not affected (performance wise especially) 2FA uses any authenticator app (the one where you scan a QR code and get a 6 digits number) to protect your account. If someone gets your password, they still cant login because they also need your phone (unlocked) to get the 6 digits (it changes every 30 secs) ---------------------------------------------------------------------- azukaar ยท 2023-10-23 09:25 I am considering Podman support but probably more next year when Cosmos is feature-complete for 1.0 Keep in mind it might be a challenge to do everything rootless but I will see what I can do ---------------------------------------------------------------------- azukaar ยท 2023-10-23 09:25 Thank you! ---------------------------------------------------------------------- azukaar ยท 2023-10-23 09:25 Thanks!! ---------------------------------------------------------------------- azukaar ยท 2023-10-23 09:26 >Truenas Haven't used it, but it looks like there are overlap. Cosmos does not yet have storage management (but soon) and uses Docker instead of VM ---------------------------------------------------------------------- azukaar ยท 2023-10-23 09:27 Too much complexity. Right now there are two containers (one for Cosmos, one for the DB) and that's already causing management issues to users, I wouldn't go the "let's make it even more complex" route May be there's another solution though ---------------------------------------------------------------------- azukaar ยท 2023-10-23 09:28 Docker is an important ingredient in the mix, to isolate the applications completely, and make things much more streamlined than traditional VM, but I understand if it's not for everyone ---------------------------------------------------------------------- pnlrogue1 ยท 2023-10-23 09:38 Excellent! Good to hear it may be on the way! ---------------------------------------------------------------------- azukaar ยท 2023-10-23 10:04 I love Podman like everyone, it's just a lot of work to integrate it consistenly while also supporting Docker ;) ---------------------------------------------------------------------- jasl_ ยท 2023-10-23 10:31 I would say you can simplify it by creating a single pod (2 containers) with cosmos and the DB all together, and this one would run on privileged mode. Even simpler would be to move to sqlite, so it's all in a single container. Then the "app store" can run unprivileged containers by default with the optional "run as privileged". In a last effort, you can run everything unprivileged but grant all the required permissions with a setup script. In my experience, the complexity doesnt grows with the number of containes but with the extra permissions required by those apps not designed to work unprivileged. I would say podman can even simplify the deployment and maintenance by using systemd quadlet hence delegating all the supervision part to systemd ---------------------------------------------------------------------- AngryMaritimer ยท 2023-10-23 12:04 Looks and feels decent. One major thing unless I am blind, Sabnzbd is not in the marketplace, but nzbget is, which has been abandoned so could lead to security risks? ---------------------------------------------------------------------- magicaldelicious ยท 2023-10-23 12:45 We found Dan Walsh everyone! ---------------------------------------------------------------------- JiggySnoop ยท 2023-10-23 12:54 would it be possible to move from portainer+ caddy to cosmos ? i'm more concerned about path mapping.i have everything in different folders. ---------------------------------------------------------------------- azukaar ยท 2023-10-23 13:02 You are correct, and Sab is due to be added ---------------------------------------------------------------------- azukaar ยท 2023-10-23 13:03 You dont need to do anything to migrate, Cosmos will just work **with** Portainer, including just picking up your existing containers ---------------------------------------------------------------------- Jasper1224 ยท 2023-10-23 20:55 So Cosmos is basically just a pre-configured docker app "store"? How do you decide which containers go into the apps menu? Why does the descriptions in Github feel like it uses boldtext for some fearmongering? ---------------------------------------------------------------------- azukaar ยท 2023-10-23 21:26 Lol that's a bit unfair of a comparison I think! :D It's more like "Portainer" + "Authelia" + "NPM" + "Wireguard" in one software. Constellation is using Nebula under the hood not Wireguard, and adds a few significant features such as the DNS and automatic config (and soon config synchronization), and I also wrote functional UI Clients for it (Nebula does not have them, except for Mobile but it's a very rough) The home screen shortcuts are based on what URLs are in the reverse proxy The statement "you cannot do it on your own" refer to the complexity of securing apps and services that are not built to be secure. Most of the selfhosted servers available, while great efforts, are very poor security wise, even big ones like Jellyfin. I guess what the statement means is, you can't just run Jellyfin and secure it "on your own" without security centric solutions, but that does not mean Cosmos alone, it can mean any hardened reverse proxy, an authentication gate like Authelia, a VPNs for remote connection, and others stuff like Crowdsec and so on... This statement is targeted at people who don't take security seriously in their setup The bold is mostly because (almost) nobody will read entirely this much text ;) ---------------------------------------------------------------------- Jasper1224 ยท 2023-10-23 22:18 Mmm, understood. Seems like Constellation and Wireguard are targeting two different use cases really, so it would be unfair to compare those also. And don't get me wrong, I'm not saying that it's "just" a "preconfigured app store", rather that it is a all-in-one deploy containers solution, which as my first comment stated, was basically what I was looking for, a Truenas Apps-alike, but standalone, so they can just handle storage, while another machine handles the services. Still would like some confirmation for NFS shares support, though I assume since it sounds like it's just a container itself, it should work as long as the host OS can pass it along. I still feel that the bold words feel a little scummy, since it makes it out to seem that it is practically impossible to do security on your own. Not that it is a lie, that securing stuff _is_ difficult, and that you gotta market/promote the service somehow, but the comments just rubs me the wrong way, like a hard sell does. Still. I do have a spare pi3 around, which meets minimum requirements. I'll install it on that to try it first, before I consider migrating my existing stuff over. ---------------------------------------------------------------------- azukaar ยท 2023-10-23 22:44 Yes you are correct wireguard's primary function is being a VPN, while constellation's primary use is to be an overlay network to access your server remotely Whatever the os file system supports, it will be fine with as it is a simple container as you mentioned And you know.. I'm not a marketing person, and I don't be pretend to be good at communicating, I'm just trying to do my best to deliver a message really Have fun with the Rb3! ---------------------------------------------------------------------- maximum_onyx ยท 2023-10-25 06:23 Installed this on my VPS a couple of days back. Look and feel fantastic and functionality like a swiss army knife. I temporarily had to turn off my container and turn back npm on. I definitely want to spend time this week to make this up and running. 1. Is there any guidance on if a subdomain is preferred over the domain ? For the first time, I used my main domain. And it somehow broke my existing let's encrypt certificate. I purged the folders and did a clean reinstall, this time on a subdomain. Works but somehow the certificate broke again. (Too many certificates or something like this) 2. Could this be because I clicked "new certificate" while I had temporarily turned off my NPM containers to try Cosmos ? ---------------------------------------------------------------------- azukaar ยท 2023-10-25 09:32 \- I think domain is preferable for home servers because then you get subdomains for apps, which are easier and can also share the auth cookies for SSO \- you probably had a cached certificate ---------------------------------------------------------------------- maximum_onyx ยท 2023-10-25 09:41 Thanks. For VPS then can I ignore the domain field during setup, and do a reverse proxy later to the cosmos container (via cosmos reverse proxy) to access it via the web ? ---------------------------------------------------------------------- azukaar ยท 2023-10-25 10:06 You need to setup the hostname in the Cosmos installer if that's what you are asking. You can put your IP or something if you dont have your domaoin yet ---------------------------------------------------------------------- maximum_onyx ยท 2023-10-25 16:00 No I have a domain already, and that's what I did during the first try. I have a blog (docker container with ghost) on XYZ.com. As soon as I installed cosmos, that blog went away and got replaced with cosmos as the host started pointing to the docker container of cosmos. This is why I would like to install cosmos on my subdomain. Question is can I install cosmos on - cosmos.xyz.com ? Or is it a requirement to have it on XYZ.com entered during setup ? ---------------------------------------------------------------------- azukaar ยท 2023-10-25 17:02 yes you can absolutely install Cosmos on a subdomain yes ---------------------------------------------------------------------- maximum_onyx ยท 2023-10-25 18:41 Perfect, thanks.