Anyone have a not insane way to manage perms for shared data directories in a containerized environment? reddit / selfhosted ---------------------------------------------------------------------- Posting____At_Night · 2023-12-05 21:21 My homeserver is running Proxmox with a separate LXC container per- service. CTs are provisioned and configured with Ansible. The easiest illustration of my problem is my media folder. It is accessible to local machines over NFS which is its own nightmare of permissions problems, and it also needs to be accessible by several different containers (deluge, sonarr, jellyfin, etc.) Currently, I am using idmapping on the unpriveleged containers to map all UIDs over 1000 to their normal values so as to not break perms on my shared folders. Not awesome for security but I'm trying to balance complexity here. For managing UIDs themselves, I just manually set them in the ansible roles for the containers, which works but is super brittle. I've tried LDAP before but it's way overkill for an environment that has effectively only one end user (me), and doesn't really fix the perms brittleness, just makes it so I don't have manually organize my UIDs. I used ACLs for a period of time to enforce proper perms but this had its issues as well. Eventually I scrapped those and just made a nightly cronjob that chowns/chmods all my shared data every night to eliminate problems. There surely must be a better way to do this. I feel like I've been taking crazy pills for years now with how complex this seemingly simple use case is. Any suggestions? ---------------------------------------------------------------------- art_of_snark · 2023-12-05 21:27 it’s called a group. probably with a more permissive umask. ---------------------------------------------------------------------- Posting____At_Night · 2023-12-05 21:31 I forgot to include in my OP that I have been using a group for this. All the services run with a user with the same media-access group and umask set to 0002. It all works currently, it's just way overcomplicated and brittle. ---------------------------------------------------------------------- adamshand · 2023-12-05 21:56 NFS assumes you have consistent users/uids across all clients. Assuming you are managing this sensibly (eg. manually or with LDAP) then you just have to deal with the fact that Unix permissions are old and simplistic. I don't run the *arrs, but I'd consider just running them all as the same UID. If you haven't discovered the SGID and sticky bit permissions yet, these can help quite a lot: - SGID: When set on a directory, any files created in the directory will have their group ownership set to that of the directory owner - Sticky bit: at the directory level, it restricts file deletion. Only the owner (and root) of a file can remove the file within that directory. A common example of this is the /tmp directory. If you are using NFSv4 on all clients, you can also use ACLs for a richer permissions model. ---------------------------------------------------------------------- HonestPrivacy · 2023-12-05 22:00 You mentioned issues with the file access control lists (although that is likely not the best solution to this problem), I'd be curious what type of issues as that is one of the main purposes of it. You'll want to set the default values for the control list so future files have the same permissions. You can also copy without keeping the permissions (mode) or ownership of the original: `cp --no-preserve=mode,ownership` With containers, you can also specify the user and group id to use (at the container level) that is mapped to the existing user so you don't run into the permission issue (in docker compose it is the user key - `user: "${UID}:${GID}"`) ---------------------------------------------------------------------- Posting____At_Night · 2023-12-05 22:01 SGID unfortunately doesn't work if you're copying a file as opposed to creating new, at least in my testing. I'm using NFSv4 on all clients but also ran into problems with ACLs. It's been a couple years since I last had that setup though so I can't remember exactly what the problem was. Honestly, what would be cool is being able to just mount a volume with some sort of auth key and ignore unix perms completely. I've looked into this a few times with various s3 compatible self hosted solutions but never found one that didn't have a crazy complicated setup process and wasn't too barebones to be useful. ---------------------------------------------------------------------- Posting____At_Night · 2023-12-05 22:08 I wish I could remember the exact problem I ran into, but yeah the opacity of ACLs made for some frustrating issues more often than I would have liked so I'm trying to avoid it for that reason if nothing else. I'm already using idmapping at the container level, but this has proven to have literally zero support in any management tools to where I had to write a hacky script to inject the lines into the lxc config files. ---------------------------------------------------------------------- adamshand · 2023-12-06 06:34 I might be misunderstanding what you mean, but I just tested and SGID correctly set the group on a file both when I touched a new file and when I copied it to another file? > Honestly, what would be cool is being able to just mount a volume with some sort of auth key and ignore unix perms completely. Just run all your containers as the same UID and chown all your files in media to that user? You could do this with Samba/CIFS because you auth when you mount all actions happen as the authenticated user. > Maybe it's time to finally try setting up a ceph cluster :/ If you're presenting it as a filesystem, you still have filesystem semantics. ---------------------------------------------------------------------- bityard · 2023-12-06 14:17 I know the pain you feel, as a sysadmin/devops type I deal with it frequently at work. It sounds like you've already considered the most obvious options. When dealing with Unix permissions, multiple applications, and various access methods, it is tough to get everything 100% correct. And that's just how it is. I'm not familiar with the software you list but when I hear "a bunch of different things need access to the same set of files with varying permissions," I think "cloud storage." There are various ways to self- host S3-compatible storage and nearly all of them let you set fine- grained permissions on buckets and users. For things that don't speak S3 natively, there are tools that let you mount a bucket as a FUSE filesystem, for example. The trade-off is that this will be FAR less performant than a native Linux filesystem, so this is not the solution you want if you must have raw speed and low latency. ---------------------------------------------------------------------- Posting____At_Night · 2023-12-06 15:21 For copy, you have to use --no-preserve with CP or it will override SGID, at least it did last time I tested. A lot of other applications like my file manager also seem to override SGID. As for presenting a volume from a ceph cluster or other s3 compatible storage, yeah, I'll have to deal with FS semantics on some level, but theoretically I can handle perms at the object/bucket level and map it onto an arbitrary UID within the container. So to each service, all the relevant files would show up rw with group:owner matching the service. ---------------------------------------------------------------------- phein4242 · 2023-12-06 18:10 Use NFSv4 and setup recursive acls ---------------------------------------------------------------------- Posting____At_Night · 2023-12-06 23:28 Every few months I look into self hosted object storage but it always looks like a maintenance nightmare. From what I've read though, Ceph supports a method of mounting a namespace as an NFS share, which if it's able to do that with reasonable perf and squash to a specific UID, might be perfect for my use case if it weren't for how complex ceph is to setup and administer. ---------------------------------------------------------------------- bityard · 2023-12-07 04:14 Ceph is a lot more than object storage though, I was thinking more along the lines of minio.