Looking for initial advice and pointers reddit / selfhosted ---------------------------------------------------------------------- deescuderoo · 2023-10-12 15:47 Hi all, I'm quite new to the self-hosted space and looking forward to get involved! It's been a bit overwhelming to navigate around since 'self- hosting' means several things depending on the use-cases and the users themselves. I want to describe a little bit my background and motivation for self-hosting, and hopefully I can get some nice pointers and advice to help me get started. I am hoping to self-host the following services/use-cases I interact with in a daily basis: * **Sync files.** There are several files I use regularly across different devices. For example, I read a lot of PDFs, sometimes I download them from my PC and then read them in my tablet. For this I use *dropbox*. I want to host the files myself, being able to two-way sync them in real time from both my PC and my mobile devices. * **Backup files.** Related, I have about +5TB of files that I need to backup. I don't need them in a regular basis, so I don't need to sync them (like, I don't need them stored in other devices), but I need them available since I occasionally work on some of them. Ideally I would be able to access a given file from anywhere. * **Google photos**. I use the service regularly with my family, but I want to host the pictures and videos in my own premises. What I would need is the ability of accessing the pictures from any device (computer or phone) from anywhere, and also upload pictures taken with our phones. I do care about good library organization (e.g. geotagging or album management) but care less about "AI features". * **Calendar**. I rely on Google calendar right now to add events and sync them across my devices. Sometimes I add them from my phone, sometimes from my computer, and I want them all synced. * **Web hosting (optional)**. I have a small static website with my personal information. Right now I host it in github, but it'd be nice (mostly for fun and for the sake of learning) to host it at home as well, if this is actually something that can be done. * **Media streaming (optional)**. I have a large chunk of media I've collected over the years (movies, music, etc.) and I'd like to be able to stream it at home and perhaps outside as well. Regarding my background, I use Linux for work and most day-to-day tasks, so I'm not new to that but I'm definitely new with everything in the self-hosting space. Based on the use-cases above, I'm looking for advice and pointers on the following: * **Hardware recommendations.** This is not *concrete* devices necessarily, but more like terms and a general idea on what I should be considering taking into account the needs above. For example, I do have an old computer I could use but storage is insufficient for backup. I probably want it on and connected all times for accessing from outside so I need low power consumption as well. There are many words and acronyms I've seen, like NUC's or NAS', but the space seems quite vast and I want to be able to navigate the space better. For the backups and photos I also want to make sure I don't lose my data, so I want to learn about how to properly backup to minimize this risk. * **Software recommendations.** Some concrete pointers are welcome but also some general advice, like, do people use containers for everything? Can the whole environment be open source? * **Safety recommendations.** Since I want to access from outside I assume there will be a lot of security issues to consider. What are the standard practices? Using VPNs? Reverse proxies? Thanks a lot! Sorry for the lengthy noob post, I hope you can help me in the right direction in order to get started. Once again, pointers alone are welcome and already quite useful! ---------------------------------------------------------------------- thekrautboy · 2023-10-12 17:11 Take one step at a time... ---------------------------------------------------------------------- deescuderoo · 2023-10-12 17:19 Yes, at least I am trying. For me the first step is acquiring the appropriate equipment, and even that has already proven to be super confusing since there is so much out there, it is *overwhelming*. I found a deal for this around my neighborhood: [https://www.qnap.com/en- us/product/ts-453%20pro](https://www.qnap.com/en- us/product/ts-453%20pro), but I need to explore further to see if this is what I really need. Also, I would strongly prefer if the solutions are Linux-based. I'm somewhat in a 'rush' to download all my files from external clouds (subscriptions expiring) and I have limited space in my current PC. I don't want to buy extra storage if instead I can buy the setup I will end up using ultimately -- but deciding on *what* exactly seems hard. ---------------------------------------------------------------------- thekrautboy · 2023-10-12 17:23 Fyi this sub here is about selfhosting **software services**, not any hardware. Try subs like /r/HomeServer /r/Homelab /r/HomelabSales /r/BuildaPC instead for that. ---------------------------------------------------------------------- black-0ut · 2023-10-12 17:44 - Syncing - Syncthing - I love this, its fast to the point its almost realtime. There are a lot of configuration options. You can run into problems if you kind of want to exclude some files. It supports it but you are likely to run into issues. - NextCloud - used it before, its kind of slow when compared to syncthing but does what its supposed to (Will also help you in your Calendar requirements.) - Backup Files - Follow the 3 2 1 rule. (3 Backups, 2 Local, 1 Remote) - If you need it available at all times you will most likely want a dedicated device for this - You can look at Truenas or FreeNas if you are looking for a free solution. I have heard good things about Unraid, its not very expensive. - If you are willing to spend more money you can buy a NAS from Synology, they are quite dependable but for the same price you will have more storage and control if you make your own NAS. - Google Photos - Immich. Have all the features you requested. But its in alpha, I believe. I am testing it at the moment. - Calendar - NextCloud. - Web Hosting - you can use Cloudflare tunnels to get this working. - Media streaming - I am using Emby and it works fine for my usecase. - Jellyfin is also good from what I have seen on this subreddit but I am not sure if it comes with a mobile client. Plus for Emby there is a plugin in Kodi so I kinda incline towards Emby. I am kinda containers for everything kind of guy because it makes maintaining and upgrading services very easy. Security - I don't think one should open gates of home network to the world if its not absolutely necessary. - Specially since good solutions exist nowadays which weren't available before using which you don't need to completely expose your machines to the internet. Ex. Zerotier, Tailscale, Argo Tunnels - Before going through the proxy and VPN route you should consider above solutions. If due to something these solutions don't make sense you can do the following - Reverse proxy - NGINX proxy manager is perfectly fine if you are just starting out. Traefik if you consider yourself an advance user. - VPN - Wireguard. - Look through guides there are many available. - Keep locking stuff till the time its a little convinient. - For every service you host, understand its risk in your environment. Selfhosting is a hobby so be prepared to face and solve issues, sometimes it will solved in minutes sometime days. When working on these projects take time to understand why you are selfhosting it, if a better service exist or is it really worth the hassle. Specially when exposing services. Happy selfhosting. ---------------------------------------------------------------------- deescuderoo · 2023-10-12 18:17 Thank you for your answer, it is very valuable. Right now I'm trying to learn and understand what hardware equipment to get, I can see some NAS' (like [https://www.qnap.com/en- us/product/ts-453%20pro](https://www.qnap.com/en- us/product/ts-453%20pro) I found a deal for) that have their own software but if my understand is correct, one should be able to set up containers that run the different services you describe above. I will also investigate further into the internet and security issues. A big motivation for me is degoogling, I want to keep my documents, calendar, pictures etc. in my own premises, but at the same time I want to access them from outside since I do that regularly. ---------------------------------------------------------------------- fenty17 · 2023-10-12 18:23 My suggestion would be to get a second hand optiplex or other small form factor pc (mine was £40 off eBay). Install Ubuntu 22.04 LTS and docker, then start with a simple docker container via docker compose and build up from there. If you are in a rush to store your files from cloud services, just get a large ssd and get the files on it. Learning self hosting is not something you want to rush. Read up first, then ask questions here and the community will be happy to help I’m sure. There are plenty of solutions for your requirements and awesome self hosted is a good source to explore these options. But ask yourself first whether you really want/need to self host certain things. Calendar for example - why would you really want to move away from your current solution? ---------------------------------------------------------------------- deescuderoo · 2023-10-12 18:33 Thanks. You make very valid points. Indeed, I don't want to rush in making certain early choices, and if I can reuse SSDs across hardware then I agree there's no harm in just getting one, installing it in my current computer, and downloading stuff there. I might end up doing this actually. I am a bit confused in terms of existing hardware. I like the idea of first trying a small laptop running an OS I am familiar with. My motivation to manage my calendar is the same as photos/video and documents in general: I want to stop delegating my data to external providers. One may say that photos/videos are more sensitive data than, say, calendars, but I want to first try and only give up when I find it unusable or not worth the trade-off. I'll keep reading on this, thanks ---------------------------------------------------------------------- black-0ut · 2023-10-12 18:34 Synology supports docker, I am not sure about QNAP. You can look it up on YouTube. If it does support or if it can at all, I am sure people must have made videos on them. Also, forgot to mention in my previous comment. Pick docker images from reputable sources. Either pick the official channel or LinuxServer.io -> https://fleet.linuxserver.io/ is a nice project. ---------------------------------------------------------------------- deescuderoo · 2023-10-12 19:19 It seems it does support it: [https://github.com/psybernoid/QNAPDocker ](https://github.com/psybernoid/QNAPDocker). Indeed, it only took a quick search. Thanks for the advice and the tip on container sources. ---------------------------------------------------------------------- GolemancerVekk · 2023-10-13 00:17 > Software recommendations. Some concrete pointers are welcome but also some general advice, like, do people use containers for everything? Can the whole environment be open source? Sure. And yes it's best to use containers for each thing and to keep the base operating system "clean" and simple. For a beginner I would recommend Debian stable for the base OS, it's a very solid base, not hard to use, and allows for flexibility in case you want to do some things on the OS rather than containers occasionally. > Calendar. I rely on Google calendar right now to add events and sync them across my devices. Sometimes I add them from my phone, sometimes from my computer, and I want them all synced. Host a calendar backend like Radicale, which has CalDAV/CardDAV support for events, tasks, notes and contacts. CalDAV and CardDAV are universal calendar/contact standards that are supported on both iOS and Android. On Android there's no built-in support for syncing except for Google's services so you need one additional app. DAVx5 is the most widely used syncing app. Once your calendar and contacts are synced into Android/iOS you can use any app that supports calendaring/contacts. If you want some recommendations I like True Phone for calling & contacts, Calengoo for events & tasks, and jtx Board for notes. Calengoo also has a desktop app with versions for Windows/Linux/Mac. Please note that jtx Board is only necessary if you want your notes hosted in CalDAV by Radicale. If you want another note solution there are other apps. > Web hosting (optional). I have a small static website with my personal information. Right now I host it in github, but it'd be nice (mostly for fun and for the sake of learning) to host it at home as well, if this is actually something that can be done. It can be done. If you have a public IP you can have your own domain name pointing at it, and you forward port 443 in your router to a reverse proxy that will redirect a subdomain to the container holding the website. Nginx Proxy Manager is fairly easy to use for this purpose. If you have a public IP but it's dynamic, many registrars nowadays offer a DNS API that you can use to updated your domain to the new IP whenever it changes. You can also delegate your DNS to a free service like deSEC.io if your registrar doesn't have an API (deSEC does). If you do not have a public IP all is not lost: Cloudflare offers a service where they put up an IP, connect it to your domain, and connect it to an encrypted tunnel from your selfhosted machine. Visitors from the internet connect to the domain and the connection goes down the tunnel to your machine, where you can do the above (reverse proxy etc.) as usual. But they require you to use their DNS services for this, and they ask you not to use it for media servers. > Media streaming (optional). I have a large chunk of media I've collected over the years (movies, music, etc.) and I'd like to be able to stream it at home and perhaps outside as well. Jellyfin/Emby/Plex are the most widely used media servers, here's a comparison: https://github.com/Protektor-Desura/Archon/wiki/Compare- Media-Servers. Jellyfin is the most open and gives you the most control, followed by Emby and then Plex (which are paid but make up for it in other features). They can host music as well as video, but if you want a specialized music server (and specialized music apps) there's Navidrome, which works with any client app that is compatible with the Subsonic standard. There are also hybrid apps that work with both Jellyfin and Navidrome. > Hardware recommendations. This is not concrete devices necessarily, but more like terms and a general idea on what I should be considering taking into account the needs above. For example, I do have an old computer I could use but storage is insufficient for backup. I probably want it on and connected all times for accessing from outside so I need low power consumption as well. There are many words and acronyms I've seen, like NUC's or NAS', but the space seems quite vast and I want to be able to navigate the space better. For the backups and photos I also want to make sure I don't lose my data, so I want to learn about how to properly backup to minimize this risk. Intel processors are preferred because they have [built-in video encoding support](https://en.wikipedia.org/wiki/Intel_Quick_Sync_Video #Hardware_decoding_and_encoding) for media transcoding. For RAM, 32 GB is nice to have but you can get away with 8 GB or even 4 GB if you don't intend to host VM's and only need containers. The OS is typically installed on SSD. Most motherboards nowadays have a NVMe slot where you can put a NVMe format SSD so you don't take up a SATA connector. But please read the motherboard manual because some boards disable a SATA connector anyway when you do that. 32 GB of space will be plenty for the OS and docker images; I have about 20 containers and still only using about half of mine. For data storage it depends a lot on how much you need and what you want to do about redundancy (some form of RAID) and backup. You should by thinking how much storage space you will need and which of the data is critical and must be backed up vs which is not the end of the world if it's lost. Obviously you will need at least one HDD that can hold all your data, but the backup HDDs can be smaller. It's good practice to set up three layers of data, the active disk (1), one backup disk inside the same machine (2), and an external disk stored separately (3), and do incremental syncs from 1 -> 2 -> 3. Some people prefer redundancy rather than backup for 1+2, so the set up a form of RAID with those disks, for example RAID 1 which mirrors the data and protects you in case one HDD suddenly dies. It's important to note that such mirroring does not protect you from deleted files because it mirrors everything. When that happens you still come to rely on an incremental backup, which in this case would be (just) the 3rd disk. If you use RAID please use the native Linux software RAID (mdadm) because it will let your disks be recognized on pretty much any Linux distro, even if you reinstall from scratch. Don't use RAID that needs to be set up from the BIOS (so called "fake" motherboard RAID) or depends on a PCI card (you're screwed if the card breaks and you can't find an exact copy). You can use more than one disk for backup, and you can store them at different places than home, and bring them in for backup once in a while. Some people fill backup HDDs once or twice a year and ship them across the country to their parents etc. For incremental data which needs to be archived over long periods of time (documents, photos, passwords etc.) you can also use Blu Ray optical disks, which have been designed to last in excess of 100 years. It's more fiddly but a lot less prone to failure than HDDs. You can't use SSDs or other types of flash memory (like USB sticks) for long term backups because flash memory starts losing data when left unpowered for long periods of time. For HDD brands, Seagate, WD and Toshiba are currently the top dogs. Ideally you should purchase their models intended for NAS use (24/7 multi-year duty) for the live disk(s), which means WD Red Plus or Pro (don't get regular Reds for the live disks), Seagate IronWolf, or Toshiba N300. [You can look up model numbers here](https://nascompares.com/answer/list-of-wd-cmr-and-smr-hard- drives-hdd/) to find out if they're SMR or CMR (normal drives). SMR is a technology which makes them pause to sort the data when handling large amounts of it, which sucks for a live disk. But you can use them for the backup disks. And for backup disks you can also use whatever is cheapest among their models because they won't run 24/7 so it doesn't matter. You should try to also avoid 10 TB drives for live disks because they have the most unfortunate combination of platters and get all the worst sides of everything (noise, power consumption, failure risk etc.) You can assemble a regular PC in a regular PC case with regular off the shelf components. It won't use *that* much power and the major advantage is that the components are super easy to replace and upgrade. HDDs are the biggest power draw but they can be set to sleep when not in use. You can buy used tiny PCs like office machines or NUCs with space for 2 HDDs but they have specialized components and often proprietary so if something breaks you will probably have to replace the whole thing. ---------------------------------------------------------------------- deescuderoo · 2023-10-13 17:04 Wow this is very useful! Thanks! I've been reading more ever since I posted this originally and your content complements very nicely. The idea of using containers mostly is very consistent with what I thought about before even searching about anything, so I'm glad to hear that here. I like Debian as a first approach. I am currently exploring pre- assembled NAS'es and also custom builds. I understand the space better and I will be making some decisions soon. Thanks again!