Best practices for a self-hosted production website? reddit / selfhosted ---------------------------------------------------------------------- kayson · 2023-10-02 17:40 I'm deploying a nodejs-based website for someone. It will be low traffic, but I want to make sure I've got all my bases covered in terms of best practices for deployment and security. This is what I've got so far: * Code is hosted in private repo on private gitea instance * build into a docker image * Separate repo for all deployment code using ansible * secrets are saved in a vault and templated to the host * Runs on dedicated host with dedicated ipv4 * host has fail2ban installed and firewalled to only allow ports 80/443 and shh * ssh hardened * non-standard port * public key auth only * images are run on docker * non-root user * one network for app+db, another for app+reverse proxy * only mapped ports are 80/443 on reverse proxy container * using swag for reverse proxy (includes fail2ban and letsencrypt) * php disabled * backups * database dumped nightly * everything synced to backblaze (wip) What else should I be doing? The one thing I know I don't have is any monitoring. I'm going to set up some kind of healthcheck, but not sure if there's anything easy to set up wrt log monitoring... Thanks! ---------------------------------------------------------------------- GolemancerVekk · 2023-10-02 18:15 > ssh hardened Also don't allow root login. > firewalled to only allow ports 80/443 and shh Don't rely on firewall alone. Also check what is actually listening on the public interface (on both TCP and UDP). It should only be 80, 443 and the SSH port on TCP, nothing else. If there's a service you actually need to be running, make it listen on a private interface or on UNIX socket if it supports that instead of TCP/UDP. If it's a service that doesn't need to run, don't run it. > Code is hosted in private repo on private gitea instance > Separate repo for all deployment code using ansible These ones should not be on the same machine that serves the live site. > database dumped nightly You need to get that off-site. And the dumps should be secured in the meantime. > Separate repo for all deployment code using ansible Not sure what's your actual deploy procedure. Does something kick in when you commit to a branch? If yes, what does it do? It's a good start. Some other points (in the spirit of best practices): * Are you checking the database dumps? * What about actually restoring the dumps them if you need to, what's the procedure there? Backblaze may not be so eager to actually give you your backups back... * Do you have a test suite that you run after each deploy? If yes, how do you handle failures? * How do you handle changes to live database data and/or structure? Do you use migrations? * If the app has static assets and/or generates static content you'd normally set up a CDN. Cloudflare can provide combined CDN/reverse tunnel solutions, but they want you to use their nameservers. Bunny.net is another good [almost] free service (they offer DNS too but it's not necessarily tied to the CDN). And some more extreme stuff, when you get into high-availability / redundancy / replication / scalability: * Backup DNS. * CDN can usually be replicated in 2+ location across the world. * You can do redundancy/replication of app and/or DB yourself with 2+ VPS and a round-robin distribution (easiest). Ideally you'd do geo- relevant routing so each region on the planet uses the closest instance – this is usually something that the cloud providers will do for you if you use their services but you can look up how to do it with Nginx if you're curious. Scalability is a whole other kettle of fish. A crude method is to once again have 2+ hosts whose CPU/RAM capabilities put together exceed predicted spike demand and to do round-robin distribution, but of course that means their resources are wasted during downtimes. You can do it for fun though, just to see it work. Ideally you'd want the proxy to be aware of the CPU/RAM consumption on each host and route based on that. You can also try doing this. Even more advanced is to spin up/down VPS instances as needed. This is complicated and typically it's something done by the cloud provider or VPS provider for you. If done manually you'd need a good algorithm and it's not easy to achieve good predictibility/reactivity that also accounts for hysteresis etc. You're probably not going to do this without some system control theory under your belt. And of course it costs extra so probably not worth the effort and money to do it live – but you could experiment on a private host with Proxmox or a cluster. ---------------------------------------------------------------------- ElevenNotes · 2023-10-02 18:46 Use podman instead of docker. ---------------------------------------------------------------------- haroldp · 2023-10-02 18:52 Sounds like a great setup! What's your DB backup retention policy? Do you just overwrite the last dump with the latest or do you retain the daily dumps for a month or whatever? It's no fun if your DB gets mangled and you only have a copy of a mangled DB. You mentioned monitoring and that is the only thing missing that really jumps out at me. Well, monitoring and *alerting*. Is the site up and listening on 80/443? Does it respond with a 200 OK page? Fast or slow? How much drive space do you have left? Is the load average high? How many messages are in the mail queue? Is the DB fast? etc, etc.. ---------------------------------------------------------------------- kayson · 2023-10-02 19:25 >What's your DB backup retention policy? Do you just overwrite the last dump with the latest or do you retain the daily dumps for a month or whatever? It's no fun if your DB gets mangled and you only have a copy of a mangled DB. I keep the last 7 nightlies. Anything older than that wouldn't really be useful since it's a store that can't afford to lose a week's worth of orders. But it's checked daily so any issues would be noticed quickly. >Is the site up and listening on 80/443? Does it respond with a 200 OK page? Fast or slow? How much drive space do you have left? Is the load average high? How many messages are in the mail queue? Is the DB fast? etc, etc.. Good points. Any suggestions on what to use for monitoring? ---------------------------------------------------------------------- haroldp · 2023-10-02 19:37 On the backups I'd say consider keeping a weekly or monthly for longer if the space they use isn't pushing up against your storage limits. Sometimes DB corruption is subtle and you'll look like a hero if you can go back three months to when, say, a thousand products started losing a shipping option each time they were updated, due to a bug in the code. I use Nagios for monitoring/alerting because it's easy for me to monitor absolutely anything. New users find Nagios very crusty though. Whatever you go with, I'd encourage you to monitor more than, "does the page load?" "We haven't been sending out emails to customers for three days" is an ugly conversation to have to have. "RAM usage is spiking" is nice to see before it becomes a performance problem. etc. ---------------------------------------------------------------------- kayson · 2023-10-02 19:41 >Also don't allow root login. Hmm that's a good point. The image I used only creates a root user, which I've been using over ssh. I did think about making a non-root user with sudo access, but then it occurred to me that maybe it's not that much more secure. Guess you'd have to type the password in so it should be. But I also have an ansible user with passwordless sudo (which I need for automatic deployment) >Don't rely on firewall alone. Also check what is actually listening on the public interface (on both TCP and UDP). It should only be 80, 443 and the SSH port on TCP, nothing else. If there's a service you actually need to be running, make it listen on a private interface or on UNIX socket if it supports that instead of TCP/UDP. If it's a service that doesn't need to run, don't run it. Good call. I only see the 3 ports. >These ones should not be on the same machine that serves the live site. Code is on a separate server. >You need to get that off-site. And the dumps should be secured in the meantime. db dumps will go to backblaze once I get that set up today. What do you mean "secured"? >Not sure what's your actual deploy procedure. Does something kick in when you commit to a branch? If yes, what does it do? For now it's manual since things are still in a somewhat beta state. When I update the code, I rebuild the image and push to my private registry. Then I run the ansible playbook. Longer term, though, the idea is that the docker image would get built automatically on app code changes, and the playbook would get run automatically on code changes or playbook changes. Related - what's the best thing to do wrt the ansible user? As mentioned above, I have it log in with a dedicated ssh key, and it has passwordless sudo. I could have a password, but then I'd have to store it somewhere for when I set up automatic deployments. I could add it to the ansible vault and to my gitea secrets which could then be used from the actions. >Are you checking the database dumps? You mean to make sure they're actually restore-able? Where's the place (meaning host) to do that? I could spin up another container to check it. >What about actually restoring the dumps them if you need to, what's the procedure there? Backblaze may not be so eager to actually give you your backups back... manual for now. I've never had issues with backblaze. The download cost is minimal since I'm Do you have a test suite that you run after each deploy? If yes, how do you handle failures? Not automatically. For now I test in a dev server, then if everything looks good just push to prod. Not sure how I would implement this. ansible? >How do you handle changes to live database data and/or structure? Do you use migrations? This is handled by the app framework. It uses migrations. I'd probably do any deployments including new framework versions manually just to be careful. >If the app has static assets and/or generates static content you'd normally set up a CDN. Cloudflare can provide combined CDN/reverse tunnel solutions, but they want you to use their nameservers. Bunny.net is another good \[almost\] free service (they offer DNS too but it's not necessarily tied to the CDN). There's not a lot of static content, nor a high peak traffic, so I'm not super worried about this. The reverse proxy is set up to send all the cache headers, so assets won't get transferred frequently. ​ Many thanks for the detailed response! ---------------------------------------------------------------------- kayson · 2023-10-02 19:42 Yeah I probably need a proper db backup container instead of the cron job I'm currently using... I'll take a look at nagios. Thanks for the suggestions! ---------------------------------------------------------------------- GolemancerVekk · 2023-10-02 21:04 > db dumps will go to backblaze once I get that set up today. What do you mean "secured"? I mean that if they sit in /tmp chmod 0666 while they're being uploaded to Backblaze it's not secure. You have to put yourself in the mindset of an attacker with access to your host machine. They should be owned by a non-priviledged user and access to that dir/files should be restricted to the user who's making the backups and/or copying them offsite. > Related - what's the best thing to do wrt the ansible user? As mentioned above, I have it log in with a dedicated ssh key, and it has passwordless sudo. I could have a password, but then I'd have to store it somewhere for when I set up automatic deployments. I could add it to the ansible vault and to my gitea secrets which could then be used from the actions. I'm not familiar enough with ansible and my deploy procedure is also different – I build the Node app into the Docker image so deploy means rebuilding the image as an unprivileged user then redeploying the container. If I want zero downtime I use two containers with a proxy in front and deploy them sequentially. But yeah, using the vault sounds sensible. > You mean to make sure they're actually restore-able? Where's the place (meaning host) to do that? I could spin up another container to check it. Depends on your purpose – simply verifying the dump vs being able to recover the live app if needed. You can load a dump into an offsite database and run some tests on the data that would satisfy you that it looks like actual live data. At the very least the attempt to load into db should tell you if it's a real db dump or a zero-length file because you ran out of disk space. If you simply attempt a dump then send whatever comes out to Backblaze and never see it again you may have some nasty surprise later. > For now I test in a dev server, then if everything looks good just push to prod. Not sure how I would implement this. ansible? Yes, the deployment mechanism (whatever it is) should run a test suite and not proceed with actual live deployment if it's not all green. Assuming you already write unit tests on dev it could be as simple as running those tests on the live-like environment. In my case it's a step in the docker image build process. It gets more complicated if they're integration tests, for that you'd need to get up a deploy that replicates the live one (complete with multiple containers if that's what your architecture is like) but not the actual live one. People who do CI/CD keep around a set of non- live-but-very-similar infrastructure for this specific purpose, typically called "staging". It's still not an 100% guarantee that staging will behave identically to live (db data being different and so on) but it's as close as possible. You *could* replicate live db data to staging in theory but it's not generally done because it would take a long time for large/complex apps. Also, taking live data out of the live environment could come with legal or regulatory issues – this also affects backups btw but those are usually covered under archival which is often required for compliance anyway so it's acceptable as long as specific procedures are followed. ---------------------------------------------------------------------- kayson · 2023-10-02 21:23 Thanks! ---------------------------------------------------------------------- DivideBackground7580 · 2023-11-01 02:23 It looks like you've covered a lot of important bases for deploying and securing your self-hosted website. Monitoring is indeed a crucial aspect to consider. As for my personal experience, I've found that Smart Proxy is a handy solution for managing proxies and IPs in a secure and efficient way. It's user-friendly and has worked well for me in various projects. However, the choice of proxy service can vary depending on a specific needs, so make sure to do some research to find the best fit! ---------------------------------------------------------------------- OkShopping2034 · 2023-11-09 06:24 Don't forget about regular software updates and patch management. Keeping everything up to date is crucial for security. Also, consider setting up a Web Application Firewall (WAF) for added protection against web-based attacks. ---------------------------------------------------------------------- kayson · 2023-11-10 17:11 Thanks! Any suggestions for WAF options? At the moment I just have nginx with fail2ban