Using Gitea (selfhosted) to do daily backups of other selfhosted servers. reddit / selfhosted ---------------------------------------------------------------------- thisiszeev · 2023-11-18 15:02 Hi All I have a selfhosted Gitea server, and I use it for a lot more than coding. I even do management of document history for my business. I love it. What I would like to do, is use it backup specific folders on other servers in my homelab. Say for example my webdev test server: I would like to daily back up /etc/ /var/www/one.example.com/ /var/www/two.example.com/ etc etc Now my knowledge on Gitea, and Git as a whole, is relatively limited to clone, add, commit, push and pull. If I setup a user for the server, then insert the ssh pub key. I would like to know, how from the terminal (via SSH to the server), I can create a new repo for folder /var/www/one.example.com/ and then do an initial commit, so that the .git folder is created locally in /var/www/one.example.com/.git/ Then I can set a cronjob to do my daily backups, but still have the magic of full file history. Also, can you configure a Repo to only keep changes back for say 90 days? (Space saving in the long run). I know there are a lot of ways to do this, but I have a very good reason for using Git, mainly, it streamlines restoring files at any point in history, and also if I need to fork a website I am developing, I can do it in Git with ease. Plus it allows me to add other users to a repo for example, and allows us to do branches etc. Currently I am backing everything up using a script I wrote, and I have a dedicated bare metal that is handling that. I get a .tar.gz for the last 7 days, the last 5 sundays and the last 3 months (1st). But this is starting to take up a lot of harddrive space. Any advice would kindly be appreciated. ---------------------------------------------------------------------- ElevenNotes · 2023-11-18 15:53 Git sure can be used to "backup" your environment like this, especially since tracking the changes in your files might be useful to you. You simply create a new project with git init and then work till you are done and git commit your changes, then git push your changes to your gitea. This can also be done automatically every n-interval. Since you already host gitea, have you ever thought about using vscode server to develop your applications? It makes the whole process very easy in every single aspect, including backups. ---------------------------------------------------------------------- thisiszeev · 2023-11-18 15:56 It have vscode, but I do a lot of my work on the server via ssh. Tweak here tweak there and I find vscode to bulky for that. So if I have a folder, /var/www/one.example.com/ with a bunch of files in it, am I right in saying that I can navigate to that folder and run git init and that will create a repo for that folder? ---------------------------------------------------------------------- ElevenNotes · 2023-11-18 15:57 vscode **server** ---------------------------------------------------------------------- thisiszeev · 2023-11-18 15:57 I am also intending to do this for all my live sites as well. ---------------------------------------------------------------------- thisiszeev · 2023-11-18 15:58 Doesn't play well with ISPConfig which I use for all my production servers. ---------------------------------------------------------------------- ElevenNotes · 2023-11-18 16:00 What? vscode server is vscode in its own container, it can do anything you can do from any shell. ---------------------------------------------------------------------- thisiszeev · 2023-11-18 16:02 Oh, I should revisit the container thing. I tried a bare metal install on top of ISPC and ISPC stopped working. ---------------------------------------------------------------------- ElevenNotes · 2023-11-18 16:03 Use containers for everything you do. As a developer you should absolutely love containers because they basically solve all our problems. ---------------------------------------------------------------------- thisiszeev · 2023-11-18 16:06 TBH I tried to get my head wrapped around it before and I failed miserably. But in another post I did recently this dude explained a lot of the frills that docker offers such as building a network in containers. I now have it on my to-do list to study it in depth. ---------------------------------------------------------------------- redditorforthemoment · 2023-11-18 16:38 This may be easier to automate with an Ansible playbook add the files to the gut repo and push them nightly at a specific time. I do this for my network infrastructure at home, Ansible will SSH to each switch, push the config to my Ansible host, then run a hit add / commit / push and only upload the changes. You could do this without Ansible using a simple cronjob, but you will run in to limitations pretty quickly if you want to do this for all of your live sites ---------------------------------------------------------------------- ThaCrrAaZyyYo0ne1 · 2023-11-18 18:03 Can I use this method with other people github repos? ---------------------------------------------------------------------- geek_at · 2023-11-18 19:55 You can use gitea actions to run any command on any server you like. Starting backups, running git pull or push.. endless possibilities ---------------------------------------------------------------------- knaak · 2023-11-18 19:56 I set up a git repo in AWS Code Commit and then I setup a script on a backup VM that I have an external HDD mounted to. I have a cron job which pulls from gitea and then pushes to CodeCommit via a remote server in git. That gives me a backup of git on the external hdd and externally via aws plus my gitea is backed up twice a week to my truenas. ---------------------------------------------------------------------- AudioOmen · 2023-11-18 20:22 Hope you are not doing that for binary files. It can become huge really fast. This is also not a good option for text files that change it's structure or content all the time. It's better to use backup solutions for backups, and version control systems for it's purpose. ---------------------------------------------------------------------- redditorforthemoment · 2023-11-18 20:43 If you have write permissions on their repo, sure. Ansible isn’t really the limitation there, it comes down to your git permissions ---------------------------------------------------------------------- lilolalu · 2023-11-18 21:18 git-annex allows managing large files with git, without storing the file contents in git. It can sync, backup, and archive your data, offline and online. Checksums and encryption keep your data safe and secure. Bring the power and distributed nature of git to bear on your large files with git-annex. git-annex is designed for git users who love the command line. For everyone else, the git-annex assistant turns git-annex into an easy to use folder synchroniser. https://git-annex.branchable.com/ ---------------------------------------------------------------------- ThaCrrAaZyyYo0ne1 · 2023-11-18 21:27 Got it! Thanks for replying bro! ---------------------------------------------------------------------- thetredev · 2023-11-19 22:35 Could setup a CI (continuous integration) pipeline for that. Basically you can setup a pipeline to checkout your git repository and use that code to do all kinds of automated jobs. Once your pipeline is completely setup, you can schedule it to run each night, each week, etc. Not sure how well gitea supports CI but GitLab does through GitLab Runner instances. Pretty easy to setup using Docker. While GitLab is supposed to be able to handle thousands of users by default (at least 4 vCPUs and 8 gigs of RAM), you can configure it to not use as many resources (2 vCPUs and 2 to 4 gigs of RAM): https://docs.gitlab.com/omnibus/settings/memory_constrained_envs.html But before you replace your gitea instance, read up on CI and see if that's your solution of choice. If so. check how you could get that running using gitea. ---------------------------------------------------------------------- thisiszeev · 2023-11-20 06:08 I am very very happy with Gitea. I think it does support CI/CD. I never knew what that was and have never bothered to look it up. But now you have sparked a small flame of interest. ---------------------------------------------------------------------- thisiszeev · 2023-11-20 06:17 I did work through the night learning how to use the "poorly" documented API for Gitea. I am now going to offer my services to that project to add more explanation to the documentation, as I am good at Technical Documentation. I now know how to use the API to create a repo, check if a repo exists, and delete a repo, amongst other things. I have made lots and lots of notes. My next step is to write code, that will automatically detect new "projects/sites/etc" on a specific server setup, ISPConfig to start, and others to follow (maybe once I do my first release, other people will contribute). So the idea is, instead of doing a rsync and then a daily/weekly/monthly tar.gz, my code will detect new websites on the server, create a repo, mount --bind the web folder, log folder, vmail folder, and also /etc/ and any other system folders you choose, inside that repo folder. It will dump the sql databases for each site, and save them in the relevant sql folder for each site. Then it will do an add, commit -m "$(date)", followed by push. I will create a post here for others who want to use it. Another advantage here is that if you need to deploy a site to a second server, you can easily do a git clone. Yes, it will pull down vmail and log as well, and you should run your db on a dedicated server in such cases, but it will allow for faster (re)deployment, than uploading a tar.gz and extracting. But for me, the main saving is space. Currently I have 7 days, 5 weeks and 6 months. This is taking up almost 2TB per server, as almost every file in each tar.gz is a carbon copy. ---------------------------------------------------------------------- thetredev · 2023-11-20 12:25 Sounds like you've got a plan there. Have fun! ---------------------------------------------------------------------- thetredev · 2023-11-20 12:36 There are many definitions for CI, and one definition for CD. Some say CI is about continously working on a certain git branch to "continuously integrate code as fast/often as possible", others define it as a way of testing feature branches via automated pipelines as an additional step before merging code into the main branch. Or something in between these two. But "CI/CD pipelines" put together are an (automated) cycle of development (code, commit, push) - testing - reviewing code - reviewing outcome in a testing deployment - accepting or rejecting changes, repeating that process until a feature is complete, then deploy to production. That one works best for web applications because you don't have to ship a device etc. But I barely touched the surface here, if you're interested you can read up on DevOps as a culture and style of working. ---------------------------------------------------------------------- thisiszeev · 2023-11-20 16:16 Dude... you just answered my prayers... this is definitely what I need. This will make my life so much better... not so much for the project my OP is for, but for all my other work... ---------------------------------------------------------------------- thisiszeev · 2023-11-20 16:19 Taken from the documentation for the version of Gitea I am using >Starting with Gitea 1.19, Gitea Actions are available as a built-in CI/CD solution ---------------------------------------------------------------------- thetredev · 2023-11-20 18:28 Guess I'll have to check it out too some day. GitLab is kinda overkill for one person :D ---------------------------------------------------------------------- thetredev · 2023-12-03 21:55 I tried Forgejo 1.21 today. Runners and docker-in-docker worked like a charm after the usual certificate problems I also run into when setting up GitLab, but that's all because of my infrastructure and has nothing to do with the tools themselves. I must say, it's really fast and even the pipelines run extremely fast. I also like the workflow definitions more than GitLab's. But I have not yet figured out how to include other workflow definitions into another workflow. GitHub has composite workflows or workflow dispatch or whatever for that, but that's not really what I'm looking for. In GitLab you just say "include this yml file from that repo located in that path" and that's it. Which is much more user friendly in my opinion. Maybe GitHub, or the Act runner (which Gitea and therefore Forgejo use under the hood) will implement that feature better in the future. This is the only thing that made me stop my testing and not consider switching (yet). If those includes get figured out, I'm happy to switch over, since I'm not using all the enterprise-grade features that you're looking for in GitLab anyway. ---------------------------------------------------------------------- thisiszeev · 2023-12-03 23:03 I am planning to migrate the Gitea installation from here at my place to a cloud server as with the constant loadshedding here in South Africa, https://youtu.be/Iiny1GrfhYM , clients are complaining that they often don't have access to the work I have done. I will definitely look into Forgejo before I do the migration. ---------------------------------------------------------------------- thisiszeev · 2023-12-04 06:06 Took a gander... looking good. If Gitea is Git with a cup of Tea, then Forgejo is Git with a 2 ltr mug of coffee. ---------------------------------------------------------------------- thetredev · 2023-12-04 14:39 Really? I thought it was the same, just without "company bloat and company politics" behind it. I like it more than Gitea in general as a tool, didn't really compare the features 1 to 1. ---------------------------------------------------------------------- thetredev · 2023-12-04 19:48 Glad to hear that! ---------------------------------------------------------------------- thetredev · 2023-12-04 21:00 I'll try [https://docs.github.com/en/actions/using-workflows/reusing- workflows](https://docs.github.com/en/actions/using-workflows/reusing- workflows) and see where this goes.