Self-hosted File Data Storage Solution reddit / selfhosted ---------------------------------------------------------------------- Estavenz · 2023-12-19 03:50 Currently, I have many thousands of files that I am looking to manage, and I am looking for a service to help me manage them. In order to help clarify what I am asking for, I'll explain the problem in more detail. There are three steps associated with adding to this collection: 1. **Collection**: This involves the act of collecting the files, which could include downloading them, creating them, or finding them offline. 2. **Storage**: How are the files stored? What data is stored alongside them? Note that self hosted data is a required option. 3. **Management**: Once the files and associated data are stored, how can you manage them? A self hosted service would be ideal. There exist many solutions for the Storage & Management steps, including OS built-in tools such as Windows File Explorer. These solutions are usually called *Digital Asset Management* (DAM) Services. There also of course exist many separate solutions for the Collection step, such as web scraping tools like [Gallery- DL](https://github.com/mikf/gallery-dl) or [yt- dlp](https://github.com/yt-dlp/yt-dlp) However, as many of you probably have encountered, it can be difficult to find solutions that work together well. Even if you do find one that works, it often heavily limits your flexibility. For example, say you want to incorporate tags as metadata associated with files for querying purposes. Many DAM services allow this functionality, but what happens if you suddenly want to switch DAM services? It may become remarkably difficult if the tags are stored in a confusing proprietary database that you'll have to parse through to get the metadata back out. Or what if you have tags that you parsed using a web scraping tool that you want to import? Some DAM services don't easily allow this and don't let you automate it (no API or access to the database), and as such you may find yourself going through the process of either automating the input yourself or manually inputting each tag (yikes). What if you also have files from different data streams such as some parsed from online and some from offline? You would have to put it all into the DAM and forget about any different metadata those files might have. Forget about having a local copy you can look through as well in most cases. Many DAMs also don't support some file types. There are also issues associated with functionality such as tagging. There is no universally accepted standard for what tags exist and what format they should be in. The closest I've found is the [Public Tag Repository](https://hydrusnetwork.github.io/hydrus/PTR.html) or [Danbooru's Tags (NSFW)](https://danbooru.donmai.us/tags). Note that the PTR is probably the closest as I cannot find a way to have access to Danbooru's tag database. Its database is also a nightmare with around 64 tables in their database and everything in SQL which requires a migration to change the schema of. As such, if you want true flexibility you have to find a way to separate the steps and be able to interchange the solutions you use for each. For collection, you should be able to use any method and scrapers that you want. For storage, files need to be stored in a universally accepted way along with their metadata that is stored either in an associated file or a database. Lastly, the management software needs to be completely separate, meaning you can switch DAMs as you please simply through importing and exporting your data. If you make a change to a file in a DAM service, it needs to be reflected in your storage solution. Note that there are solutions that try to cover all three steps, such as [Hydrus Network](https://github.com/hydrusnetwork/hydrus) which is a robust solution that acts as a DAM and also as a downloader with users being able to make custom downloaders. However, I've found myself using different downloaders for unsupported sites and struggling to find a way to automate the importing of tags. It also gets difficult when the biggest community made downloaders (e.g. for Danbooru) do not parse everything you want. It also does not support all of the functionalities that someone may want such as easily seeing file children/parents/siblings. The only real solution I can think of for this is to store files in a universal file system (simple hierarchical file system such as exFAT) and to store metadata associated with each file's hash in a database that can be flexibly updated to reflect data stored or created in DAM services (exporting files). This means JSON or similar is likely the best option. This solution would also require custom importers and exporters to be made for each DAM service to allow full flexibility. The exporter would need to be able to sync with the universal database. Creating this kind of solution would be difficult and I would like to avoid this if possible. Not to mention if your DAM uses machine learning and likely uses vector databases which would be kinda hard to integrate into JSON. My question is if there any tools or standards that accomplish this without needing to create this on my own. Are there existing open source self hosted services that meet this kind of flexibility? Is there an existing standard on what file data there is? tldr: Are there any open source & self hosted Digital Asset Management services that allow for true flexibility (i.e. switching DAM services or for storing different kinds of metadata)? Are there any standards to know how to store files? Any help is appreciated. Please feel free to recommend DAMs as I am also looking for better services. [Hydrus Network](https://github.com/hydrusnetwork/hydrus) is currently my overall choice. If you want to look at criteria that some others have for robust DAMs, [this](https://softwarerecs.stackexchange.com/questions/47756/digital- asset-management-for-photography) is a good post. ---------------------------------------------------------------------- adamshand · 2023-12-19 04:40 You might have better luck over at https://www.reddit.com/r/DataHoarder/ ---------------------------------------------------------------------- Estavenz · 2023-12-19 04:43 I also posted over there, I just wanted to see if you guys knew of any self-hosted services ---------------------------------------------------------------------- Eirikr70 · 2023-12-19 08:54 I'm sorry I didn't take the time to read your post until the end. Maybe [Cozy](https://cozy.io/en/) meets your needs. ---------------------------------------------------------------------- ItsPwn · 2023-12-19 21:08 Wold You consider running a baremetal software on a spare computer or running a virtual machine with storage ? I could propose , guide through how to make Synology working in home enviroment without buying overpriced hardware ---------------------------------------------------------------------- Estavenz · 2023-12-20 00:50 Is Synology's File Explorer nice? Do they have a dedicated DAM with functionality? I do already have a Debian NAS set up and all my devices are connected through Tailscale so I can access most self- hosted DAMs from anywhere. I didn't know Synology had more file management features than Windows, I'll have to check it out in a VM. Thanks for the suggestion ---------------------------------------------------------------------- Estavenz · 2023-12-20 00:55 It does seem like a nice cloud solution, I'll check it out to see if the file management features meet my needs. Thanks for the suggestion