AI Model Storage?
reddit / selfhosted

----------------------------------------------------------------------
loadgalax · 2023-12-25 23:07

i need something to storage a large amount of models like
[Civitai](https://civitai.com) but lightweight

I try [Gitea](https://gitea.com) with [LFS](https://git-lfs.com) and
deleting the old commits with
[This](https://clearinsights.io/blog/how-to-remove-all-git-commit-
history/) the LFS files remain in the Gitea Server so useless files
taking space any Selfhosted alternative ?

----------------------------------------------------------------------
Dazzling_Ad1507 · 2023-12-26 01:55

So a frontend? or just raw storage? You can look at filebrowser for
just plain model storage on a HDD or SSD; or a more sophisticated
option like: https://dvc.org/

----------------------------------------------------------------------
loadgalax · 2023-12-26 02:22

Ty for the response I will try that in a sec and I'm searching for a
frontend I was thinking in using Minio for the storage but a don't
know any frontend for Minio/s3 and the web ui is not useful for me
because if I search for a file I want more info about them if is a
satable diffusion model, lora, text model, etc

----------------------------------------------------------------------
ChewbiScornu · 2023-12-26 21:17

Hi ! I've been working in the field for a while now. To answer the
storage question, most of what is done right now is S3. You'll find
some people with peculiar use cases wanting to put models in their
Hadoop ecosystem or something else, but it's mostly S3. Models are
mostly set of relatively small to medium files, so you'll have your
best experience with that technology.

As for minio it's an excellent solution for that ! It's easy to
maintain, scales well and it comes with most of the features you want
to achieve. It also has a pretty ok UI, but as you said it won't help
you much with your tagging problem. If minio ( just like AWS S3 ) does
allow you to add metadata to your files, the UI won't help you much to
sort through those. So I would advise you to use your object path as a
way to filter your models.

The comment above also recommended DVC who does some lifting for you
in terms of integration, but it's basically something that you run
above S3 ( or some other storage solution ) to integrate your models
with git. You won't find any "catalog" solution there.

If you feel a bit courageous, you can try to integrate clearml who
offers a solution for experiment tracking AND model management. It's
really easy to use, but it has dependencies to mongo, redis and
elastic. Which renders it a bit overkill for small self hosted setups.

TLDR: Stick to minio for now.

----------------------------------------------------------------------
loadgalax · 2023-12-26 23:42

Im testing Minio as a LFS Backend in gitea and same problem LFS not
deleting
surely im missing something but until i figure out im stuck with
Minio with cli :(