Image moderation for gore / violent content
reddit / selfhosted

----------------------------------------------------------------------
ponytroutlover · 2023-10-14 12:54

Hi,
I have created a webapp where users can post images, but I do not want
to allow violent/gore content. I know there are many API alternatives,
but I couldn't find any self hosted github library/project to detect
violent content (All I found was NSFW filters).

Do you if a viable self-hosted solution exists today ?

----------------------------------------------------------------------
Shmiggles · 2023-10-14 13:12

User-Generated Content is an increasingly regulated space. Under the
UK's soon-to-be-promulgated Online Safety Bill, any Internet service
that is accessible to UK users and is deemed in scope by the Office
for Communications will be subject to onerous regulations under threat
of hefty penalties. The EU is working on similar legislation, and
other Western powers will be soon to follow.

The big social media companies - Meta, X, Reddit, and so on - have
massive teams of human moderators, because automating content
moderation is not yet possible. An early iteration of this you can
read up on is the Scunthorpe Problem. While that was solved with
better regexes, a similar problem, the Forbidden Meatball Problem,
persists on most platforms, including Reddit. That's just text,
without the added complications of interpreting images.

Your possible ways forward are:

* limit usage to people you know and trust personally; or
* moderate all submissions yourself until you can afford to employ a
team to do it for you.

----------------------------------------------------------------------
lilolalu · 2023-10-14 14:21

Of course it's possible to flag potential problematic content vis AI,
this is offered by paid services from azure i.e. but I am sure there
are OpenSource image Moderation models. The question is how many
things they will falsely flag and how much they will let pass. The
tools exist though.

----------------------------------------------------------------------
lilolalu · 2023-10-15 08:04

https://github.com/topics/violence-detection