[HN Gopher] Discord Indexes Trillions of Messages
       ___________________________________________________________________
        
       Discord Indexes Trillions of Messages
        
       Author : todsacerdoti
       Score  : 23 points
       Date   : 2025-04-24 19:13 UTC (3 hours ago)
        
 (HTM) web link (discord.com)
 (TXT) w3m dump (discord.com)
        
       | lysace wrote:
       | LLM training input treasure? Signal-to-noise ratio is going to be
       | lowish though.
        
         | DrillShopper wrote:
         | If you want to train a groomer then Discord messages are the
         | way to do it.
        
           | lithos wrote:
           | If we're acting like Redditors: "you can just block Minecraft
           | Discords, and remove most of the grooming"
        
         | YetAnotherNick wrote:
         | I think discord's data has insane value because it has real
         | time reasoning steps much more than any big social media.
         | Obviously you might want to filter out low information messages
         | like "Hello" but model based filtering for signal is already
         | solved more or less.
         | 
         | Discord has lot of very technical channels like the one which
         | solved BB(5) after decades of research.
        
           | leo-notte wrote:
           | Agreed. discord has a unique structure with threaded
           | conversations, context carryover, and back and forth
           | reasoning that you don't get from places like twitter or even
           | reddit. it's especially useful for training models on
           | collaborative problem solving or exploratory dialogue.
           | filtering is a challenge but definitely solvable with current
           | tools.
        
       | coldblues wrote:
       | Indexes trillions of messages and refuses to delete any of them
       | and comply with GDPR. Deleting your account DOES NOT delete your
       | messages, they remain on the platform for an indefinite period of
       | time. The only way to get them to delete your messages is to go
       | through this process
       | (https://discordomicon.github.io/removal/overview/) and you're
       | still likely to go through several support tickets before they
       | comply with your request.
        
       | JackSlateur wrote:
       | Tldr: they used a single massive elasticsearch cluster. Now, they
       | are using multiple elasticsearch cluster, running in k8s. And
       | redis has been replaced by pubsub (GCP).
       | 
       | Since sounds like expensives design but ok, I guess
        
       ___________________________________________________________________
       (page generated 2025-04-24 23:02 UTC)