[HN Gopher] Processing 40 TB of code from ~10M projects with a s...
       ___________________________________________________________________
        
       Processing 40 TB of code from ~10M projects with a server and Go
       for $100 (2019)
        
       Author : KolmogorovComp
       Score  : 38 points
       Date   : 2022-10-03 19:59 UTC (3 hours ago)
        
 (HTM) web link (boyter.org)
 (TXT) w3m dump (boyter.org)
        
       | [deleted]
        
       | layer8 wrote:
       | Average number of Makefiles is 1518.1769808494607? That can't be
       | right?
        
       | layer8 wrote:
       | It would be interesting to plot some of the statistics by age of
       | repository.
        
       | thethirdone wrote:
       | Does anyone know why "pawn" is considered an offensive word? It
       | seems likely it ranks highly on the offensive word list because
       | of chess engines.
        
         | ok_dad wrote:
         | I also noticed "knob" and "xxx" near the top of that list (and
         | others) for which I understand why they are considered
         | "offensive" in some contexts, but in code those are quite non-
         | offensive (unless, of course, the context is offensive, haha).
        
         | yaddaor wrote:
         | Phonetically it sounds like "porn".
        
       | throwawaygog6 wrote:
       | >>> If someone wants to host the raw files to allow others to
       | download it let me know. It is a 83 GB tar.gz file which
       | uncompressed is just over 1 TB in size.
       | 
       | Anyone knows if it is possible to download similar data set for
       | youtube and reddit? I have ideas for search engine based on it,
       | but I don't want to write/maintain scraper scripts.
        
         | testesttest4 wrote:
         | I have most of the historical reddit data except ~year I use to
         | train ML models. Let me see if I can find a public link for
         | you...
        
       | [deleted]
        
       | KolmogorovComp wrote:
       | Oops linked to an anchor in the body and cannot edit. Correct
       | link: https://boyter.org/posts/an-informal-survey-of-10-million-
       | gi...
        
       | jwilk wrote:
       | Previously:
       | 
       | https://news.ycombinator.com/item?id=21121735 (80 comments)
        
       ___________________________________________________________________
       (page generated 2022-10-03 23:01 UTC)