[HN Gopher] Ask HN: What does the data engineering behind LLMs l...
       ___________________________________________________________________
        
       Ask HN: What does the data engineering behind LLMs look like?
        
       I've seen a lot of discussion about key aspects of LLMs like ML
       (research, architecture), Infrastructure (GPUs, Cloud), and Product
       (ChatGPT et al) but not much on the data engineering side. A lot of
       hand waving like you "just" train on the entire public Internet.
       There must be a ton of complexity here, as well.  What is the
       difference between web scraping and crawling? They are not simply
       indexing websites, these systems must be extracting and storing
       vasts amount of data from those crawled sites (hence Reddit,
       Twitter, etc calling foul). Do these systems rely on tons of proxy
       IPs?  There's probably not too much going on after ingestion beyond
       storing all this data as text or image in an optimal format for the
       training system(s) to use.
        
       Author : lostpharoah
       Score  : 7 points
       Date   : 2023-04-19 21:46 UTC (1 hours ago)
        
       | Closi wrote:
       | The other element here is quality content - you probably aren't
       | just training on public internet data for commercial LLMs,
       | hopefully you can train on scanned books too, and closed academic
       | journals, radio transcripts, photograph stores, map data,
       | codebases, technical documentation...
        
       ___________________________________________________________________
       (page generated 2023-04-19 23:03 UTC)