Post B2LKpCAdp2GOWw0u6C by anthropy@mastodon.derg.nz
 (DIR) More posts by anthropy@mastodon.derg.nz
 (DIR) Post #B2LKpA9VKIAKGpSy36 by anthropy@mastodon.derg.nz
       2026-01-16T12:18:43Z
       
       0 likes, 0 repeats
       
       I sometimes see people saying "you can't train LLMs on the output of LLMs"There's a lot to be said here. But I do want to say that it's actually fine, and done quite a lot, but with some extra contextual bits (like labels), going from larger to smaller models. We call this process "distillation".It does have to be done carefully, because without proper labeling it can derail a model, but a lot of things can derail a model tbh 😅
       
 (DIR) Post #B2LKpCAdp2GOWw0u6C by anthropy@mastodon.derg.nz
       2026-01-16T12:24:22Z
       
       1 likes, 0 repeats
       
       training an LLM properly is really hard, but not for the reasons you might expect.You want these models to 'generalize', that is, you do NOT want them to just regurgitate the text they're given, you want them to create parameters that build an internal model of the world, giving them a functional understanding of concepts beyond just the words.This allows them to predict the logical consequences of a sentence, rather than just the next likely word.