[HN Gopher] Something weird is happening with LLMs and chess
___________________________________________________________________
Something weird is happening with LLMs and chess
Author : crescit_eundo
Score : 57 points
Date : 2024-11-14 17:05 UTC (5 hours ago)
(HTM) web link (dynomight.substack.com)
(TXT) w3m dump (dynomight.substack.com)
| PaulHoule wrote:
| Maybe that one which plays chess well is calling out to a real
| chess engine.
| singularity2001 wrote:
| this possibility is discussed in the article and deemed
| unlikely
| margalabargala wrote:
| I don't see that discussed, could you quote it?
| probably_wrong wrote:
| Note: the possibility is not mentioned in the article but
| rather in the comments [1]. I had to click a bit to see it.
|
| The fact that the one closed source model is the only one
| that plays well seems to me like a clear case of the
| interface doing some of the work. If you ask ChatGPT to count
| until 10000 (something that most LLMs can't do for known
| reasons) you get an answer that's clearly pre-programmed. I'm
| sure the same is happening here (and with many, many other
| tasks) - the author argues against it by saying "but why
| isn't it better?", which doesn't seem like the best argument:
| I can imagine that typical ChatGPT users enjoy the product
| more if they have a chance to win once in a while.
|
| [1] https://dynomight.substack.com/p/chess/comment/77190852
| sobriquet9 wrote:
| This is likely. From example games, it not only knows the rules
| (which would be impressive by itself, just making the legal
| moves is not trivial). It also has some planning capabilities
| (plays combinations of several moves).
| pseudosavant wrote:
| LLMs aren't really language models so much as they are token
| models. That is how they can also handle input in audio or visual
| forms because there is an audio or visual tokenizer. If you can
| make it a token, the model will try to predict the following
| ones.
|
| Even though I'm sure chess matches were used in some of the LLM
| training, I'd bet a model trained just for chess would do far
| better.
| ChrisArchitect wrote:
| [dupe] https://news.ycombinator.com/item?id=42138276
| digging wrote:
| Definitely weird results, but I feel there are too many variables
| to learn much from it. A couple things:
|
| 1. The author mentioned that tokenization causes something
| minuscule like a a " " at the end of the input to shatter the
| model's capabilities. Is it possible other slightly different
| formatting changes in the input could raise capabilities?
|
| 2. Temperature was 0.7 for all models. What if it wasn't? Isn't
| there a chance one more more models would perform significantly
| better with higher or lower temperatures?
|
| Maybe I just don't understand this stuff very well, but it feels
| like this post is only 10% of the work needed to get any meaning
| from this...
| azeirah wrote:
| Maybe I'm really stupid... but perhaps if we want really
| intelligent models we need to stop tokenizing at all? We're
| literally limiting what a model can see and how it percieves the
| world by limiting the structure of the information streams that
| come into the model from the very beginning.
|
| I know working with raw bits or bytes is slower, but it should be
| relatively cheap and easy to at least falsify this hypothesis
| that many huge issues might be due to tokenization problems
| but... yeah.
|
| Surprised I don't see more research into radicaly different
| tokenization.
| cschep wrote:
| How would we train it? Don't we need it to understand the heaps
| and heaps of data we already have "tokenized" e.g. the
| internet? Written words for humans? Genuinely curious how we
| could approach it differently?
| viraptor wrote:
| That's not what tokenized means here. Parent is asking to
| provide the model with separate characters rather than
| tokens, i.e. groups of characters.
| DrNosferatu wrote:
| What about contemporary frontier models?
___________________________________________________________________
(page generated 2024-11-14 23:00 UTC)