[HN Gopher] Inductive or deductive? Rethinking the fundamental r...
___________________________________________________________________
Inductive or deductive? Rethinking the fundamental reasoning
abilities of LLMs
Author : belter
Score : 90 points
Date : 2024-09-02 00:49 UTC (22 hours ago)
(HTM) web link (arxiv.org)
(TXT) w3m dump (arxiv.org)
| hydrox24 wrote:
| Is there a good reason to exclude abductive reasoning from an
| analysis like this? It's even considered by at least one of the
| referenced papers (Fangzhi 2023a).
|
| Abductive reasoning is common in day-to-day life. It seeks the
| best explanation for some (often incomplete) observations, and
| reaches conclusions without certainty. I would have thought it
| would be important to assess for LLMs.
| soferio wrote:
| https://en.wikipedia.org/wiki/Abductive_reasoning
|
| Is abductive inference synonymous with bayesian inference?
| keiferski wrote:
| Here's a good page to read on that:
|
| https://plato.stanford.edu/entries/abduction/#AbdVerBayConTh.
| ..
| randcraw wrote:
| I like to think of abductive reasoning as the basis for
| science that explains natural processes that happened in the
| past -- like astronomy and geology and evolution -- where
| experiments are too big to conduct or processes too slow to
| observe in real-time. So we propose mechanistic explanations
| for nonobvious outcomes like the formations of stars, or
| motion of large land mass via plate tectonics or glaciation,
| or long-range organism speciation over millennia. That's the
| role for abduction, to explain how all that happened.
| User23 wrote:
| No, but agreement with priors is one way one might choose
| between possibilities.
|
| For example suppose you go outside and the streets are wet.
| Perhaps it rained, or perhaps someone drove a fire truck
| around spraying water all over the streets. You might select
| the former because of its higher prior probability.
| refulgentis wrote:
| My instinct is it is a distinction without a difference in this
| context. i.e. if deductive is "I watched the cue ball hit the 8
| ball, therefore, the 8 ball is moving" and abductive is "the 8
| ball is moving towards me, therefore the cue ball must have hit
| it. I cannot claim to have deduced this because I did not
| observe it", LLMs cannot observe the situation, so any
| deduction (in the binary induction/deductive sense) must be
| done by abduction.
| jcims wrote:
| LLMs don't feel like a whole brain, they feel like the impulsive
| side that gets filtered by other processes.
| eightysixfour wrote:
| While there is sometimes an exaggeration of the differences, I
| have always found LLMs to behave like (and have many of the
| weaknesses of) the left hemisphere of the brain as described in
| books like "The Master and His Emissary." Considering the left
| is the language center, this shouldn't be surprising.
| AdieuToLogic wrote:
| > LLMs don't feel like a whole brain ...
|
| Because they are not.
|
| LLM's are a better form of Bayesian inference[0] in that the
| generated tokens are statistically a better fit within a larger
| context.
|
| 0 - https://en.wikipedia.org/wiki/Bayesian_inference
| stavros wrote:
| This strikes me as saying "Brains aren't brains, they're
| merely a collection of neurons that output some voltage in
| response to their input voltage".
| suprjami wrote:
| You're comparing the physical item with the reasoning
| capabilities of that item.
|
| A popular LLM joke at the moment is "How many Rs in
| strawberry?"
|
| You, a human being, understand this question. The purpose
| of the activity is to count the letter R in that word and
| give the one correct answer.
|
| LLMs don't understand any of that. They don't know what
| words are, what letters are, what numbers are, what
| counting is, that a question can have one definite correct
| answer or many answers, what a question is, nor what an
| answer is.
|
| They break your input into tokens and then look at the most
| likely set of output tokens given your input. That's all.
|
| If you train a model on enough sensible input tokens and
| sensible responses then the output will mostly seem human
| and sensible. But it's never reasoning.
| gaganyaan wrote:
| They very much do understand words, letters, etc. Why do
| you think otherwise? Because of one trick question?
| suprjami wrote:
| You are fooled into thinking that producing a set of
| output tokens which mimic actual human input and output
| tokens displays understanding. It doesn't.
|
| "How many Rs in strawberry?" is not a trick question.
| That's about as straight and factual as a question could
| be.
| og_kalu wrote:
| >"How many Rs in strawberry?" is not a trick question.
|
| To a system that sees letters and words, sure. To a
| system that doesn't, you're just asking a blind man to
| count how many apples are on the table and patting
| yourself on the back for his failure. It just doesn't
| make any sense to begin with.
|
| And this is before the fact that human intelligence is
| rife with absurd seeming failure modes and cognitive
| biases.
|
| It's frankly very telling that these token related
| questions are the most popular kind of questions for
| these discussions.
|
| This is no testable definition of reasoning or
| intelligence that will cleanly separate LLMs and Humans.
| That is the reality today. And it should make anybody
| pause.
| kortilla wrote:
| > you're just asking a blind man to count how many apples
| are on the table
|
| Which input is it you think an LLM doesn't have to be
| able to count the number of letters in a word you
| literally provide it
| isaacfung wrote:
| The text is converted to embeddings after tokenization.
| The neural networwk only sees vectors.
|
| Imagine the original question is posed in English but it
| is translated to Chinese and then the LLM has to answer
| the original question based on the Chinese translation.
|
| It's a flaw of the tokenization we choose. We can train
| an LLM using letters instead of tokens as the base units
| but that would be inefficient.
| daveguy wrote:
| By that definition the LLM literally does not see
| anything. LLMs predict tokens. That's it.
| isaacfung wrote:
| It sees embeddings that is trained to encode semantic
| meanings.
|
| The way we tokenize is just a design choice. Character
| level models(e.g. karpathy's nanoGPT) exist and are used
| for educational purpose. You can train it to count number
| of 'r' in a word.
|
| https://x.com/karpathy/status/1816637781659254908?lang=en
| gaganyaan wrote:
| The LLM sees tokens, and predicts next tokens. These
| tokens encode a vast world, as experienced by humans and
| communicated through written language. The LLM is seeing
| the world, but through a peephole. This is pretty neat.
|
| The peephole will expand soon, as multimodal models come
| into their own, and as the models start getting mixed
| with robotics, allowing them to go and interact with the
| world more directly, instead of through the medium of
| human-written text.
| gaganyaan wrote:
| Humans have many well-documented cognitive shortcomings
| and failure modes. Does that mean we don't reason either?
| suprjami wrote:
| We aren't talking about a human mind which operates in
| unexpected or ways which don't conform with society.
| Don't bring an appeal to neurodivergence to an LLM
| argument.
|
| We are talking about a computer program whose operation
| we understand, can observe, and can debug. Modern LLMs
| certainly take a lot of human effort to do this, but it
| can be done.
| gaganyaan wrote:
| This isn't about neurodivergence. Every single human
| suffers from a long list of cognitive shortcomings. We
| just call them things like "optical illusions" and find
| them interesting, but don't then go on to make silly
| claims like "humans can't reason".
| quantadev wrote:
| Intelligence is a vast spectrum of capabilities. There
| are people who function normally but cant't recognize
| faces. There are people who can't see or understand
| anything on their left side. Even savantism is an example
| where some brains have nearly superhuman capabilities in
| one area, but are almost mentally retarded in other
| areas, like common sense, and yes even letter counting.
| All the letter counting failure shows is that maybe all
| LLMs lean a bit more towards autistic brains than towards
| the actual average human brain.
| jcims wrote:
| Ask it this.
|
| "Please spell the word strawberry with a phonetic
| representation of each letter then tell me how many 'r's
| are in the word strawberry?"
|
| It's gotten the answer right every time I've tried.
| gaganyaan wrote:
| Another one is to just ask it to spell it out with dashes
| in between each letter. If anyone's wondering, it then
| gets it right because that changes the tokenization,
| changing how the model actually "sees" it.
| davnn wrote:
| ChatGPT answer: There are *two* "R"s in the word
| "strawberry."
| AdieuToLogic wrote:
| >> A popular LLM joke at the moment is "How many Rs in
| strawberry?"
|
| > ChatGPT answer: There are _two_ "R"s in the word
| "strawberry."
|
| Given enough instances of the "LLM joke" in a training
| data set, the joke itself having a consistent form
| (sequence of tokens) and likely followed by the answer
| having a similarly consistent form (sequence of tokens),
| the probability of the latter being produced as quoted is
| high.
| imtringued wrote:
| There is a lot of broken English on the internet and yet
| LLMs are better at English than the average native
| speaker. This failure mode has nothing to do with the
| training data.
| viraptor wrote:
| > They break your input into tokens and then look at the
| most likely set of output tokens given your input. That's
| all.
|
| That's humans as well, possibly. This description gets
| repeated over and over and is factually correct, but we
| still don't know if brains do anything more. The "it's
| not reasoning" may be true, but it doesn't follow from
| just that description.
| seanmcdirmid wrote:
| > They break your input into tokens and then look at the
| most likely set of output tokens given your input. That's
| all.
|
| That isn't right: the pre processor provides a lot of
| material on strawberries and counting r's, which is then
| pretending to the question...and then they predict the
| next sentence as an answer to the question. The model by
| itself doesn't know anything, it it just a statistical
| processor of context, just tokenizing the question and
| using the model to predict the answer would actually give
| you less than a wrong answer, it would be gibberish. It
| messes up on the question because the context it
| retrieves based on the question text isn't useful in
| producing the correct answer.
| imtringued wrote:
| >because the context it retrieves based on the question
| text isn't useful in producing the correct answer.
|
| "retrieves" is the wrong word. Each token (in GQA a small
| tuple of tokens is summarized into a single token),
| becomes an element in the KV cache. The dot product of
| every token with every other token is taken (matrix
| multiplication) and then a new intermediate token is
| produced using softmax() and multiplication by the V
| matrix. What the attention mechanism is supposed to do is
| combine two tokens and form a new token. In other words,
| it is supposed to perform the computation of a function
| f(a,b) = c. The attention layer is supposed to see "count
| r" and "strawberry" and determine the answer 3.
|
| Well, at least in theory. Given the combination "count r"
| and "r", it is practically guaranteed that the attention
| mechanism succeeds. What this tells us is that the
| tokenization of the word "strawberry" is causing the
| model to fail, since it doesn't actually see the letters
| on the character level. So it is correct to say that the
| attention mechanism does not have the correct context to
| produce the correct answer, but it is wrong to say that
| "retrieval" is necessary here.
|
| The reason why it doesn't make sense to label what is
| happening as "reasoning" is that the model does not
| consider its own limitations and plans around them. Most
| of the work so far has been to brute force more data and
| more FLOPS, with the hope that it will just work out
| anyway. This isn't necessarily a bad strategy as it
| follows the harsh truth learned from the bitter lesson,
| but the bitter lesson never told us that we can't improve
| LLMs through human ingenuity, just that human ingenuity
| must scale with compute and training data. For example,
| the human ingenuity of "self play" training (as opposed
| to synthetic data) works just fine, precisely because it
| scales so well.
|
| Instead of complaining so much about humans trying to
| "gotcha" the LLMs, what we really ought to build is an
| adversarial model that learns to "gotcha" LLMs
| automatically and include it in the training process.
| AdieuToLogic wrote:
| Large Language Model algorithms do not reason.
|
| They are statistical text generators, whose results are defined
| by their training data set. This is why the paper cited reads
| thusly: Despite extensive research into the
| reasoning capabilities of Large Language Models (LLMs),
| most studies have failed to rigorously differentiate
| between inductive and deductive reasoning ...
|
| There is no differentiation because what was sought is the
| existence of what does not.
|
| The authors then postulate: This raises an
| essential question: In LLM reasoning, which poses a greater
| challenge - deductive or inductive reasoning?
|
| There is no such thing as "LLM reasoning." Therefore, the
| greatest challenge is accepting this fact and/or that
| anthropomorphism is a real thing.
| astrange wrote:
| They are not statistical text generators. They are black boxes
| wrapped in an interface part of which is a statistical text
| generator.
|
| You can write programs into transformer models and run them
| deterministically; they aren't "statistical".
| godelski wrote:
| Can you cite the deterministic part? And I don't think fixing
| your seeds count as making a model not statistical, though
| yes deterministic.
|
| It's also still statistical if it gives you the same answer
| 99999/100000 times. You can definitely tune these things to
| be much more certain about certain problems but it tends to
| decrease capabilities in others.
|
| I also don't like the term "black box." While they aren't
| transparent they aren't completely opaque either. A big
| reason I didn't like the term is that I feel it also
| encourages people to not research this more. While we don't
| completely know what's going on I see no reason we can't. Is
| there some proof they their calculations are an unbreakable
| encryption or is it just a hard problem. I'm pretty sure it's
| the latter, and I think it's good to differentiate "we're
| dumb" from "indeterminate"
| taylorius wrote:
| The issue with comprehending LLM's internal workings is
| just that they're so big. The operation of the network
| itself is deterministic - but it's output is a set of
| probabilities over the value of the next token in the
| stream. At that point you have to choose one, based on that
| probability distribution, and then start the process again.
| That choice is where the randomness comes in.
| godelski wrote:
| Just to clarify, are you disagreing with me? Correcting
| me? Adding context for others? I guess I'm missing the
| intent
| taylorius wrote:
| Just adding context, elaborating a little on the
| deterministic / statistical aspects of an LLM. I don't
| disagree with what you said, I'm afraid - so not much
| prospect for a back and forth :-)
| astrange wrote:
| > Can you cite the deterministic part? And I don't think
| fixing your seeds count as making a model not statistical,
| though yes deterministic.
|
| I was thinking temperature=0, but what more do you need
| than that?
|
| (nb there is also some randomness from non-associative FP
| operations happening in different orders)
|
| > I also don't like the term "black box." While they aren't
| transparent they aren't completely opaque either.
|
| A white box is a subset of a black box here. The important
| part is that the sampler/text generation is wrapped around
| the transformer model and is not actually the same thing as
| it.
| godelski wrote:
| > I was thinking temperature=0, but what more do you need
| than that?
|
| Temperature isn't a thing exclusive to LLMs or ML. It's a
| general technique from statistics where you're modifying
| the variance of the distribution (also see truncation).
| And as you're aware, temperature zero isn't actually
| temperature zero.
|
| So say such a thing makes something non-statistical is
| the same as saying prime numbers don't exist because any
| prime p * 0 is 0. > A white box is a
| subset of a black box here
|
| No. A white box and black box sit at opposite ends of a
| spectrum which is just saying how much we understand
| something. Complexity isn't what makes something a black
| box, it's our lack of understanding that does. Of course
| this tends to coinside, but there's plenty of extremely
| complex things we have a great understanding of, even if
| not to infinite precision.
|
| Please take your passion and use it to drive you to learn
| about these things deeper. It's actually the same advice
| I give to the LLM fanboys who think it's AGI. But trying
| to dispel that myth without correct information makes it
| harder. It muddies the waters. These topics are extremely
| complex and the issue with the ML field is itself a
| hypocritical claim that it is easy (scale is all you need
| while calling these machines black boxes). I mean we can
| talk about the elephant in the room but certainly Von
| Neumann has said enough.
| astrange wrote:
| > No. A white box and black box sit at opposite ends of a
| spectrum which is just saying how much we understand
| something.
|
| So it's a subset, because "understanding something" is a
| refinement of the state of "not understanding something"
| - you can just pretend you don't know what's happening in
| that part of the system and it is now a black box.
| desumeku wrote:
| Semantic word games aren't a crowd-winning form of
| argument.
| godelski wrote:
| It's worse when you play that semantic game to make your
| point stronger (changing your original meaning) and
| willfully misinterpreting the other person. Everyone
| loves being willfully misinterpreted
| stavros wrote:
| I really dislike these non-sequitur arguments of "LLMs do not
| reason, because <detail on how they work>", as if a detail on
| how they work unequivocally proves the impossibility of
| reasoning.
| quantadev wrote:
| I've noticed that on HackerNews about 80% of all debates or
| discussions where people disagree, boils down to a
| disagreement about the definition of a word.
| fedeb95 wrote:
| feel free to extend to other contexts. It was basically
| Socrates' argument.
| throwawaytemp29 wrote:
| That is certainly true, for some definition of 80%.
| slashdave wrote:
| I see. Well, I claim my pet rock can think. What do you mean
| it doesn't have brain cells, what kind of argument is that?
| fragmede wrote:
| My pet silicon rock was processed by TSMC in a design by
| Nvidia, and does this trick called inference that very much
| does look like thinking. Who made yours?
| Y_Y wrote:
| I deem your rock to be disqualified on the basis of its
| performance clearly having been enhanced by doping.
| Quiark wrote:
| "This alien super-computer made from entangled iron atoms
| clearly is not thinking because it doesn't have neurons"
|
| you mean like this?
| desumeku wrote:
| ChatGPT - Our very own alien super-computer!
| gaganyaan wrote:
| Define reason before making grandiose claims.
| quantadev wrote:
| I think "reasoning" is the best word in the English language
| that I know of to describe what LLMs are doing. Lots of
| people disagree because for about 2000 years that word always
| implied consciousness, and LLMs definitely don't have
| consciousness.
| gaganyaan wrote:
| Define consciousness before making that claim either.
| quantadev wrote:
| I'm fine with the standard dictionary definition in this
| case. :P
| gaganyaan wrote:
| There's as many definitions as there are dictionaries,
| but let's try picking "awareness". LLMs are aware of many
| things, so they are conscious of those things, meaning
| they display some level of consciousness.
| quantadev wrote:
| Nice one. Now do "qualia"
| gaganyaan wrote:
| Qualia is a silly god of the gaps style argument. There's
| no difference between "seeing red" and "having your
| neurons tickled so that you think you're seeing red".
|
| People really don't understand consciousness, but it's
| actually quite simple when you realize that the question
| isn't "is this conscious?", it's "what is this conscious
| of?". A rock is not conscious of anything, because it has
| no state that represents anything. A sunflower is
| conscious of the sun's position, because it has state
| that represents that. A cat is conscious of much more,
| such as "this mouse is food". An LLM is conscious of much
| more, but less than a human.
|
| The LLM is a model that is conscious of many things, as
| experienced through the medium of human text.
| quantadev wrote:
| The fact that we can tell even from an EEG (a grossly
| inaccurate measurement of brain activity) whether a
| person is conscious or unconscious, indicates there's a
| physical reality to it, which might be able to be
| understood by science some day.
|
| I think Qualia is a wave phenomena, and all wave
| phenomena simultaneously can be described as "existing"
| and "not existing" as follows:
|
| Think of a football stadium "wave". It has a beginning,
| ending, location, velocity, etc., all being
| scientifically measurable observables. Yet I ask you
| "Does the wave exist, or is it merely people in various
| states of arm positions?" There is no right or wrong
| answer. This is the nature of waves.
|
| So if you want to argue consciousness "does not exist" I
| say fine, that is a valid "perspective", just like the
| denial of the existence of any other wave phenomena is a
| valid perspective. However keep in mind also that the
| entire universe is also made of waves.
| HeatrayEnjoyer wrote:
| It's not possible to know if _anyone_ has consciousness,
| other than them themselves.
|
| We must be *very* careful before making authoritative
| statements about who and what has qualia.
| godelski wrote:
| The logic goes the other way. The claim that is
| extraordinary is that a rock with electricity has
| consciousness. Nothing else we've built has shown
| evidence of consciousness and there's good explanations
| for the results we see without needing consciousness.
|
| Sure, it's still debatable for us humans (the term isn't
| well defined) but that doesn't change the direction of
| required proof.
| mistermann wrote:
| Any assertion has a burden of proof, you are likely
| referring to cultural/colloquial norms/beliefs on the
| matter.
|
| There is cultural/normative truth, and then there are
| various stricter/technical interpretations (using forms
| of logic other than binary, which causes reality to
| appear very differently), and making such distinctions is
| typically culturally "discouraged" (it is rude, pedantic,
| _not what HN is for_ , etc).
| godelski wrote:
| Consciousness is not binary.
|
| The burden of proof does have direction. It's in the
| direction of the more complicated thing. Do you think a
| rock is conscious? What about lightning? If you're
| 99.999% of people, then the answer is no. So if you want
| to claim AI is conscious you need make the claim where
| inanimate becomes animate. And you know what they say
| about extraordinary claims... But personally I don't
| believe in ghosts, even those in machines
| gaganyaan wrote:
| See my other comment too, but I agree. Rocks and
| lightning aren't conscious of anything, because they have
| no state that represents anything. Amoeba are conscious
| of some things, because they have state that represents a
| tiny slice of the world. Cats are conscious of much more,
| then LLMs and humans in turn.
|
| It's not about animate vs inanimate, it's about models
| that represent something.
|
| There's also some confusion because we have limited
| language here. A person can be "unconscious", but that
| doesn't mean they've lost their world model. It's just
| currently not executing.
| quantadev wrote:
| Unconscious brains appear to be functioning normally as
| far as all of science can tell, except for one big
| difference: Brain Wave frequencies. Only the lower
| frequencies exist. Higher frequencies are scientifically
| proven to be required for consciousness. You can call
| that a meaningless correlation that happens to be totally
| inconsequential, but most neuroscientists strongly
| disagree with that.
|
| EDIT: I mean neuroscientists know brain frequency is more
| than an accidental correlation with consciousness.
| godelski wrote:
| > because they have no state that represents anything.
|
| Are you sure about that? Both have lots of representation
| to them. Rocks have their atomic structures and you can
| easily view this as code. You can also view lightning as
| a representative of the intersection of cloud and ground
| states.
|
| The point I'm making is that representation is not
| enough. Storage (or memory) is not enough even recall is
| not enough. If it is, then any mechanical process is.
| Because you can build computation machines out of nearly
| anything. You can build memory out of almost anything.
| And the hello world python program I wrote has both!
|
| You can argue scale, but we see animals with much smaller
| scale than these ML models perform unambiguous reasoning
| tasks and critical thinking. You can then argue
| architecture and data, but then that's not just scale.
|
| The problem with all these people making claims is that
| they don't consider base assumptions and they don't
| follow the consequences of the model (your mental model
| of what is consciousness) is and must mean! So many
| people argue that they are working from "first
| principles" but I don't see anyone writing down their
| fucking axioms.
|
| Of course you should try to prove what you want, but not
| at the cost of not being wrong. Doing so will prevent you
| from achieving what you want. I'm just so tired of people
| shooting themselves in the foot and calling it medicine.
| We need to stop with all the bullshit because I actually
| want to build AGI
| gaganyaan wrote:
| It's not the silicon that's conscious of things, it's the
| model that happens to use that silicon.
| godelski wrote:
| Again, at what point does the program become conscious?
| ```python def main(): print("Hello
| gaganyaan") if __name__ == '__main__':
| main() ```
|
| That program is not conscious.
|
| So... The burden of proof is about what level of
| complexity is sufficient.
|
| Sure, no one has this answer. But also despite a few
| billion people believing in ghosts, including some saying
| a large portion of that saying they've _seen_ ghosts, the
| burden of proof is to show ghosts exist, not showing
| ghosts don 't exist. The reason being that one is
| falsifiable while the other isn't.
|
| Don't confuse "the absence of proof is not proof of
| absence" with "the absence of evidence is not evidence of
| absence." If you do you might start a war in the middle
| east.
| quantadev wrote:
| Computer chips don't have any brain waves. Even some
| theoretical computer chip able to run a "Perfect in Every
| Way" simulation of a brain including even simulating the
| physics of the actual brain waves...still has no brain
| waves. Nothing but 1s and 0s turning on and off, and not
| even turning on and off in anything resembling a wave
| pattern at all.
|
| This is why people who believe the "Consciousness is a
| Computation" view never want to discuss brain waves,
| despite the overwhelming evidence that brain waves is
| precisely the thing out of which consciousness is
| composed.
| Fellshard wrote:
| The grandiose claim would be that LLMs have reason to begin
| with. It is quite normal for things to not have reasoning
| capacity.
| aurareturn wrote:
| Brains do not reason - they are a neural network whose results
| are defined by their experiences, memories, and sensory inputs.
|
| I'm tired of reading comments from people who keep repeating
| that LLMs don't think, don't reason, isn't intelligence because
| it is not human. If your definition of the above is because
| it's not human, it's quite useless as a comment. We know LLMs
| aren't biological human brains. So what?
|
| Define what reasoning is to you. Then tell us why LLMs don't
| reason and why it matters.
| cscurmudgeon wrote:
| Not OP. But:
|
| 1. Reasoning is ability to at least carry out proofs in FOL
| (first-order logic). FOL can simulate Turing Mach
|
| 2. LLMs are formally equivalent to only a subset of FOL.
|
| Why is this important? To model human mathematics, you need
| at least first-order logic.
|
| These arguments have been around for decades, e.g., Penrose.
| I am tired of people bringing up strawmen arguments ("Not
| intelligent because not human!")
| scarmig wrote:
| Is there a proof that collections of human neurons are
| capable of carrying out proofs in first order logic, in
| full generality?
| fmbb wrote:
| Is anyone trying to prove that all humans can?
| gaganyaan wrote:
| Can a cat reason? It knows nothing of FOL or math, but can
| be very smart when figuring out how to catch a mouse.
| Infinity315 wrote:
| The term 'reason' is being overloaded here.
|
| The type of 'reason' a cat uses is different from the
| 'reason' used in math
|
| The information a cat uses is incomplete whereas the
| information used in math and logic is theoretically all
| accessible.
|
| The reasoning a cat uses allows for inconsistencies with
| its model because of its use of incomplete information
| whereas no inconsistencies are permissible in math and
| logic.
|
| Formally speaking, math uses deductive reasoning whereas
| the cat uses abductive reasoning.
| logicchains wrote:
| Transformers with chain of thought can use FOL:
| https://arxiv.org/abs/2310.07923
| AdieuToLogic wrote:
| > Brains do not reason ...
|
| They can, at times, and do so best when emotion is not
| involved.
|
| > I'm tired of reading comments from people who keep
| repeating that LLMs don't think, don't reason, isn't
| intelligence because it is not human.
|
| LLM's represent a category of algorithm. Quite elegant and
| useful in some circumstances, but an algorithm none the less.
| A quick search produced this[0] example discussing same.
|
| Another reference, which may not be authoritative based on
| whatever most recent edit the link produces, is[1]:
| A large language model (LLM) is a computational model
| capable of language generation or other natural language
| processing tasks. As language models, LLMs acquire these
| abilities by learning statistical relationships from vast
| amounts of text during a self-supervised and semi-supervised
| training process.
|
| > Define what reasoning is to you.
|
| Reasoning was the process I went through to formulate this
| response, doing so with intent to convey meaning as best as I
| can, and understand as best as possible the message to which
| I am replying.
|
| > Then tell us why LLMs don't reason and why it matters.
|
| LLM's do not possess the ability to perform the process
| detailed above.
|
| This is why it matters.
|
| 0 - https://github.com/rasbt/LLMs-from-scratch
|
| 1 - https://en.wikipedia.org/wiki/Large_language_model
| viraptor wrote:
| > Reasoning was the process I went through to ...
|
| That's not a useful definition for judging whether LLMs
| reason or not. It's not something we can measure on an
| objective level and introduces another concept of intent
| which is just as vague as reasoning.
|
| Specifically, an LLM can produce a similar message to what
| you posted. Everything else about that process is not
| defined well enough to differentiate you.
| Vampiero wrote:
| Ok, that's cool, now ask GPT to solve any programming or
| logic problem at all and maybe you'll start to understand
| why you can reason and why it can't.
| viraptor wrote:
| It can solve some problems and not others. But that would
| be again a different definition of reasoning, not what
| the parent wrote. And it would exclude animals/humans as
| reasoning, because they can't solve all logic problems.
| imtringued wrote:
| Artificial intelligence is still intelligence, even if it is
| just a shallow copy of human intelligence.
|
| What irritates me when I see comments like yours is that
| precise knowledge of weaknesses of LLMs is necessary to
| either improve LLMs, so most of the people who claim LLMs
| reason or are already AGI basically deny the ability to
| improve them, since they are already perfect. Research into
| studying the limitations of the current generation of AI is
| unwanted and by extension so is the next generation of AI.
| godelski wrote:
| > They are statistical text generators, whose results are
| defined by their training data set
|
| Honestly I'm pissed at the research community. It's fraud. If
| you don't know what's in the training data you simply cannot
| differentiate reasoning from memorization.
|
| Beyond that, the experiments are almost certainly in the
| training data. Like come on! I feel like I'm going crazy here.
| How can any coder not think LLMs are training on oct and hex!?
|
| https://news.ycombinator.com/item?id=41422751
| og_kalu wrote:
| Of course they've been trained on oct and hex.
|
| The question is would the results be largely valid if this
| was done on a human who had learnt how to perform base 8, 9,
| 11 etc arithmetic instead ?
|
| I mean, they're clearly not trying to test the ability to
| derive base arithmetic from scratch.
| anon373839 wrote:
| > Large Language Model algorithms do not reason.
|
| My belief is that what LLMs do is best described as
| approximating the outputs of reasoning. This is different from
| reasoning itself and also different from simply regurgitating
| the most similar training examples.
|
| I also think it's extremely challenging for people to
| understand the significance of LLM output because language is
| intertwined with meaning. So if you construct a valid and
| seemingly responsive sequence of words in response to a prompt,
| there is a strong illusion of purpose behind those words, even
| if they were assembled in a completely naive fashion.
| levitatorius wrote:
| This! One simple argument is that language is NOT a magical
| reasoning substance in itself, but a communication medium. It
| is medium for passing (a) meaning. So first there is a
| meaningful thought (worth of sharing), then an agent puts a
| SIGNIFIER on that meaningful thought, then communicates it to
| the recipient. Communication medium can be a sentence, it can
| also be an eyewink or a tail wiggle. Or a whistle. The
| "language" can be created on the spot, if two subjects get a
| meaning of signifier by intuition (e.g. I look at the object,
| you follow my gaze).
|
| So the fallacy of the whole LLM field is the belief that
| language has some intrinsic meaning. Or if you mix the
| artifacts of language in some very smart way, the meaning
| will emerge. But it doesn't work if meaning occurs before the
| word. The text in books has no reasoning, it was authors. The
| machine shuffling the text fragments does not have a
| meaningful thought. The engineer which devised a shuffling
| machine had some meaningful thought, the users of the machine
| have same thoughts, but not the machine itself. To put it
| another way, if there was an artificial system capable of
| producing meaningful thoughts, it is not a presence of
| language which produces a proof, it's communication.
| Communication requires an agent (as in "agency") and an
| intent. We have neither in LLM. As to the argument that we
| ourselves are mere stochastic parrots - of course we can
| produce word salads, or fake mimics of coherent text, it is
| not a proof that LLM IS the way our minds work. It is just a
| witness to the fact language is a flexible medium for the
| meanings behind - it can just as well be used for cheating,
| pretending, etc.
| mistermann wrote:
| > So the fallacy of...
|
| The text from this point on seems to have lost contextual
| awareness of what preceded it (which was excellent imho).
| godelski wrote:
| > One simple argument is that language is NOT a magical
| reasoning substance in itself, but a communication medium.
|
| I'm wildly impressed by how many people think language is
| thinking. My best guess is they're conflating inner speech
| with thinking. But if you can't figure out that the words
| you vocalize aren't an isomorphic representation of the
| things you try to convey, well... It's hard for me to
| believe you've spent enough time thinking about what it
| means to think. Miscommunication is quite common and so
| there's sufficient feedback to learn this without being
| explicitly taught. Then again, there are people in the
| world that I fear...
| seanmcdirmid wrote:
| LLMs are statistical text generators whose results depend on
| the model and the context given. They have gotten so good
| because the context they can effectively operate over keeps
| getting really big. If you take the model and operate on a
| small context, you will get very uninteresting results.
|
| The only reason it seems like it is reasoning is because it's
| probably stuffing a lot of reasoning in its context, and
| regurgitating that out in ways that are statically weighted
| with other things in the context on what is being reasoned
| about.
|
| Frankly, even most commenters on HN don't get how LLMs operate,
| thinking the model itself is what knows about different bases
| like hex and oct, when really, it searched up a bunch of
| material on different bases to include in the context before
| the model was ever activated.
| sfink wrote:
| Confession: I haven't read the paper.
|
| But any mention of LLM reasoning ability ought to address the
| obvious confound: the LLM is trained on examples of deductive
| reasoning, inductive reasoning, abductive reasoning, SAT-solver
| reasoning, geniuses' musings, etc. If they replicate one of those
| examples, then should that be called "reasoning" of any sort or
| not? Regurgitating those examples may even involve some
| generalization, if the original topics of an example are swapped
| out (perhaps by a nearby topic in latent space).
|
| Given that it appears they're training and testing on synthetic
| problems, this objection probably does not apply to their actual
| results. But given the fuzziness it creates for the definition of
| "reasoning" of any sort, I would have expected some working
| definition of reasoning in the paper's abstract.
|
| Training on Moby Dick and thus being able to regurgitate text
| from Moby Dick does not mean the LLM is capable of writing a new
| Moby Dick-like book. (Thankfully; one is more than enough!)
| viraptor wrote:
| The tasks used are artificially created and don't exist in the
| training sets. For example there's very little practical math
| in base 11 on the internet, or English with explicitly mixed up
| but rule based grammar.
| WaitWaitWha wrote:
| Neither. LLMs are just really, really good pattern matchers with
| enormous set of patterns.
| randcraw wrote:
| The conclusions of the authors that LLMs can reason inductively
| very well runs counter to what I've read elsewhere. A big part of
| doing induction is the ability to generalize a shared pattern
| from multiple disparate examples, recognizing the essential
| elements that are necessary and sufficient to satisfy that
| pattern's operators' constraints. To date, I've seen consensus
| that LLMs can match verbs or basic relational operators across
| examples, thereby associating the mechanisms in similar events
| that lead to similar outcomes. But extending that facility
| further, to employing predicate logic operators, or even the
| simpler propositional ones appears to fall largely outside LLM
| capabilities. To suggest then that LLMs can then perform higher-
| order reasoning skills yet, like the modeling of contrapositives,
| this seems quite a stretch.
| golergka wrote:
| I've just successfully chatted to ChatGPT about equivalence or
| at least high similarities between QFT, neural networks and
| cellular automata (referencing Wofram's work). Does that
| pattern matching count?
| Vampiero wrote:
| No but GPT is really good at fooling laymen who are not
| experts of a field, and it stands to reason that it just fed
| you a bunch of bs
| xwolfi wrote:
| And you were able to verify, of course, that anything new or
| surprising to you (as in, not a simple derivation of your own
| prompts), was true ?
|
| I noticed that if I ask it to tell me how good
| cryptocurrencies are, it'll do it, and then if I say I
| disagree and they're wrong, it'll simply switch and agree
| with me as well. The thing has no care for truth, no opinion
| of its own, no ability to insist, and just feeds you whatever
| is statistically close to your own questions.
| godelski wrote:
| I'm really tired of these papers and experiments.
|
| You cannot test reasoning when you don't know what's in the
| training set. You have to be able to differentiate reasoning from
| memorization, and that's not trivial.
|
| Moreso, the results look to confirm that at least some
| memorization is going on. Do we really not think GPT has
| extensively been trained on arithmetic in base 10, 8, and 16?
| This seems like a terrible prior. Even if not explicitly, how
| much code has it read that performs these tasks. How many web
| pages, tutorials, Reddit posts cover oct and hex? They also
| haven't defined zero shot correctly. Arithmetic in these bases
| aren't 0-shot. They're explicitly in distribution...
|
| I'm unsure about base 9 and 11. It's pretty interesting to see
| that GPT 4 is much better at these. Anyone know why? Did they
| train on these? More bases? Doesn't seem unreasonable but I don't
| know.
|
| The experimentation is also extremely lacking. The arithmetic
| questions only have 1000 tests where they add two digits. This is
| certainly in the training data. I'm also unconvinced by the
| syntax reasoning tasks since the transformer (attention)
| architecture seems to be designed for this. I'm also unconvinced
| these tasks aren't in training. Caesar ciphers are also certainly
| in the training data.
|
| The prompts are also odd and I guess that's why they're in the
| appendix. For example, getting GPT to be better at math or many
| tasks by having it write python code is not novel.
|
| There's some stuff here but this really doesn't seem like a lot
| of work for 12 people from a top university and a trillion dollar
| company. It's odd to see that many people when the experiments
| can be run in a fairly short time.
| Infinity315 wrote:
| I think the results still tell us something.
|
| Discrepancies in mathematical ability between the various bases
| would seem to suggest memorization as opposed to
| generalization.
| om8 wrote:
| How to you define memorization and reasoning? There is a large
| grey area in between them. Some say that if you can memorize
| facts and algorithms and apply them to new data, it is a
| memorization. Some say that it is reasoning.
|
| More than that -- It's not clear that what humans do is not
| "just" a memorization. We can always look at human experience
| mechanisticly and say that we don't think -- we just memorized
| thinking patterns and apply them when speaking and "thinking"
| akomtu wrote:
| A lot of reasoning is similar to interpolation within a
| sparse set of observations. Memorization is rounding up to
| the nearest known example. Basic guess is linear
| interpolation. And reasoning is about discovering the
| simplest rule that explains all the observations and using
| this rule to extrapolate.
| godelski wrote:
| > It's not clear that what humans do is not "just" a
| memorization.
|
| While I agree that there is a lot of gray in-between I think
| you are misrepresenting my comment. And I'm absolutely
| certain humans do more than memorization. Not all humans, but
| that's not the bar. Some humans are brain damaged and some
| are in fact babies (and many scientific do agree that
| sentience doesn't appear at birth).
|
| If you doubt me I very much encourage you to dive deeper into
| the history of science and get doing deep deep knowledge on
| any subject. Because you'll find this happen all the time.
| But if you apply a loose enough definition to memorization
| (that isn't one that would be generally agreed upon if you
| used it's logical conclusions) then yeah, everything is
| memorization. But everything is foo if I define everything to
| be foo, so let's not.
| imtringued wrote:
| > Some say that if you can memorize facts and algorithms and
| apply them to new data, it is a memorization. Some say that
| it is reasoning.
|
| Memorizing facts and algorithms is memorization. The rest of
| what you are talking about is not.
|
| Applying existing knowledge on new data without deriving new
| information is generalization. An example of this is the case
| of a semantic segmentation model classifying a car that it
| has never seen. If the model was not trained on birds, it
| will never classify a bird as a bird.
|
| Computation of decidable problems is a large, possibly the
| largest subset of reasoning. Most humans do not struggle with
| solving decidable problems, the problem is that they are slow
| and can only solve small problem sizes, but most problems
| encountered in practice aren't one large decidable problem,
| but a long chain of many small, dozens to hundreds of
| heterogeneous problems that are seamlessly mixed with one
| another. LLMs struggle with decidable problems that are out
| of distribution, but you can give a human instructions on how
| to do something they have never done before and they will
| follow them with no problem.
|
| > More than that -- It's not clear that what humans do is not
| "just" a memorization.
|
| I hope it is clear that I did not memorize this message I am
| writing here and that it is the unique result of processes
| inside my brain that were not captured in the training
| process of an LLM.
|
| >We can always look at human experience mechanisticly and say
| that we don't think -- we just memorized thinking patterns
| and apply them when speaking and "thinking"
|
| Again you are trying to twist this in an absurd direction.
| Let's come up with a teleoperated humanoid robot on Mars that
| is controlled by a human on Earth. The robot acts exactly
| like a human does. Does this mean the robot is now capable of
| reasoning and thinking like a human, simply because it is
| replaying a recording of the human's body and speech? This is
| the argument you are making. You're arguing that the robot's
| ability to replay a human's actions is equivalent to the
| processes that brought about that human action.
| godelski wrote:
| > Let's come up with a teleoperated humanoid robot on Mars
|
| One example I've always liked is from Star Trek. They got
| holodecks and no one thinks those are sentient people even
| though they are adaptive.
|
| I don't care what Iilya said, mimicking a human does not
| make a human. It may look like a duck, swims like a duck,
| and quack like a duck, then it's _probably_ a duck, but you
| haven 't ruled out an advanced animatronic. In fact, I'm
| betting right now people could make an animatronic that
| would convince most people it is a duck because most people
| just don't know the nuances of duck behavior.
| hanrelan wrote:
| You'll probably find this talk [1] interesting. They control
| all the training data for small LLMs and then perform
| experiments (including reasoning experiments).
|
| [1] Physics of LLMs:
| https://www.youtube.com/watch?v=yBL7J0kgldU&t=7s
| Animats wrote:
| We can tell some of what's in the training set. One of the
| answers for the inductive reasoning test begins "begin from the
| rightmost digit". Look that phrase up in Google. It shows up in
| Chegg, Course Hero, and Brainly content for elementary
| arithmetic. If you bash on those how-to articles, available for
| bases 2 and 10, you can probably generate the pattern for base
| 8.
|
| This looks like an LLM doing the usual LLM thing - finding
| relevant items and combining them to fit. This doesn't require
| the level of abstraction and induction the authors impute to
| the LLM. Ordinary LLM behavior explains this, once you've found
| the relevant training data.
|
| People often solve problems that way too, of course.
| Terr_ wrote:
| That reminds me of an old paper about "Benny's Rules", a
| case-study focused on a kid who seemed to be doing better
| than average in math tests when it came to final answers...
| but for all the wrong reasons, using an inferred set of
| arbitrary text manipulation rules.
|
| The intent was to point out that the educational approach was
| flawed, but I think there are interesting parallels to token
| processing in LLMs, which--unlike a human child--are built in
| such a way that crazy partial-fit rules are likely their only
| option.
|
| > Benny believed that the fraction 5/10 = 1.5 and 400/400 =
| 8.00 because he believed the rule was to add the numerator
| and denominator and then divide by the number represented by
| the highest place value.
|
| https://blog.mathed.net/2011/07/rysk-erlwangers-bennys-
| conce...
| Animats wrote:
| This is a problem with some tests. The students may detect
| a pattern in the test answers which reflects the work of
| those generating the answers, not the content.
|
| See this article on SAT test prep.[1] The requirement that
| only one answer can be right means that wrong answers have
| easily identifiable properties.
|
| [1] https://blog.prepscholar.com/the-critical-fundamental-
| strate...
| xiphias2 wrote:
| Transformers are amazing pattern matchers and terrible use of
| GPUs for reasoning, which is mostly search + execution of highly
| non-linear programs (lambda calculus).
|
| I love seeing Victor Taelin experimenting with parallizing these
| programs (with HVM and other experiments with proof languages),
| but it's sometimes a bit sad how much time researchers take in
| making papers about existing things instead of trying to improve
| the state-of-the art in something that's most probably missing
| from the current models.
| Datagenerator wrote:
| The human mind wonders and takes time to dream autonomously.
| Perhaps the llm.c we need for the next breakthrough addresses
| rounds of meditation in it's training in order to provoke more
| reason alike features to the NextGen LLM.
| calf wrote:
| Why does it have to be an either-or? Maybe LLMs are doing a bit
| of both, in a weird hybrid way; it is both doing a statistical
| calculation and yet the network is parameterized to do some very
| rudimentary (and nonhuman) reasoning computations. That's
| plausible to me, and explains why the reasoning is so hard to
| isolate... Just like how looking at a human brain it is hard to
| isolate the reasoning capacities.
| bob1029 wrote:
| LLMs are incredible about mapping to a space of already seen
| things. When this space is unimaginably large, you can be fooled
| for a long time.
|
| But, they clearly struggle with generalization and rule
| following. This failure to generalize (extrapolate, deduce,
| compute) is why we still can't fire all of our DBAs.
|
| Has anyone encountered an LLM-based text-to-SQL engine that
| actually gets the job done? I think that's your best canary. I
| stopped caring somewhere around "transpose these 2 letters of the
| alphabet" not working consistently.
| moktonar wrote:
| Asking them to make ASCII art is the final test, to me.
___________________________________________________________________
(page generated 2024-09-02 23:01 UTC)