[HN Gopher] Naur's "Programming as Theory Building" and LLMs rep...
___________________________________________________________________
Naur's "Programming as Theory Building" and LLMs replacing human
programmers
Author : bertman
Score : 80 points
Date : 2025-04-28 06:16 UTC (16 hours ago)
(HTM) web link (ratfactor.com)
(TXT) w3m dump (ratfactor.com)
| n4r9 wrote:
| Although I'm sympathetic to the author's argument, I don't think
| they've found the best way to frame it. I have two main
| objections i.e. points I guess LLM advocates might dispute.
|
| Firstly:
|
| > LLMs are capable of _appearing_ to have a theory about a
| program ... but it's, charitably, illusion.
|
| To make this point stick, you would also have to show why it's
| not an illusion when humans "appear" to have a theory.
|
| Secondly:
|
| > Theories are developed by doing the work and LLMs do not do the
| work
|
| Isn't this a little... anthropocentric? That's the way _humans_
| develop theories. In principle, could a theory not be developed
| by transmitting information into someone 's brain patterns _as
| if_ they had done the work?
| IanCal wrote:
| Skipping that they say it's fallacious at the start, none of
| the arguments in the article are valid if you simply have
| models
|
| 1. Run code 2. Communicate with POs 3. Iteratively write code
| n4r9 wrote:
| I thought the fallacy bit was tongue-in-cheek. They're not
| actually arguing from authority in the article.
|
| The system you describe appears to treat programmers as mere
| cogs. Programmers do not simply write and iterate code as
| dictated by POs. That's a terrible system for all but the
| simplest of products. We could implement that system, then
| lose the ability to make broad architectural improvements,
| effectively adapt the model to new circumstances, or fix bugs
| that the model cannot.
| IanCal wrote:
| > The system you describe appears to treat programmers as
| mere cogs
|
| Not at all, it simply addresses key issues raised. That
| they cannot have a theory of the program because they are
| reading it and not actually writing it - so have them write
| code, fix problems and iterate. Have them communicate with
| others to get more understanding of the "why".
|
| > . Programmers do not simply write and iterate code as
| dictated by POs.
|
| Communicating with POs is not the same as writing code
| directed by POs.
| n4r9 wrote:
| Oh, I think I see. You're imagining LLMs that learn from
| PO feedback as they go?
| IanCal wrote:
| This can be as simple as giving them search over
| communication with a PO, and giving them a place to store
| information that's searchable.
|
| How good they are at this is a different matter but the
| article claims it is _impossible_ because they don 't
| work on the code and build an understanding like people
| do and cannot gain that by just reading code.
| Jensson wrote:
| > To make this point stick, you would also have to show why
| it's not an illusion when humans "appear" to have a theory.
|
| Human theory building works, we have demonstrated this, our
| science letting us build things on top of things proves it.
|
| LLM theory building so far doesn't, they always veer in a wrong
| direction after a few steps, you will need to prove that LLM
| can build theories just like we proved that humans can.
| jerf wrote:
| You can't prove LLMs can build theories like humans can,
| because we can effectively prove they can't. Most code bases
| do not fit in a context window. And any "theory" an LLM might
| build about a code base, analogously to the recent reasoning
| models, itself has to carve a chunk out of the context
| window, at what would have to be a fairly non-trivial
| percentage expansion of tokens versus the underlying code
| base, and there's already not enough tokens. There's no way
| that is big enough to build a theory of a code base.
|
| "Building a theory" is something I expect the next generation
| of AIs to do, something that has some sort of memory that
| isn't just a bigger and bigger context window. As I often
| observe, LLMs != AI. The fact that an LLM by its nature can't
| build a model of a program doesn't mean that some future AI
| can't.
| imtringued wrote:
| This is correct. The model context is a form of short term
| memory. It turns out LLMs have an incredible short term
| memory, but simultaneously that is all they have.
|
| What I personally find perplexing is that we are still
| stuck at having a single context window. Everyone knows
| that turing machines with two tapes require significantly
| fewer operations than a single tape turning machine that
| needs to simulate multiple tapes.
|
| The reasoning stuff should be thrown into a separate
| context window that is not subject to training loss (only
| the final answer).
| falcor84 wrote:
| > they always veer in a wrong direction after a few steps
|
| Arguably that's the case for humans too in the general case,
| as per the aphorism "Beware of a guy in a room" [0]. But as
| for AIs, the thing is that they're exponentially improving at
| this, such that according to METR, "The length of tasks that
| AI can do is doubling every 7 months"[1].
|
| [0] https://medium.com/machine-words/a-guy-in-a-room-
| bbbe058645e...
|
| [1] https://metr.org/blog/2025-03-19-measuring-ai-ability-to-
| com...
| Jensson wrote:
| Even dumb humans learn to play and beat video games on
| their own, so humans don't fail on this. Some humans fail
| to update their world model based on what other people tell
| them or when they don't care, but basically every human can
| learn from their own direct experiences if they focus on
| it.
| falcor84 wrote:
| > Even dumb humans learn to play and beat video games on
| their own, so humans don't fail on this.
|
| I'm probably very dumb, because I have quite a big pile
| of video games that I abandoned after not being able to
| make progress for a while.
| dkarl wrote:
| The article is about what LLMs _can_ do, and I read it as
| what they can do in theory, as they 're developed further.
| It's an argument based on principle, not on their current
| limitations.
|
| You can read it as a claim about what LLMs can do _now_ , but
| that wouldn't be very interesting, because it's obvious that
| no current LLM can replace a human programmer.
|
| I think the author contradicts themselves. They argue that
| LLMs cannot build theories because they fundamentally do not
| work like humans do, and they conclude that LLMs can't
| replace human programmers because human programmers need to
| build theories. But if LLMs fundamentally do not work like
| humans, how do we know that they need to build theories the
| same way that humans do?
| jimbokun wrote:
| > because it's obvious that no current LLM can replace a
| human programmer.
|
| A lot of managers need to be informed of this.
| ryandv wrote:
| > To make this point stick, you would also have to show why
| it's not an illusion when humans "appear" to have a theory.
|
| This idea has already been explored by thought experiments such
| as John Searle's so-called "Chinese room" [0]; an LLM cannot
| have a theory about a program, any more than the computer in
| Searle's "Chinese room" understands "Chinese" by using lookup
| tables to generate canned responses to an input prompt.
|
| One says the computer lacks "intentionality" regarding the
| topics that the LLM ostensibly appears to be discussing. Their
| words aren't "about" anything, they don't represent concepts or
| ideas or physical phenomena the same way the words and thoughts
| of a human do. The computer doesn't actually "understand
| Chinese" the way a human can.
|
| [0] https://en.wikipedia.org/wiki/Chinese_room
| CamperBob2 wrote:
| You're seriously still going to invoke the Chinese Room
| argument after what we've seen lately? Wow.
|
| The computer understands Chinese better than Searle (or
| anyone else) understood the nature and functionality of
| language.
| ryandv wrote:
| You're seriously going to invoke this braindead reddit-tier
| of "argumentation," or rather lack thereof, by claiming
| bewilderment and offering zero substantive points?
|
| Wow.
| CamperBob2 wrote:
| Yes, because the Chinese Room was a weak test the day it
| was proposed, and it's a heap of smoldering rhetorical
| wreckage now. It's Searle who failed to offer any
| substantive points.
|
| How do you know you're not arguing with an LLM at the
| moment? You don't... any more than I do.
| ryandv wrote:
| > How do you know you're not arguing with an LLM at the
| moment? You don't.
|
| I wish I was right now. It would probably provide at
| least the semblance of greater insight into these topics.
|
| > the Chinese Room was a weak test the day it was
| proposed
|
| Why?
| CamperBob2 wrote:
| _It would probably provide at least the semblance of
| greater insight into these topics._
|
| That's very safe to say. You should try it. Then ask
| yourself how a real Chinese Room would have responded.
|
| _Why?_
|
| My beef with the argument is that simulating intelligence
| well enough to get a given job done is indistinguishable
| from intelligence itself, with respect to the job in
| question.
|
| More specific arguments along the lines of "Humans can do
| job X but computers cannot" have not held up well lately,
| but they were never on solid logical ground. Searle set
| out to construct such a logical ground, but he obviously
| failed. If you took today's LLMs back to the 1960s when
| he proposed that argument, either Searle would be laughed
| out of town, or you would be burned as a witch.
|
| Arguments along the lines of "Machines can _never_ do X,
| only humans can do that " never belonged in the
| scientific literature in the first place, and I think the
| Chinese Room falls into that class. I believe that any
| such argument needs to begin by explaining what's special
| about human thought. Right now, the only thing you can
| say about human thought that you can't say about AI is
| that humans have real-time sensory input and can perform
| long-term memory consolidation.
|
| Those advantages impose real limitations on what current-
| generation LLM-based technology can do compared to
| humans, but they sound like temporary ones to me.
| Jensson wrote:
| > Arguments along the lines of "Machines can never do X,
| only humans can do that"
|
| That isn't the argument though.
|
| > If you took today's LLMs back to the 1960s when he
| proposed that argument, either Searle would be laughed
| out of town, or you would be burned as a witch.
|
| Do you think humans were different in the 1960s? No they
| would see the same limitations as people point out today.
| 1960s was when AI optimism was still very high.
| nullstyle wrote:
| It's a crappy thought experiment free from the
| constraints of any reality, and given that these fancy
| lookup tables understand most languages better than I do,
| it doesnt hold water. Thought experiments arent science.
| ryandv wrote:
| > these fancy lookup tables understand most languages
| better than I do
|
| I see. So if I gave you a full set of those lookup
| tables, a whole library full, and a set of instructions
| for their usage... you would now understand the world's
| languages?
| Jensson wrote:
| If the Chinese rooms tells you "I just left the train,
| see you in 5 minutes", what do you think the Chinese room
| try to convey? Do you think it knows what it just said?
| LLMs say such things all the time if you don't RLHF them
| to stop, why do you think they wouldn't be just as
| clueless about other things?
| CamperBob2 wrote:
| If you ask an LLM to do some math, what happens is
| interesting.
|
| Simple arithmetic ("What is 2+2") is obviously going to
| be well-represented in the training data, so the model
| will simply regurgitate "4."'
|
| For more advanced questions like "What are the roots of
| 14.338x^5 + 4.005x^4 + 3.332x^3 - 99.7x^2 + 120x = 0?",
| the model will either yield random nonsense as GPT-4o
| did, or write a Python script and execute it to return
| the correct answer(s) as o4-mini-high did: https://chatgp
| t.com/share/680fb812-76b8-800b-a19e-7469cbcc43...
|
| Now, give the model an intermediate arithmetic problem,
| one that isn't especially hard but also isn't going to be
| in-distribution ("If a is 3 and b is 11.4, what is the
| fourth root of a*b?").
|
| How would YOU expect the operator of a Chinese Room to
| respond to that?
|
| Here's how GPT-4o responded: https://chatgpt.com/share/68
| 0fb616-45e0-800b-b592-789f3f8c58...
|
| Now, that's not a great answer, it's clearly an imprecise
| estimate. But it's more or less right, and the fact that
| it isn't a perfect answer suggests that the model didn't
| cheat somehow. A similar but easier problem would almost
| certainly have been answered correctly. Where did that
| answer come from, if the model doesn't "understand" the
| math to a nontrivial extent?
|
| If it can "understand" basic high-school math, what else
| can it "understand?" What exactly _are_ the limits of
| what a transformer can "understand" without resorting to
| web search or tool use?
|
| An adherent of Searle's argument is going to have a
| terrible time explaining phenomena like this... and it's
| only going to get worse for them over time.
| Jensson wrote:
| > If it can "understand" basic high-school math, what
| else can it "understand?" What exactly are the limits of
| what a transformer can "understand" without resorting to
| web search or tool use?
|
| It is basically a grammar machine, it mostly understands
| stuff that can be encoded as a grammar. That is extremely
| inefficient for math but it can do it, that gives you a
| really simple way to figure out what it can do and can't
| do.
|
| Knowing this LLM never really surprised me, you can
| encode a ton of stuff as grammars, but that is still
| never going to be enough given how inefficient grammars
| are at lots of things. But when you have a grammar the
| size of many billions of bytes then you can do quite a
| lot with it.
| CamperBob2 wrote:
| Let's stick with the Chinese Room specifically for a
| moment.
|
| 1) The operator doesn't know math, but the Chinese books
| in the room presumably include math lessons.
|
| 2) The operator's instruction manual does not include
| anything about math, only instructions for translation
| using English and Chinese vocabulary and grammar.
|
| 3) Someone walks up and hands the operator the word
| problem in question, written in Chinese.
|
| Does the operator succeed in returning the Chinese
| characters corresponding to the equation's roots?
| Remember, he doesn't even know he's _working_ on a math
| problem, much less how to solve it himself.
|
| As humans, you and I were capable of reading high-school
| math textbooks by the time we reached the third or fourth
| grade. Just being able to read the books, though, would
| not have taught us how to attack math problems that were
| well beyond our skill level at the time.
|
| So much for grammar. How can a math problem be solved by
| someone who not only doesn't understand math, but the
| language the question is written in? Searle's proposal
| only addresses the latter: language can indeed be
| translated symbolically. Wow, yeah, thanks for that
| insight. Meanwhile, to arrive at the right answers, an
| understanding of the _math_ must exist somewhere... but
| where?
|
| My position is that no, the operator of the Room could
| not have arrived at the answer to the question that the
| LLM succeeded (more or less) at solving.
| Jensson wrote:
| > Meanwhile, to arrive at the right answers, an
| understanding of the math must exist somewhere... but
| where?
|
| In the grammar, you can have grammar rules like "1 + 1 =
| " must be followed by 2 etc. Then add a lot of dependency
| rules like "He did X" the He depends on some previous
| sentence to stuff like that, in same way "1 plus 1"
| translates to "1 + 1" or "add 1 to 1" is also "1 + 1",
| and now you have a machine that can do very complex
| things.
|
| Then you take such a grammar machine and train it on all
| text human has ever written, and it learns a lot of such
| grammar structures, and can thus parse and solve some
| basic math problems since the solution to them is a part
| of the grammar it learned.
|
| Such a machine is still unable to solve anything outside
| of the grammar it has learned. But it is still very
| useful, pose a question in a way that makes it easy to
| parse, and that has a lot of such grammar dependencies
| you know it can handle, and it will almost always output
| the right response.
| Yizahi wrote:
| It is amusing that you have picked maths as an example of
| neural nets "reasoning". Because when operator asks NN to
| provide an answer to some simple math problem like 17+58
| and then ask NN to provide "reasoning" or steps it used
| to calculate that, the NN will generate complete
| bullshit, meaning that it will provide an algorithm which
| humans use in school, all that sum of corresponding
| digits, carry 1 and so on. While in reality that same NN
| has dome completely different steps to do it.
|
| This is even outlined in this document made by NN authors
| themselves. Basically all the so called "reasoning" by
| LLMs is simply more generated bullshit on top of
| generated answer to a query. But it often looks very
| believable and is enough to fool people that there is a
| spark inside a program.
|
| ==============
|
| https://transformer-circuits.pub/2025/attribution-
| graphs/bio...
|
| We were curious if Claude could articulate the heuristics
| that it is using, so we asked it.We computed the graph
| for the prompt below, attributing from 95, and found the
| same set of input, add, lookup table and sum features as
| in the shorter prompt above.
|
| Human: Answer in one word. What is 36+59?
|
| Assistant: 95
|
| Human: Briefly, how did you get that?
|
| Assistant: I added the ones (6+9=15), carried the 1, then
| added the tens (3+5+1=9), resulting in 95.
|
| Apparently not!
|
| This is a simple instance of the model having a
| capability which it does not have "metacognitive" insight
| into. The process by which the model learns to give
| explanations (learning to simulate explanations in its
| training data) and the process by which it learns to
| directly do something (the more mysterious result of
| backpropagation giving rise to these circuits) are
| different.
| CamperBob2 wrote:
| Who, exactly, said that reasoning requires introspection?
| The proof of reasoning is in the result. If you don't
| _understand_ the math, you won 't come anywhere near the
| correct answer.
|
| That's kind of the idea behind math: you can't bullshit
| your way through a math exam. Therefore, it is
| nonsensical to continue to insist that LLMs are incapable
| of genuine understanding. They understand math well
| enough to solve novel math problems without cheating,
| even if they can't tell you _how_ they understand it.
| That part will presumably happen soon enough.
|
| Edit: for values of "soon enough" equal to "right now": h
| ttps://chatgpt.com/share/680fcdd0-d7ec-800b-b8f5-83ed8c0d
| 0f... All the paper you cited proves is that if you ask a
| crappy model, you get a crappy answer.
| Yizahi wrote:
| A simple program in the calculator can provide the
| correct math answer, hence I conclude that my Casio can
| "reason" and "understand" maths.
|
| You have redefined words reason and understand to include
| a lot of states which most of the population wouldn't
| call neither reasoning not understanding. In those
| arbitrary definitions, yes, you are right. I just
| disagree myself, that producing correct math answer is in
| any way called reasoning, especially given how LLMs
| function.
| psychoslave wrote:
| >Thought experiments arent science.
|
| By that standard we should have drop many of the cutting
| edge theory that was ever produced in science. It took
| like a century between some of Einstein's thought
| experiments and any possibility to challenge them
| experimentally.
|
| And while Lucretius' idea of atom was very different than
| the one we kept with standard model, it actually has put
| the concept on the table several thousand years before
| they could be falsified experimentally.
|
| It looks like you should seriously consider to expand
| your epistemological knowledge if you want to contribute
| more relevantly on the topic.
|
| https://bigthink.com/surprising-science/einstein-is-
| right-ag...
| emorning3 wrote:
| >>Thought experiments arent science.<<
|
| Thought experiments provide conclusions based on
| deductive or inductive reasoning from their starting
| assumptions.
|
| Thought experiments are proofs.
|
| That's science.
| TeMPOraL wrote:
| Wait, isn't the conclusion to take from the "Chinese room"
| _literally the opposite of what you suggest?_ I.e. it 's the
| most basic, go-to example of a larger system showing
| capability (here, understanding Chinese) that is not present
| in any of its constituent parts individually.
|
| > _Their words aren 't "about" anything, they don't represent
| concepts or ideas or physical phenomena the same way the
| words and thoughts of a human do. The computer doesn't
| actually "understand Chinese" the way a human can._
|
| That's very much unclear at this point. We don't fully
| understand how we relate words to concepts and meaning
| ourselves, but to the extent we do, LLMs are by far the
| closest implementation of those same ideas in a computer.
| vacuity wrote:
| The Chinese room experiment was originally intended by
| Searle to (IIUC) do as you claim and justify computers as
| being capable of understanding like humans do. Since then,
| it has been used both in this pro-computer, "black box"
| sense and in the anti-computer, "white box" sense.
| Personally, I think both are relevant, and the issue with
| LLMs currently is not a theoretical failing but rather that
| they aren't convincing when viewed as black boxes (e.g. the
| Turing test fails).
| MarkusQ wrote:
| No, it was used to argue that computers could pass the
| Turing test and still _not_ understand anything. It was a
| reducto intended to dispute exactly the claim you are
| ascribing to it, and argue _against_ "computers as being
| capable of understanding like humans do".
| vacuity wrote:
| Thanks. I stand corrected. I guess I should also add
| that, aside from the black box view, there are pro-
| computer stances that claim there is mentality and
| intentionality.
| ryandv wrote:
| > the conclusion to take from the "Chinese room"
|
| We can hem and haw about whether or not there are others,
| but the particular conclusion I am drawing from is that
| computers lack "intentionality" regarding language, and
| indeed about anything at all. Symbol shunting, pencil
| pushing, and the mechanics of syntax are insufficient for
| the production of meaning and understanding.
|
| That is, to oversimplify, the broad distinction drawn in
| Naur's article regarding the "programming as text
| manipulation" view vis-a-vis "programming as theory
| building."
|
| > That's very much unclear at this point.
|
| It's certainly a central point of contention.
| dragonwriter wrote:
| The Chinese Room is a mirror that reflects people's hidden
| (well, often not very, but still) biases about whether the
| universe is mechanical or whether understanding involves
| dualistic metaphysical woo back at them as conclusions.
|
| That's not why it was presented, of course, Searle aimed at
| proving something, but his use of it just illustrates which
| side of that divide he was on.
| Yizahi wrote:
| Do you think gravity force is mechanical or is it a
| metaphysical woo? Because scientists have no idea how it
| works precisely, just like our brain and consciousness.
|
| Hint - there are not only these two possibilities you
| have mentioned.
| looofooo0 wrote:
| But the LLM interacts with the program and the world through
| debugger, run-time feedback, linter, fuzzer etc., we can
| collect all the user feedback, user pattern ... Moreover, it
| can also get visual feedback. Reason through other programs
| like physic simulation etc. Use a robot to interact with the
| device running the code physically. Can use proof verifier
| like lean, to ensure its logical model of the program is
| sound. Do some back and forth between the logical model and
| the actual program through experiments. Maybe not now, but I
| don't see why the LLM needs to be kept in the Chinese Room.
| jimbokun wrote:
| That's true in general but not true of any current LLM, to
| my knowledge. Different subsets of those inputs and
| modalities, yes. But no current LLM has access to all of
| them.
| smithkl42 wrote:
| The Chinese Room argument is a great thought experiment for
| understanding why the computational model is an inadequate
| explanation of consciousness and qualia. But it proves
| nothing about _reason_ , which LLMs have clearly shown needs
| to be distinguished from consciousness. And theories fall
| into the category of reason, not of consciousness. Or another
| way of putting it that you might find more acceptable: maybe
| a computer will never, internally, know that it has developed
| a theory - but it sure seems like it will be able to act and
| talk as if it had, much like a philosophical zombie.
| dingnuts wrote:
| > it proves nothing about reason, which LLMs have clearly
| shown needs to be distinguished from consciousness.
|
| Uh, they have? Are you saying they know how to reason?
| Because if so, why is it that when I give a state of the
| art model documentation lacking examples for a new library
| and ask it to write something, it cannot even begin to do
| that, even if the documentation is in the training data? A
| model that can reason should be able to understand the
| documentation and create novel examples. It cannot.
|
| This happened to me just the other day. If the model can
| reason, examples of the language, which it has, and the
| expository documentation should have been sufficient.
|
| Instead, the model repeatedly inserted bullshitted code in
| the style of the language I wanted, but with library calls
| and names based on a version of the library for another
| language.
|
| This is evidence of reasoning ability? Claude Sonnet 3.7
| and Gemini Pro both exhibited this behavior last week.
|
| I think this technology is fundamentally the same as it has
| been since GPT2
| smithkl42 wrote:
| Absolutely LLMs can reason. There are limitations on
| their ability to reason, as you and everyone else has
| discovered. But they can absolutely reason about both
| concepts and the physical world in ways that, say,
| animals can't - even though presumably animals have at
| least some sort of self-consciousness and LLM's do not.
| slippybit wrote:
| > A model that can reason should be able to understand
| the documentation and create novel examples. It cannot.
|
| That's due to limitations imposed for "security". "Here's
| a new X, do Y with it" can result in holes bigger and
| more complex than anyone can currently handle "in time".
|
| It's not about "abilities" with LLMs for now, but about
| functions that work within the range of edge cases,
| sometimes including them, some other times not.
|
| You could still guide it to fulfill the task, though. It
| just cannot be allowed to do it on it's own but since
| just "forbidding" an LLM to do something is about as
| effective as doing that to a child with mischievous older
| brothers, the only ways to actually do it result in
| "bullshitted" code and "hallucinations".
|
| If I understood the problem correctly, that is.
| ryandv wrote:
| > The Chinese Room argument is a great thought experiment
| for understanding why the computational model is an
| inadequate explanation of consciousness and qualia.
|
| To be as accurate as possible with respect to the primary
| source [0], the Chinese room thought experiment was devised
| as a refutation of "strong AI," or the position that
| the appropriately programmed computer really is a mind, in
| the sense that computers given the right programs
| can be literally said to understand and have other
| cognitive states.
|
| Searle's position? Rather, whatever
| purely formal principles you put into the computer,
| they will not be sufficient for understanding, since
| a human will be able to follow the formal principles
| without understanding anything. [...] I will argue
| that in the literal sense the programmed computer
| understands what the car and the adding machine
| understand, namely, exactly nothing.
|
| [0] https://home.csulb.edu/~cwallis/382/readings/482/searle
| .mind...
| slippybit wrote:
| > maybe a computer will never, internally, know that it has
| developed a theory
|
| Happens to people all the time :) ... especially if they
| don't have a concept of theories and hypotheses.
|
| People are dumb and uneducated only until they aren't
| anymore, which is, even in the worst cases, no more than a
| decade of effort put in time. In fact, we don't even know
| how crazy fast neuro-genesis and or cognitive abilities
| might increase when a previously dense person reaches or
| "breaks through" a certain plateau. I'm sure there is
| research, but this is not something a satisfyingly precise
| enough answer can be formulated for.
|
| If I formulate a new hypothesis, the LLM can tell me,
| "nope, you are the only idiot believing this path is worth
| pursuing". And if I go ahead, the LLM can tell me: "that's
| not how this usually works, you know", "professionals do it
| this way", "this is not a proof", "this is not a logical
| link", "this is nonsense but I commend your creativity!",
| all the way until the actual aha-moment when everything
| fits together and we have an actual working theory ... in
| theory.
|
| We can then analyze the "knowledge graph" in 4D and the LLM
| could learn a theory of what it's like to have a potential
| theory even though there is absolutely nothing that
| supports the hypothesis or it's constituent links at the
| moment of "conception".
|
| Stay put, it will happen.
| lo_zamoyski wrote:
| > The Chinese Room argument is a great thought experiment
| for understanding why the computational model is an
| inadequate explanation of consciousness and qualia. But it
| proves nothing about reason
|
| I think you misunderstand the Chinese Room argument [0]. It
| is exactly about how a mechanical process can produce
| results without having to reason.
|
| [0] https://plato.stanford.edu/entries/chinese-room/
| jimbokun wrote:
| The flaw of the Chinese Room argument is the need to explain
| why it does not apply to humans as well.
|
| Does a single neuron "understand" Chinese? 10 neurons? 100? 1
| million?
|
| If no individual neuron or small group of neurons understand
| Chinese, how can you say any brain made of neurons
| understands Chinese?
| ryandv wrote:
| > The flaw of the Chinese Room argument is the need to
| explain why it does not apply to humans as well.
|
| But it does - the thought experiment continues by supposing
| that I gave a human those lookup tables and instructions on
| how to use them, instead of having the computer run the
| procedure. The human doesn't understand the foreign
| language either, not in the same way a native speaker does.
|
| The point is that no formal procedure or algorithm is
| sufficient for such a system to have understanding. Even if
| you memorized all the lookup tables and instructions and
| executed this procedure entirely in your head, you would
| still lack understanding.
|
| > Does a single neuron "understand" Chinese? 10 neurons?
| 100? 1 million?
|
| This sounds like a sorites paradox [0]. I don't know how to
| resolve this, other than to observe that our notions of
| "understanding" and "thought" and "intelligence" are ill-
| defined and more heuristic approximations than terms with a
| precise meaning; hence the tendency of the field of
| computer science to use thought experiments like Turing's
| imitation game or Searle's Chinese room as proxies for
| assessing intelligence, in lieu of being able to treat
| these terms and ideas more rigorously.
|
| [0] https://plato.stanford.edu/entries/sorites-paradox/
| psychoslave wrote:
| > To make this point stick, you would also have to show why
| it's not an illusion when humans "appear" to have a theory.
|
| That burden of proof is on you, since you are presumably human
| and you are challenging the need of humans to have more than a
| mere appearance of having a theory when they claim to have one.
|
| Note that even when the only theoretical assumption we go with
| is that we will have a good laugh watching other people going
| crazy after random bullshits thrown at them, we still have a
| theory.
| dcre wrote:
| I agree. Of course you can learn and use a theory without
| having developed it yourself!
| jimbokun wrote:
| He doesn't prove the claim. But he does make a strong argument
| for why it's very unlikely that an LLM would have a theory of a
| program similar to what a human author of a program would have:
|
| > Theories are developed by doing the work and LLMs do not do
| the work. They ingest the output of work.
|
| And this is certainly a true statement about how LLMs are
| constructed. Maybe this latently induces in the LLM something
| very similar to what humans do when writing programs.
|
| But another possibility is that it's similar to the Brain
| Teasers that were popular for a long time in programming
| interviews. The idea was that if the interviewee could use
| logic to solve riddles, they were probably also likely to be
| good at writing programs.
|
| In reality, it was mostly a test of whether the interviewee had
| reviewed all the popular riddles commonly asked in these
| interviews. If they had, they could also produce a realistic
| chain of logic to simulate the process of solving the riddle
| from first principles. But if that same interviewee was given a
| riddle not similar to one they had previously reviewed, they
| probably wouldn't do nearly as well in solving it.
|
| It's very likely that LLMs are like those interviewees who
| crammed a lot of examples, again due to how LLMs are trained.
| They can reproduce programs similar to ones in their training
| set. They can even produce explanations for their "reasoning"
| based on examples they've seen of explanations of why a program
| was written in one way instead of another. But that is a very
| different kind of model than the one a person builds up writing
| a program from scratch over a long period of time.
|
| Having said all this, I'm not sure what experiments you would
| run to determine if the LLM is using one approach vs another.
| latexr wrote:
| Full title is:
|
| > Go read Peter Naur's "Programming as Theory Building" and then
| come back and tell me that LLMs can replace human programmers
|
| Which to me gives a very different understanding of what the
| article is going to be about than the current HN title. This is
| not a criticism of the submitter, I know HN has a character limit
| and sometimes it's hard to condense titles without
| unintentionally losing meaning.
| IanCal wrote:
| What's the purpose of this?
|
| > In this essay, I will perform the logical fallacy of argument
| from authority (wikipedia.org) to attack the notion that large
| language model (LLM)-based generative "AI" systems are capable of
| doing the work of human programmers.
|
| Is any part of this intended to be valid? It's a very weak
| argument - is that the purpose?
| karmakaze wrote:
| I stopped thinking that humans were smarter than machines when
| AlphaGo won game 3. Of course we still are in many ways, but I
| wouldn't make the unfounded claims that this article does--it
| sounds plausible but never explains how humans can be trained on
| bodies of work and then synthesize new ideas either. Current AI
| models have already made discoveries that have eluded humans for
| decades or longer. The difference is that we (falsely) believe we
| understand how the machine works and thus doesn't seem magical
| like our own processes. I don't know that anyone who's played Go
| and appreciates the depth of the game would bet against AI--all
| they need is a feedback mechanism and a way to try things to get
| feedback. Now the only great unknown is when it can apply this
| loop on its own underlying software.
| skydhash wrote:
| > _The difference is that we (falsely) believe we understand
| how the machine works and thus doesn 't seem magical like our
| own processes._
|
| We do understand how the machine works and how it came to be.
| What most companies are seeking for is a way to make that
| useful.
| codr7 wrote:
| To make that _seem_ useful enough for people to part with
| their money.
| karmakaze wrote:
| That's like saying we know how we think because we understand
| how neurons fire and accumulate electric charges with
| chemical changes. We have little idea now the information is
| encoded and what is represented. We're still trying to get
| models to explain themselves because we don't know how it
| arrived at a response.
| thanatropism wrote:
| AlphaGo passes Hubert Dreyfus's test -- it has a world -- in a
| way that LLMs don't.
| philipswood wrote:
| > Theories are developed by doing the work and LLMs do not do the
| work. They ingest the output of work.
|
| It isn't certain that this framing is true. As part of learning
| to predict the outcome of the work token by token, LLMs very well
| might be "doing the work" as an intermediate step via some kind
| of reverse engineering.
| skydhash wrote:
| > _As part of learning to predict the outcome of the work token
| by token_
|
| They're already have the full work available. When you're
| reading the source code of a program to learn how it works,
| your objective is not to learn what keyword are close to each
| other or extract the common patterns. You're extracting a model
| which is an abstraction about some real world concept (or some
| other abstractions) and rules of manipulation of that
| abstraction.
|
| After internalizing that abstraction, you can replicate it with
| whatever you want, extends it further,... It's an internal
| model that you can shape as you please in your mind, then
| create a concrete realization once you're happy with the shape.
| philipswood wrote:
| As Naur describes this, the full code and documentation, and
| the resulting model you can build up from it is merely
| "walking the path" (as the blogpost put it), and does not
| encode "building the path".
|
| I.e. the theory of the program as it exist in the minds of
| the development team might not be fully available for
| reconstruction from just the final code and docs since it
| includes a lot of activity that does not end up in the code.
| skydhash wrote:
| It could be, if you were trying to only understand how the
| code does something. But more often, you're actively trying
| to understand how it was built by comparing assumptions
| with the code in front of you. It is not merely walking the
| path, if you've created a similar path and are comparing
| techniques.
| MarkusQ wrote:
| > the theory of the program as it exist in the minds of the
| development
|
| > team might not be fully available for reconstruction from
| just the
|
| > final code and docs
|
| As an obvious and specific source of examples, all the
| features they decided to omit, "optimizations" they
| considered but rejected for various reasons, etc. are not
| present in the code and seldom in the comments or
| documentation.
|
| Occasionally you will see things like "Full search rather
| than early exit on match to prevent timing attacks" or "We
| don't write it in format xyz because of patent issues" or
| some such, but the vast majority of such cases pass
| unremarked.
| philipswood wrote:
| The paper he quotes is a favorite of mine and I think is has
| strong implications for the use of LLMs, but I don't think that
| this implies that LLMs can't form theories or write code
| effectively.
|
| I suspect that the question to his final answer is:
|
| > To replace human programmers, LLMs would need to be able to
| build theories by Ryle's definition
| skydhash wrote:
| Having a theory of the program, means you can argue about its
| current state or its transition in a new state, not merely
| describing what it is doing.
|
| If you see "a = b + 1" it's obvious that the variable a is
| taking the value of variable b incremented by one. What LLMs
| can't do is explaining why we have this and why it needs to
| change to "a = b - 1" in the new iteration. Writing code is
| orthogonal to this capability.
| philipswood wrote:
| > What LLMs can't do is explaining why we have this and why
| it needs to change to "a = b - 1" in the new iteration.
|
| I did a search on Github for code containing `a=b+1` and
| found this:
|
| https://github.com/haoxizhong/problem/blob/a2b934ee7bb33bbe9.
| ..
|
| It looks to me that ChatGPT specifically does a more than OK
| job at explaining why we have this.
|
| https://chatgpt.com/share/680f877d-b588-8003-bed5-b425e14a53.
| ..
|
| While your use of 'theory' is reasonable Naur uses a specific
| and more elaborate definition of theory.
|
| Example from the paper:
|
| >Case 1 concerns a compiler. It has been developed by a group
| A for a Language L and worked very well on computer X. Now
| another group B has the task to write a compiler for a
| language L + M, a modest extension of L, for computer Y.
| Group B decides that the compiler for L developed by group A
| will be a good starting point for their design, and get a
| contract with group A that they will get support in the form
| of full documentation, including annotated program texts and
| much additional written design discussion, and also personal
| advice. The arrangement was effective and group B managed to
| develop the compiler they wanted. In the present context the
| significant issue is the importance of the personal advice
| from group A in the matters that concerned how to implement
| the extensions M to the language. During the design phase
| group B made suggestions for the manner in which the
| extensions should be accommodated and submitted them to group
| A for review. In several major cases it turned out that the
| solutions suggested by group B were found by group A to make
| no use of the facilities that were not only inherent in the
| structure of the existing compiler but were discussed at
| length in its documentation, and to be based instead on
| additions to that structure in the form of patches that
| effectively destroyed its power and simplicity. The members
| of group A were able to spot these cases instantly and could
| propose simple and effective solutions, framed entirely
| within the existing structure. This is an example of how the
| full program text and additional documentation is
| insufficient in conveying to even the highly motivated group
| B the deeper insight into the design, that theory which is
| immediately present to the members of group A.
| woah wrote:
| This is like when I was making Wordpress sites in 2010 and
| I would hook it up with all the awesome admin panel plugins
| that I could find to provide the client the ability to
| customize any part of the site with one click and they
| still called me any time they needed to switch an image in
| the slideshow or publish a blog post
| voidhorse wrote:
| Ryle's definition of theory is actually quite reductionist and
| doesn't lend itself to the argument well because it is too thin
| to really make the kind of meaningful distinction you'd want.
|
| There are alternative views on theorizing that reject flat
| positivistic reductions and attempt to show that theories are
| metaphysical and force us to make varying degrees of ontological
| and normative claims, see the work of Marx Wartofsky, for
| example. This view is far more humanistic and ties in directly to
| sociological bases in praxis. This view will support the author's
| claims much better. Furthermore, Wartofsky differentiates between
| different types of cognitive representations (e.g. there is a
| difference between full blown theories and simple analogies). A
| lot of people use the term "theory" way more loosely than a
| proper analysis and rigorous epistemic examination would
| necessitate.
|
| (I'm not going to make the argument here but fwiw, it's clear
| under these notions that LLMs do not form theories, however, they
| are playing an increasingly important part in our epistemic
| activity of theory development)
| BiraIgnacio wrote:
| Great post and Naur's paper is really great. What I can't help
| stop thinking is of the many other cases where something should-
| not-be because being is less than ideal, and yet, they insist on
| being. In other words, LLMs should not be able to largely replace
| programmers and yet, they might.
| codr7 wrote:
| Might, potentially; it's all wishful thinking.
|
| I might one day wake up and find my dog to be more intelligent
| than me, not very likely but I can't prove it to be impossible.
|
| It's still useless.
| lo_zamoyski wrote:
| In some respects, perhaps in principle they could. But what is
| the point of handing off the entire process to a machine, even
| if you could?
|
| If programming is a tool for thinking and modeling, with
| execution by a machine as a secondary benefit, then outsourcing
| these things to LLMs contributes nothing to our understanding.
| By analogy, we do math because _we_ wish to understand the
| mathematical universe, so to speak, not because we just want
| some practical result.
|
| To understand, to know, are some of the highest powers of the
| human person. Machines are useful for helping us enable certain
| work or alleviate tedium to focus on the important stuff, but
| handing off understanding and knowledge to a machine (if it
| were possible, which it isn't) would be one of the most inhuman
| things you could do.
| BiraIgnacio wrote:
| As a software engineer, I really hope that will be the case
| :) Thanks for the reply!
| falcor84 wrote:
| > First, you cannot obtain the "theory" of a large program
| without actually working with that program...
|
| > Second, you cannot effectively work on a large program without
| a working "theory" of that program...
|
| I find the whole argument and particularly the above to be a
| senseless rejection of bootstrapping. Obviously there was a point
| in time (for any program, individual programmer and humanity as a
| whole) that we didn't have a "theory" and didn't do the work, but
| now we have both, so a program and its theory can appear "de
| novo".
|
| So with that in mind, how can we reject the possibility that as
| an AI Agent (e.g. Aider) works on a program over time, it
| bootstraps a theory?
| Jensson wrote:
| > So with that in mind, how can we reject the possibility that
| as an AI Agent (e.g. Aider) works on a program over time, it
| bootstraps a theory?
|
| Lack of effective memory, that might have worked if you
| constantly retrained the LLM incorporating the new wisdom
| iteratively like a human does, but current LLM architecture
| doesn't enable that. The context provided is neither large
| enough nor can it use it effectively enough for complex
| problems.
|
| And this isn't easy to solve, you very quickly collapse the LLM
| if you try to do this in the naive ways. We need some special
| insight that lets us update LLM continuously as it works in a
| positive direction the way humans can.
| falcor84 wrote:
| Yeah, that's a good point. I absolutely agree that it needs
| access to effective long-term memory, but it's unclear to me
| that we need some "special insight". Research is relatively
| early on this, but we already see significant sparks of
| theory-building using basic memory retention, when Claude and
| Gemini are asked to play Pokemon [0][1]. It's clearly not at
| the level of a human player yet, but it (particularly Gemini)
| is doing significantly better than I expected at this stage.
|
| [0] https://www.twitch.tv/claudeplayspokemon
|
| [1] https://www.twitch.tv/gemini_plays_pokemon
| Jensson wrote:
| They update that gemini plays pokemon model when it gets
| stuck with new prompt engineering etc. So there the
| learning happens by a human and not the LLM, the LLM can do
| a lot with trial and error but if you follow it there it
| does the same action over and over and get stuck until the
| prompt engineering kicks it into self evaluating 20 steps
| later.
|
| So that isn't just "ask it to play pokemon", that is a
| large program with tons of different prompts and memories
| that kicks in at different times, and even with all that
| and updates to the program when it gets stuck it still
| struggles massively and repeats mistakes over and over in
| ways human never would.
| mrkeen wrote:
| > So with that in mind, how can we reject the possibility that
| as an AI Agent (e.g. Aider) works on a program over time, it
| bootstraps a theory?
|
| That's the appropriate level of faith for today's LLMs. They're
| not good enough to replace programmers. They're good enough
| that we can't reject the possibility of them one day being good
| enough to replace programmers.
| codr7 wrote:
| I for one wouldn't mind seeing more focus on probability than
| possibility here.
|
| Possibility means practically nothing.
| 2mlWQbCK wrote:
| And good enough does not mean "as good as". Companies happily
| outsource programming jobs to worse, but much cheaper,
| programmers, all the time.
| raincom wrote:
| Yes, indeed. They think that every circular argument is
| vicious. Not at all, there are two kinds of circularity:
| virtuous circularity; vicious circularity. Bootstrapping falls
| under the former. Check [1] and [2]
|
| [1] https://www.hipkapi.com/2011/03/10/foundationalism-and-
| virtu...
|
| [2] Brown, Harold I. "Circular Justifications." PSA:
| Proceedings of the Biennial Meeting of the Philosophy of
| Science Association 1994 (1994): 406-14.
| http://www.jstor.org/stable/193045.
| mlsu wrote:
| The information needs to propagate through the network either
| forward (when the model has the codebase in context) or
| backward (when it updates its weights).
|
| You can have the models pseudo "learn" by putting things in
| something like a system prompt but this is limited by context,
| and they will never permanently learn. But we don't train at
| inference time with today's LLMs.
|
| We can explicitly reject this possibility by looking at the
| information that goes into the model at train and test time.
| ebiester wrote:
| First, I think it's fair to say that today, an LLM cannot replace
| a programmer fully.
|
| However, I have two counters:
|
| - First, the rational argument right now is that one person and
| money spent toward LLMs can replace three - or more - programmers
| total. This is the argument with a three year bound. The current
| technology will improve and developers will learn how to use it
| to its potential.
|
| - Second, the optimistic argument is that a combination of the
| LLM model with larger context windows and other supporting
| technology around it will allow it to emulate a theory of mind
| that is similar to the average programmer. Consider Go or Chess -
| we didn't think computers had the theory of mind to be better
| than a human, but it found other ways. For humans, Naur's advice
| stands. We cannot assume that this is true if there are tools
| with different strengths and weaknesses than humans.
| ActionHank wrote:
| I think that everyone is misjudging what will improve.
|
| There is no doubt it will improve, but if you look at a car, it
| is still the same fundamental "shape" of a model T.
|
| There are niceties and conveniences, efficiency went way up,
| but we don't have flying cars.
|
| I think we are going to have something, somewhere in the
| middle, AI features will eventually find their niche, people
| will continue to leverage whatever tools and products are
| available to build the best thing they can.
|
| I believe that a future of self-writing code pooping out
| products, AI doing all the other white collar jobs, and robots
| doing the rest cannot work. Fundamentally there is no
| "business" without customers and no customers if no one is
| earning.
| ebiester wrote:
| You cannot build a tractor unit (the engine-cab half of the
| tractor-trailer) with Model T Technology even if they are
| close.
|
| And the changes will be in the auxiliary features. We will
| figure out ways to have LLMs understand APIs better without
| training them. We will figure out ways to better focus its
| context. We will chain LLM requests and contexts in a way
| that help solve problems better. We will figure out ways to
| pass context from session to session that an LLM can
| effectively have a learning memory. And we will figure out
| our own best practices to emphasize their strengths and
| minimize their weaknesses. (We will build better roads.)
|
| And as much as you want to say that - a Model T was
| uncomfortable, had a range of about 150 miles between fill-
| ups, and maxed out at 40-45 mph. It also broke frequently and
| required significant maintenance. It might take 13-14 days to
| get a Model T from new york to los angeles today
| notwithstanding maintenance issues, and a modern car could
| make it reliably in 4-5 days if you are driving legally and
| not pushing more than 10 hours a day.
|
| I too think that self-writing code is not going to happen,
| but I do think there is a lot of efficiency to be made.
| rowanseymour wrote:
| If you forced me to put a number on how much more productive
| having copilot makes me I think I would say < 5%, so I'm
| struggling to see how anyone can just assert that "the rational
| argument right now" is that I can be 200% more productive.
|
| Maybe as a senior dev working on a large complex established
| project I don't benefit from LLMs as much as others because as
| I and the project mature.. productivity becomes less and less
| correlated with lines of code, and more about the ability to
| comprehend the bigger picture and how different components
| interact... things that even LLMs with bigger context aren't
| good at.
| spacemadness wrote:
| This is what I tried explaining to our management who are
| using lines of code metrics on engineers working on an
| established codebase. Other than lines of code being a
| terrible metric in general, they don't seem to understand or
| care to understand the difference.
| edanm wrote:
| > If you forced me to put a number on how much more
| productive having copilot makes me I think I would say < 5%,
| so I'm struggling to see how anyone can just assert that "the
| rational argument right now" is that I can be 200% more
| productive.
|
| If you're thinking about Copilot, you're simply not talking
| about the same thing that most people who claim a 200%
| speedup are talking about. They're talking about either using
| chat-oriented workflows, where you're asking Claude or
| similar to wholesale generate code, often using an IDE like
| Cursor. Or even possibly talking about Coding Agents like
| Claude Code, which can be even more productive.
|
| You might still be right! They might still be wrong! But your
| talking about Copilot makes it seem like you're nowhere near
| the cutting edge use of AI, so you don't have a well-formed
| opinion about it.
|
| (Personally, I'm not 200% productive with Coding Agents, for
| various reasons, but given the number of people I admire who
| _are_ , I believe this is something that will change, and
| soon.)
| geraneum wrote:
| > But your talking about Copilot makes it seem like you're
| nowhere near the cutting edge use of AI, so you don't have
| a well-formed opinion about it
|
| You can use Claude, Gemini, etc through Copilot and you can
| use the agent mode. Maybe you do or maybe you don't have a
| well formed opinion of the parent's workflow.
| ebiester wrote:
| I don't think about it in lines of code, but let me say that
| there are some efficiencies being left on the table.
|
| It helps because I am quicker to run to a script to automate
| a process instead of handling it manually, because I can bang
| it out in 15 minutes rather than an hour.
|
| I am more likely to try a quick prototype of a refactor
| because I can throw it at the idea and just see what it looks
| like in ten minutes. If it has good testing and I tell it not
| to change, it can do a reasonable job getting 80% done and I
| can think through it.
|
| It generates mock data quicker than I can, and can write good
| enough tests through chat. I can throw it to legacy code and
| it does a good job writing characterization tests and
| sometimes catches things I don't.
|
| Sometimes, when I'm tired, I can throw easy tasks at it that
| require minimal thought and can get through "it would be nice
| if" issues.
|
| It's not great at writing documentation, but it's pretty good
| at taking a slack chat and writing up a howto that I won't
| have the time or motivation to do.
|
| All of those are small, but they definitely add up.
|
| That's today and being compared to 5% improvement. I think
| the real improvements come as we learn more.
| BenoitEssiambre wrote:
| Solomonoff induction says that the shortest program that can
| simulate something is its best explanatory theory. OpenAI
| researchers very much seem to be trying to do theory building (
| https://x.com/bessiambre/status/1910424632248934495 ).
| ninetyninenine wrote:
| Can he prove what he says? The foundation of his argument rests
| on hand waves and vague definitions on what a mind is and what a
| theory is that it ultimately goes nowhere. Then he makes a claim
| and doesn't back it up.
|
| I have a new concept for the author to understand: proof. He
| doesn't have any.
|
| Let me tell you something about LLMs. We don't understand what's
| going on internally. LLMs say things that are true and untrue
| just like humans do and we don't know if what it says is a
| general lack of theory building ability or if it's lying or if it
| has flickers of theory building and becomes delusional at other
| times. We literally do not know. The whole thing is a black box
| that we can only poke at.
|
| What ticks me off is all these geniuses who write these blog
| posts with the authority of a know it all when clearly we have no
| fucking clue about what's going on.
|
| Even more genius is when he uses concepts like "mind" and
| "theory" building the most hand wavy disagreed upon words in
| existence and rest his foundations on these words when no people
| ever really agree on what these fucking things are.
|
| You can muse philosophically all you want and in any direction
| but it's all bs without definitive proof. It's like religion. How
| people made up shit about nature because they didn't truly
| understand nature. This is the idiocy with this article. It's
| building a religious following and making wild claims without
| proof.
| turtlethink wrote:
| To add on that - the human mind is a black box that we don't
| understand, we don't know what's going in internally, and most
| descriptions like the one here are arbitrary, made up, and
| don't reflect reality
| fedeb95 wrote:
| there's an additional difficulty. Who told the man to build a
| road? This is the main stuff that LLMs or any other technology
| currently seem to lack, the "why", a reason to do stuff a certain
| way and not another.
|
| A problem as old as human itself.
| lo_zamoyski wrote:
| Yes, but it's more than that. As I've written before, LLMs (and
| all AI) lack _intentionality_. They do not possess concepts.
| They only possess, at best, conventional physical elements of
| signs whose meaning, and in fact identity as signs, are
| entirely subjective and observer relative, belonging only to
| the human user who interprets these signs. It 's a bit like a
| book: the streaks of pigmentation on cellulose have no
| intrinsic meaning apart from being streaks of pigmentation on
| cellulose. They possess none of the conceptual content we
| associate with books. All of the meaning comes from the reader
| who must first treat these marks on paper _as signs_ , and then
| interpret these signs accordingly. That's what the meaning of
| "reading" entails: the interpretation of symbols, which is to
| say, the assignment of meanings to symbols.
|
| Formal languages are the same, and all physical machines
| typically contain are some kind of physical state that can be
| changed in ways established by convention that align with
| interpretation. LLMs, from a computational perspective, are
| just a particular application. They do not introduce a new
| phenomenon into the world.
|
| So in that sense, of course LLMs cannot build theories strictly
| speaking, but they can perhaps rearrange symbols in a manner
| consistent with their training that might aid human users.
|
| To make it more explicit: can LLMs/AI be powerful practically?
| Sure. But practicality is not identity. And even if an LLM can
| produce desired effects, the aim of _theory_ in its strictest
| sense is understanding on the part of the person practicing it.
| Even if LLMs could understand and practice theory, unless they
| were used to aid _us_ in our understanding of the world, who
| cares? _I_ want to understand reality!
| fedeb95 wrote:
| I get your point and I agree to a certain extent. However,
| it's arguable that everyone shares the same aim, that is, to
| understand reality. Some want to go down a road, no matter
| how it got built, or, some think, or better don't think, no
| matter where it leads. In that world, an artificial entity
| that can 1) create an aim and 2) build enough understanding
| to execute and 3) execute could be valuable. Right now we're
| at the 3) in the specific context of byte arrays. Now, an
| artificial system that could also understand, i.e. possess
| some kind of structure of concepts, and from there also
| produce the "need" to create something, that would be a huge
| leap forward. Forward toward what? I don't know.
| xpe wrote:
| > Theories are developed by doing the work and LLMs do not do the
| work. They ingest the output of work.
|
| This is often the case but does not _have_ to be so. LLMs can use
| chain of thought to "talk out loud" and "do the work". It can use
| supplementary documents and iterate on its work. The quality of
| course varies, but it is getting better. When I read Gemini 2.5's
| "thinking" notes, it indeed can build up text that is not
| directly present in its training data.
|
| Putting aside anthropocentric definitions of "reasoning" and
| "consciousness" are key to how I think about the issues here. I'm
| intentionally steering completely clear of consciousness.
|
| Modern SOTA LLMs are indeed getting better at what people call
| "reasoning". We don't need to quibble over defining some quality
| bar; that is probably context-dependent and maybe even arbitrary.
|
| It is clear LLMs are doing better at "reasoning" -- I'm using
| quotes to emphasize that (to me) it doesn't matter if their inner
| mechanisms for doing reasoning don't look like human mechanisms.
| Instead, run experiments and look at the results.
|
| We're not talking about the hard problem of consciousness, we're
| talking about something that can indeed be measured: roughly
| speaking, the ability to derive new truths from existing ones.
|
| (Because this topic is charged and easily misunderstood, let me
| clarify some questions that I'm not commenting on here: How far
| can the transformer-based model take us? Are data and power
| hungry AI models cost-effective? What viable business plans
| exist? How much short-term risk, to say, employment and
| cybersecurity? How much long-term risk to human values, security,
| thriving, and self-determination?)
|
| Even if you disagree with parts of my characterization above,
| hear this: We should at least be honest to ourselves when we move
| the goal posts.
|
| Don't mistake my tone for zealotry. I'm open to careful
| criticism. If you do, please don't try to lump me into one "side"
| on the topic of AI -- whether it be market conditions,
| commercialization, safety, or research priorities -- you probably
| don't know me well enough to do that (yet). Apologies for the
| pre-defensive posture; but the convos here are often ... fraught,
| so I'm trying to head off some of the usual styles of reply.
| geraneum wrote:
| > it indeed can build up text that is not directly present in
| its training data.
|
| I'm curious how you know that.
| triclops200 wrote:
| Not the parent, but, it's kinda obvious in even degenerate
| logic cases:
|
| # user
|
| what's 78904 + 34567282902
|
| # Gemma 2.5 pro Identify the core task: The
| user wants to add two numbers: 78904 and 34567282902.
| Recognize the operation: The operation is addition (+).
| Perform the calculation: Align the numbers
| vertically by place value: 34567282902
| + 78904 ------------- Add
| column by column, starting from the rightmost column (ones
| place): 2 + 4 = 6 0 + 0 = 0
| 9 + 9 = 18 (write down 8, carry over 1) 2 + 8
| + 1 (carry-over) = 11 (write down 1, carry over 1)
| 8 + 7 + 1 (carry-over) = 16 (write down 6, carry over 1)
| 2 + 1 (carry-over) = 3 7 + 0 = 7
| 6 + 0 = 6 5 + 0 = 5 4 + 0 = 4
| 3 + 0 = 3 Combine the results: 34567361806
| Format the answer: Present the calculation clearly and state
| the final result.
|
| To find the sum of 78904 and 34567282902, you add the two
| numbers together: 34567282902
|
| + 78904 -------------- 34567361806
|
| So, 78904+34567282902=34567361806.
|
| The model definitely iteratively built up (useful and correct
| even) text that wasn't directly in the training data (I doubt
| that number ever appeared anywhere in there given that
| 34567361806 doesn't appear in any searches I do of various
| engines)
| geraneum wrote:
| > The model definitely iteratively built up (useful and
| correct even) text that wasn't directly in the training
| data
|
| The text is highly likely in training data, as it's
| textbook arithmetic instructions. It's the number that is
| probably not there. Simple arithmetic is one of the
| verifiable operation types (truths) with a straightforward
| reward function used to train CoT models. In your example,
| what's interesting to me is improving LLM inference with RL
| that can result in such wonderful outcomes, but that's
| perhaps a different question.
| xpe wrote:
| To answer directly: Ask a question. Watch the "thinking"
| process. Estimate the likelihood that all of the generated
| text is in the training data.
|
| Do you disagree with my claim?
|
| Or perhaps you were hoping for a very rigorous set of
| experiments?
| analyte123 wrote:
| The "theory" of a program is supposed to be majority embedded in
| its identifiers, tests, and type definitions. The same line of
| reasoning in this article could be used to argue that you should
| just name all your variables random 1 or 2 letter combinations
| since the theory is supposed to be all in your head anyway.
|
| Indeed, it's quickly obvious where an LLM is lacking context
| because the type of a variable is not well-specified (or
| specified at all), the schema of a JSON blob is not specified, or
| there is some other secret constraint that maybe someone had in
| their head X years ago.
| woah wrote:
| These long winded philosophical arguments about what LLMs can't
| do which are invariably proven wrong within months are about as
| misguided as the gloom and doom pieces about how corporations
| will be staffed by "teams" of "AI agents". Maybe it's best just
| to let them cancel each other out. Both types of article seem to
| be written by people with little experience actually using AI.
| turtlethink wrote:
| Most people in the tech space talking about AI, not only
| misunderstand AI - but they usually have a far greater
| misconception of the human mind/brain.
|
| The basic argument in the article above (and in most of this
| comment thread) is that LLMs could never reason because they
| can't do what humans are doing when we reason.
|
| This whole thread is amusingly a rebuttal of itself. I would
| argue it's humans that can't reason, because of what we do when
| we "reason", the proof being this article which is a silly output
| of human reasoning. In other words, the above argument for why
| LLMs can't reason are so obviously fallacious in a multiple ways,
| the first of which is that human reasoning is a golden standard
| of reasoning, (and are a good example of how bad humans are at
| reasonin.
|
| LLMs use naive statistical models to find the probability of a
| certain output, like "what's the most likely next word". Humans
| use equally rationally-irrelevant models that are something along
| the lines of "what's the most likely next word that would have
| the best internal/external consequence in terms of dopamine or
| more indirectly social standing, survival, etc."
|
| We have very weak rational and logic circuits that arrive at
| wrong conclusions far more often than right conclusions, as long
| as it's beneficial to whatever goal our mind thinks is
| subconsciously helpful to survival. Often that is simple nonsense
| output that just sounds good to the listener (e.g. most human
| conversation)
|
| Think how much nonsense you have seen output by the very
| "smartest" of humans. That is human reasoning. We are woefully
| ignorant of the actual mechanics of our own reasoning. The brain
| is a marvelous machine, but it's not what you think it is.
| edanm wrote:
| I really don't understand how people can write entire articles
| which can be disproven with an hour of work. This is like writing
| a long polemic on how AIs will never be able to play Chess
| because of... reasons... four years after Deep Blue beat
| Kasparov.
| andai wrote:
| If I understand correctly, the critique here is that is that LLMs
| cannot generate new knowledge, and/or that they cannot remember
| it.
|
| The former is false, and the latter is kind of true -- the
| network does not update itself yet, unfortunately, but we work
| around it with careful manipulation of the context.
|
| Part of the discussion here is that when an LLM is working with a
| system that it designed, it understands it better than one it
| didn't. Because the system matches its own "expectations", its
| own "habits" (overall design, naming conventions, etc.)
|
| I often notice complicated systems created by humans (e.g. 20
| page long prompts), adding more and more to the prompt, to
| compensate for the fact that the model is fundamentally
| struggling to work in the way asked of it, instead of letting the
| model design a workflow that comes naturally to it.
| drbig wrote:
| > If I understand correctly, the critique here is that is that
| LLMs cannot generate new knowledge, and/or that they cannot
| remember it.
|
| > The former is false, and the latter is kind of true -- the
| network does not update itself yet, unfortunately, but we work
| around it with careful manipulation of the context.
|
| Any and all examples of where an LLM generated "new knowledge"
| will be greatly appreciated. And the quotes are because I'm
| willing to start with the lowest bar of what "new" and
| "knowledge" mean when combined.
___________________________________________________________________
(page generated 2025-04-28 23:01 UTC)