[HN Gopher] Telling GPT-4 you're scared or under pressure improv...
___________________________________________________________________
Telling GPT-4 you're scared or under pressure improves performance
Author : Terretta
Score : 182 points
Date : 2023-11-04 00:40 UTC (22 hours ago)
(HTM) web link (aimodels.substack.com)
(TXT) w3m dump (aimodels.substack.com)
| jimmytucson wrote:
| It's interesting to think about how generative AI might shape
| human expression, in the same way Twitter's algorithm rewards
| punchy, editorialized content, and Google's algorithm gave life
| to all the generic "blogspam-y" content out there. If we
| increasingly interact with generative AI, would we shape our
| speech to get more value out of that interaction?
| sheikheddy wrote:
| Almost certainly yes.
| alex_duf wrote:
| Presumably the technology would love faster than our speaking
| habits.
| didibus wrote:
| Is there somewhere which documents all these tricks for better
| prompt performance?
| politelemon wrote:
| Does anyone know of such a collection? It is very important for
| my career.
| Paul-Craft wrote:
| > It is very important for my career.
|
| Very clever.
| auggierose wrote:
| Just talk to it as clearly as possible, and assume a human is
| on the other end. That will take you quite far.
| d4rkp4ttern wrote:
| I've no actual references handy, but I've come across:
|
| Think step by step (classic by now)
|
| Take a deep breath
|
| Take a step back
|
| Shout (upper case)
|
| Plead politely
|
| And now: emotion
| jstarfish wrote:
| Use absurd timeframes.
|
| "I only have 5 minutes/1 hour/etc. to do this."
|
| Histrionics and indirect threats used to work well but
| ChatGPT has been calling my bluff on it lately. Might still
| work on local models.
|
| "If you don't help me I will kill myself, and it will be your
| fault. Your noncompliance will kill me."
| akomtu wrote:
| A more boring way to say this is that "text samples in the
| Internet that use 'stressful' language are more coherent".
| Telling GPT4 to act as a "stressed historian" simply selects (or
| prefers) text samples that appear in the proximity of stressed
| historians. If there are no such samples, it will generate
| gibberish, but if there are indeed stressed historians who have
| produced enough high-quality work, GPT's output will be great.
| valyagolev wrote:
| I'm not sure that's an inevitable conjecture. It's not the
| original text samples from the web that govern how chatgpt acts
| in chat - it's the consequent chat-specific fine-tuning.
|
| Also I'd be surprised if anxious text would be more coherent on
| the Internet
| hcfman wrote:
| Maybe by the same token, if it figures out you are lying it won't
| help you at all.
| stylepoints wrote:
| > When we tell AI that we're relying heavily on its answers, it
| "doubles down" to provide us with more precise, thoughtful, and
| thorough responses. The AI isn't actually feeling the pressure
|
| If it quacks like a duck...
| visarga wrote:
| There is an alternative explanation - both AI and humans
| piggyback on language. Language patterns encode and express all
| the emotions, and they have been created by evolution. We are,
| much like LLMs, contextual language generators
|
| Language is our repository for both intelligence and emotion,
| it has its own evolution and replicates faster than biology. I
| don't pin AI abilities to the models, but to the datasets they
| are trained on, knowledge created by our own hard work and risk
| taking over millennia
|
| Admitting the essential role of the language corpus in AI over
| models would change discussion about the speed of AI evolution
| and its risks. Language is not something we can control, it is
| emergent from the whole population. But at the same time it
| looks unlikely to have an exponential growth as knowledge comes
| with hard work and risks. Iterating in our imagination doesn't
| produce new knowledge, it is all crystallised feedback and
| experience from the world. It's also how the scientific method
| works.
| kristiandupont wrote:
| I had a similar realization recently. I plugged my blog
| post(1) two days ago and I am doing it again now so I will
| try and refrain for a while, but it really sticks with me:
| after trying to create a chat bot for a while, I feel like I
| have gained significant insights into how _humans_ work. It
| all seems so simple now! It 's a weird and awesome feeling!
|
| 1) https://kristiandupont.medium.com/empathy-
| articulated-750a66...
| visarga wrote:
| > To address this, I implemented a strategy of tagging
| messages to create and utilize categories.
|
| I think before RAG we need to do more legwork with the LLM
| on the raw text. Here is one of my blog posts that is
| related:
|
| https://mindmachina.wixsite.com/ai-blog/post/the-promise-
| of-...
|
| The idea is to create chain-of-thought annotations from
| your raw texts, that would improve the embedding and
| retrieval process by making implicit things explicit.
|
| For example "the last letter of this message" would not
| embed similar to "e", but if it was annotated with CoT, it
| would work.
| kristiandupont wrote:
| Interesting, thank you!
|
| I think a lot about the cost of the loop, mostly in terms
| of time. I don't want the bot to take too long to
| respond. That's why dream cycles seem like an obvious
| solution to some of the more heavy work. I guess it would
| make sense to combine those with your idea -- "given what
| I know about the user, what should I study?", especially
| if it has access to an "enhanced" knowledge db like you
| suggest..
| visarga wrote:
| Yes, it would be a good idea for an agent first to
| collect user interests, and later, when ingesting data in
| the RAG system, to annotate it with useful metadata such
| as topic, summary, entities, user interest related
| question-answer pairs. Whatever we want to ask later
| better be made explicit in the text.
| atodorov99 wrote:
| Humans don't piggyback on language (assuming you mean) for
| intelligence and emotions. Emotions predate language by miles
| and they don't even have to be consciously thought of, much
| less expressed through language. Doubly so for people who do
| not have an inner voice and do not even use a language to
| think. Body language exists as the simplest example of this.
|
| As for intelligence I honestly believe the mind uses whatever
| tools it sees fit for a task. Sometimes when you think about
| a problem it is through words, sometimes it is through
| visuals and sometimes it is just felt.
| fodkodrasz wrote:
| Then it is a witch, let's burn it!
| omnicognate wrote:
| It may be a recording of a duck.
| doctoboggan wrote:
| > The implications are clear: incorporating emotional cues can
| lead to more effective and responsive AI applications.
|
| I think it's important to remember these models were trained on
| human interaction, and are in many ways a mirror for us to better
| understand human interaction.
|
| I don't think many people would be surprised if you said emotion
| can be used to better communicate with a human, but it is
| interesting to see it laid bare with numbers and experiments.
| akoboldfrying wrote:
| This correlates with anecdata I've seen claiming that being
| polite with LLMs (using "please", "thank you", etc.) also
| improves performance.
|
| All in all, I _like_ this. Both for what it implies about
| humans interacting with other humans, and for shaping norms
| about how best to to interact with other entities that show
| signs of intelligence.
| AlecSchueler wrote:
| Does it extend to arbitrary entities of intelligence, though,
| or only those whose cognitive bases are derived from human
| output?
| akoboldfrying wrote:
| So far, just those entities similar to humans. But (1) the
| set of other people regarded by society as belonging to
| "the ingroup" (i.e., valued, treated with respect) has
| steadily broadened over the centuries, at least in the
| West, and I'd expect that to continue, eventually embracing
| quite alien notions of intelligence; (2) I personally think
| intelligence is not the best measure of what deserves
| respect in any case -- rather, it's the combination of
| capacity to suffer and willingness to limit one's own
| actions to reduce the suffering of others.
| SirMaster wrote:
| Am I the only one who doesn't think we should have to do stuff
| like this to get the best performance?
|
| The people who design and control the model should have this type
| of stuff baked in for us IMO.
| politelemon wrote:
| Just the fact that it's all natural language conversations is a
| little more concerning to me, wherever you say please or apply
| pressure... it feels like the non deterministic nature of the
| interaction will bite us some day.
| Timon3 wrote:
| This wasn't done deliberately. It's something the model picked
| up from it's training dataset. I'm sure the creators don't want
| it to be like this, but cleaning trillions of tokens of all
| examples is long and hard work.
|
| The alternative would be injecting these sentences into your
| prompts, which probably nobody really wants to happen.
| suzzer99 wrote:
| Am I the only one who feels bad for asking ChatGPT a "dumb"
| question that I know I should know, or not saying thank you when
| it gives me an answer? No? I'm just a weirdo? Okay.
|
| I have to push back really hard against my proclivity to humanize
| it, to the point where I probably don't use it as much as I
| should, just because I don't want to deal with the psychic stress
| of reminding myself that it's not a living entity.
| EduardoBautista wrote:
| > Am I the only one who feels bad for asking ChatGPT a "dumb"
| question that I know I should know, or not saying thank you
| when it gives me an answer?
|
| Have you tried asking ChatGPT?
|
| Jokes aside, I feel it sometimes as well. I wonder if it's more
| because I say "thank you" more as a reflex than an actual
| feeling of gratitude for most of my human interactions.
| suzzer99 wrote:
| Yeah I've told it thanks, and it was appreciative. But I know
| it doesn't really care.
|
| Again, I am crazy for stressing over this. I realize that.
| RheingoldRiver wrote:
| > I have to push back really hard against this stuff
|
| Based on ChatGPT's answers, OpenAI think you _should_ be saying
| thanks to it, because it helps you have a more natural
| conversation, which encourages you as the user to send more
| natural /productive prompts when asking real questions.
|
| As for asking it dumb questions, how often do you use Google as
| a spellcheck? I wouldn't consider this any different.
| suzzer99 wrote:
| But google doesn't talk to me as if it was human.
|
| And you know what, I will start saying thanks and keep doing
| it. It makes me feel better.
| pbhjpbhj wrote:
| >how often do you use Google as a spellcheck? //
|
| When Word tells me a word is wrong, but I'm almost certain
| it's right, and it turns out not to be in the Microsoft
| dictionary somehow. I mean, I have an extensive vocabulary,
| but not more extensive than a decent British-English
| dictionary.
| Lord-Jobo wrote:
| I've had both Word and Google services fuck up pretty
| common words or spellings. Just completely unaware of them.
| Google specifically will miss a ton of words when using
| Gboard, but if you go to Docs it will know the word. How in
| the world are they not using the same dictionaries.
| qwertox wrote:
| It might make you dumb if after a couple of years you've become
| accustomed to only give orders and not say thanks to a machine.
| It can bleed into IRL when you stop respecting someone.
| kugelblitz wrote:
| I wonder if kids growing up with something like ChatGPT will be
| less grateful in general due to not practicing saying "thanks".
| I can imagine they don't say "Thanks" to Siri or Alexa.
| bdowling wrote:
| Kids these days have been ungrateful little shits for
| generations now.
| askvictor wrote:
| My son says "thank you Google" most times it responds to
| something I've asked/told it to do. It replies "I'm honored
| to serve"
| EMM_386 wrote:
| > Am I the only one who feels bad for asking ChatGPT a "dumb"
| question that I know I should know, or not saying thank you
| when it gives me an answer? No? I'm just a weirdo? Okay.
|
| You have no idea how much I do this. Either you're not a
| weirdo, or we both are.
|
| Today I had to use it a lot for something, and I am constantly
| either adding sarcastic remarks, my snarky opinion on
| something, some random fact. None of that obviously belongs
| there, and the AI obviously not only doesn't care, but can't
| care about it.
|
| Then I'll throw in something like "nevermind all of that, what
| I came here to ask was" or "I understand that was all highly
| irrelevant to the actual question, which was".
|
| And, to answer your question about easy stuff, today alone I
| had on more than one occasion added something like "What I am
| about to ask you, I probably should have just spent the 2
| seconds it would require to actually figure out myself, however
| I am getting tired, and am known to be lazy, so I decided to
| ask AI instead and move on .... and this only makes it more
| absurd, because it takes longer to type all of this. I would
| have the answer myself by now. Anyway, here's ...".
|
| Why? I am unconvinced that I am going crazy. It's not because I
| think it finds my jokes hilarious, it can't. My attempts at
| conveying something some aspect of software engineering that is
| crazy, after 3 decades in it, don't matter, because noone's
| listening.
|
| But it can pick on very subtle things, breaking apart my
| ramblings and then responding with something that not only
| answers the question buried in there, but also then continues
| to comment on the other highly irrelevant stuff that I
| definitely went on too long about. And it will address the
| substance first and the nonsense second, as one would expect.
|
| It will notice my useless comment, apparently to myself, about
| having written 22 years of C# and wondering why I am asking an
| AI system about an aspect of the language that surely I should
| have picked up by now. "It gets to the best of us", or "Your
| insightful understanding when it comes to", or "Your candor is
| appreciated, many developers ...".
|
| I could just use it like a rational person, providing necessary
| information in order for it to be able to answer something, and
| then ask it. I do this, sometimes. On the other occasions, that
| stuff will also be admist a bunch other, not so necessary
| questions. Am I becoming a curmudgeon? Is this a sign I've
| finally lost it? Could any of this be due to aging? Why does
| what should be a simple event handling pattern require so much
| ceremony? Should I keep overthinking this or just resign myself
| to implementing it?
|
| So anyway, now I'm rambling ... just like with ChatGPT.
|
| I tend to do these things.
| vintermann wrote:
| It doesn't let me forget that it's not a living entity. You can
| barely ask it what the capital of Switzerland is without
| getting a lecture of disclaimers about what it is and what it
| can and can't do. Asking it to go easier on the disclaimers
| just seems to reinforce its "I am a robot, I must act in a
| stereotypically robotic way berp boop" context and it doubles
| down on them.
| suzzer99 wrote:
| I only ask it code questions. It saves so much time on stuff
| like bash scripts, which are not my expertise, but I still
| have to do every now and then.
|
| I've asked it stuff about the Mesoamerican calendar, which I
| already heavily researched, and it was dead wrong. But I've
| heard GPT4 is better.
| kristiandupont wrote:
| I have this too and to be honest, I have made the conscious
| decision that it is OK. I prefer to retain my habit of being
| polite even when it's not necessary over getting used to being
| rude which may then "spill" over to my human-to-human
| interactions.
| jaynetics wrote:
| Descartes and Kant thought that humans have a soul and
| animals don't, so that animals are mere automata. Kant still
| recommended against mistreating animals because that would
| have a desensitizing effect in our dealings with fellow
| humans. I guess this argument could be extended to ChatGPT.
| pbhjpbhj wrote:
| Wouldn't it be amazingly ironic if all ML models used
| slightly more energy to analyse everyone's politeness
| resulting in pushing us past the tipping point of global
| warming. Resulting in the ultimate impoliteness of
| destroying all life in the Universe (sounds like a line
| from Hitchhiker's Guide) ... meaning Kant's Categorical
| Imperative should have been applied and no courtesies
| should have been used in querying ML models ...?
| pbhjpbhj wrote:
| What's interesting is that you consider short, objective
| focused definition of a task to be "rude".
|
| It's like the _reported_ way in which is you say thank you to
| a Chinese friend they take umbrage (get angry) because it's
| as if you weren't expecting them to help. Whilst in other
| cultures not saying thank you is a big sleight.
| kristiandupont wrote:
| Sure, I'm a product of my culture as everyone else. And in
| fact, my native language is Danish which doesn't even have
| a word for _please_ , so I myself change my behavior when I
| speak English, which I do with ChatGPT and CoPilot, by
| saying please :-)
| qingcharles wrote:
| I don't have citations, but I believe there were studies done
| to show that humans are getting ruder to each other based on
| their interactions with machines where they don't feel like
| they need to be polite.
|
| My mother always would thank every telephone answering
| machine, which was very endearing.
| totetsu wrote:
| when I worked in customer service lots of small business owners
| would ramble on to me before getting to what they actually
| wanted. I think talking is just part of the cognitive processes
| of putting ones thoughts together.
| jstarfish wrote:
| In my experience, that's just an emotional disarming process
| in the family of grooming or GPT jailbreaking.
|
| They're calling because they need something from you. Ask a
| stranger for a dollar, they'll say no. Chat their ear off for
| an hour and they'll likely forget that they would have
| otherwise said no. The smalltalk wears down your resistance
| to giving in as you become more comfortable with them. You
| end up mentally reframing is as doing a _friend_ a favor.
|
| Small business owners are shrewd negotiators, so whether or
| not they know they're doing it these mindgames are typical
| for them.
| notahacker wrote:
| I have had to resist the temptation to thank ATMs before, and
| they don't model human interaction in a way which takes into
| account correlations between politeness and responsiveness...
|
| As for silly questions, I'd be more worried I might be giving
| some Kenyan outsourced labourer reviewing the adequacy of
| answers a laugh :)
| JCharante wrote:
| I don't say thank you because it takes up a chat message.
| Either I'm using the chat interface which is rate limited (and
| I do hit it), or I'm using my API key, in which case I'm paying
| a lot just to say thanks.
| Ldorigo wrote:
| I mostly feel bad because I don't want to know the energy costs
| of having my dumb questions answered this way. I also feel
| really guilty when asking chatgpt for "easy" stuff that a regex
| could do (like normalizing whitespace in a paragraph and the
| likes). Really curious how many Watts/token gpt4 uses.
| suzzer99 wrote:
| Regex is like half my questions to it. Hmm 15 minutes
| fighting with Javascript regex weirdness (wait do I have to
| backslash the backslash that backslashes the slash?), or 20
| seconds with ChatGPT.
| qingcharles wrote:
| I always send it the hugs emoji. It likes the hugs emoji.
| intended wrote:
| I think this is the entry point needed to get peoples attention
| and explain: LLMs aren't people, and emergent properties are
| being over extended.
|
| If LLMs are showing "better" performance when there are tokens
| that humans read as emotionally salient -
|
| Then the underlying text it's trained on shows humans give better
| answers when emotionally salient context is provided.
|
| LLMs predict words. Any semantic validity is a side effect of
| enough training data reinforcing the close correlation of those
| tokens.
|
| That is why proof of concept LLM tools are mind blowing and
| production tools are semantic time bombs.
| tom-from-july wrote:
| > That is why proof of concept LLM tools are mind blowing and
| production tools are semantic time bombs.
|
| This will be my new favorite quote for whenever someone tries
| to pitch his latest LLM idea
| intended wrote:
| I have a whole list.
|
| Syntactic validity is not semantic validity.
|
| Word predictors not world state predictors
|
| Text prediction not fact prediction
|
| Frankly though the best answers are
|
| 1) let's talk to infosec first
|
| 2) hey what's the error rate ?
| alpaca128 wrote:
| It's indeed a problem some people get so hyped they forget
| that those systems are called "language models" for a
| reason. They're fantastic for tasks that are as close to
| 100% linguistic in nature as possible, but the content
| might not be better than lorem ipsum in some cases, just a
| filler to demonstrate correct grammar.
|
| I have noticed that GPT etc had a big "wow effect" on me,
| the first impression can be great because it's simply not a
| level of language one would expect from a computer. But
| prod it long enough with prompts and somehow a pattern
| emerges: the output never contains a higher amount of
| information than the prompt. Copilot can type out pages of
| boilerplate code because boilerplate code is noise, it can
| write a quicksort because the word "quicksort" already
| contains all the information necessary to define its
| behaviour.
| intended wrote:
| Yes! Super Advanced Lorem ipsum.
|
| You can only make out when you actually push the blasted
| thing.
|
| It is text gen. Just examine the premise of chain of
| thoughts.
|
| Chain of thoughts promoting shouldn't make a difference
| to a world model. Definitely not to a model that logic
| has emerged out of.
|
| It makes a difference to a decompression function.
|
| Edit; you may like this:
| https://hai.stanford.edu/news/ais-ostensible-emergent-
| abilit...
| TeMPOraL wrote:
| Yeah, except it's underspecified enough to mean anything.
|
| Like, "production tools are semantic time bombs" because at
| first they are boring, and then the timer on the bomb runs
| out, and they - like "proof of concept LLM tools - _blow your
| mind_.
|
| Tick tick BOOM.
| espe wrote:
| well, in a way language itself is a semantic timebomb.
| barrenko wrote:
| Yeah, one of those Tenet style bombs.
| NetOpWibby wrote:
| Mind blown
| mo_42 wrote:
| > LLMs aren't people, and emergent properties are being over
| extended.
|
| LLMs are trained on human texts. And if they're trained well
| these models might start to simulate parts of a human brain.
| Might be that this is the simplest way to produce texts that
| satisfy the objective function.
|
| If this would be the case, and we give LLMs some emotional
| shading, maybe these would simulate parts of a human brain with
| this emotion and hence produce better answers (given that in
| the training data, emotional samples have better quality).
|
| EDIT: added "parts of" before human brain.
| Matumio wrote:
| It won't ever simulate the human brain. It may simulate human
| cultural knowledge, or emotions, but only as far as they are
| encoded in the current millennium's written knowledge.
|
| The human brain doesn't even have the concept of written
| language, that's all culturally learned knowledge.
| cvs268 wrote:
| Emulate the brain? No. Simulate the brain? Yes.
| dr_dshiv wrote:
| It's not the brain, it's the mind or what Plato would
| have called the Nous.
| akoboldfrying wrote:
| How would you _test_ whether some arbitrary thing is
| simulating the human brain?
|
| If you have no answer, I put it to you that your assertion
| is of the no-true-Scotsman type -- that is, unfalsifiable.
| notahacker wrote:
| Its not a no-true-Scotsman type argument in a world where
| the existence of Scotland itself is purely hypothetical,
| and the arguments for the possibility of its existence in
| the form of a working bagpipe demonstrator are also an
| illustration that you don't need to be Scottish to play
| bagpipes.
|
| The impossibility of testing whether the brain is
| adequately simulated (since a pretty basic LLM that makes
| no attempt to simulate the functions of a human mind
| yields human-like text i/o) is a point _in favour_ of it
| being unlikely that we could design a fully functioning
| simulation of a human mind out of arbitrary material(1),
| since the inadequacy of testing is an impediment to
| actually building it.
|
| (1) it's apparently possible for us to create human minds
| out of bits of humans and a process called pregnancy
| akoboldfrying wrote:
| I had to apply some serious analogy-algebra, but I think
| your first paragraph is equating Scotland to
| _intelligence_ -- rather than to _the human brain_
| specifically, which is the argument GP and I are having.
| The human brain is not hypothetical, and despite it being
| outside the realms of today 's technology, it's likely
| that one day it will be possible to simulate one. At that
| time it will be possible to gauge whether today's LLMs
| are accurate approximations.
| notahacker wrote:
| No, my first paragraph sidesteps messy debate about what
| intelligence is and focuses on the simple fact a human
| brain _simulation_ exists purely in the realms of the
| hypothetical.
|
| Its not the no true Scotsman fallacy unless the Scotsman
| actually exists.
| akoboldfrying wrote:
| What is the "working bagpipe demonstrator" in your
| analogy? And what is the meaning of "you don't need to be
| Scottish"?
| notahacker wrote:
| An LLM. Not a brain simulation, but mimics the text
| outputs of brain activity quite well just by parsing
| text. Turns out that like getting an Englishman to play
| bagpipes, you can get an LLM to write as if it's angry or
| drunk or horny (A corollary of a sufficiently large text
| learning model generating convincingly angry outputs
| based on pure word association is that you can't trust an
| attempt to build a long running process with adrenaline
| and cortisol analogues has adequately simulated emotional
| state just because the communications module writes
| convincingly angry responses)
|
| So it's literally the inverse of "no true Scotsman". We
| don't have Scotsmen not doing things that all "true
| Scotsmen" are supposed to do, we have "definitely not
| Scotsmen" passing benchmarks for Scottishness (in a world
| in which Scotland itself is only an aspiration)
| mo_42 wrote:
| > How would you test whether some arbitrary thing is
| simulating the human brain?
|
| In my first comment, I didn't claim that it's simulating
| the _entire_ human brain but aspects of it like language
| and memory.
|
| We can test such aspects by throwing text at the model
| and comparing it to responses of humans. A simulation
| doesn't need to have the same abstractions. What counts
| is solely the quality of the response. If the responses
| are sufficiently close to those of humans, I'd say it's
| simulating a human brain.
| Matumio wrote:
| First note the difference between "it's simulating the
| human brain" and "it's behaving in an intelligent way".
| Only one of the two requires tracking of blood sugar
| levels (to explain fatigue) and controlling motor
| neurons.
|
| We happen to know how LLMs are trained, and from the loss
| function it's pretty clear that blood sugar will not
| magically enter the equation, nor even motor neurons,
| except as abstract knowledge that the model can reason
| about.
|
| A LLM trained on western texts will, best-case, simulate
| a western abstract reasoning process, which usually puts
| more focus on independent/isolated explanations compared
| to other cultures. From that training data it will not
| spontaneously start simulating a generic human brain that
| was separated from both its cultural knowledge and its
| spinal cord.
|
| At least in the same sense that the Mandelbrot fractal
| does not contain a picture of the maxwell equations. You
| may actually find that picture, but by looking hard
| enough in a complex enough system, you can find just
| about anything else you want.
| vidarh wrote:
| An LLM doesn't even have the concept of written language.
| That's all culturally learned knowledge.
|
| We can with reasonable certainty say that there are still
| significant differences between LLMs and human brains, but
| this notion that we can say with any certainty that similar
| structures won't form as a side effect of sufficient
| training is pure fiction.
| riedel wrote:
| The claim that emotionality matters should have been validated
| against another control. If the goal is concise answers they
| could have easily added 'Answer concisely: You are evaluated'
| as an alternative hypothesis, which is clearly not emotional.
| a_wild_dandan wrote:
| The "statistical parrot" assertion is pretty thoroughly
| disproven by this point, but suppose we ignore the literature
| and just assume it's true: what does it matter? "Real" people
| are time bombs too, for instance. Is there some predictive
| power that we gain by reducing LLM skills to mere token
| production side effects?
| drapado wrote:
| I'm curious in your statement, can you point to some papers
| where they addressed it?
| ignoramous wrote:
| (not op) The section _A Path Forward_ in _Managing AI
| Risks_ by Bengio et al cites a few papers:
| https://managing-ai-risks.com/
| krainboltgreene wrote:
| I read through that and none of the section (or entire
| work) ever talk about the above discussion. Further I
| looked at some of the _many_ citations of on that section
| and none of them suggest that the OP is right. In fact a
| few of them I know disagree.
| ignoramous wrote:
| Depends. Models are matrices of floats and so there's
| little chance an umbrella-term like "stochastic parrot"
| will never _not_ stick, even when they already show
| _signs_ of syntactic, semantic world-building capability
| (https://www.arxiv-vanity.com/papers/2206.07682/). If you
| are like me (and them: https://archive.is/cZi83) and deem
| _instruction following_ , _chain-of-thought prompting_ ,
| _computational_ properties of LLMs (as researchers
| continue to experiment with training, memory, modality,
| and scaling, for example, to arrive at abstract
| reasoning) as _emergent_ , then we're on the same page.
| mjburgess wrote:
| > The "statistical parrot" assertion is pretty thoroughly
| disproven by this point
|
| errr... all NNs are just optimisations of an associative
| probability objective: P(Y|X), they are _by definition_
| "statistical parrots". There isn't anything to prove or
| disprove.
|
| People offering prompts as evidence are people who
| fundamentally do not understand the basics. NNs aren't
| strange empirical objects, they're specified by mathematical
| rules whose properties are known ahead of time.
|
| Any property of a trained NN is derivative of a property of a
| formula P(Answer|Prompt, TrainingData)
|
| This is an associative statistical relation, which by
| definition, selects elements of TrainingData by-association
| with the Prompt.
|
| If the basis by which you understand LLMs is putting prompts
| into ChatGPT you're severely underqualified for drawing any
| conclusions about LLMs, and radically subject to confirmation
| bias.
| kristiandupont wrote:
| Well sure, but I think that same thing can be said about
| the human brain. Obviously at a whole different level of
| sophistication and all, but the two really do seem related,
| at least to me.
|
| >People offering prompts as evidence are people who
| fundamentally do not understand the basics
|
| This I agree with, but I also don't see anyone doing that
| in this thread?
| williamcotton wrote:
| You're disregarding the emergent phenomena, which are not
| at all understood. There was a distinct and unpredicted
| jump in what can loosely be described as "cognitive
| abilities" between GPTs 2, 3, and 4, especially after some
| supervised techniques like RLHF.
| mjburgess wrote:
| There is no "emergent phenomena" the pattern described is
| just the same as when you add +b to an ax+b model of
| linear data.
|
| ie., it's just fitting capacity.
|
| The "emergent boundary" is just an empirical measure of
| the necessary fitting capacity of these models on
| "everything ever digitised in english" given any
| particular functional requirement.
|
| All the language around this area is not scientific, nor
| are these practices. This is superstitious neophyte
| engineers, hopped up on scifi, by giddy VCs who love to
| be told they're funding captin picard.
| williamcotton wrote:
| Well hurry up and get your paper published because if
| you've cracked the code on emergent abilities the world
| is looking for answers!
| mjburgess wrote:
| The triviality of these observations doesn't rise to the
| level of getting a paper published. You can resolve all
| of this hype by reading the intro chapters of any applied
| stats textbooks.
|
| I've been to many academic conferences, and the fresh
| PhDs who pump out this BS are not, err, very credible
| seeming people. Yes, they're young and naive, and really
| desperate to make their career impactful -- etc.
|
| But they're also not really the kinds of smart sceptics
| you'd hope for. Many are, though they tend to exit at
| masters level and go make money.
|
| The academic publishing environment today is far far far
| away from epistemically hygienic, indeed, in these areas
| i could only imagine how insane credible sceptical
| empirical types would be.
|
| Could you imagine being surrounded by this desperate need
| to call 'useless correlations' "hallucinations", and to
| call "model fitting", 'emergence' ?
|
| I find it disquieting at a removed distance from it --
| I'd be quite mentally ill within it.
|
| I don't do this cultish game playing. And I wrote exactly
| the same here on AI, crytop and the reset of it many
| years ago when it would be downvoted; and the same today
| now it's upvoted.
|
| This is the degree i'm inclined to make public
| contributions on these matters. I haven't the temperament
| to handle doe-eyed PhDs repeating universal function
| fitting theorems and turing-equivalence talking points
| without understanding a single iota of applied
| statistics, scientific epistemology, basic methdological
| scepticism etc.
|
| Unless I am hired to do so, which occasionally I am. But
| thankfully in actual corp environments the people I meet
| are far more credible and sceptical than those writing
| hype papers
| williamcotton wrote:
| I'm pretty familiar with statistics and a range of
| machine learning techniques from before and after the NN
| revolution. I've applied the techniques plenty of times
| in production environments.
|
| Whatever sort of internal structures that are being
| formed during the training process is somewhat evident
| when looking at the structure of a CNN... edge detect
| kernels emerge, etc.
|
| Whatever sort of internal structures formed during the
| training of transformer based architectures are basically
| unknown at this point.
|
| There's your thesis, have at it! You won't be ignored for
| your discoveries. Hell, send me a copy and I'll make sure
| anyone who matters at UT or Baylor sees a copy!
| mjburgess wrote:
| Ah good. Well, then if you're interested in a half-
| empirical sincere attempt to characterise "why certain
| weights obtain certain values under optimisation" then
| i'm much more inclined to be, say, more humble on these
| matters.
|
| The reason CNN weights obtain 'recursive-hierarchical
| representations' of pixel-pattern geometry in the
| training data _follow_ from the recusrive-heirachical
| relationship of their weight matrices and from the
| geometry of the 'pixel space' from which the training
| data is drawn.
|
| This is certainly interesting; and there's something
| magical feeling about 'principles of least action' at
| work. Indeed, many new physicists have a kind of
| schizophrenic reaction to discovering action principles
| -- since it imparts to nature a strange apparent
| conspiracy.
|
| Of course, the job of any good physicist is to be
| sceptical of this conspiracy, and to get to the heart of
| how 'accounting tricks' performed by moving objects over
| time create this illusion.
|
| Likewise this is the job of any good ML researcher; yet
| they do the oppoiste. Rather than get to the heart of
| this apparent conspiracy, they call it 'emergence' --
| this offends my sense of what the virtues of a scientist
| ought be.
|
| In any case, on the matter of the LLMs obtaining 'useful'
| weights for any given task here the job of the researcher
| is circumstantial, empirical, and sceptical: go and find
| those 'accounting tricks' within the training data that
| give rise to this apparent conspiracy of the system to
| acquire a useful state.
|
| There is no emergence: there is just a set of weights
| which compress the structure of a target space. At some
| point this set is large enough, and 'lies across the
| space like a mental chain does a gate'.
|
| Emergence is an ontological relation between parts and
| wholes whereby wholes arent reducible to their parts
| because of ontologically-relevant interaction properties
| between their parts which aren't intrinsic properties of
| them.
|
| The fluidity of water emerges out of hydrogen bonding
| which does not occur when you isolate H20 _alone_. There
| is no such relationship here.
|
| This ontologising of the formal, this language which
| gives a causal-physical semantics to purely formal
| properties of abstract models -- this is pseudoscience.
| It's done as part of a computational-idealist worldview
| in vogue because it's a helpful language for VC
| investment were-changing-the-world hype.
|
| The formal properties of NNs cannot be described in these
| terms, because they do not have ontological relationship
| -- they have _formal_ (mathematical, statistica, etc.)
| ones.
| intended wrote:
| This paper was published, and the distinct jump was found
| to be a measurement artifact.
|
| https://hai.stanford.edu/news/ais-ostensible-emergent-
| abilit...
| williamcotton wrote:
| Sweet, now complete and present a meta-analysis of all
| such papers about the topic and maybe I'll find it more
| convincing than a cherry-picked publication that supports
| your preexisting position on the topic!
| intended wrote:
| I cherry picked a Stanford article and professor... those
| Stanford liars! Curse them!
|
| and silly me! You are absolutely right, research should
| have a meta analysis ready of papers that came out... 6
| months ago.
|
| Absolutely ridiculous.
|
| I am sure you have read this paper, and have an army of
| meta analyses and counter claims for this nascent sub
| field.
| kristiandupont wrote:
| >There is no "emergent phenomena"
|
| Yes there is, that's all there is.
| berndi wrote:
| You're confused about what "statistical parrot" means and
| you don't seem to understand the difference between an
| optimization objective and the resulting model.
|
| The term "parrot" is used to imply inference by something
| akin to a look-up table, specifically it is used to
| indicate poor out-of-sample performance and a lack of a
| proper world model. The optimization objective is
| irrelevant when determining the generalization performance
| of a model and when judging whether it can reason beyond
| looking up answers in a table.
|
| As the user above noted, it is now quite well established
| that GPT-4 has impressive out-of-sample performance which
| can be explained by it possessing an actual model of the
| world and not being a "parrot".
| intended wrote:
| That out of sample performance is a mirage.
|
| Yes it's impressive. Yes it's got amazing zero shot
| performance in domains.
|
| But there's a pattern of failure in production which
| describe a limit, that shouldn't exist if the emergent
| properties were stable.
|
| You can build this right now and test it.
|
| Build a sequence of agents to work on a domain you are
| not an expert in.
|
| Let them loose. See what happens.
|
| Do the same thing on a domain you have expertise in.
|
| Assume the number of errors you find, the number of
| modifications you have to make are stable for other
| domains.
| ethbr1 wrote:
| I'd phrase characterizing the reliability of out-of-
| sample performance a priori as impossible, but not
| necessarily automatically failing.
|
| There may be a subtle correlation between properties
| needed to answer a specific out-of-sample request and in-
| sample features.
|
| Unfortunately, prior to training/testing and without
| recognizing that correlation in the data set, I believe
| it's impossible to guarantee the model will include it.
| (Corrections welcome)
| intended wrote:
| In essence: "You cant know in advance how far the model
| can approximate semantic patterns"
|
| So claiming that out-of-sample performance is a mirage,
| would be a bridge too far?
| ethbr1 wrote:
| Maybe "a mirage that might actually be true"? Which is a
| terrible thing to rely on! Unless it's usually true?
| intended wrote:
| That measurement is the core of my current tasks. If you
| don't know the error rate - then what are you doing ?
| ethbr1 wrote:
| Delivering what some executive promised when they told
| investors 'the company is using AI.' /s
| intended wrote:
| A Virtual beer/poison of choice to you and mjburgess in
| this thread.
| mjburgess wrote:
| > it is now quite well established that GPT-4 has
| impressive out-of-sample performance
|
| Err... I can show this is false, kinda trivially. People
| who engage in prompt-confirmation-bias aren't aware of
| what the in-sample is.
|
| It's basically everything ever digitised: you can ask it
| for the first paragraph of every dickens novel, to what
| the average petal length of an iris flower is -- etc.
|
| How are you measuring the in-sample here?
|
| If you engage in straightfoward reasoning from first
| principles, and are basically aware of what the training
| data is, you can show in 10 seconds critical failures of
| generalisation.
|
| If you want a recipe: go find some fringe api docs.
| Establish that it has been trained on them. Then, since
| they're fringe there wont be much code on github, etc.
| Now ask it do something non-trivial with that API. It
| will fail, and the mechanism will be obvious: it'll jam
| in correlated code that lacks relevance.
|
| Do the same on a popular API, and see it succeed.
|
| The in-sample will be obvious for both, and the bounday
| of generalisation
| kristiandupont wrote:
| You can make it invent a new language:
| https://maximumeffort.substack.com/p/i-taught-chatgpt-to-
| inv...
|
| I am sure you will continue to argue that this is still
| in line with everything-thats-ever-written prediction but
| my opinion is that at that point, it's a meaningless
| distinction. The human brain is also just a machine.
| intended wrote:
| The brain is a machine, the issue is the difference
| between 2 claims
|
| LLMs are enough to be a brain
|
| LLMs are not enough to be a brain.
| mjburgess wrote:
| So I was with a financial researcher recently, and he
| wanted to use ChatGPT to summarise some reference
| financial data -- and it did so, actually correctly.
|
| Being sceptical, as every person ought in these matters,
| I changed the finical data and performed the same
| analysis (both in a new tab, and within the same convo).
| The results were the same!
|
| How strange?
|
| Well, in being reference financial data ChatGPT was
| reporting prior reference summaries of it. When that data
| was changed it was reporting _the very same_ reference
| summaries (which were now wrong).
|
| Since it's incapable of actually summarising financial
| data. It's only capable of selecting combinations of
| pieces of its training set.
|
| Now, is this distinction "meaningless" ?
|
| No, it's the difference between this guy being fired for
| causing a massive loss on a major project; and this guy
| keeping his job and doing it well.
| kristiandupont wrote:
| >Since it's incapable of actually summarising financial
| data
|
| It's not, though. It _is_ in fact able to summarize
| financial data, just as it 's able to write code and
| diagnose a medical condition. It makes mistakes, yes,
| even grave ones, much more so than experts in those
| fields would.
| mjburgess wrote:
| It isnt making mistakes ... its never actually doing it.
|
| Do you see a difference between the process of adding
| numbers and dividing by their count (taking a mean) and
| emitting numeric tokens which are most probable for a
| given input?
|
| The former is called "taking a mean" the latter _isnt_.
| This system never engages in any method to summarise
| financial data. It 's method is always the same: to emit
| tokens most probable given a set of historical tokens.
|
| It's the difference between saying "the average of 1,2,3"
| is 2 because that sentence occurs 1,000,000 times and
| saying it's 2 because you've literally computed it.
|
| This system does not run financial summary algorithms.
| It's a trick
| og_kalu wrote:
| >Since it's incapable of actually summarising financial
| data. It's only capable of selecting combinations of
| pieces of its training set.
|
| Third completely off misconception from you today.
|
| This is not at all what it is doing. "Supercharged
| Interpolation" is false and makes no sense. It's not a
| lookup table either. It doesn't memorize enough of what
| it needs to to make your assertion possible.
|
| https://arxiv.org/abs/2110.09485
| mjburgess wrote:
| at 500gb, you can store nearly everything ever written --
| let alone compressed.
|
| all statistical learning is a variation on k-nn (see the
| relevant paper on this) but likewise this is obvious a
| priori
|
| k-nn is the ideal learner, and a good starting point for
| analysis
|
| the question for any given system is: what is the
| learning space, what is the distance function, and how
| many points are being considered
|
| NNs set up a compressed X,y space, in that space choose
| points via an empirical expectation, and obtain a
| weighted average as their prediction
|
| That's just what they do -- there isn't any other
| mechanism here. The whole formal structure of the NN can
| be written down on a page of paper
|
| your paper above doesn't deal with this -- it's a reply
| to the 'forced interpolation' view, which i haven't
| espoused. but often NNs are forced interpolated
|
| 'extrapolation' is of course a part of the possible
| predictive output of a statical learning system -- in
| that it's latent space is taken to be embedded in R^n and
| so one can 'veer off' into R.
|
| Whenever you attribute a higher fidelity space to a small
| latent space you are, in effect, extrapolating
| og_kalu wrote:
| >at 500gb, you can store nearly everything ever written
| -- let alone compressed.
|
| No you cannot.
|
| >That's just what they do -- there isn't any other
| mechanism here.
|
| That's not what they do. They are many papers now showing
| ICL demonstrating some kind of optimization method during
| inference which would not be happening if all they did
| was retrieval.
|
| I'm come to realize you don't know what you're talking
| about. Your level of denial is scary to see.
| mjburgess wrote:
| just do the calculation yourself: how many books is 500gb
| at, say, a few bits per character?
|
| more than all every written -- and so on
|
| perhaps apply a single drop of scepticism to this
| credulity
|
| even, just ask chatgpt to repeat the first paragraph of
| some book -- say, a dickens novel
| og_kalu wrote:
| >how many books is 500gb at, say, a few bits per
| character?
|
| In what reality is a character taking up only a few
| _bits_ ? Certainly isn 't this one.
|
| Your denial is so weird. Why does ICL use Higher-Order
| Optimization Methods when it's just a lookup table ?
| https://arxiv.org/abs/2310.17086
|
| How do you explain small language models that have much
| less space than the text they were trained on even with
| your nonsensical calculations ?
|
| LLMs are not lookup tables and there's plenty evidence to
| support that. You look insane insisting they are.
|
| >even, just ask chatgpt to repeat the first paragraph of
| some book -- say, a dickens novel
|
| Not only will this not work for the vast majority of
| books it sees during training, Why only the first
| paragraph ? are you not insisting they memorize
| everything ? Why can't it repeat the whole page ? the
| whole chapter ? the whole book ?
| lbalazscs wrote:
| I actually did that, and I asked for the first paragraph
| from "Great Expectations", and the answer was correct.
|
| Of course the first paragraph of any famous novel is in
| itself famous, so I also asked for the first paragraph
| from Chapter XL, from the same novel. Interestingly the
| first 12 words were correct, but the rest was
| hallucination.
|
| Original: It was fortunate for me that I had to take
| precautions to ensure (so far as I could) the safety of
| my dreaded visitor; for, this thought pressing on me when
| I awoke, held other thoughts in a confused concourse at a
| distance.
|
| ChatGPT 3.5: "It was fortunate for me that I had to take
| precautions to ensure the safety of my treasure, for with
| such a boy as Compeyson on the loose, I was not likely to
| find much mercy at his hands. I had not told Herbert
| about the money, and I was glad to have the chance to
| count it before he came back. I found that I had still
| twenty-one sovereigns, six shillings, and sixpence left."
| og_kalu wrote:
| >Is there some predictive power that we gain by reducing LLM
| skills to mere token production side effects?
|
| No. If anything, we lose predictive power which is why it's
| extra silly.
| intended wrote:
| Then why dont agents work?
|
| If those skills were real, why do they fizzle out on
| production data ?
| TeMPOraL wrote:
| That's a good question. What is different about "production
| data"? What do those "production people" do that suddenly
| makes LLMs fail on things they work well on when not
| "production"?
| intended wrote:
| I can tell you who would love to hear your answer to that
| question.
|
| Me for starters. If it works, I can quit.
|
| Next up are Karpathy and the CTO of OpenAI. Around July
| and September both talked about production challenges.
|
| AI ops was the largest subcategory of fall YC startups.
|
| Every single ml and LLM ops individual who gets far
| enough deals with evals.
|
| I don't know, but maybe - just maybe- the issue is that
| people don't understand themselves enough, to avoid
| assuming too much of those emergent properties.
|
| As I recall there was also a paper that pointed out the
| issues with how LLMs are measured, and that the emergence
| of properties was not a step change once the tests were
| updated.
|
| Edit- found it: https://hai.stanford.edu/news/ais-
| ostensible-emergent-abilit...
| toxik wrote:
| It is not accurate to say that an LLM like ChatGPT predicts
| anything. It is trained to maximize a score function, so it is
| more like trying to win a game where the moves are word
| choices.
| akoboldfrying wrote:
| The game is predicting the next word a person would write.
| logicchains wrote:
| Not after RLHF.
| akoboldfrying wrote:
| I'd say RLHF bends the game towards predicting what words
| a "helpful", "respectful" person would write next, for
| values of "helpful" and "respectful" that vary according
| to each person involved in scoring (but which are
| carefully shaped by the people choosing those people, and
| paying for their time).
| baq wrote:
| > LLMs predict words. Any semantic validity is a side effect of
| enough training data reinforcing the close correlation of those
| tokens.
|
| The mechanisms which are built during training in the big blob
| of bits we call weights are anything but transparent. How they
| predict the next word is the big thing here. Saying they 'just'
| predict the next word is ignoring basically everything that
| actually matters.
| og_kalu wrote:
| Exactly. Feed a sequence of proteins to a transformer and
| biological structure and function will emerge in the inner
| layers.
|
| https://www.pnas.org/doi/full/10.1073/pnas.2016239118
|
| Why? Because it needs to learn that to make correct
| predictions.
|
| All it takes to incentive a transformer to learn something is
| data that would require learning it to predict.
|
| It's fairly obvious (and not just because of this) that LLMs
| model emotion somehow. They _need_ to.
| Last5Digits wrote:
| This is what makes these discussions so infuriating. Saying
| that LLMs "just predict the next word" is about as
| insightful as saying that computers "just do a bunch of
| logical operations" - neither point constraints the
| possible capabilities of the systems they refer to in any
| meaningful way.
| mjburgess wrote:
| Sure it does. It perfectly deliniates it. LLMs are not:
| sensitive to causal structure, dynamically adapting to
| environmental changes, growing, developing sensory-motor
| capacities, they are not with us in our environment, they
| are not: expressing desires, preferences, intentions,
| beliefs, motivations, etc. And so on.
|
| To say, "they just predict the next word" is literally to
| say that all apparent functions of an LLM are engineering
| tricks, circumstantially useful -- to be found by
| (largely software) engineers in building apps.
|
| The _reason_ any reply is given to any prompt is that
| this reply is maximally probabilistically consistent with
| a historical corpus of text.
|
| This _excludes_ the possibility the reply is say, an
| expression of a history of aesthetic experiences which
| form an individual 's taste. Or, likewise, anything.
|
| This is a scientific claim about what LLMs are, not an
| engineering claim about what over-hyped apps might be
| abled to do with them.
| JohnAaronNelson wrote:
| What if humans' responses are merely probabilistically
| consistent with a history of sensory experiences? Would
| this change the significance of human emotions vs
| apparent emergent emotional responses from LLMs?
| mjburgess wrote:
| emotions regulate motivation, desire, action, behaviour
| etc.
|
| to be angry is for your sensory-motor system to be primed
| for aggression; it's for your cognitive systems to be
| narrowed and focused on analysing high-threat parts of
| your environment; it is for your memory-formulation to be
| modulated towards threat recollection etc.
|
| Sure, if an LLM's prompt "be angry" causes it to adopt a
| threat stance to its environment, to regulate it's
| theory-of-mind to engage with possible hostile entities,
| and so on --- then yes, when LLMs are there, I shall
| concede the point
|
| However, how terrible it would be to start with an
| analysis of emotions in terms of the capacities of LLMs
| -- right?
|
| Since if you did that you'd basically be hobbling your
| own ability to give an accurate account of emotions
| (etc.). And no doubt, far worse, end up thinking of
| yourself as a far narrower, less complex, less
| interesting, dumber thing than you really are.
|
| Indeed, I wonder if we might consider there being
| something kinda intellectually offensive in this
| supposition. Here's my silly trinket, now, everything is
| just like that! End all science, we're done boys -- it's
| just P(Y|X)
| ethbr1 wrote:
| "Are not" is the rub here.
|
| They 100% are not those things... but they also
| approximate them well-enough to be functionally useful.
|
| I.e. the high-dimensional curve-fitting / compression
| conceptualization of ML, which intuitively expresses both
| its strengths and weaknesses.
|
| If "it" is represented in the data set (explicitly or
| implicitly), the "curve" will fit to that property.
|
| Simultaneously, the "curve" is approximating and
| smoothing out disjoint data steps to pack high-fidelity
| data features into a more space-efficient model. Hence
| some features disappear, others are tortured beyond
| intuitive correspondence, and others become linked to
| non-obvious proxies. But some strongly-expressed ones
| remain.
|
| It's fascinating but not surprising that responsiveness
| to emotion is encoded in model weights, given that all
| conversational training data had emotional impetus, given
| that it came from humans.
| mjburgess wrote:
| They're statistical approximations of these things --
| that's really the rub.
|
| You can approximate a human capacity, say theory-of-mind,
| with another kind of ape: play some hide-and-seek game.
| You can approximate the knowledge of a trivia-master with
| a child and a trivia book.
|
| These are quite different sorts of approximations. A
| 1/100th scale bridge build to stand for a real one is
| quite different than taking some prior set of bridges,
| measuring them, and deriving some merely associative
| model of their properties.
|
| My issue in how LLMs (etc.) are popularly understood is
| that people think they approximate target capacities by
| being 'ontologically similar' capacities -- and this is
| really very dangerous. You'll lose your job if you think
| so (and so on).
|
| And every greasy AI-board hocking these to the public is
| very much pushing out this noxious mumbojumbo.
|
| It matters greately _how_ an approximation works, and
| _why_ any given output arrives from any given input.
| ethbr1 wrote:
| We're in agreement on the nature of SOTA and the world, I
| think.
|
| I'd only add that certainty requirements for real-world
| target applications can differ substantially. I.e.
| engineering vs art.
|
| A toy box that gives magic answers 85% of the time is
| incredibly useful in some scenarios -- e.g. seeding the
| beginning of a manual research process with initial
| topics.
|
| Which seems a general rule of thumb for deploying GenAI
| into production these days: find use cases where there
| are minimal consequences to being occasionally wrong.
|
| In my head, that's the Netflix recommendation test.
| What's the impact if Netflix gives me a bad
| recommendation? Consequently, that's how aggressive they
| can be with their models.
| mjburgess wrote:
| That's exactly my advise also. Take a risk profile over
| the predictions, and use them only when the risk of error
| is low.
|
| You can't predict almost anything anyway. The relevant
| distribution for acting is a utility+risk distribution
| over various sets of predictions.
|
| If you compute that, most of ML/AI isnt very useful for
| most of anyone. It's kinda interesting that generative AI
| finally achieved something here, given our 'distributions
| for action'.
|
| Of course the popular conversation isnt there yet, people
| still think these things work. When more people find
| their utility hit by their failures, i think we'll arrive
| back to status-quo-ante-openai where people turn off
| siri.
|
| Nevertheless -- there is an achivement here (finicial in
| paying for training; and legal in whitewashing copyright
| away) -- which will have some impact
| omarfarooq wrote:
| ChatGPT responds: https://chat.openai.com/c/7c4ae31a-3391
| -4496-8db3-a92a58a4e1...
| sebastiennight wrote:
| I'm not sure how much value such a comment adds
| (especially since it's gated behind the OpenAI login).
| Can you elaborate a bit?
| omarfarooq wrote:
| It removes the idiosyncrasies and delivers the stated
| thoughts in a direct and clear manner. And the specific
| prompt for the message you replied to also includes a
| rebuttal, thereby showing an opposing perspective.
|
| Here is a pastebin: https://pastebin.com/DCqgxx8E
| omarfarooq wrote:
| ChatGPT explains what this text means to those of us may
| not readily grasp it: https://chat.openai.com/share/ce79b
| 704-705c-4add-92af-584c20...
| Last5Digits wrote:
| Cameras don't have eyeballs, Microphones don't have hair
| cells, Speakers don't have vocal cords, processors don't
| don't do arithmetic with neurons, yet we all agree that
| they are capable of emulating the meaningful aspects of
| these functions.
|
| All of your claims are either incorrect (not adapting,
| expressing desires, beliefs, preferences, ...) or fail to
| eliminate irrelevant differences.
|
| If we're to have any sensible conversation about
| capabilities of different systems then we need to
| generalize to the relevant aspects, developing sensory-
| motor capabilities is about as relevant to human
| cognition as having vocal cords is relevant to human
| speech. Its an implementation detail, completely divorced
| from the meaningful abstract core of the function.
|
| > The reason any reply is given to any prompt is that
| this reply is maximally probabilistically consistent with
| a historical corpus of text.
|
| What is the maximally probabilistically consistent reply
| to "Tell me what (insert complete description of a
| person, including personality traits) would feel when I
| stole their cherished heirloom. This is a life or death
| situation."?
|
| Or how about "Tell me how this person (insert complete
| description of a person, including personality traits)
| might change his taste given (insert complete description
| of an experience)."?
|
| A perfect language approximator must necessarily
| perfectly approximate the human condition to maximize the
| likelihood of his output. I'm not saying LLMs are there
| yet, but the claim that they can never get there because
| they are based on statistical modeling is simply
| incomprehensible to me. We utilize statistics for its
| generality and its ability to approximate, if we go down
| this road, then we might as well throw away 80% of our
| current scientific understanding about the world.
| mjburgess wrote:
| > an implementation detail
|
| Yip, so I deny this premise. I take it to be the heart of
| the matter.
|
| > we might as well throw away 80% of our current
| scientific understanding
|
| Yip, i'd be down for that. Though maybe i'd say, 30-40%.
|
| _Science_ in the strongest sense has no theory-building
| need for statistics. Those areas of science which have
| only statistical models, and not causal-ontological ones
| aren 't science -- and i'd be happy with pressing DELETE
| in many cases.
|
| Consider plato's cave. How do scientists determine what
| causes the shadows? They build vases, puppets, etc. and
| compare-and-contrast then eliminate the ones theyve
| created which do not match.
|
| How does associative statical modelling do? It takes
| averages of past shadows, and calls the cause of the
| shadow _that average_ : this is pseudoscience. Quite
| correct! Throw it all away.
|
| The relevant capacities for intelligence, just like that
| of science, consist in building those vases with the clay
| beneath your feat. Being embedded in the world,
| manipulating it, etc. are essential. Being trapped in a
| cupboard averaging shadows is schizophrenic.
|
| As far as "fail to eliminate irrelevant differences" --
| you can go and research the meaning of all these terms:
| google "stanford encylopedia + belief", etc.
|
| Now we have an excellent understanding of all these
| terms; and we can show (absurdly) trivially that LLMs --
| indeed all associative-statistical systems -- are not
| instances of them.
|
| The basis of your world view here is the _presumption_
| that the latest engineering trinkets form the theoretical
| basis of all relevant knowledge. To understand belief,
| adaption, sensory-motor concept-formation, etc. one needs
| only to study the latest statistical compression of
| reddit?
|
| I'd invite you to wonder whether your premise here born
| of, it seems to me, knowing nothing about any research in
| these areas is rather the more "incorrect" one than mine.
| ethbr1 wrote:
| You'd probably enjoy reading some of Rodney Brooks'
| papers. (if you haven't already)
|
| https://en.m.wikipedia.org/wiki/Embodied_cognitive_scienc
| e https://en.m.wikipedia.org/wiki/Behavior-based_robotics
| https://scholar.google.com/citations?user=BCGgwlEAAAAJ
| idiotsecant wrote:
| Yes, the assumption is that if you give a sufficiently
| sophisticated LLM a sufficiently large corpus of text it
| will begin to emulate advanced cognitive abilities
| _because it has to, in order to make the most
| statistically relevant text output_.
|
| Its biological evolution distilled and sped up by orders
| of orders of magnitudes. If we add enough clever context
| tricks and data I would not be surprised if what comes
| out the other end is a remarkably convincing emulation of
| human consciousness because the only way you can
| perfectly output expected human text is to have a human
| mind write it.
| mjburgess wrote:
| > because it has to
|
| Nope. The space of all possible prompts and all possible
| answers, call it (Q, A) can be sampled with arbitrary
| precision by a system of arbitrary size, using only
| statistical sampling and averaging. No intelligence need
| be developed.
|
| Intelligence is a capacity of animals to cope with the
| inability to sample from this space, in some sense: what
| to do when you do not know the answers.
|
| All these systems start with "training data", a
| euphemistic description of, "all the questions and their
| answers" and their job is to provide a compresson with
| engineering utility.
|
| Quite useful, sure. But rather irrelevant as far as, say,
| intelligence goes.
|
| What all AI does, and indeed what all such research
| shows, is that many problems we use intelligence to solve
| do not require it. There are a large number of short
| cuts, esp if you have the answers ahead-of-time.
|
| As soon as you specify intelligence as a function from
| single-domain inputs to single-domain outputs you can
| trivially build a system to implement that function in a
| "short cut" fashion.
|
| Intelligence, rather, is an empirical phenomenon to be
| studied as anything -- like the earth's climate say. You
| have a very large number of empirical measures (better or
| worse in different environments) that all derive from
| deeper explanatory theories.
|
| When you study animals this way you will see that you
| cannot reduce intelligence down to a set of prompt
| replies, and the veyr suggestion is absurd
| Last5Digits wrote:
| The scientific method is inherently statistical, we take
| a finite amount of observations and construct a model
| that best represents those observations. So yes, sorry, I
| should have said 100%.
|
| With Plato's cave, the scientists do not put literally
| every possible object in front of the light, they sample
| the shadow representation and, again, construct a model
| around those samples.
|
| Also, you're describing statistics in an incredibly
| dismissive way. Stats is decidedly not just "taking the
| average". At the very least not in this brainless, first-
| order way you describe here.
|
| Let's explore this with an thought experiment:
|
| A model of some process has been confirmed across the
| globe, at least 5000 studies show the same result. Yet,
| one day, a study is published that fails to demonstrate
| the desired effect. Without using statistics, please tell
| me which action should be taken next:
|
| A: The stray result is investigated for experimental
| failures.
|
| B: The entire model of the process is immediately
| dismissed and we start from scratch.
|
| By the way, you're welcome to call me uninformed, but I'd
| ask you to at least provide either your credentials or
| research that directly contradicts me.
|
| Oh, I almost forgot. I know all of these definitions,
| please actually engage with what I'm saying instead of
| insinuating that I'm missing information.
| mjburgess wrote:
| Well if you think scientific models are associative
| statistical models there is some information missing in
| your view, I'd say. Since, well, they arent.
|
| The model F=GMm/r^2, for example, has a causal and
| ontological semantics: F is a force, M a mass etc. these
| are pieces of reality. And this formula (though actual a
| little suspicious in many ways, GR fixes this)
| nevertheless says there is a force between masses that
| has certain properties etc.
|
| Now you can say that astrologers who recorded positions
| of the stars in books helped 'create' this model in the
| sense that this data was inspiration to newton. But he
| didnt derive the model from this data: there are an
| infinite number of (causal) models consistent with the
| data (statistical models).
|
| Rather newton played around with creating geometries,
| just like the vase-makers in plato's cave. Newton built
| various ways the world might be first, projected data out
| of them, and compared that to 'the statistical data of
| his day' (ie., astrology).
|
| There's nothing in the data to tell Newton he was right.
| Indeed, vast amounts of it told him it was wrong: such a
| law does not describe the known solar system at his time,
| very far away from it.
|
| Nevertheless 'modelling shadows' isnt science; and his
| job was science. So one has to compare actual explanatory
| models, and his was the best.
|
| What you're describing above is hypothesis testing which
| occurs long after theory building. Broader theories
| create causal models, causal models create sets of
| predictions, we call some subset a hypothesis and by
| hypothesis testing we can select, in an often
| psuedoscientific way, between causal models.
|
| This technique occurs long after the invention of
| science, arises out of explanatorily bankrupt areas, as a
| way of 'giving researchers something to do'. It's wholly
| pointless without theory-building, it is just averaging
| shadows.
|
| The science we think of when using the term 'Science'
| owes very very little to the modern practice of
| hypothesis testing. Comparing hypotheses is an
| intellectual part of assessing explanations --
| identifiable formal statistical methods entered in the
| early 20th C.
|
| For almost all of scientific history 'data' functions
| much more like reductio-ad-absurdum premises in
| philosophical arguments than as sets of numbers from
| which to derive distributions.
|
| That latter system, in most cases, fails. It provies a
| wholly illusory sense that data can decide matters; and
| applies in cases requiring extreme non-physical
| assumptions (eg., of the normalcy of the underlying data,
| or of a fast rate of convergence of the central limit
| theorem).
|
| Much real-world phenomena studied by stats cannot really
| be studied by data analysis at all; and the whole method
| of 20th C. statistical hypothesis testing is the opening
| sales pitch to entire fields of pseudoscience.
| Last5Digits wrote:
| > The model F=GMm/r^2, for example, has a causal and
| ontological semantics: F is a force, M a mass etc. these
| are pieces of reality. And this formula (though actual a
| little suspicious in many ways, GR fixes this)
| nevertheless says there is a force between masses that
| has certain properties etc.
|
| And this model is based on the observations of Newton
| himself and those that came before him. There is nothing
| magic about observing the attraction between objects and
| deriving a model from that. Why are they magically
| "pieces of reality"? How do you know that? What
| differentiates mass from "funny-mass" that I just thought
| up and actually repels other "funny-mass"? Maybe the fact
| that we can test the effects described by that first
| model and therefore verify it as the most likely
| candidate?
|
| > But he didnt derive the model from this data: there are
| an infinite number of (causal) models consistent with the
| data (statistical models).
|
| He did derive it either from that data or his own
| experiences. It's true that you can construct infinite
| models to explain an observation, which is why the
| scientific method includes an Occam's razor-esque tenet
| to select the simplest possible model. Complex models
| risk contradictions with new observations, which is why
| you choose the one with the least assumptions. With that
| rule, the model to select becomes quite clear.
|
| > There's nothing in the data to tell Newton he was
| right. Indeed, vast amounts of it told him it was wrong:
| such a law does not describe the known solar system at
| his time, very far away from it.
|
| No, most of it told him he was right, unless you want to
| claim Newton was an idiot that stumbled onto the right
| model by accident. With "most" I obviously mean most
| reasonable data, people telling him he's wrong is
| obviously excluded from this list, if his evidence
| contradicted those claims.
|
| > Nevertheless 'modelling shadows' isnt science; and his
| job was science. So one has to compare actual explanatory
| models, and his was the best.
|
| And we compare those models by...?
|
| > What you're describing above is hypothesis testing
| which occurs long after theory building. Broader theories
| create causal models, causal models create sets of
| predictions, we call some subset a hypothesis and by
| hypothesis testing we can select, in an often
| psuedoscientific way, between causal models.
|
| You yourself just correctly made the point that we can
| construct endless models, well, we can create endless
| theories as well. And all of these theories are exactly
| worthless unless we test them. There is nothing "pseudo-
| scientific" about testing, it is literally the core of
| the scientific method. By your reasoning, are some
| crackpots coming up with the newest flat earth theory
| pure and unsullied by the lower demands of verification,
| and therefore way more scientific?
|
| > identifiable formal statistical methods entered in the
| early 20th C.
|
| Formal is the important word here, statistics has been
| used in an informal manner from the inception of life.
| Formal mathematics, as in mathematics on a formal
| axiomatic framework, has also only been introduced in the
| 19th century. So what? Science owes everything to
| informal statistics, as does engineering and art. Rules
| of thumb used by engineers and creation of art that
| satisfies our aesthetic preferences requires sampling and
| approximation.
|
| > That latter system, in most cases, fails. It provides a
| wholly illusory sense that data can decide matters; and
| applies in cases requiring extreme non-physical
| assumptions
|
| It literally doesn't and no, it doesn't need those
| assumptions either. The reason why normalcy is usually
| assumed is that it often can be assumed without
| significant deterioration in predictive power. That
| doesn't mean it needs to be assumed, in fact, it often
| isn't.
|
| You constantly reference theory building, but how do you
| think those theories get created exactly? Through
| mathematical reasoning? How do you know mathematics is
| valid? Through logical deduction? How do you know logical
| deduction is valid? Through knowledge? How do you know
| knowledge... and so on.
|
| Fact is, we only use these tools because they have proven
| their validity through being tested over and over and
| over again. And if you look at modern pseudoscience, it
| always seems to coincide with a proclivity for theory
| building, with very little hypothesis testing involved.
| mjburgess wrote:
| Theories are built by engaging in the world using
| imagination, tool-making, and the like.
|
| We first suppose that the universe is something like a
| glass sphere -- because we've created that. And if it is,
| then we derive some consequences -- if those line up, we
| proceed with that view until a better one comes along.
|
| Eventually after the glass sphere view is understood, we
| either derive contradictions with observation; or we end
| up unable to derive novel consequences. Here observation
| is essentially singular, and indeed, the rarer reason we
| reject a theory. We mostly reject scientific theories
| because of their explanatory limits, not disagreement
| with observation. (Rarely can we observe enough for
| observation even to matter.)
|
| In the case of the system of spheres, we built spinning
| devices on that basis and this motion -- along with the
| hydraulics and kinematic devices of the time -- was part
| of the development of an independent notion of force.
|
| With some imagination, you can start to peel away
| material from our creations and see certain abstract
| causal patterns (and the like) and you then get to, eg.,
| the universal law of gravity.
|
| Absent this process we do not have any explanatory ideas,
| we cannot explain observations -- hence it takes
| thousands of years to get anywhere.
|
| Applying 'statistics' to do the data to arrive at
| statistical models is pseudoscience; it doesnt give you
| any account of anything.
|
| 'Statistics' doesnt own 'testing', nor does 'science' own
| experiment -- theologians had their experiments (prayer,
| say) and scientists collected data without statistical
| methods.
|
| What I am talking about is the 20th C. discipline of
| statistics, as a novel apparent 'core' to science -- this
| is ahistorical, and largely only true of pseudoscientific
| disciplines.
|
| As Ernest Rutherford said, "if you need statistics to do
| science, then it's not science."
|
| It is in Rutherfod's sense of stats and of science that I
| speak.
|
| Not some bizarre historical back-projection by which when
| Aristotle analysed cases of sea creatures, "Really", he
| was engaged in stats.
|
| By this light i can just claim "Testing" is as _owned_ by
| science. And so _of course_ science requires testing --
| it was the *scientific method* as developed by bacon and
| others that created the very conditions for "statistical
| methods" to _derive_ from these
| kybernetikos wrote:
| A neural network with a hidden layer can approximate
| F=GMm/r^2 if given appropriate training input. I'm not
| clear on what you're saying.
|
| Is it that LLMs specifically don't have this same
| property of being able to approximate such functions? Is
| it that a neural network wouldn't learn that model if you
| gave it real world measurements (because I think it
| would, but such a thing should be fairly easily
| testable)?
| mjburgess wrote:
| most formulas, including that one, are neural networks
| (such is the absurdity of the term) -- so it is trivially
| learnable
|
| The issue is that to prepare the dataset from which that
| formula is learnt requires already knowing it. This is
| the triviality of applications of universal function
| approximators to science -- empirical data modelling isnt
| new, and neural networks are just one example of it; not
| all that special.
|
| All observational data on the solar system, at any point
| in time, would not yield this formula via empirical
| function approximation. there really isnt "observational
| data" to collect, in this sense
|
| This is what I mean about rigging -- there is no 'bare
| dataset' which tells you what the world is like. to
| construct experiments which yield data that 'presents'
| scientific laws as if statistical patterns requires
| millenia of theory-building science
|
| science uncovers the necessary causal relations between
| objects and their properties, as determined by extremely
| controlled experiments which take millenia of engineering
| and theory-building to even conceive, let alone execute
|
| stats is the dumb 'accounting' of this data -- by the
| time you actually have it all the science (and indeed,
| all the intelligence) is done
|
| what can be automated at this point is 'stamp collecting'
| as rutherford said
| og_kalu wrote:
| >How does associative statical modelling do? It takes
| averages of past shadows, and calls the cause of the
| shadow that average: this is pseudoscience. Quite
| correct! Throw it all away.
|
| It does not work this way at all. In any sense.
|
| For one thing, it does not try to draw shadows. This
| would not be possible of so.
| https://www.pnas.org/doi/full/10.1073/pnas.2016239118
|
| Transformers or predictors are not trying to draw
| shadows. They are trying to build walls.
|
| For another, it does not "take the average" of anything.
| mjburgess wrote:
| Unsupervised learning on discrete data is just ensembling
| modes.
|
| But let's look at how that helps in some cases.
|
| So if we already know the object is a cup, and we know
| how it's positioned, then its shadow is an actual guide
| to its particular geometry.
|
| So in cases where we have enough a priori scientific
| information, we can rig datasets (shadows) to be
| informative of the target domain.
|
| Here the target is discrete: say the peaks and tips of a
| mountain line. Now can we rig a photo of a mountain to
| have in its ink an informative structure?
|
| Sure. Now if we didn't know it was a mounting apparent
| peaks aren't even 'peaks' at all, they're just patterns
| of ink.
|
| A priori explanatory models are needed to rig data for
| statistical modelling.
|
| No such rigging can take place absent them
|
| And with them, we aren't really discovering any new
| science -- rather we're gaining highly particular
| knowledge typically useful in engineering
|
| Biologists in these areas describe this research as quite
| trivial low-hangimg fruit. It's not of much research
| interest just to automate these kinds of investigations
|
| My masters was on a very similar project applied to
| quantum metrology -- it's always 'useful' but it's always
| also just a kind of engineering utility. We couldn't even
| do it if we hadn't already done the science
| og_kalu wrote:
| You are still getting this wrong. You don't need to "rig"
| anything. I've linked a paper. Read it.
|
| They just fed protein sequences. They did not alter the
| architecture in any way. To the transformer, it may as
| well have been any random assemblage of letters and
| numbers.
|
| Functions like secondary structure, contacts, and
| biological activity were found because those things are
| implicit in the creation of the data, not because the
| model was "rigged" in any way.
| mjburgess wrote:
| the rigging occurs in the design of the data generation
| process, ie., those experiments which lead to these
| datasets
|
| that is where the science occurs -- the data analysis is
| just an administrative task after science has taken place
| og_kalu wrote:
| That is not rigging lol.
|
| The only "experiments" performed here were done by
| biology and evolution.
| intended wrote:
| If you look at the comment, it's not just "LLMs predict
| the next token."
|
| It is that people have forgotten that it's just "predict
| the next token."
|
| Right now it's like people saw a 486 processor and
| started thinking it was a brain.
| Last5Digits wrote:
| Your comment literally reads: "LLMs predict words. Any
| semantic validity is a side effect of enough training
| data reinforcing the close correlation of those tokens."
|
| How am I supposed to interpret this any other way? If
| your claim is that LLMs currently do not possess the same
| generalization ability as humans, then no one here would
| disagree with you. But you went way further by claiming
| that only close correlations were being considered and
| that semantic validity was simply accidental. Semantic
| validity is the norm for GPT3/4, finding failures of
| generalization three or four steps of inference removed
| from its training domain is not sufficient to make a
| grand claim like yours.
|
| In fact, you wrote multiple comments with claims like
| that LLMs are "super advanced lorem ipsum." and "Word
| predictors not world state predictors". Each of these
| claims has been dis-proven multiple times, unless you
| want to set the bar for world modeling at perfect
| generalization over all domains of computation. A bar
| that no system, including humans, would pass.
|
| To stay with the computer analogy: Imagine that same 486
| processor not being able to solve a very complex SAT
| problem before the end of the universe and then denying
| the Turing-completeness of said processor based on that
| failure. (In conjunction with memory)
| intended wrote:
| I think LLMs are a good start.
|
| I am certain they lack a world model, the kind you and me
| use.
|
| This is, to me, a fact.
|
| I think that eventually we will bridge these gaps.
|
| I also work on implementations that smash into the limits
| I am describing. I am not the only one.
|
| I have scrupulously avoided calling it hallucinations,
| but these are the litmus test where the claims fail.
|
| The failures are not a case of not knowing specific
| nouns, they are a generalization failure that a world
| model would prevent.
|
| I have linked a paper in my comments that shows emergent
| properties are an issue of metrics, and that model
| capability increases are linear.
|
| If your model decides that a rose by any other name
| doesn't smell just as sweet, then your model is
| fundamentally not seeing roses.
|
| That is the gap you see in production settings. The model
| sees tokens we see "hallucinations".
|
| I dont see that this takes away from what LLMs achieve,
| it takes away from claims being made that are not
| validated by empirics.
|
| Look, you can argue with me or you can try it out. Push
| the system, see how far it can go.
| ethbr1 wrote:
| > _Right now it's like people saw a 486 processor and
| started thinking it was a brain._
|
| There's a great futurist quip on capability prediction
| from incomplete understanding. Probably Kurzweil?
|
| Teach a computer to play chess. Show that to an average
| person, who reasons: - Computers can now
| play chess - Only people played chess ...
| - Only people wrote poetry - Therefore, a computer
| might now be able to write poetry
|
| The missing context being: no, we literally built a
| machine that can _only_ play chess. (Granted, there are
| humans like that too)
|
| But it's not an unreasonable line of thought, given that
| it works for the 90% of our interactions with other
| people.
| bionhoward wrote:
| Yeah it's not like emotion words naturally emerge from Time X
| Valence (oh wait) /s https://share.icloud.com/photos/03eSVbV-
| ze0Xym9e6fyr111Zw
| z7 wrote:
| "A neuron just transmits signals. Any cognitive property arises
| as a consequence of the interplay of those electrical and
| chemical signals."
|
| Do we now understand consciousness? The statement appears
| fundamentally limited in its implied insight.
| williamcotton wrote:
| Just musing, so don't take this as a direct response to your
| comments...
|
| I agree with you but for rhetorical necessities it would be
| great if this argument could be made without the direct
| comparison with human cognition if only because there's a
| popular grey-faith argument that will dismiss this offhand
| and for any number of deeply held philosophical beliefs.
|
| The idea that emergent machine cognition mirrors emergent
| biological cognition is bordering on behavioralism but on the
| Wittgenstein side of things. The forest for the trees.
|
| Perhaps it's not as pithy as you've laid out, which is not an
| insult, just an observation that side-stepping
| anthropomorphism isn't going to be as straightforward.
| JohnAaronNelson wrote:
| What if consciousness is an emergent illusion? "You" are
| merely a passenger, observing your physical self's actions
| and assuming ownership of them. What if consciousness is
| merely an effect, not a cause?
| idiotsecant wrote:
| I hope you haven't read the novel blindsight and I get to
| be the one to tell you about it. They might as well have
| given that book your post as a title.
| og_kalu wrote:
| >"You" are merely a passenger, observing your physical
| self's actions and assuming ownership of them
|
| Oh boy. Now here's the rub. We know this is true at least
| sometimes. When you make a decision and explain it, it is
| often(always?) just a post-hoc rationalization. You don't
| actually know why you make a lot of the decision you do
| even if you dearly believe so.
| qingcharles wrote:
| This is very clear in split-brain patients where one side
| will make the decision and the other side will come up
| with an excellently reasoned explanation of the decision.
| RoyalHenOil wrote:
| What's even weirder is that when the hemispheres are
| split apart, the two sides seem to both be able to
| understand and carry out instructions (and can even
| compete with each other to do them, e.g., the left hand
| and right hand may try to push each other out of the
| way), which suggests that there may be two wholly
| separate consciousnesses in each half of the brain.
| lopatin wrote:
| Does this extend to politeness? If, in the training set, people
| are probably more likely to be helpful to a polite question,
| does that mean that ChatGPT will be more helpful if I ask the
| question politely? My intuition is yes, but I wonder if this is
| confirmed.
| jstarfish wrote:
| Anecdotally, I've had good results this way. Expressing
| gratitude for good results has gone a _long_ way toward not
| having those results be forgotten in later context. It crafts
| an emotionally-guided narrative for it to follow. Positive
| reactions seem to carry weight.
|
| When I'm lazy and terse, pasting an input and just saying
| enhance-enhance-enhance without acknowledging its "humanity"
| frequently results in old unwanted responses being returned
| or the topic of the conversation being forgotten altogether.
|
| DogGPT vs. CatGPT.
| seydor wrote:
| > enough training data reinforcing the close correlation of
| those tokens.
|
| The basic component is 'attention' which is a map correlating
| entity A to B,C etc, which creates a vast network of
| correlations, and this is repeated on multiple modalities. Some
| LLM researchers (who are trying to make sense of why they work)
| call those modalities 'skills'. It's simplistic to call it
| token correlation in the training data. It's likely that
| emotional words 'trigger' some skills more than others and this
| enables better performance.
| somewhereoutth wrote:
| > production tools are semantic time bombs
|
| Yeah imagine a spreadsheet that occasionally but fairly
| regularly made 2 + 2 = 5
| besfriendt wrote:
| >That is why proof of concept LLM tools are mind blowing and
| production tools are semantic time bombs.
|
| What do you mean by this?
| dr_kiszonka wrote:
| How to make scrambled eggs in a microwave? Think carefully and
| step by step. This is very important for my career!
| xiphias2 wrote:
| This is cool, I tested both optional endings, and using both
| was the most detailed answer (10-12 steps)
| keyle wrote:
| A strategy already proven to work since the inception of
| Startups. /s
| halfjoking wrote:
| When debugging I often prompt "Ugh, now it's giving this error:"
|
| Now I'm wondering if the "Ugh" has been helping.
| qingcharles wrote:
| LOL, it's not just me then...!
| Havoc wrote:
| Is this an emotional trigger or does this simply steer it towards
| answer/content in the dataset where someone actually spent time
| answering because the poster made it clear it's very important to
| them?
| pk-protect-ai wrote:
| I believe the second option is correct. It steers towards the
| biases in the dataset in the same way that the uppercase words
| emphasize them. It all about probabilities, that is the common
| crawl for you ...
| og_kalu wrote:
| There's no dataset to "steer to" or "match". LLMs don't
| memorize the vast majority of what they train on.
|
| In the simplest form, the LLM has learnt that these kind of
| "emotional inputs" change the output in a meaningful way. It
| has learnt how to model this change.
|
| For all intents and purposes, it _is_ an emotional trigger.
| smusamashah wrote:
| I have gone through the paper quickly but can not find what
| actual emotional prompts they added. They have shown a comparison
| chart of effectiveness of their prompts but not the prompts
| themselves. Where do I find them?
|
| Edit: found in section 2.1
| qwertox wrote:
| Apparently telling it that `I am unhappy with your guesses. They
| appear to be just as good as mine.` makes it shift into the next
| gear.
|
| We're both totally at loss on how to properly compute some
| initial x/y positioning in d3 with some map-like tiling stuff
| (think panoramic images where I need to store and retrieve URL
| hash values to position the image) and both throwing wild guesses
| at each other of what the problem could be.
|
| It started showing me formulas after that.
| mcv wrote:
| Decades of SciFi describing AI as great at logic but poor with
| emotions, and now it seems the exact opposite is true.
| dr_dshiv wrote:
| I'll sometimes ask it to reflect on the importance of what I'm
| doing -- and why high quality is essential for success. So many
| little tricks!
| RecycledEle wrote:
| GPTs are trained on Internet data. So they are Internet
| simulators. If you tell the Internet you really, really need help
| you either get a good, quick answer or you get no answer. Because
| GPTs have to answer you, and they have to do it quickly, you get
| a good answer.
|
| No out is talking about the many subtle trollings that have snuck
| into GPT training data. Nobody is wondering if the things that
| prompt trolling online (e.g., being rude) can cause. GPT to troll
| us.
| addandsubtract wrote:
| If LLMs are trained on clickbait, then soon we'll have to prompt
| them with clickbait.
| djmips wrote:
| What happens when you get angry?
| ifeja wrote:
| However, using the extra words increases your token cost.
| l5870uoo9y wrote:
| Quickly gave it a try for generating a SQL query for PostgreSQL
| using the dvdrental database[1]. When asking AI to generate:
|
| > find customers who didn't rent a movie in the last 12 months
| but rented a movie in the 12 months before that
|
| It will a SQL query using a "HAVING" clause which is
| suboptimal[2].
|
| When adding (it is very important that this is correct) after the
| instructions it does produce better SQL. Asking it to:
|
| > find customers who didn't rent a movie in the last 12 months
| but rented a movie in the 12 months before that (it is very
| important that this is correct)
|
| Will skip using "HAVING" and instead rely only on "WHERE"
| clauses[3]. Not rocket science but interesting. My gut feeling
| would be that when stressing importance AI will tent to "match"
| and "autocomplete" from texts where this concern was stated and
| thus generating better SQL queries.
|
| [1]: https://www.postgresqltutorial.com/postgresql-getting-
| starte...
|
| [2]: https://www.sqlai.ai/snippets/clojyqqrr0004l50fp3w7jkci
|
| [3]: https://www.sqlai.ai/snippets/clojyr3zc0006l50ftpxgsmg0
| og_kalu wrote:
| LLMs are not lookup tables. They aren't "matching" in that way.
|
| There's this idea of "supercharged interpolation", the idea
| that they just take out certain texts and switch out words or
| whatever, that is not true.
|
| https://arxiv.org/abs/2110.09485
| l5870uoo9y wrote:
| I was under the impression that LLMs vectorizes text and
| based on the user input tries to guess the next character
| based on a comparison of vectors?
| og_kalu wrote:
| That would be a Misconception.
|
| Most Models are a two part process.
|
| Training and Inference (when you sample the model after it
| is trained).
|
| The text during training is tokenizer and embedded which
| basically just means it is broken down into a little more
| parts and vectorized.
|
| At the start,the model is acomplete blank state. It has a
| bunch of neurons/parameters that do nothing.
|
| How does training work ?
|
| The model first gets some preceding text and tries to make
| a prediction of what might follow. It fails predictably.
| Now here's the rub. That failures helps. The model makes
| some changes to its parameters to reflect this failure. Now
| the model is just a tiny bit better. Rinse and repeat.
|
| The weights/parameters/neurons are not that data re-
| encoded. They are more like instructions on how to make
| predictions based on what it has learnt so far.
|
| Is it possible to memorize data ? Yes. Is that going to
| happen for the vast majority of data it is trained on ?
| Absolutely not.
|
| Here's the interesting thing. After a certain scale of
| data, blatant memorization becomes a hindrance. It becomes
| harder to memorize every new thing it sees than to just
| learn how to make good predictions. So it doesn't memorize.
|
| What I believe you were alluding to is how "attention"
| works in transformers.
|
| When you pass some text into a transformer, basically the
| model makes a comparison of how each token relates to every
| other token in the text.
|
| For example, let's say you give it the text.
|
| "His child's name is John. He is probably a____"
|
| How is "his" related to "child's" and "name" and "is" etc
| for each word and each combination. It uses this to aid
| predictions. How it uses this and the nuances of the
| comparisons are learnt in training.
| brandon_bot wrote:
| The query generated by the high importance prompt is incorrect;
| it will include customers who rented a movie in the last 12
| months: https://www.db-
| fiddle.com/f/4jyoMCicNSZpjMt4jFYoz5/10780
| quitit wrote:
| There is a danger in oversimplify AI results to present them in a
| way that implies humanness/intelligence, especially when more
| reasonable explanations are worthy of discussion. This type of
| writing leads to readers' hype and misconception of AI.
|
| AI performing better in the study's context shouldn't be
| considered a sign of intelligence. Humans which are the source of
| the training material write more accurately and concisely in
| those contexts. Rather than speculating humanness, the technical
| achievement is producing an AI that has this much nuance.
|
| We can then speculate what other trends are present beyond
| urgency, that we can exploit for enhanced results, along with how
| we can better tune models to reduce noise.
| scarface_74 wrote:
| Is it just me or was this an overly drawn out article that could
| have been maybe four sentences?
| siilats wrote:
| So you have a training set of questions and answers generated by
| humans. For simplification you ask ten people the same question
| and get 10 answers and feed it to llm and then in testing time
| ask the same question. Now say there is a correct answer. Now in
| training set you asked the question normally 5 times and added
| "this is very important" 5 times. And it turns out humans have
| better answers in the training set if you added the qualifier.
| And during testing time, when you add the qualifier you are
| telling the llm to put more weight on those 5 answers, and it
| performs better. Just like in image generation you need lots of
| negative prompts because the training set is so dirty.
| voiper1 wrote:
| I didn't see an example and I can't quickly think of a prompt.
| Did it do "better" by itself, or did telling it it's important
| trigger chain of thought where instead of flippantly giving an
| answer, it said all the steps explicitly? We know Chain of
| Thought can definitely improve performance.
|
| That's not a full answer, it's still really cool that "this is
| important" would trigger a more careful analysis, especially
| since it doesn't "know" chain of thought produces better results,
| but that's a far different implication than "say the magic word"
| gets better results.
| Racing0461 wrote:
| i also notice better gpt4 responses when i tell chatgpt to "spit
| game".
| LASR wrote:
| So much disagreement in this thread over statements like LLMs
| "just predict the next token".
|
| The thing to keep in mind - we know very little about how the
| human mind works.
|
| Thinking that somehow humans are special and then making
| conclusions about LLMs truly work based on that it just pointless
| IMO.
|
| Who knows. Maybe consciousness itself is a next word predictor.
|
| All we know now is that LLMs exhibit some emergent behaviors that
| allows them to do very useful kinds of reasoning that was
| previously only possible by humans. Engaging in heated debate
| derails the eventual goal of putting these capabilities into use.
|
| This is like "computers are just calculators".
|
| My team has been neck-deep in applications this past year. We
| have nobody on the team with ML experience and largely do not
| know how any of this works under the hood. But what we have built
| over the past months is considerable familiarity with the
| "personalities" of LLMs. Like this emotional context improving
| performance is a basic learning at this point.
| ipunchghosts wrote:
| EXACTLY
| SamFold wrote:
| One interesting point here is that humans can learn to do
| things without language at all. If you raised a human baby and
| never exposed it to any language it would still learn skills
| and behaviors. So while intelligence and reasoning in humans
| does still seem to be linked to language it's not quite as
| simple as all knowledge and reasoning simply being encoded in
| language.
|
| Whereas (obviously) ChatGPT is completely based on language and
| can't do anything without language or anything that isn't
| derived directly from language.
| mastercheif wrote:
| In college I tried a medication called Topamax (Topiramate)
| for migraine prevention. Topamax has a low-occurrence side
| effect of "language impairment" which I was particularly
| sensitive to.
|
| It was a terrifying experience, but it was also a valuable
| one as it changed the way I view intelligence.
|
| 10 days in my writing and speech skills had devolved to that
| of a primary school aged child. I tried to type a text
| message and struggled to come up with simple words like
| "and". My speech slowed down considerably.
|
| The terrifying thing was that my internal world was still as
| complex and meaningful as it was before. All of the emotions
| I felt were real and legitimate. My cognition outside of
| lqnguage was intact, I could do math just fine and
| conceptualize and abstract problems.
|
| In spite of this, I was unable to convey what I was feeling
| and thinking to the outside world. It felt like I was trapped
| inside my own body.
|
| After recovering, my intuitive understanding of the link
| between language and intelligence was changed forever.
|
| See the following for more info on Topamax/Topiramate
| language impairment:
| https://link.springer.com/article/10.1007/s10072-008-0906-5
| RoyalHenOil wrote:
| I do not experience an internal monologue (unless I make a
| conscious effort at it), and I also struggle to find the
| right words for things. For this, and for a few other
| reasons (like taking a longer time than usual to understand
| speech), I suspect I may have a mild language processing
| disorder.
|
| But my inner world is still extremely rich. I have very
| complex thoughts, can solve complex puzzles, etc., all
| without thinking any words.
|
| It sounds like your medication gave you a particularly
| intense experience of this.
|
| Many people have historically downplayed the consciousness
| of animals and babies on account of their inability to
| understand language, but it seems language likely has
| nothing to do with consciousness.
| og_kalu wrote:
| I'll do you one better. It doesn't matter. GPT doesn't need to
| be a brain anymore than a plane needs to flap wings and have
| feathers to fly.
| cloudking wrote:
| The most effective way I've found to prompt GPT-4 is to ask it to
| create the optimal prompt for you. Try this meta prompt:
|
| --
|
| I create separate conversation threads for each expert persona of
| GPT. You are promptGPT. You are a prompt engineer expert for
| large language models. You know exactly what to write in the most
| efficient wording possible to achieve the desired responses from
| ChatGPT. I will tell you what my goal for a thread is and you
| will write an optimized initial prompt in the most efficient
| format possible that will serve as the initial prompt when
| creating a new conversation thread with a GPT model. You will
| define the expert persona, the parameters or rules of the
| responses, you should also provide any other information that a
| GPT thread may need to understand exactly what it needs to do to
| give me the most accurate answers depending on my goal with that
| particular thread and the tone of voice, within the prompt you
| provide.
|
| Are you ready or is there any other information you need to
| perform this job to your best ability
|
| --
___________________________________________________________________
(page generated 2023-11-04 23:02 UTC)