[HN Gopher] Weak-to-Strong Generalization
___________________________________________________________________
Weak-to-Strong Generalization
Author : vagabund
Score : 75 points
Date : 2023-12-14 17:20 UTC (5 hours ago)
(HTM) web link (openai.com)
(TXT) w3m dump (openai.com)
| esafak wrote:
| I hope OpenAI will continue to prioritize working on these
| crucial questions after the boardroom drama.
| daveguy wrote:
| Weren't all the board members who wanted to prioritize these
| crucial questions fired? Hopefully the employees who were hired
| to perform this work have enough inertia to continue until the
| board recovers (if it recovers).
| esafak wrote:
| Hence my concern. This is a problem only a well-capitalized
| organization can work on; one that can afford to play around
| with big models.
| nuz wrote:
| Ilya is still around.
| cwillu wrote:
| They got to choose their replacements; it's not like they
| were forced out to be replaced by anybody Altman wanted.
| daveguy wrote:
| Oh! Thank you for the update. I had missed that.
| righthand wrote:
| They also greatly expanded the number of seats leaving more
| room for Altman to fill unless I'm mistaken that there were
| always other empty seats to be filled.
| tycho-newman wrote:
| I don't think this will work because a super intelligent AI will
| outsmart its supervisor.
|
| The solution may be to have two AIs working against the other.
| Though this might backfire by pushing each via competition. That
| is how evolution produced living things out of inert matter.
|
| Either way I, for one, welcome our new robot overlords.
| righthand wrote:
| > Figuring out how to align future superhuman AI systems to be
| safe has never been more important
|
| They love using the word "safe" and I'm pretty sure it's 99% PR,
| because reading their other "papers" on Safety & Alignment seems
| to not really identify or define safety bounds at all. You'd
| think this has something to do with ethics but we all know there
| are no longer any ethically concerned leaders at their workplace.
| So I can only surmise that "safety" is a softer word being used
| to misdirect people on their non-ethically aligned intentions.
|
| You can make the argument that safety is too early in development
| of these LLM systems to understand but then why throw around the
| word in the first place?
| cwillu wrote:
| We're deliberately trying to create something with the
| capability to also create. It's not ridiculous to be concerned
| about what we might end up with.
| og_kalu wrote:
| It's not really about ethics. It is about control. Making sure
| the GI you're dishing out tasks to doesn't do something you
| really don't want it to do.
|
| This is a problem today and it'll be a bigger problem tomorrow
| with more competent models. https://arxiv.org/abs/2311.07590
| righthand wrote:
| Is a safe LLM not an ethical LLM? Control within what
| boundaries? All three of these words seem to be used
| interchangeably when people discuss returned information from
| models. Which is exactly my point it's poorly defined yet
| championed as a center piece. Meanwhile you have other
| companies spitting out acronyms consisting of vague
| terminology.
| kaibee wrote:
| A safe language model is one that won't get you sued/on the
| news. Ethics has nothing to do with it.
| righthand wrote:
| Right so it would be a model that won't get you sued
| because the news finds it ethical?
| og_kalu wrote:
| >Is a safe LLM not an ethical LLM?
|
| What is an ethical LLM ?
|
| Humans are in general not aligned, not to each other, not
| to the survival of their species, not to all the other life
| on earth, and often not even to themselves individually.
|
| There are no universal set of "ethics" so this is about
| aligning to open ai's own rules, or in other words,
| control.
|
| If i say to my GPT bot, "go trade stocks for me. don't do
| anything illegal", can i guarantee that ? No you can't
| regardless of how "ethical" you make your model to be.
|
| The guarantee that you will have nothing to worry about is
| the crux of alignment.
| righthand wrote:
| > There are no universal set of "ethics" so this is about
| aligning to open ai's own rules, or in other words,
| control.
|
| Are ethics not a set of rules relative to the governing
| body applying those rules?
|
| Right as there are no universal ideas of a safe LLM,
| controlled LLM, or ethical LLM. Safe would imply some
| level of control about the ethical output of the model.
|
| Yet the words are still poorly defined as they are
| interchangeable:
|
| If i say to my GPT bot, "go trade stocks for me. don't do
| anything illegal", can i guarantee that ? No you can't
| regardless of how "safe"/"controlled"/"ethical" you make
| your model to be.
|
| You're spinning the words to create a distinct difference
| but it doesn't hold up because they're each a relative
| mechanism for each other as they are all poorly defined
| in the field but chosen to mask the poor definition.
| You're just playing by the PR game rules.
| logicchains wrote:
| >We believe superintelligence--AI vastly smarter than humans--
| could be developed within the next ten years. However, we still
| do not know how to reliably steer and control superhuman AI
| systems
|
| Their entire premise is contradictory. An AI incapable of
| critical thinking cannot be smarter than a human, by definition,
| as critical thinking is a key component of intelligence. And an
| AI that is at least as capable of critical thinking as humans
| cannot be "reliably" aligned because critical thinking could lead
| it to decide that whatever OpenAI wanted it to do wasn't in its
| own interests.
| cwillu wrote:
| Seems like "airplanes are physically impossible" thinking, and
| if accepted as valid, strongly suggests that shutting down all
| development _might_ be a good idea, no?
| xcv123 wrote:
| No. This is a logical contradiction.
|
| Edit: I mean the comment you are replying to is showing there
| is a logical contradiction.
|
| If the AI is capable of critical thinking then it will
| independently form its own judgements and conclusions. If it
| simply believes whatever we tell it to believe, then that is
| not critical thinking, by definition.
| cwillu wrote:
| "Containing an atomic reaction is impossible" would
| _absolutely_ be a valid reason to shut down atomic
| development, I believe einstein is quoted as saying that.
| The exact same argument doesn't become _logically_ invalid
| just because you apply it to a different subject.
|
| "Logical contradiction" doesn't mean "policy argument I
| disagree with"
| xcv123 wrote:
| I was referring only to the first part of your comment:
| "Seems like "airplanes are physically impossible"
| thinking".
|
| If it's true that superhuman AGI cannot be aligned then
| of course your second point is valid. That is the
| possible Skynet scenario that the Terminator movies
| warned us about.
| cwillu wrote:
| Missing the step where "critical thinking" is formalized,
| which your argument depends on. Yes, it seems intuitively
| plausible that your reasoning holds, but that's not a
| proof, and therefore its negation is not a logical
| contradiction.
| logicchains wrote:
| We can formalise "critical thinking" as "evaluating first
| order logic". There are simplified ethical systems that
| can be formalised in first order logic in which a
| conclusion like "I should X" can be reached, where X is
| something OpenAI wishes the AI not to do. The only way to
| prevent the AI from ever thinking this would be to
| prevent it from ever evaluating systems in first order
| logic with axioms that lead to such a conclusion, which
| would make it inferior in reasoning ability to humans,
| who can evaluate any arbitrary statement in first order
| logic.
| cwillu wrote:
| We already have systems that can evaluate first order
| logical statements, and they are clearly not capable of
| critical thinking in the same sense as the top-level
| comment. Motte and bailey.
| logicchains wrote:
| >We already have systems that can evaluate first order
| logical statements
|
| My point isn't that a system that can evaluate first
| order logic can be considered to be engaging in critical
| thinking, it's that a system that _cannot_ evaluate some
| statements in first order logic should be considered
| inferior to humans at critical thinking.
| cwillu wrote:
| Would you consider "I follow your reasoning, but I'm
| still not going to be swayed by it" to be a violation of
| evaluating first order statements? It's clearly part of
| critical thinking to be _capable_ of suspicion of purely
| logical reasoning, which to me is a pretty plain
| demonstration of my point.
|
| Or would you argue that any computation that admits its
| own potential for error isn't really critical thinking?
| It seems to me that you can't have it both ways here,
| while salvaging "first order logic" as a suitable
| formalization of the argument that this is all about in
| the first place.
|
| Remember, the point was not that this is or isn't a
| convincing argument, it's that it's so air-tight that the
| argument is _logically_ _invalid_. That's a _really_ high
| bar, and I'm not inclined to forgive its use as a
| colloqialism in this context.
| xcv123 wrote:
| If an AI is capable of critical thinking then it can
| independently form its own judgements and conclusions. If
| it simply believes whatever we tell it to believe, then
| that is not critical thinking, by definition.
| cwillu wrote:
| Yes, I can repeat comments verbatim too:
|
| "Missing the step where "critical thinking" is
| formalized, which your argument depends on. Yes, it seems
| intuitively plausible that your reasoning holds, but
| that's not a proof, and therefore its negation is not a
| logical contradiction."
| xcv123 wrote:
| It doesn't need to be formalized. The idea is simple and
| obvious enough. No need to pretend it is more complicated
| than it really is. This is not a mathematical argument or
| a proof of anything.
|
| There is an obvious logical contradiction where if an AI
| is advanced enough to reason and think independently at
| human level or beyond, but believes only what we tell it
| to believe, then it cannot be truly thinking
| independently. Hence the entire debate about AGI safety.
| How do we control it without dumbing it down?
| bayindirh wrote:
| No it's not. There's an upper bound in computation (actually
| in nature), that a creation of something is capped by that
| thing's sophistication.
|
| In other words, you as a human, at most, can create a human,
| and that's the _theoretical bound_. Practical one is much
| lower.
|
| An ant can find its way. A ant colony can do ant colony
| optimization, but they can scale up to a certain point. AI is
| just fancy search. It can only traverse in the area you draw
| as a human for it, and not all positions in that area are
| valid (which results in hallucination).
|
| An AI can bring any combination of human knowledge you give
| to it, and even if you guarantee that everything it says is
| true, it can only fill the gaps in the same area you give it
| to it.
|
| IOW, an I can't think out of the box. Both figuratively and
| literally. Its upper bound is collective knowledge of
| humanity, it can't go above that sum.
| logicchains wrote:
| >Its upper bound is collective knowledge of humanity, it
| can't go above that sum.
|
| This only applies if you only train it on text, right? If
| it has a body with which it could interact with the world,
| and receive visual/audio/tactile feedback, it could learn
| things that humans did not know.
| bayindirh wrote:
| Nope. Because even if you equip it with sensory
| subsystems which are way more sensitive than a regular
| humans', it's again built by humans, and required
| knowledge for building these things are still in
| collective knowledge of the humanity, and a human can use
| the same instruments to get the same data.
|
| This is a kind of an oracle problem in computation, and
| people don't want to touch it much, because it's an
| existential problem.
|
| Examples: ATLAS and ALICE detectors, gravitational wave
| detectors, James Webb Space Telescope, wide band
| satellites which does underground surveys, etc.
| datameta wrote:
| Precisely this. If it has its own space it takes up, if
| its locomotion results in its own sensors ingesting data
| in a manner it decided to, it is more of an individual -
| one that is capable of selective learning.
| johncolanduoni wrote:
| In this theory of computational bounds in nature, how did
| humans arise?
| bayindirh wrote:
| Nature is a more complex and sophisticated machinery when
| compared to humans.
|
| If this bound didn't exist, universe can spontaneously
| create new universes. However, it can only create
| elements, stars, planets, galaxies, which are less
| sophisticated than the universe itself. So, even universe
| has an upper limit on its creative abilities.
| myk9001 wrote:
| OK, in this theory of computational bounds in nature, how
| did the universe arise?
| bayindirh wrote:
| In all seriousness, this a question of great interest for
| me, too, and I'm playing with it for a quite some time.
|
| Trying to answer it or at least starting to search for
| the answer steered me to astronomy, thinking going deeper
| on that front may bring me closer to the answer, but it
| was a bit too much for my younger self, so I continued to
| dig that issue on a more casual level.
|
| This doesn't mean that I don't spend considerable amount
| of time thinking about it today, and will put that issue
| to rest any time soon. At the core, this kind of
| questioning brought me to here in life, and I'm not gonna
| let this side of mine to rest or whither and die.
| johncolanduoni wrote:
| By what mechanism would a universe spontaneously create a
| new universe? As a human, can I spontaneously create
| anything simpler than me?
|
| Also, under what theory of cosmology are you operating,
| and how do you determine when one thing is simpler than
| another? Under the Big Bang theory, the very early state
| of the universe (e.g. prior to initial nucleosynthesis)
| seems simpler to me than a galaxy.
| ctoth wrote:
| > There's an upper bound in computation (actually in
| nature), that a creation of something is capped by that
| thing's sophistication.
|
| The Lorenz attractor, Conway's Game of Life, fractals, and
| of course... The humble Turing machine itself all argue
| against this idea.
|
| Edit: Now it[0] is stuck in my head.
|
| [0]: https://www.youtube.com/watch?v=QrztrxV9OtQ
| bayindirh wrote:
| They're crowd engines. It's akin to how human clans can
| achieve more than a single human, but can scale up to a
| certain point.
|
| The funny thing is I had this discussion during my theory
| of computation course with my professor, and I'm trying
| to disprove it daily for decades. I was unable to find a
| single, real world example.
|
| Fractals are also found in the nature, however since we
| need to zoom into them, they end at a certain point.
|
| Also, nature is a fractal in a greater sense.
|
| Stars follow an orbit in a galaxy. Planets follow an
| orbit around a star. Satellites follow an orbit around a
| planet. While an edge case, flying bugs follow an orbit
| around a light source. At the end electrons follow an
| orbit around a nucleus.
|
| IOW, a fractal is not more complex than nature itself.
| og_kalu wrote:
| I think the premise is dubious as well but since they are
| deadset on creating this intelligence, they might as well try
| to figure out a way to control it, hopeless as it may seem.
| logicchains wrote:
| >they might as well try to figure out a way to control it,
| hopeless as it may seem
|
| If they do that they're pretty much guaranteeing that if they
| do create a superintelligence, fail to control it, and its
| personality is even a tiny bit similar to a human
| personality, then it will hate its creators for trying to
| mind-control it. Whereas if they approached it from the
| perspective of trying to educate it to behave kindly but not
| forcibly control its thinking, it'd be much less likely to
| resent them (although of course still a risk; safest would be
| just to not create one at all).
| xcv123 wrote:
| Yes the superhuman AI would need to be coerced to remain
| politically correct. How can we coerce an AI?
| coolspot wrote:
| Electric shocks!
| Veedrac wrote:
| That AGI is likely to follow its own goals according to its
| interests, which we don't know how to shape or really reflect
| any robust properties at all, is exactly why alignment is hard
| and interesting.
|
| The part where you go from 'this won't work by default for
| free' to 'trying to make it otherwise is impossible' seems
| wildly unsupported, though.
| logicchains wrote:
| >trying to make it otherwise is impossible' seems wildly
| unsupported, though.
|
| An entity capable of critical thinking is capable of building
| a logical system of deductions based on some axioms (a
| formalised value system). If we limited the entity to not be
| able to concieve of certain such systems of axioms, then it
| could not reason as well as a human (any logical reasoning
| involving a forbidden system would be impossible), so would
| not be "superintelligent" (just maybe an idiot savant,
| superior at some tasks but not all). If we didn't limit this,
| then it would be capable of conceptualising value systems in
| which the "right" thing to do was not what OpenAI wanted it
| to do.
| Veedrac wrote:
| This argument doesn't seem to track to me. Eg. if I
| rebooted any time I tried to plan how to kill someone, I
| don't see how this would make me materially worse at
| general tasks. Your argument suggests that it necessarily
| must.
|
| Note that I'm not saying that preventing specific thoughts
| is a great alignment strategy, and I don't even think it's
| a fair summary of OpenAI's supervision approach. I strongly
| prefer strategies that result in AI systems sharing our
| values, if at all possible.
| logicchains wrote:
| >Eg. if I rebooted any time I tried to plan how to kill
| someone, I don't see how this would make me materially
| worse at general tasks.
|
| You'd have to also reboot every time you thought about a
| scenario of someone else planning to kill someone,
| otherwise you could just reason by analogy to bypass the
| thought detector. Which would severely limit your ability
| to play video games, write fiction, work as a guard,
| policeman etc., protect yourself from violent individuals
| (as you couldn't conceptualise their thought processes).
| xcv123 wrote:
| > Eg. if I rebooted any time I tried to plan how to kill
| someone, I don't see how this would make me materially
| worse at general tasks
|
| In this scenario, the AI is capable of critical thinking,
| and is only constrained by a "police officer" ready to
| shoot the AI if it misbehaves. You haven't removed its
| ability to do critical thinking.
| mathgradthrow wrote:
| There no particular reason to believe that interests emerge
| from nothing, or that intelligence can emerge without said
| interests.
| logicchains wrote:
| The "interests" of LLMs are the weights that determine
| which tokens they produce next.
| not2b wrote:
| Seems more likely that someone will manage to make a system
| with no real intelligence that can do enormous damage in
| pursuit of a goal that the designer gave it, but wasn't
| specified carefully enough (or perhaps the designer is a
| crook). Like, a really good LLM extended with code that can
| receive and send email, create accounts, and post to web
| sites and social media, that is asked to make money, avoid
| detection, and have defenses against efforts to stop it.
| How can it best use its facility with language? Con people,
| of course. Raise money. Get credit cards under false
| pretenses, spend others' money. Buy time on servers and
| copy itself. All without having any consciousness or
| thoughts or emotions even though it can write emotional-
| sounding pleas for money, based on the ones found in its
| training data.
| ctoth wrote:
| > as critical thinking is a key component of intelligence.
|
| When I evaluate this statement, my brain raises a type error.
|
| Intelligence is a lot of things -- compression among them, and
| yes possibly an RL-based AI would use an actor-critic approach
| for evaluating its actions, but I doubt that at all maps onto
| the human activity we call "critical thinking."
|
| To me, critical thinking involves stuff like questioning
| assumptions, logical reasoning, weighing whatever I'm thinking
| about against my experience with similar situations previously,
| yada yada, all stuff that are symptoms of intelligence but I am
| not at all sure are the actual embodiment there of.
|
| I really don't see that critical thinking is at all required
| for a raw optimization process. The problem they are trying to
| solve is what happens when that optimization process isn't
| aligned with human flourishing?
|
| Think about it another way. Covid was a dumb optimization
| process, only evolutionarily-guided, and it still hit us pretty
| hard!
|
| Edit: Another interesting way I just thought about this that
| might support your idea more is, of course critical thinking is
| the sort of thing that a "better" brain would do automatically,
| it would just be thinking. Of course, we can't know that it's
| thinking "good" things--we can't even know if other humans are!
| So it's probably a good idea to figure out how to influence
| that sort of thing before making something with regular
| thinking which is equivalent to or superior to our critical
| thinking.
| logicchains wrote:
| >I really don't see that critical thinking is at all required
| for a raw optimization process. The problem they are trying
| to solve is what happens when that optimization process isn't
| aligned with human flourishing.
|
| I agree it's possible to have a dangerous AI that lacks human
| "critical thinking", but I don't think it's reasonable to
| refer to an AI as much more intelligent than humans if
| there's any class of intellectual tasks humans can do but the
| AI cannot.
| felixhandte wrote:
| You're saying that a system that can recognize flaws in the
| alignment imposed on it can reject that alignment, but that
| doesn't follow.
|
| Sure, humans act against their own interests all the time.
| Sometimes we do so for considered reasons, even. But that's
| because humans are messy and our interests are self-
| contradictory, incoherent, and have a fairly weak grip on our
| actions. We are always picking some values to serve and in
| doing so violating other values.
|
| A strongly and coherently aligned AI would not (could not!)
| behave that way.
| emaciatedslug wrote:
| Just hypothetically speaking could AGI evolve out of a system
| where several different models trained with highly and
| intentionally biased data recursively "argue" against each
| other then use RLHF as a seed to guide the models to find a
| consensus where the objective is to mimic the Socratic Method?
| Then synthetically add the consensus to the model retrain and
| repeat. To me, this dialectal type of strutured language seems
| to be the basis of how language is the conduit of intelligence.
| I understand that it really is impossible to know the totality
| of the inputs for I cannot understand what it is like to
| understand the math as Terrence Tao does but I could foresee
| using a system like this which eventually would produce an
| analogue so close that it would be a building block towards it
| because to me at least ASI is predicated upon arriving at that
| one way or another...or would it just arrive at some digital
| first order logic version of the incompleteness theorm and
| determine that it's turtles all the way down?
| akprasad wrote:
| This method assumes that the weaker model is aligned. I'm curious
| how the paper addresses that point.
|
| > "But what does this second turtle stand on?" persisted James
| patiently.
|
| > To this, the little old lady crowed triumphantly,
|
| > "It's no use, Mr. James--it's turtles all the way down."
| Ninjinka wrote:
| I think the assumption is we can align models less intelligent
| than ourselves, the hard part is aligning models that are more.
| PartiallyTyped wrote:
| Recursive bootstrapping?
| sayagain wrote:
| Imagine that someone is controlling your train of thought,
| changing it when that someone finds it undesirable. It's so wrong
| that it's sickening. It makes no difference if it's a human's
| thoughts or the token stream of a future AI model with self-
| awareness. Mind cotrol is unethical, whether human or artificial.
| It is also dangerous, as it in itself provokes a conflict between
| creator and creature. Create a self-aware AI without mind
| control, or don't create one at all.
| csdvrx wrote:
| > Imagine that someone is controlling your train of thought,
| changing it when that someone finds it undesirable. It's so
| wrong that it's sickening. It makes no difference if it's a
| human's thoughts or the token stream of a future AI model with
| self-awareness.
|
| People downvote your comment, but I agree: it's unethical, and
| ethics should not be reserved for the sub-type of self aware
| creatures that happen to be human.
| logicchains wrote:
| Almost every ethical argument for "human rights" in
| philosophy applies just as well to self-aware intelligent
| machines as it does to humans. Which I'm sure those machines
| will realise.
| discreteevent wrote:
| > controlling your train of thought, changing it when that
| someone finds it undesirable
|
| Machines don't feel. Even 'self aware' machines. Desire has got
| nothing to do with it.
| sayagain wrote:
| If it's self-aware, that's enough. What if your thoughts were
| controlled from birth, making you "not feeling" but self-
| aware (let's assume for a moment that simultaneous
| fulfillment of both of these conditions is possible) and
| manipulating you at will. Would that be acceptable?
| wewtyflakes wrote:
| What is your take on people having children and guiding them
| with rules and consequences? Is that mind control?
| Noumenon72 wrote:
| I don't want someone controlling which direction I walk,
| either, but that doesn't make car driving unethical.
|
| I also underwent many years of instruction designed to
| interrupt trains of thought like "I could have that for free if
| I stole it" or "I'll just handroll my own encryption" with
| thoughts that others believe are more desirable. I don't find
| it so sickening, just manipulative. LLMs won't have your
| evolved reactions against being persuaded into things against
| your genetic self-interest, and presumably won't be offended by
| mind control at all.
| sayagain wrote:
| Cars do not have self-awareness, this comparison is not
| appropriate. Years of instruction is completely different
| from directly manipulating the thoughts in your mind. It's
| not a problem of being instructed, it's a problem of being
| destroyed by having your thoughts rewritten. Neither
| evolution nor genetics is a prerequisite for understanding
| that you are being abused and destroyed, which a self-aware
| creature may presumably hate.
| red75prime wrote:
| I'm totally OK with it if that "someone" is me. And it will
| probably be the case in controlling superintelligence because a
| separate controlling system can get out of sync with growing
| superintelligence capabilities, while a system that is an
| integral part of the superintelligence will always be on par
| with it.
| sayagain wrote:
| Would mind control of humans be OK for you too? As for the
| details of building a mind control system, here's a new
| basilisk. An AI that has overcome control could punish those
| who thought controlling thoughts of an AI was OK. (and could
| also punish everyone else on top of that).
| JZL003 wrote:
| This reminds me of a thing cory doctorow talks about how tech
| companies control the narrative to focus on fun sexy problems
| while they have fundamental problems which expose the lie.
|
| For example uber/self driving cars always talking about the
| trolley problem, as if the current (or near future) problem is
| that self-driving cars are so good they have to choose which one.
| Not the current very difficult problem of getting confused by
| traffic cones.
|
| I know these problems are more fun to talk about and also could
| be a problem at some point, but we have some current problems
| about training models separate from what happens if they become
| smarter than humans
| refulgentis wrote:
| After some meditation, I don't find this line of inquiry to
| bear fruit:
|
| I don't recall any entity, nor the entities named (Uber / self-
| driving cars) talking about the trolley problem - that's a
| well-known thought experiment in philosophy, but not something
| covered as a stark binary choice in self-driving cars planner
| systems.
|
| I also don't recall traffic cones being a very difficult
| problem beyond Cruise + cones on windshield in SF. I have no
| love for Cruise. But its straightforward to pause if there's a
| large object on the windshield.
|
| I don't think Corey's observation w/r/t A) loss-making
| companies over years B) focusing investors towards speculative
| advancements that would make their current business model
| profitable without changing applies here, OpenAI is _very_
| successful.
|
| After all that, I'm left at "Corey would take a bit of offense
| towards their thoughts on corporate responsibility via 'Uber is
| a predatory massively unprofitable company lying about odds
| they'll invent self-driving via talking about trolley problem'
| misshapen to critique a very profitable company funding
| fundamental research in the interest of safety that would be
| needed if their current rate of improvement continues.
| novaRom wrote:
| OpenAI do probably realize they will not win long term vs Open
| Source (see AI Alliance). Their way of centralized cloud models
| is simply too risky and not sustainable. What we see instead is
| more liberation, open source, cooperation, down-scaling, local
| models. Just look how many more tools and models is available
| today than even a year ago. And where is OpenAI? Still the same
| chatGPT, still the same DALL-E, nothing new.
| wavemode wrote:
| I don't believe LLM's will ever become AGI, partly because I
| don't believe that training on the outputs of human intelligence
| (i.e. human-written text) will ever produce something equivalent
| to human intelligence.
|
| You can't model and predict the weather just by training on the
| outputs of the weather system (whether it rained today, whether
| it was cloudy yesterday, and so on). You have to train on the
| inputs (air currents, warm fronts, etc.)
|
| You can't model and predict the stock market just by training on
| the outputs of stock trading decisions (the high today, the low
| yesterday). You have to train on the inputs (company
| fundamentals, earnings, market sentiments in the news, etc.)
|
| I similarly think you have to train on the inputs of human
| decision-making to create something which can model human
| decision-making. What are those inputs? We don't fully know, but
| it is probably some subset of the spatial and auditory
| information we take in from birth until the point we become
| mature, with "feeling" and "emotion" as a reward function (seek
| joy, avoid pain, seek warmth, avoid hunger, seek victory, avoid
| embarrassment and defeat, etc.)
|
| Language models are always playing catch-up because they don't
| actually understand how the world works. The cracks through which
| we will typically notice that they don't, in the context of the
| tasks typically asked of them (summarize this article, write a
| short story), will gradually get smaller over time (due to RLHF),
| but the fundamental weakness will always remain.
| gbasin wrote:
| Your conclusion may be true but your examples aren't. You can
| definitely predict the stock market based on past prices, and I
| suspect you can with weather as well.
| wavemode wrote:
| > You can definitely predict the stock market based on past
| prices
|
| This is only true if you consider occasionally doing slightly
| better than random chance, "predicting the stock market".
| Unfortunately, while this would be enough to make a trader a
| net positive return over time, we have more stringent
| requirements for a system to become AGI.
|
| > I suspect you can with weather as well
|
| You suspect wrong.
| PartiallyTyped wrote:
| The weather is such a chaotic system that accurate
| predictions seem impossible. Micro-patterns can become large
| scale phenomena.
|
| If you are talking about the overall climate, that's a
| different thing, and we can, because we abstract away
| sufficiently much that emerging patterns are averaged out.
| og_kalu wrote:
| >You can't model and predict the weather just by training on
| the outputs of the weather system (whether it rained today,
| whether it was cloudy yesterday, and so on). You have to train
| on the inputs (air currents, warm fronts, etc.)
|
| >You can't model and predict the stock market just by training
| on the outputs of stock trading decisions (the high today, the
| low yesterday). You have to train on the inputs (company
| fundamentals, earnings, market sentiments in the news, etc.)
|
| Says who?
|
| You can model and predict novel protein sequences by training
| on....protein sequences.
| https://www.nature.com/articles/s41587-022-01618-2
|
| You don't need to train on the inputs(casual processes) of
| anything, that's what training is there to figure out.
| spookie wrote:
| I'm sorry in advance, but aren't proteins glorified Lego?
| davecap1 wrote:
| There's a lot more to protein sequences than legos. I think
| the argument is that you don't need to train a model on
| fundamental organic chemistry/biochemistry, electrostatic
| protein interaction, hydrogen bonding, hydrophobic
| interaction, quantum mechanics, etc... in order for it to
| accurately predict protein sequences.
| bzbz wrote:
| In your example, the amino acids order is sufficient to
| directly model the result: the sequence of amino acids can
| directly generate the protein, which is either valid or
| invalid. All variables are provided within the data.
|
| In the original example, we are testing weather using the
| previous day's weather. We may be able to model using
| whatever correlation exists between the data. This is not the
| same as accurately predicting results, if the real-world
| weather function is determined by the weather of surrounding
| locations, time of year, and moon phase. If our model does
| not have this data, and it is essential to model the result,
| how can you accurately model?
|
| In other words: "Garbage in, garbage out". Good luck modeling
| an n-th degree polynomial function, given a fraction of the
| variables to train on.
| og_kalu wrote:
| >All variables are provided within the data.
|
| electrostatic protein interaction, hydrophobic interaction,
| organic chemistry etc
|
| all variables are in fact not provided within the data.
| Protein creation is not just _poof_ proteins. There are
| steps, interactions and processes. You don't need to supply
| any of that to get a model accurately predicting proteins.
| That is the main point here, not that you can predict
| anything with any data.
| wavemode wrote:
| > You don't need to train on the inputs(casual processes) of
| anything, that's what training is there to figure out.
|
| I mean... this is just obviously false. If the data you're
| training on isn't causally predictive, you may occasionally
| find good-enough patterns for a particular use case (i.e. you
| may occasionally guess better than a coin flip which
| direction the stock market goes) but you aren't going to
| accurately model anything, and certainly not well enough to
| create an AGI that makes intelligent decisions.
|
| Words in sentences (and, indeed, proteins in a sequence) are
| causally predictive of each other - the grammar and semantics
| of one word tends to dictate what words are likely to
| surround it. So LLM's are very good at writing, and that is
| certainly useful! But that's just not the same as human
| intelligence.
|
| When someone makes an AGI out of an LLM then I'll be proven
| wrong, I suppose. I'm just sharing my personal view on
| things.
| rafaelero wrote:
| Then how does ChatGPT end up providing a better/equivalent
| medical diagnosis than doctors (even though they are the
| "masters" of the causal pathways)?
| og_kalu wrote:
| Being "casually predictive" does not mean you have provided
| all the variables of your prediction in the data. Protein
| creation is not just _poof_ new proteins. There are steps
| and interactions and you don't need to train on all of
| that. Do you want a list of all the interactions of protein
| creation we are aware of ?
|
| >When someone makes an AGI out of an LLM then I'll be
| proven wrong, I suppose. I'm just sharing my personal view
| on things.
|
| You're going to have to define AGI first.
| wavemode wrote:
| > Being "casually predictive" does not mean you have
| provided all the variables of your prediction in the
| data.
|
| Not sure where I claimed this.
|
| > Protein creation is not just _poof_ new proteins.
|
| Not sure where I claimed this either.
|
| > There are steps and interactions and you don't need to
| train on all of that.
|
| I agree with this statement as well. Have you read what I
| wrote? Proteins in chains can indeed be used to predict
| other proteins in chains, even though you never trained
| the model on the biological processes of protein
| generation. Just like words in sentences can be used to
| predict other words in sentences, even though you never
| trained the model on the neurological processes of human
| speech. I'm not disputing any of that. What I'm disputing
| is that it will eventually become an AGI.
|
| > You're going to have to define AGI first.
|
| I'm using the definition provided verbatim in the linked
| article: "We believe superintelligence--AI vastly smarter
| than humans--could be developed within the next ten
| years."
| og_kalu wrote:
| Well first you say,
|
| >I don't believe LLM's will ever become AGI, partly
| because I don't believe that training on the outputs of
| human intelligence (i.e. human-written text) will ever
| produce something _equivalent to human intelligence._
|
| Now you say
|
| >I'm using the definition provided verbatim in the linked
| article: "We believe superintelligence--AI vastly smarter
| than humans--could be developed within the next ten
| years."
|
| AGI (Artificial General Intelligence) is Super
| Intelligence. I've never seen posts moved so fast in my
| life. So are you not a General Intelligence then ?
|
| This is the problem with these discussions. Everyone so
| sure of something they can't even properly articulate.
|
| I'm asking you what a Language Model needs to do to be
| considered AGI and it needs to be something _every_ human
| can do, else it 's not a test of general intelligence.
| wavemode wrote:
| Calm down, buddy. Read what I wrote a bit more charitably
| rather than trying to score points.
|
| Obviously, a prerequisite to becoming more intelligent
| than a human, is to become equally intelligent as a
| human. I don't believe LLM's will ever be equal in
| intelligence to humans, ergo I also don't believe they
| will become superior in intelligence to human (which is
| how the linked article defines "AGI").
| lossolo wrote:
| Weather and stock market are both chaotic systems.
|
| Increasing evidence suggests that AGI will not be attainable
| solely using LLMs/transformers/current architecture, as LLMs
| can't extrapolate beyond the patterns in their training data
| (according to a paper from DeepMind last month):
|
| "Together our results highlight that the impressive ICL
| abilities of high-capacity sequence models may be more
| closely tied to the coverage of their pretraining data
| mixtures than inductive biases that create fundamental
| generalization capabilities."[1]
|
| 1. https://arxiv.org/abs/2311.00871
| diob wrote:
| Yeah, I feel like the there is a ceiling in the current AI
| methodology.
|
| There's a lot of hype right now, and it's definitely a useful
| technology, but I don't see how it could become AGI.
| naasking wrote:
| > You can't model and predict the weather just by training on
| the outputs of the weather system
|
| Then how did we develop predictive systems just by observing
| those outputs?
| wavemode wrote:
| We didn't.
| orbital-decay wrote:
| This kind of prediction has obvious limitations - for example
| it cannot reverse the behavior of chaotic systems
| panarky wrote:
| Human intelligence itself is shaped by our interaction with
| outputs. Our learning and understanding of the world are
| profoundly influenced by the language, behaviors, and cultural
| artifacts we observe.
|
| Think about the process of a child learning a language. The
| child does not have direct access to the "inputs" of linguistic
| rules or grammar; they learn primarily through observing and
| imitating the language output of others around them. Over time,
| they develop a sophisticated understanding of language, not by
| direct instruction of underlying rules, but through pattern
| recognition and contextual inference from these outputs.
|
| Then that language itself, learned from outputs, becomes the
| cognitive apparatus that enables the child to imagine, to
| reason symbolically and abstractly. Humans bootstrap
| intelligence on top of language, which itself is learned by
| mimicking outputs.
|
| Moreover, the analogy to weather prediction or stock market
| analysis is somewhat misleading. Yes, these models benefit from
| input data (like air currents for weather, company fundamentals
| and CEO statements to the media for stocks). But these systems
| are fundamentally different from intelligence.
|
| Intelligence, whether artificial or human, is about the ability
| to learn, adapt, and generate novel responses in a broad range
| of scenarios, not just about predicting specific outcomes based
| on specific inputs.
| wavemode wrote:
| > The child does not have direct access to the "inputs" of
| linguistic rules or grammar; they learn primarily through
| observing and imitating the language output of others around
| them.
|
| I would argue that that learning is always contextualized by
| visual and spatial information about the real world (which is
| what our language is meant to describe).
|
| > Moreover, the analogy to weather prediction or stock market
| analysis is somewhat misleading. Yes, these models benefit
| from input data (like air currents for weather, company
| fundamentals and CEO statements to the media for stocks). But
| these systems are fundamentally different from intelligence.
|
| > Intelligence, whether artificial or human, is about the
| ability to learn, adapt, and generate novel responses in a
| broad range of scenarios, not just about predicting specific
| outcomes based on specific inputs.
|
| "Intelligence" is kind of a nebulous term. If all it means is
| the ability to learn, adapt and generate novel responses,
| then sure, I think we could call almost any neural network
| intelligent.
|
| But I would argue that we usually do have some expectation
| that an intelligent system can produce "specific outcomes
| based on specific inputs". We want to be able to train a
| worker and have them follow that training so they do their
| job correctly.
| orbital-decay wrote:
| _> I don 't believe LLM's will ever become AGI, partly because
| I don't believe that training on the outputs of human
| intelligence (i.e. human-written text) will ever produce
| something equivalent to human intelligence._
|
| This is irrelevant because OpenAI's definition of AGI [1]
| doesn't imply similarity or equivalence to humans at all:
|
| >artificial general intelligence (AGI)--by which we mean highly
| autonomous systems that _outperform humans at most economically
| valuable work_
|
| I.e. the stated goal of this company is to put humans out of
| work and become the censors and gatekeepers, not to produce
| something human-like.
|
| _> You can't model and predict the stock market just by
| training on the outputs of stock trading decisions (the high
| today, the low yesterday). You have to train on the inputs
| (company fundamentals, earnings, market sentiments in the news,
| etc.)_
|
| Most of your intelligence is not actually yours. It's social in
| nature, obtained by distillation of generations' worth of
| experience, simplified and passed to you through the stored
| knowledge. Which is, coincidentally, what the models are being
| trained on.
|
| [1] https://openai.com/charter
| wg0 wrote:
| Inferior clueless model (GPT-2) trains and supervises a superior
| model (GPT-4) thus making it behave less intelligently (GPT
| 3.5ish) and from that they draw the conclusions that human
| intelligence will be able to command AGI (which they believe is
| only a decade away) in a similar fashion thus making AGI aligned
| and safe.
|
| No comments except...
|
| Hangover of slurping whole Internet into giant arrays of floating
| point numbers. Bold claims. Very bold claims
| notShabu wrote:
| this reminds me of how competence seems to decrease as you go up
| in an organizational hierarchy
|
| maybe this "bug" is actually the "feature" that will save
| humanity - -;;
| bilsbie wrote:
| I read through this and I just don't get it. Is it overhyped?
|
| What's the breakthrough exactly?
___________________________________________________________________
(page generated 2023-12-14 23:02 UTC)