[HN Gopher] The real data wall is billions of years of evolution
___________________________________________________________________
The real data wall is billions of years of evolution
Author : walterbell
Score : 77 points
Date : 2024-10-03 18:23 UTC (5 days ago)
(HTM) web link (dynomight.substack.com)
(TXT) w3m dump (dynomight.substack.com)
| spacebacon wrote:
| Lots of good thinking in this article. A few things come to mind
| before we hit a data wall.
|
| 1. Sensor all things
|
| 2. Waves upon waves
|
| 3. Dynamic or Living Semiotic Graphs. Bring your own terminology.
|
| 4. General Artificial Synesthesia.
| Nevermark wrote:
| Important concept for model building:
|
| You don't need more data when the data you have characterizes a
| problem well. More data is simply redundant and resource wasting.
| In this case, talking like people about things people talk about
| is covered well by current data sets. Saying we can't get more
| data is really saying we have collected at least enough data.
| Probably more than we need.
|
| Lots of room to improve models though:
|
| Using convolution for vision learning didn't create/require more
| data than training fully connected matrices. And it considerably
| increased models efficiency and effectiveness on the same amount
| of data. Or less.
|
| Likewise, transformers have a limited window of response. Better
| architectures with open ended windows will be able to do much
| more. Likely more efficiently and effectively. Without any more
| data. Maybe with less.
|
| Maybe in a few decades we will reach a wall of optimal models. At
| the rate models are improving now that doesn't appear to be
| anytime close.
|
| Finally, once we start challenging models to perform tasks we
| can't, they will start getting data directly from reality. What
| works, what doesn't. Just as we have done. The original source of
| our knowledge wasn't an infinite loop of other people talking
| back to the beginning of time.
| kridsdale3 wrote:
| I wonder if we'll reach the physical nanostructure wall of
| silicon long before that, and then all progress will have to be
| algorithmic efficiency gains. The era of Metal Muscle will end
| and we will return to the era of smart people pondering in
| coffee shops.
| EliBullockPapa wrote:
| Even if transistors reach physical limits, there's always
| different materials and architecture optimizations. We also
| know the human brain has far more intelligence per watt than
| any transistor architecture I know of. The real question is
| if those will be commercially worth researching.
| dimatura wrote:
| I believe this is part of the argument in the post - the
| "architecture" of the nervous system (and the organism it is an
| inseparable part of) is itself largely a product of evolution.
| Its already optimized to deal with the challenges the organism
| needs to survive/reproduce, and depending on the organism, with
| little or even no data.
| lolc wrote:
| It's weird to me how the article in the very first paragraph
| mentions brute force, but as data problem. As if we hadn't seen a
| staggering raise in ops and memory bandwidth. Machines have been
| able to hold the textual history of humans in short-term memory
| for a while. But the ops they could perform on that have been
| limited. Not much point telling a 2005 person to "add data". What
| were they going to do? Wait 20 years to finish a round we do in a
| week now?
|
| It's very clear to me that the progress we observe in machine
| intelligence is due to brute processing power. Of course
| evolution of learning algorithms is important! But the main
| evolution that drives progress is in the compute. Algorithms can
| be iterated on that much faster if your generations are that much
| shorter.
|
| Why are all these AI companies falling over each other to buy the
| best compute per watt humans have ever produced? Because compute
| is king and our head was optimized by evolution to be very
| efficient at probabilistic computing. That's where machines are
| catching up.
|
| The mark of intelligence is to not need much data at all.
| jcgrillo wrote:
| > The mark of intelligence is to not need much data at all.
|
| I think part of the answer might be information filtering. The
| eye can detect single photons, but by the time that information
| from 10^16 photons/s entering the eyeball gets to the meat CPU
| it's been filtered down to something relevant and manageable.
| And at no part in that pipeline is any component operating at
| more than like 100Hz.
|
| So fine tuning the filters to match the processor--and all the
| high fidelity sensors--simultaneously sounds like a job for
| evolutionary search if ever there was one. But this is the wild
| ass guess of someone who doesn't actually know much about
| biology or machine learning so take with a big chunk of salt.
| kridsdale3 wrote:
| We're also able to do many kilowatts worth of digital
| inference equivalent processing, instantaneously, for a
| couple watts at most in cost.
| jcgrillo wrote:
| Right, and we're doing it with highly specialized analog
| hardware, not the general purpose digital kind. Maybe
| there's a thread to pull on there as well?
| marstall wrote:
| michael levin talks about "intelligence at every scale". he has a
| recent study where he found some of the hallmarks of intelligence
| in an off-the-shelf sorting algorithm. individual cells by
| themselves certainly have signs of intelligence, such as memory,
| attention, the ability to recognize that a strategy has failed
| and come up with another, etc.
| JohnMakin wrote:
| Language Models do not work like the human brain. Continuing to
| compare the two like there is an analogy at all is doing far more
| harm than good.
| nielsbot wrote:
| Right. I think the tl-dr of the article is: AI needs a
| different type of machine. And the "learnings" of millions of
| years of evolution is how to build it.
|
| I do wonder if humans will hit upon a real AI solution soon. We
| developed flying machines in < 100 years. They don't work like
| birds but they do fly.
| joe_the_user wrote:
| _...doing far more harm than good..._
|
| Odd turn of phrase. Thinking LLMs work like brains may be
| holding back an advance to full AGI but is that a harm or good?
| I'm not against all powerful models but "build and deploy this
| stuff as fast as possible with minimal consequence
| consideration" definitely seems like a harm to me. Perhaps the
| Sam Altmans of the world should keep believing LLMs are
| "brains".
| JohnMakin wrote:
| I guess it would depend on how you view AGI. I personally do
| not believe AGI is possible under current or near-future
| technology, so it is not really a concern to me. Even the
| definition of "AGI" is a little murky - we can't even
| definitely nail down what "g" is in humans, how will we do
| that with a machine?
|
| Anyway, that aside, yes, your general understanding of my
| comment is correct - if you _do_ believe in AGI, this kind of
| framing is harmful. If you don 't believe AGI, like me, you
| will think it is harmful because we're inevitably headed into
| another AI winter once the bubble bursts. There are actual
| very useful things that can be done with ML technology, and
| I'd prefer if we keep investing resources into that stuff
| without all this nonsensical hype that can bring it crashing
| down at any moment.
|
| An additional concern of mine is that continuing to make
| comparisons this way makes the broader populace much more
| willing to trust/accept these machines implicitly, rather
| than understanding they are inherently unreliable. However,
| that ship has probably already sailed.
| simne wrote:
| I think, we just don't have right data. What I mean, human is not
| pure brain, but first year of life learn physiology of himself,
| and this physiology is very important part of human intelligence,
| but it is unwritten, even some opinions named it unconscious.
|
| Current AI learning is not even deaf, but something like learn
| Dadaism (or other philosophy) without understanding of human
| being, with some much simpler life philosophy (single cell).
| starmaan wrote:
| In Information theory terms, evolution contributes 1 bit of
| entropy per generation (itila book mckay). Not sure what are
| those claims about DNA or physics.
| quicon wrote:
| Interesting wildcard ideas in the article, but I don't think we
| can understand how the brain works using computing concepts. For
| a nice discussion on DNA being the "blueprint" of life I
| recommend Philip Ball's "How life works".
| visarga wrote:
| Started good by mentioning the data wall, but finished bad. It's
| not the DNA, we share most of it with other species. DNA can't
| contain human culture, and if it could, we would have been as
| capable 200K years ago as today.
|
| It's the one thing we have and they don't - language. During our
| 200K years our species has accumulated concepts, models, methods
| and values. We put them in language form and transmitted them
| across generations.
|
| The main problem of course is search. We search for
| understanding, and do it collectively. A single lifetime would
| not suffice, it takes humanity across many generations to travel
| the road from caves to AI. That is why language is key, allows
| for iterative improvement. It also articulates search space, goal
| space and action space. It is symbolic and discrete, that allows
| for exact replication. DNA shares the same qualities with
| language - isn't it interesting?
|
| Imagine you make a discovery, but having no language you can't
| reproduce the steps again, because it's all fuzzy. Or you can't
| teach anyone else. Language, with its symbolic structure is
| necessary to reproduce complex sequences of steps. Without it our
| exploration would not yield the fruits of exploitation. We would
| not benefit for unreproducible discovery.
|
| I am against all kinds of essentialism. Chomsky thinks we have
| innate grammar, but forgets about co-adaptation of language for
| learnability in children, and about learning from our five senses
| - they provide a better explanation than innateness.
|
| Searle explains our special status by biology, we have biological
| brains that's why we genuinely understand, he rejects distributed
| understanding but can't explain how come no single neuron
| understands on its own anything.
|
| Chalmers thinks there is an inexplainable experience (qualia),
| but doesn't consider relational embeddings that can model
| qualities in experience. Relational embeddings are created by
| relating experiences against other experiences, creating their
| own high dimensional semantic space. No, it's not panpsychism,
| the whole universe doesn't need to be conscious.
|
| And this time, in the article the magic is attributed to DNA.
| It's not that, it is search. We search and learn, learn and
| transmit, cooperate and reuse. Its not even a brain thing. It's
| social. We need more than one brain to cover this path of
| cultural evolution. Progress and language are not based in
| individuals but in societies.
|
| My point is that from now on we hit the data wall. Imitation is
| thousands or millions of times easier than real innovation and
| discovery. AI will have to pay the same exploration price, it
| will have to learn from the world. Of course, new discoveries are
| not written in any books. They have to be searched. And search is
| an activity dependent on the environment we search in. Not in
| brains alone, or even DNA. AI will search and create new data,
| but it will be a slow grind.
| cma wrote:
| > and about learning from our five senses
|
| You can be born without 5 and still learn very well. If sound
| and vision are both missing before a critical age (e.g. Helen
| Keller lost hers at 19mo) it can affect cognitive development.
| dboreham wrote:
| > DNA can't contain human culture
|
| Furthermore: My conjecture is that DNA doesn't contain any
| "training data". There's no data path for information to get
| from an organism's learnings about its surroundings in
| generation N, into the DNA for generation N+1. DNA is just
| plans for how to wire up the I/O devices. Everything we think
| we see that seems like "instinct" will turn out to be explained
| by a combination of the wiring to I/O devices, and early
| training.
| overtomanu wrote:
| DNA does not contain the training data, but it can be
| transmitted to next generation in other ways, like alteration
| of gene expression.
|
| Reference:
|
| Transgenerational epigenetic inheritance - Wikipedia
|
| https://en.wikipedia.org/wiki/Transgenerational_epigenetic_i.
| ..
| ccozan wrote:
| In a thought experiment, I would really like to see chatGPT,
| Claude, Gemini, etc, talk to each other in a way that the
| prompts are primed with something like "you are not alone in
| this AI room, formulate questions and answers that help
| exchanging ideas".
|
| Also , an observation to language: even bacteria have a
| lauguage, the chemical language ( quorum sensing ). Meaning,
| maybe we just need to create the necessary medium for current
| LLMs , they will start talking to each other and creating their
| own language.
| mistermann wrote:
| This is interesting:
|
| https://x.com/repligate
| modeless wrote:
| > Current language models are trained on datasets fast
| approaching "all the text, ever". What happen when it runs out?
|
| Robots.
|
| To reduce hallucinations our AI models need more grounding in the
| real world. No matter how smart an AI is it won't be able to
| magically come up with answer to any possible question just by
| sitting and thinking about it. AIs will need to do experiments
| and science just as we do.
|
| To maximize the amount of data AIs can train on, we need robots
| to enable AIs to do their own science in the physical world. Then
| there is no limit to the data they can gather.
| SketchySeaBeast wrote:
| Doesn't this then turn into a problem of sample quantity? You
| would need to shift into a quality mindset because with a robot
| you can't perform a billion iterations, you're locked into much
| more complex world with unavoidably real time interactions.
| Failure is suddenly very costly.
| modeless wrote:
| With a million robots you can perform a billion iterations.
| We won't need a billion iterations on every task; we will
| start to see generalization and task transfer just as we did
| for LLMs once we have LLM-scale data.
|
| You are right that failure is costly with today's robots. We
| need to reduce the cost of failure. That means cheaper and
| more robust robots. Robots that, like a toddler, can jump off
| a couch and fall over and still be OK.
|
| Tying back to the article, this is the real evolutionary
| advantage that humans have over AIs. Not innate language
| skills or anything about the brain. It's our highly
| optimized, perceptive, robust, reliable, self-repairing,
| fail-safe, and efficient bodies, allowing us to experiment
| and learn in the real physical world.
| dingnuts wrote:
| > robust, reliable, self-repairing, fail-safe, and
| efficient bodies
|
| you must be young and healthy because I cannot imagine
| using any of these words to describe this continuously
| decaying mortal coil in which we are all trapped and doomed
| guitheeengineer wrote:
| AI's advantage would be that their learning can be shared
|
| For example if Robot 0002 learns that trying to move a pan
| without using the handle is a bad idea, Robot 0001 would get
| that update (even if it came before)
| SketchySeaBeast wrote:
| But that ends up with weirdly dogmatic rules because it's
| not always a bad idea to move a pan without using the
| handle, it's just in some situations. It still takes a ton
| of potentially destructive iterations to be sure of
| something.
| guitheeengineer wrote:
| Yea its tricky and costly. I believe we should bet on
| specificity to make this more optimal.
|
| I know the trend with AI is to keep the scope generic so
| it can tackle different domains and look more like us,
| but I believe that even if we reach that, we'll always
| come back to make it better for a specific skill set,
| because we also do that as humans. No reason for an AI
| driver to know how to cook.
|
| If we narrow the domain as much as possible it will cut
| the number of experiments it needs to do significantly
|
| Edit: I wonder if its even going to be useful to devote
| so much resources into making a machine as similar as us
| as possible. We don't want a plane to fly like a bird,
| even if we could build it.
| kridsdale3 wrote:
| Then we will continue to have a Temperature variable in
| the Action Models.
| Mistletoe wrote:
| My roomba can't do the whole room without screwing up or
| getting stuck, it feels like we are eons away from a robot
| being able to do what you describe autonomously.
| modeless wrote:
| A few short years ago we were eons away from passing the
| Turing test.
| AStonesThrow wrote:
| Imagine if NASA-JPL had an LLM connected to all their active
| spacecraft, and at the terminal you could just type, "Hey
| V'Ger, how are conditions on Phobos over the past Martian
| Year?"
| dimatura wrote:
| I think it's reasonable to argue that data acquired via a
| sensorimotor loop in an embodied agent will go beyond what you
| can learn passively from a trove of internet data, but this
| argument goes beyond that - the "data" in evolution is
| "learned" (in a fashion) not just from a single agent, but from
| millions of agents, even those that didn't survive to replicate
| (the "selection", of course, being a key part of evolution).
|
| A neat thing about the kind of artificial robots we build now
| is that the process can be massively sped up compared to the
| plodding trial and error of natural evolution.
| enasterosophes wrote:
| People were already talking about Big Data in the 90s. If you
| send this idea back in time to 2005, people wouldn't be stunned
| by the revolutionary innovations it would unlock. They would say,
| "oh, someone else on the Big Data hype train."
| kridsdale3 wrote:
| As much as we can plot out and understand exponential growth
| curves, I'm pretty sure people in 2005 would still be shocked
| to hear about GPUs with hundreds of gigs of RAM, with
| bandwidths around a Tbps, and not just one per server, but
| hundreds of industrial greenhouse sized buildings with a
| million of them, each, consuming entire nuclear plants of
| generation output.
|
| Also you could blow their minds with a 24 TB HDD. It's as nuts
| as telling a 2024 person about a 1 PB HDD in a regular PC.
| randcraw wrote:
| This essay sort of waves a hand at the sub-symbolic roots of
| knowledge that lie beneath text and that babies spend several
| years mastering before they are ever exposed to text. IMHO the
| proper measure of that latent knowledge is qualitative, not
| quantitative.
|
| It's the tacit 'grounded knowledge' of the world that's present
| in humans that has the potential to fully fill in LLMs' causal
| blank in their text-based superficial info. This kind of
| knowledge is threadbare in today's LLMs, but essential to form a
| basis for further self-education in any intelligent agent. I know
| STaR and RLHF have been suggested as synthetic means to achieve
| that experimental end, but I'm not sure they're sufficient to
| connect the dots between LLMs' high-level book learning and human
| babies' low-level experiment-based intuition for cause and
| effect. But adding yet more text data is surely NOT the way to
| span that chasm.
| pessimizer wrote:
| It's not DNA, it's embodiment in general. People learn an
| enormous amount in the process of existing and moving through
| space, and they hang all of their abstract knowledge on this
| framework.
|
| Related: it's a belief of mine that bodily symmetry is essential
| for cognition; having duplicate reflected forms that can imitate,
| work against, and coordinate with each other, like two hands,
| gives us the ability to imagine ourselves against the environment
| we're surrounded by. Seeing, sensing and being in full control of
| two things that are almost exactly the same, but are different
| (the two halves of one's body) gives us our first basis for the
| concept of comparison itself, and even of boundaries and the
| distinguishing of one thing from another. I believe this is
| almost the _only_ function of external symmetry; since
| internally, and mostly away from sensory nerves, we 're wildly
| asymmetrical. Our symmetry is the ignition for our mental
| processes.
|
| So I'm not in a DNA data wall camp, I'm in an embodiment data
| wall camp. And I believe that it will be solved by embodying
| things and letting them learn physical intuitions and
| associations from the world. Mixing those nonverbal physical
| metaphors with the language models will improve the language
| models. I don't even think it will turn out to be hard. Having
| eyes that you can move and focus, and ears that you can direct
| will probably get you a long way. With 2 caveats: 1) our DNA does
| give us hints on what to be attracted to; there's no reason for a
| model to look or listen in a particular direction, we have
| instincts and hungers, and 2) smell and touch are really really
| rich, especially smell, and they're really hard to implement.
|
| Incidentally: the article says that we've been optimized by
| evolution for cognition, but what could have been optimized was
| _child-rearing._ Having an instinct to _train_ might be more
| innate and extensive than any instinct to _comprehend._ Human
| babies are born larval, and can 't survive on their own for years
| if not decades. Training is not an optional step. Maybe the
| algorithms are fine, and our training methods are still hare-
| brained? We're training them on language, and most of what is
| written is wrong or even silly. Being able to catch a ball is
| never wrong, and will never generate bad data.
| aithrowawaycomm wrote:
| This simply does not pass the smell test. If you want to
| analogize biological brains to ANNs then clearly evolution
| refines the _architecture_ of the "natural neural network," not
| the _data._ No ANN training involves adding artificial neurons,
| defining new types of artificial neurons, etc, but that clearly
| describes the biological history of the brain.
|
| Taking this article literally, the brain hasn't really changed
| much since nematodes, it's just absorbed a ton of data. That's
| transparently stupid. All species, including humans and worms,
| have evolved over billions of years to get where they are today.
| Why is it that only humans get to access the billions of years of
| data? I am guessing the author has childish views about humans
| being "the most evolved" species. But all species are more
| evolved than they were 1m years ago.
|
| This entire article relies on a dumb bait-and-switch. It's an
| incoherent analogy which seems motivated by a desire to simply
| excuse away the shortcomings of transformers.
| danielmarkbruce wrote:
| > clearly evolution refines the architecture of the "natural
| neural network," not the data
|
| This doesn't pass the smell test. Common sense and experience
| suggest animals have both built in functionality and a built in
| ability to learn and they seem to work together. Babies don't
| learn to cry for food, it works right out of the box. Some
| animals work with practically zero parental guidance.
|
| This is a pretty silly statement overall. Just being alive and
| paying attention should give someone a sense for how silly it
| is.
| aithrowawaycomm wrote:
| I genuinely don't understand your point at all. I am not
| contesting that instincts exist, and I am not sure what you
| think the disagreement is. Let me rephrase:
|
| There have been two sides to ANN development: architecture
| and data. If we're analogizing brains to ANNs like the author
| is doing, then brains also have an "architectural" component
| and a "data" component. But we need to be clear that
| evolution (including individual mutations and epigenetics)
| shapes the architecture, while lived experience shapes the
| data. The author is claiming that the experience of billions
| of years of evolution should actually go into the "data"
| column, somehow, and that human brains at birth have actually
| learned from the experience of rodent-lizards that lived 200
| million years ago. This is just idiotic, and it does not help
| that the architecture is completely ignored. Again the
| problem is a fundamental bait-and-switch in the analogy
| itself.
|
| Instinctual behavior is much better-explained by evolution
| influencing the architecture of the brain, rather than waving
| your hands and saying "data, lots of it!" The crying baby is
| much better explained by an ancestor to all birds and mammals
| who had a neurological mutation that made it whine (perhaps
| softly) when it was hungry, not because that lizard finally
| accumulated enough "evolutionary data" to learn how to cry.
| (What would that data be?) The neurological mutation is of
| course purely speculative, but it is _plausible,_ which is
| not the case for what the author is saying.
|
| The author claims quite directly that humans learn from
| billions of years of animal experiences in a very similar way
| to how GPT learns from billions of lines of text, which is a
| preposterous claim that requires extraordinary evidence. No
| evidence is provided. I am confident that this Substack is
| quackery.
___________________________________________________________________
(page generated 2024-10-08 23:01 UTC)