[HN Gopher] Deep Neural Nets: 33 years ago and 33 years from now...
       ___________________________________________________________________
        
       Deep Neural Nets: 33 years ago and 33 years from now (2022)
        
       Author : gsky
       Score  : 258 points
       Date   : 2023-08-26 00:13 UTC (22 hours ago)
        
 (HTM) web link (karpathy.github.io)
 (TXT) w3m dump (karpathy.github.io)
        
       | Nevermark wrote:
       | The most fundamental change is the difference in _what_ models
       | are being trained on.
       | 
       | Little images of characters is a trivia type problem, very
       | different from training on the linguistic and visual
       | communication of essentially the whole human race.
       | 
       | Another 33 years of expanded computing resources won't be
       | training models to mimic the behavior and knowledge of humanity.
       | 
       | That problem ( _us!_ ) will have been reduced to a toy problem
       | long before then.
        
         | visarga wrote:
         | I think AI models will evolve by generating synthetic data,
         | filtering and improving it, and then retraining. Possibly with
         | external systems in the loop - code execution, search, human,
         | simulation or robot. Quality won't degrade because there will
         | be a lot of effort put into data filtering and diversity. We
         | can always improve on a model by giving it more time.
         | 
         | Model architecture doesn't matter compared to the dataset. Any
         | model from a class can learn the same skills from the same
         | data, but change the data and they all change their abilities -
         | the intelligence is in the data.
         | 
         | The future is data engineering, not model architecturing. Human
         | culture, by analogy, evolves faster than human biology. The
         | data is evolving faster than the model. And we are seeing a
         | drastic reduction in novel architectures in AI, diverse
         | datasets applied to the same transformer models in recent
         | years. Even among the transformers, very few variants are
         | largely used, thousands of them abandoned.
         | 
         | I like to think of it as language evolution by memetics being
         | the real engine behind intelligence. We and AI are riding the
         | language exponential together.
        
           | eru wrote:
           | > Model architecture doesn't matter compared to the dataset.
           | Any model from a class can learn the same skills from the
           | same data, but change the data and they all change their
           | abilities - the intelligence is in the data.
           | 
           | You might be right in the same sense that big-O notation is
           | 'right'. Constant factor can matter; especially once you have
           | to take energy use into account.
        
         | ResearchCode wrote:
         | Come close to solving the toy problem of autonomous driving
         | first, we're still waiting.
        
           | Nevermark wrote:
           | I don't know. I find pessimistic views, like you are
           | expressing, very strange.
           | 
           | My Tesla drives and navigates itself most of the time. 90-95%
           | at least, just not 100%.
           | 
           | As apposed to cars 10 or more years ago which didn't do any
           | of that.
           | 
           | To me it is much like the "God of the Gaps" when tremendous
           | progress on a big problem is dismissed negatively, due to the
           | (continuously shrinking) gaps of what it can't do.
        
             | ResearchCode wrote:
             | We are already five years late in autonomous vehicles
             | replacing all truck drivers. We will see if we even have
             | autonomous driving "long before" 33 years have passed.
             | "AGI" (rebranded AI after "AI" failed to deliver) will of
             | course still not be a thing.
        
               | Nevermark wrote:
               | I have no idea what "late" technology means.
               | 
               | And not delivering AGI yet is a problem?
               | 
               | What are these broad technology schedule based criticisms
               | founded on?
               | 
               | I really want to understand this viewpoint!
               | 
               | Hopefully not the over-optimism of anyone who uses
               | optimistic timelines as a motivational force. That's not
               | real data. Or a suitable benchmark for human progress.
        
             | cma wrote:
             | Waymo and Cruise are operating without drivers which is
             | much more impressive, even if in limited areas.
        
               | Nevermark wrote:
               | I don't know if it is much more impressive than Tesla,
               | given Tesla's "limited areas" don't seem very limiting in
               | my experience.
               | 
               | But I think all these complementary takes on the problem,
               | with significant year-to-year progress by all three
               | firms, are fantastic.
               | 
               | That used to be considered a fast learning curve!
        
       | breckinloggins wrote:
       | I really enjoyed this article. My only critique is that the 2055
       | predictions are "meta-linear". In other words: the author avoids
       | the (probable) mistake of taking our current tech and linearly
       | regressing the numbers 33 years forward, but the predictions
       | still suggest a kind of "worldline symmetry" with the present
       | date at the origin.
       | 
       | It's quite possible that none of these predictions will come true
       | simply because the timeframe is large enough for many
       | unanticipated breakthroughs and roadblocks.
       | 
       | Maybe someone will figure out a much, much simpler foundational
       | architecture than "perceptrons++", maybe we'll all be training
       | clouds of 3D gaussians, maybe quantum computers will finally take
       | off and we don't even have the nouns for the building blocks
       | we'll use.
       | 
       | On the negative side perhaps we hit a hard scaling limit (in
       | hardware or training) that we didn't see coming. Or a
       | civilizational setback.
       | 
       | All that said, though, if I were a betting man I wouldn't exactly
       | wager against the article's conclusions; they're probably the
       | best we can extrapolate knowing only the past and present state
       | of affairs.
        
         | Nevermark wrote:
         | I think you are right, the next 33 years are likely to be very
         | different.
         | 
         | I would lean to them being even more dramatic, due to the
         | opportunity to advance algorithms, not just resources.
         | 
         | On the more obvious side, most libraries are not yet taking
         | full advantage of many known gradient optimization techniques.
         | It's been so much easier to just add data & processing that
         | there is an overhangs of tools to still apply.
         | 
         | And large successful models are telling us important things.
         | 
         | For instance, it is clear that language models are learning a
         | kind of logic of language similar to how we process thoughts,
         | allowing highly disparate types information to be woven
         | together sensibly.
         | 
         | At some point, identifying the nature of that processing could
         | radically simplify language processing.
         | 
         | That is just one opportunity for radical architecture and
         | algorithm advances, and it would be revolutionary.
        
       | [deleted]
        
       | version_five wrote:
       | This was really good. The only thing I didn't see explicitly
       | discussed, although I guess it's obvious, is that what's
       | different 33 years later is the inputs the models operate on. The
       | '89 sota model used 16x16 greyscale images, today we have single
       | digit megapixel color images, in 30 years, a desktop will be able
       | to train Clip in 90 seconds, but what will the sota models be
       | trained on?
        
         | eigenvalue wrote:
         | We might have megapixel images that we can easily get with
         | phone cameras, but virtually all vision models in common use
         | take 224x224 resolution images as input, or maybe 384x384.
         | Anything higher resolution than that just gets resampled down.
         | It seems that you are better off using your compute budget on a
         | bigger "brain" than on better "eyes" for now.
        
           | version_five wrote:
           | I don't think that's current. Certainly the object detection
           | models work on bigger images, and the datasets they're
           | pretrained on e.g. coco are not 224x224. I think standard
           | models pretrained on imagenet, like the Resnets usually have
           | everything resized to 224x224, and so they favor this kind of
           | scaling.
        
         | ramblerman wrote:
         | Millions of hours of data captured by headsets like the vision
         | pro?
         | 
         | Not sure all the things it captures, but a model could be
         | trained on the combination of audio/video/spatial/iris/what
         | have you...
        
         | retrac wrote:
         | Human behaviour in a way far more general than which token we
         | might next type. To mimick humans as closely as might be
         | possible with the basic deep learning method, train something
         | that can predict human behaviour in general. Training would
         | require billions to quadrillions of hours of video and audio
         | and probably many other inputs, from many different people,
         | engaged in the full variety of human activity.
        
           | quickthrower2 wrote:
           | Humans have a brain that physically changes especially in
           | childhood, so that is potentially a massive advantage.
        
           | reducesuffering wrote:
           | Why? An adult by 25 only has 146k hours of video experience
           | "training," most of it repeated, derivative, and
           | unproductive. And their encoded genes can be observed in
           | their genome, so don't need to be retrained by millions of
           | years of evolution.
        
             | thaw13579 wrote:
             | Much of that time also includes physical interaction with
             | the world, which makes it far more valuable because it can
             | improve performance in a focused way.
        
             | [deleted]
        
             | canjobear wrote:
             | Neural nets seem to learn much slower than humans. Even
             | GPT-2 has seen orders of magnitude more tokens of language
             | than a human experiences in a lifetime. At least as far as
             | language is concerned, humans are able to extract a lot
             | more information from their training data.
        
               | mkaic wrote:
               | Humans are also extensively pretrained by billions of
               | years of evolution, so by starting from scratch GPT is
               | admittedly disadvantaged from the get-go.
        
               | georgeg23 wrote:
               | But the human genome is only 3 gigabytes and the vast
               | majority of that is unlikely to be encoding brain
               | structure.
        
               | retrac wrote:
               | "Only" 3 gigabytes.
               | 
               | The lambda calculus (a system we know is capable of
               | infinite self-complexity, learning, etc. with the right
               | program) can be described in a few hundred bits. And a
               | neural net can be described in the lambda calculus in
               | perhaps a few thousand bits.
               | 
               | Also, we have no idea how "compressed" the genome is.
        
               | georgeg23 wrote:
               | Basic structure can be encoded yes (and obviously is
               | given brains have consistent structure), but the weights
               | or parameters, presuming that brains learn via synaptic
               | weights, obviously do not fit in the genome.
               | 
               | Compression still must obey information theory.
        
               | canjobear wrote:
               | True, but we don't know how to recreate that kind of
               | pretraining.
        
             | edgyquant wrote:
             | Humans do not train on video, the idea of a video, or even
             | a frame, is a high level abstraction within the human
             | brain.
        
             | alanbernstein wrote:
             | How better to learn to do menial physical tasks like house
             | cleaning, and produce picking?
        
       | dang wrote:
       | Related:
       | 
       |  _Deep Neural Nets: 33 years ago and 33 years from now_ -
       | https://news.ycombinator.com/item?id=30673821 - March 2022 (5
       | comments)
        
       | antirez wrote:
       | Always refreshing reading Andrej Karpathy: the more he knows
       | stuff, the more he explores the fundamentals of the science of ML
       | in a direct and simple way. The field is full of papers that for
       | a very hard to reproduce gain in some new convoluted architecture
       | (in the hope to beat some state of art result) will happily fill
       | 50 useless pages trying to make their works "serious".
        
       | mark_l_watson wrote:
       | Great article. I lived through the early days of artificial
       | neural networks. I was on a DARPA advisory panel for neural
       | network tooling in the mid 1980s, wrote the first version of the
       | SAIC ANSim commercial product, and created the simple back-prop
       | model that was deployed in the bomb detector my company built
       | under contract to the FAA. I also managed a 'conventional' deep
       | learning team at Capital One 5-6 years ago.
       | 
       | My world has been very exciting in the last 18 months. I spend as
       | much time as I can exploring self hosted LLMs, APIs from Hugging
       | Face, OpenAI, etc.
       | 
       | My mind is blown even thinking about tech 33 years from now!
        
       | bilsbie wrote:
       | It's interesting in that time we almost completely lost interest
       | in neural networks and then came back around to them.
        
         | sroussey wrote:
         | I had to retake my AI class at university several times because
         | I just didn't agree on the "AI is symbolic search" aspect.
         | 
         | Now though, I'm sure people are taking LLMs and putting them
         | together to do forward and backward chaining.
        
         | jacquesm wrote:
         | Thank Hinton for that. It's a pity we don't have a Nobel for
         | software.
         | 
         | But a Turing award is pretty neat as well.
        
         | Sharlin wrote:
         | In this case there are good reasons for the resurgence, but
         | that's really the case with pretty much anything software-
         | related. Except the fashion cycles tend to be shorter with more
         | mainstream technologies.
        
       | m-i-l wrote:
       | So should we spend the next 33 years doing the same things, just
       | with more data and more compute power? That would be the logical
       | conclusion of the breathless "I can't believe it is finally
       | happening in my lifetime" and "we just need bigger models and
       | more data" enthusiasm for LLMs when they first appeared. But can
       | we really simply brute force our way to AGI?
       | 
       | Remember, 33 years ago "connectionist AI" wasn't the dominant AI
       | paradigm, and "symbolic AI" wasn't the only other approach either
       | - there were others, like "robotic functionalism" (the idea that
       | you couldn't have true intelligence with interacting with the
       | physical world). Maybe in 33 years some of these other approaches
       | will have a resurgence, perhaps in combination with connectionist
       | approaches. Or maybe they'll even be some entirely new approach.
        
       | Funtech007 wrote:
       | [flagged]
        
       | fnord77 wrote:
       | > Our datasets and models today [2055] look like a joke. Both are
       | somewhere around 10,000,000X larger.
       | 
       | will there really be 10 million times 400 million images floating
       | around then?
        
         | ramblerman wrote:
         | I think you are limiting yourself by thinking of the dataset of
         | the future as just being more and bigger images.
         | 
         | Perhaps it will be trained on whole videos, or a combination of
         | different inputs from agents that move about in the real world
         | / or a video game.
        
           | Lucasoato wrote:
           | Maybe the real game changer in the future will be the ability
           | to train the same model on very different kind of inputs like
           | video, images, text, audio... Imagine also all these data
           | cleaning tasks are already automated, you just need to feed
           | the model PDFs and automatically a support model will extract
           | all the relevant metadata... or probably you'll just be able
           | to select a set of books from an online library and your
           | model will train on them as well (of course for a non trivial
           | subscription lol)
        
         | sroussey wrote:
         | Generate as many as you need.
        
           | sroussey wrote:
           | Oh, also curious... today, how many individual image frames
           | from video are there just from Tesla vehicles?
        
           | alpaca128 wrote:
           | Training models from generated content degrades them over
           | time.
        
             | pyinstallwoes wrote:
             | Yet science fiction pushes civilization towards novelty
        
             | Philpax wrote:
             | The generated results can come from other means - for
             | example, pretraining on rendered CG imagery is quite
             | popular in the computer vision world, especially for
             | problems where acquiring ground truth data in the real
             | world is quite difficult.
        
         | djantje wrote:
         | Maybe, but the input in 2055 will be more something in the form
         | of continuous/realtime data input streams.
        
         | ben_w wrote:
         | 10e6*400e6/8e9/365/18 = 76 images per person per waking hour;
         | it's not implausible given how many cameras there are and how
         | many moments people might snap to share with remote friends --
         | I can easily believe we'll have always-on video chat with
         | multiple people in AR glasses by that point.
        
           | anonzzzies wrote:
           | Most images are not shared though; just snapped. In the past
           | you had photo albums no-one ever looked in. And that weren't
           | that many pics; now , whenever, people (old and young) take
           | 100s of pictures, on iPhones often by holding the button so
           | it zaps 100s of them in a few seconds.
        
             | ben_w wrote:
             | > Most images are not shared though
             | 
             | Not _yet_.
             | 
             | As the joke goes:
             | 
             | People in the 60s:
             | 
             | I better not say that or the government will wiretap my
             | house
             | 
             | People today:
             | 
             | Hey wiretap, do you have a recipe for pancakes?
        
             | cma wrote:
             | Maybe you won't receive your "world coin" universal income
             | dividend unless you livestream 24/7.
        
       | alexmuro wrote:
       | It's crazy how little has changed and how much had changed. I
       | remember what a revelation "the unreasonable effectiveness of
       | RNNs" was when I was read it and it feels like we live in a
       | different world.
        
         | benreesman wrote:
         | I think we could collectively more constructive and sober
         | conversation if we kept that 2015 bit of work as a sort of
         | baseline.
         | 
         | The new stuff is better, by a lot, and with implications more
         | to come.
         | 
         | But those of us paying attention then had a frame of reference
         | where "so much better it's crazy" still stops short of "it's
         | out of control".
         | 
         | It's a lot better.
        
       | dsign wrote:
       | It's not clear that compute will scale as it did for the next 33
       | years. But it doesn't really need to.
       | 
       | I read the article and I was thinking "my God, I remember I used
       | MSE that weekend in my pet ML project and it really didn't work
       | out that well; wrong loss function." Our current crop of LLMs, or
       | the one next year, will be perfectly able to tell me how I can
       | improve my code and graphs, which means that I can deploy some
       | expert-level techniques that otherwise would be "locked" to me by
       | 50000 hours of "mastery acquisition".
       | 
       | A part of me is telling me that we humans are doomed, and that in
       | 33 years we would have created a world in which we humans are
       | irrelevant. But another part tells me that if we avoid that fate
       | and all the other dooms, the future might just be quite bright.
        
         | somewhereoutth wrote:
         | > or the one next year
         | 
         | We have heard, and will continue to hear, this sort of thing
         | rather a lot. The last 5 yards are the hardest, but without
         | them the previous 5 miles are of limited utility.
        
       | phkahler wrote:
       | >> The original network trained for 3 days on a SUN-4/260
       | workstation.
       | 
       | This is exactly why I didn't start experimenting with this stuff
       | back then. I read some articles and had the interest, but having
       | no access to existing training data or "fast" computers was
       | really a show stopper. This article really convinced me that the
       | amazing results today are mostly due to hardware advances.
       | 
       | I will add my own view that 1) hardware will not be advancing
       | anywhere near so much in the future. And 2) training and
       | inference have to be done together like real brains do. Then the
       | AI will learn from experience while deployed and you can clone
       | the best ones later.
        
         | 0xDEF wrote:
         | >This article really convinced me that the amazing results
         | today are mostly due to hardware advances.
         | 
         | For LLMs that is true. But many other things like Whisper,
         | Stable Diffusion etc. could in theory have been made a decade
         | earlier.
        
       | mk_stjames wrote:
       | Something else I find exciting, starting with one of the
       | reflections-
       | 
       | The original training took 3 days on a Sun 4/260 workstation; I
       | can't find specifics but I believe that era of early SPARC
       | workstations would likely pull about 200 watts in total (the CPU
       | wasn't super high powered but the whole system, running with the
       | disks and the monitor etc would pull about that).
       | 
       | So 200 watts * 72 hours = 14400 watt-hours of energy.
       | 
       | Karpathy trained the equivalent on a Macbook, not even fully
       | utilized, in 90 seconds. Likely something around 20 watts * 0.025
       | hours = 0.5 watt-hours.
       | 
       | An energy efficiency improvement of nearly 30000x.
        
         | Yajirobe wrote:
         | > watt-hours
         | 
         | You mean joules (up to a constant factor)?
        
           | renonce wrote:
           | A watt-hour is 3600 joules but watt-hours or kilowatt-hours
           | is commonly used because it's easier to calculate.
        
         | Aardwolf wrote:
         | 30k doesn't even sound like that much to me given Moore's law.
         | I'd expect more improvement since 1989. Supercomputer
         | performance increased more than a million since then
        
           | fooker wrote:
           | Amdahl's law
        
           | cma wrote:
           | 33 years ago is 2000/1999
        
             | fHr wrote:
             | quickest maffs
        
             | Philpax wrote:
             | Um... you might want to check your tens digit.
        
           | swores wrote:
           | My (wrong) intuition on reading your comment was that you
           | were over-estimating the expected growth in performance over
           | that time period, but actually after checking the maths based
           | on Moore's Law, i.e. doubling every two years (though of
           | course I understand that was a rough estimate, more of a
           | concept prediction than expected to be precise) you're right
           | so I'll share the maths for anyone else whose intuition might
           | be as poor as mine:
           | 
           | Doubling every 2 years = compound annual growth rate (CAGR)
           | of ~41.42%                 CAGR = ((End Value / Start
           | Value)^(1 / Number of Years)) - 1            ((2 / 1)^(1 /
           | 2)) - 1 = 0.41421356237
           | 
           | Therefore in 34 years since then:                 1 * (1 +
           | 0.41421356237)^34 = ~131,072
           | 
           | So x30k is ~4.4x less than 131k. Then again, that's
           | equivalent to ~x1.833 every two years, compared to Moore's
           | Law of x2 every two years, so only ~8% less growth per two
           | years, which coming back to the fact that Moore's Law is a
           | rough estimate concept not an exact fact, doesn't seem to far
           | off!
        
             | jacquesm wrote:
             | The rest of that difference can easily be explained by the
             | difference in the class of hardware used. A desktop made
             | today vs a laptop is roughly that factor 4. Not sure if
             | back then there would have been laptops that you could have
             | done this on for a more apples-to-apples comparison.
             | 
             | Modern laptops give great efficiency, when I went for solar
             | power here the first thing to go was the desktop computer.
             | I still have it, but it hasn't run in over a year and the
             | elderly thinkpad that is now my daily driver uses far less
             | power and still has enough compute to serve my modest
             | needs. But if I would dive into something requiring much
             | more compute I'd have to start the desktop again.
             | Unfortunately power management is not such that computers
             | can really throttle down to 'miser mode' when you don't
             | need it, it's a good step but not as good as the jump
             | between desktop and laptop.
        
               | swores wrote:
               | Hadn't thought of that, good point
        
               | DesiLurker wrote:
               | also the 'memory wall', remember memory b/w did not grow
               | at pace with moore's law. sure there are ways to mitigate
               | it but that eat into chip budget and reflect when real
               | world performance is calculated.
        
               | jacquesm wrote:
               | Yes, true and in a way that wall is still there. The way
               | GPUs are limited in how much RAM they have because there
               | is a way to sell you that memory at a multiple of the
               | cost.
               | 
               | Imagine a GPU with a 128G or even 256G slot based memory
               | section that is sold unpopulated. 8 SODIMM slots or so.
        
             | 2Gkashmiri wrote:
             | imagine that we are discussing "proving" a law with
             | historical data from our POV but at the time, it must've
             | seem like a theory at best or comical at the least.
             | 
             | 8% less growth is not the point. The "law" has stood the
             | test of time which says something about the guy and his
             | vision
        
           | ben_w wrote:
           | It isn't much, but as the link says, the neural network they
           | were reimplementing is _too small_ to take advantage of
           | modern hardware.
        
         | fouronnes3 wrote:
         | This is very interesting, because I've always thought that all
         | NN performance should be measured in a unit with energy in the
         | denominator.
        
           | eru wrote:
           | It totally depends on what you want to use a measure for.
           | Just like neither height or volume alone will tell you what
           | will fit in your car.
           | 
           | By any measure that puts energy used by the brain in the
           | denominator, humans are probably dumber than ants. But that
           | doesn't mean those measures are always accurate.
           | 
           | (For contemporary neural networks, you also have to
           | distinguish training costs from inference costs.)
        
             | nojvek wrote:
             | To add more context, humans are 100W biological machines.
             | Brain is ~20% of that power - 20W.
             | 
             | The greatest form of general intelligence at 20W.
             | 
             | A MacBook Air is ~30W.
             | 
             | https://www.jackery.com/blogs/knowledge/how-many-watts-a-
             | lap...
        
               | beebmam wrote:
               | You're leaving out the training requirements
        
               | renonce wrote:
               | It is "the greatest" because we only appreciate
               | intelligence that we ourselves understand. A 0.0001W
               | calculator calculates arithmetic faster than any human
               | brain.
        
             | chubot wrote:
             | I dispute that, if the metric is a chess game between an
             | ant and a human
        
           | quickthrower2 wrote:
           | For inference that could be useful, but the energy is not for
           | the model it is for at least the tuple of: model, model
           | architecture and compilation, and hardware chosen.
        
       | ilaksh wrote:
       | Maybe a better idea of the future is to look at what LeCun is
       | working on now as a future program. He wants to change quite a
       | lot in order to move towards more animal-like cognitive
       | abilities.
       | 
       | LeCun is not even really interested in supervised learning
       | anymore, for example.
       | 
       | https://youtu.be/vyqXLJsmsrk?si=8n0ylC6qdLX06CmY
       | 
       | Note that the talk is not really primarily about ChatGPT even
       | though that's in the title. The new ideas are a little bit in.
       | The beginning of the talk is just him explaining how unimpressed
       | he is with LLMs. Which I think is a misjudgement but that doesn't
       | mean his plan doesn't have merit.
        
         | golol wrote:
         | I don't understand why Yann is so focused on his "Animals are
         | smarter than AI" analogy. If compute wasn't so limited,
         | couldn't we just train a transformer on video, audio and text
         | data? I don't see why it would not learn the basic physical
         | structure of our world just like a language transformer learns
         | the grammatical and other structures of language. Then with
         | this prettaines transformer you can build an agent and use some
         | reinforcement learning too. I feel very confident that this
         | would mirror the level of intelligence of non-human animals
         | quite well.
        
           | nojvek wrote:
           | Animals are indeed smarter than AI for certain aspects of
           | intelligence (physical intelligence).
           | 
           | Evolution has optimized animal brain and bodies to survive
           | and take care of the next generation. They have a good grasp
           | of environment, where they are, where food is, where
           | predators are, basic communication if they live as a group.
           | Babies grow up and start learning.
           | 
           | Our current AI is extremely power hungry compared to a brain.
           | Cruise & Waymo put large power hungry supercomputers in cars.
           | The computing system costs 100k+. They still make silly
           | mistakes like crashing into fire trucks, blocking roads,
           | driving into wet cement etc.
           | 
           | ChatGPT and friends make silly mistakes for trivial math
           | problems that require a few hierarchical planning steps.
           | 
           | All in all, brains have some form of symbolic computation and
           | reasoning that we haven't been able to replicate with current
           | AI algorithms.
           | 
           | I'm not saying we'll never be able to but current AI is
           | really hyped. Kinda like crypto boom of 2019.
           | 
           | There are some really hard algorithmic problems to be solved.
           | 
           | Google, Microsoft, Meta could have 1000x more computing power
           | and data, however in the grand space or all algorithms there
           | exists a learning algorithm that is probably >10000X more
           | efficient at generalized modeling and reasoning than what we
           | have.
           | 
           | The proof that we (20W biological generally intelligent
           | computers) exist validates the hypothesis that there is a lot
           | of advancement we can still do at the algorithm part.
        
           | jerpint wrote:
           | There have been many attempts to do multi modal pre training,
           | the difficulty is finding the right combination of data for
           | it to be "useful" and "scalable". It's not trivial to just
           | train a transformer on video, text, audio, etc. mainly due to
           | O(N^2) on the token counts, time components with video, etc
        
           | quickthrower2 wrote:
           | I know intuition is often wrong, but to me a mix of dedicated
           | visual processing AI and language AI and sound AI all somehow
           | interacting with each other would be a good way to make an
           | "animal" like AI rather than throw loads of attention heads
           | at everything all at once.
        
         | version_five wrote:
         | The odds of the next material advance in ML/AI coming from one
         | of its pioneers are zero. Not to say we shouldn't listen to
         | what LeCun has to say (the other ones have basically lost it)
         | but focusing on him is a bad way of imagining the future.
        
       | vendiddy wrote:
       | AI layperson here. Is there something like an MRI scan for a
       | neural network?
       | 
       | Imagining I could take a foundation model, run it on my
       | specialized task, and measure which regions of the neural network
       | light up.
       | 
       | Then I could carve out unused regions of the network to create a
       | more lightweight model.
       | 
       | Or is this a silly idea?
        
       | RivieraKid wrote:
       | Possible future development I'm excited about is something like
       | GPT but for real world interaction - i.e. robots that take input
       | from sensors and are able to physically navigate and manipulate
       | the world.
       | 
       | Fine-tuning would then be used for specific environments (human
       | hair, an apartment) and robots (robotic barber, cleaning robot).
        
         | ultra_nick wrote:
         | Google put an LLM into a robot a few months ago. This is the
         | second paper I've seen on it.
         | 
         | https://arxiv.org/abs/2306.08647
        
       ___________________________________________________________________
       (page generated 2023-08-26 23:02 UTC)