[HN Gopher] Goldman on Generative AI: doesn't justify costs or s...
___________________________________________________________________
Goldman on Generative AI: doesn't justify costs or solve complex
problems [pdf]
Author : edzitron
Score : 75 points
Date : 2024-07-05 20:17 UTC (2 hours ago)
(HTM) web link (web.archive.org)
(TXT) w3m dump (web.archive.org)
| nabla9 wrote:
| Great report.
| great_psy wrote:
| Is there more than the cover to this ? On mobile I only see one
| page of the PDF.
| dekhn wrote:
| Yes, here's the original URL which should download a full
| multi-page PDF:
| https://www.goldmansachs.com/intelligence/pages/gs-research/...
|
| It's mostly speculative narrative with a fair number of data-
| driven charts. I wouldn't spend much time on it unless you like
| financial analysis with hand-waving.
| geor9e wrote:
| The PDF has 31 pages.
| dekhn wrote:
| Except for a short window around the release of GPT-4 (especially
| the inflated claims around beating expert trained humans at legal
| and math tests, as well as "replacing google"), I think people
| have more or less right-sized their expectations for large
| language models and generative AI. Clearly it can do interesting
| and impressive things but it's not superintelligence, and the
| folks predicting we're just around the corner have been
| recognized once again as shysters, hucksters, and charlatans. It
| doesn't help that state of the art ML researches have gotten so
| good at over-hyping the actual abilities of their technology.
|
| However, I do think we'll continue to see impressive advances in
| the areas of media consumption and production, with complex
| reasoning on hard problems being a likely area of improvement in
| the near (1 decade) future. While I once never expected to see
| something like HAL in my lifetime, I feel that many aspects of
| HAL (voice recognition, ship automation, and chess-playing) have
| been achieved, if not fully integrated into a single agent. We
| can expect most applications to be banal- the giants who have the
| largest data piles will train models that continue to optimize
| the addictivity of social media, and click-thru rates of ads.
|
| I am also quite impressed at the recall of information by
| language models for highly factual and well-supported things
| (computer reviews in particular).
| karaterobot wrote:
| I agree with what you say above, but my perception is that most
| people still view the current crop of models as a step or two
| away from superintelligence. That superintelligence, or AGI, is
| a matter of continued improvement along the current lines,
| rather than along entirely different lines.
| torginus wrote:
| I like to think of the 'car factory' analogy - it's populated
| by robots that are in some respects far superior to humans,
| and are doing 90% of the labor. Some ancient futurist, not
| having seen one before, could correctly predict that 9 out of
| 10 jobs will be done by robots, and arrive at the incorrect
| conclusion that robots have rendered humans obsolete.
|
| In actuality, humans are still needed for the 10% the robots
| can't do well, or serve to enhance the productivity of
| humans.
|
| I predict AI is like this and going to be for a while - it
| can clearly do some stuff well and sometimes better than
| humans, but humans will have their niches for a while.
| dekhn wrote:
| I call this the "filter changing problem". No matter how
| complex you make the technology, somebody still has to
| change the oil filter (or do whatever other maintainence is
| required to keep the system running). Sort of like ML-SRE,
| for those who are familiar with the concept.
| dekhn wrote:
| I don't think we can really say what path would lead to
| superintelligence (for whichever definition you desire) in
| the near future. Perhaps it is technically possible to
| achieve merely by making an embodied agent with enough
| different tricks in a single model (which I see as a matter
| of continued improvement along current lines), or maybe it
| requires several new things we haven't conceived yet.
|
| Personally, my area of interest is scientific discovery.
| Could a model not dissimilar from what we have today, if
| asked a cogent question, not answer it with an experiment
| that could be carried out? For example, one of the most
| important experiments, Avery-MacCleod, which proved (to the
| extent that you can prove anything in biology) that DNA, not
| protein, was the primary element of heredity, is not all that
| complicated, and the mechanical details seem nearly in reach
| of modern ML techniques. Similarly, could the ML model
| provide a significant advance in the area of understanding
| the molecular function in intimate detail of proteins as
| determined by their structure (which AlphaFold does not do,
| yet), complete with experimental instructions on how to
| verify these hypotheses? As of this time, my review of modern
| ML methods for science suggest we have made some advances,
| but still have not passed the "phase transition"
| demonstrating superscientist-level understanding of any
| field. But perhaps it will just fall out naturally from
| improved methods for media generation/parsing and ad
| targeting.
|
| I continue to remain hopeful that within my remaining 20-40
| or so years (I'm a typical american male, age 51, with a
| genome that contains no known risk factors) I will see
| something like what Vinge describes in
| https://edoras.sdsu.edu/~vinge/misc/singularity.html in a way
| that is demonstrable _and_ safe, but honestly, I think it
| could go in any number of directions from "grim meat-hook
| future" to "unexpected asteroid takes out human life on the
| planet, leaving tardigrades to inherit the earth" to
| "kardyshev-scale civilization".
| aurareturn wrote:
| I don't think anyone thought it was super intelligence.
|
| I think it's impressive that we went from LLMs not being useful
| at all to GPT3.5 shocking the world to GPT4 becoming super
| useful for many things in around 7 months time.
|
| LLM progress have slowed down a bit. But I think we're just
| getting started. It's still really early. It's only been 1 year
| since GPT4 came out. Even at the level of GPT4, scaling it
| would have immense benefits. But my sense is that we'll have a
| few more levels of great leaps in LLM capabilities that will
| shock people in the next 3-4 years.
| dekhn wrote:
| My text written above makes it quite clear that I don't think
| most people were saying GPT-4 was superintelligence, but the
| implication, especially from the charlatans, was there.
|
| I'm referring to the OpenAI white paper on GPT4 that shows
| exam results. https://cdn.openai.com/papers/gpt-4.pdf figure
| 4, and surrounding text.
|
| Clearly not superintelligence, as I would define it (see my
| other comment about scientific discovery, which I consider
| the best test), but these are tests actual humans take, and
| we know how most humans would score on these tests (thru some
| amount of memorization, along with parsing word problems and
| doing some amount of calculation). But many people who looked
| at the testing results concluded that GPT-4 was actually a
| reasoning agent, or that reasoning agents were just around
| the corner.
|
| The press picked up on that, and my LinkedIn stream was
| absolutely filled with second-class influencers who thought
| that superhuman capabilities were not far away. For a while
| there, looking at some of the test results specifically on
| moderately challenging math problems, I suspected that LLMs
| had some sort of reasoning ability.
| aurareturn wrote:
| No one said it was super intelligence. I've never seen
| anyone say that about GPT4 in the media or on Hacker
| News/Reddit/X.
|
| Yes, I'm sure if you google "GPT4 super intelligence",
| you'll find a stupid source that says it is. But I've never
| seen anyone reputable say it is.
| Voloskaya wrote:
| > and the folks predicting we're just around the corner have
| been recognized once again as shysters, hucksters, and
| charlatans
|
| Why? Can you see the future? No one (serious) was claiming that
| GPT-4 is superintelligence, it's about the rate of improvement.
|
| There has only been 6 years between GPT-1 and GPT-4, and each
| iteration brought more and more crazy emergent behaviour. We
| still don't see any sign of the scaling laws slowing down.
|
| I work in ML research, and personally don't believe ASI is just
| around, but I talk everyday to researcher that believe so, they
| don't say that to swindle anyone's money (they have extremely
| well paid 9 to 5 jobs at Goog/MS/OAI, they aren't trying to
| raise money from VCs), they only believe so due to the rate of
| improvement.
|
| Claiming, barely 18 months after GPT-4, when we haven't yet
| seen any result from the next jump in scale, that it's all
| baloney is a bit premature.
|
| Btw in research time, 10 years from now is << around the corner
| >>.
|
| Now for the VC-chasing folks, their motivation is an entirely
| different story.
| dekhn wrote:
| I'm pretty good at estimating the future; I started working
| in ML around 1993 and my last work in ML was on TPU hardware
| at Google (helping researchers solve deep problems when
| hardware goes wonky), and a number of my ideas (like
| AlphaFold's capabilities) were predicted by me at CASP in
| ~2003.
|
| I just continue to think that Vinge was a bit optimistic both
| on the timeline and acceleration rate. Everybody who cares
| about this should read
| https://edoras.sdsu.edu/~vinge/misc/singularity.html and
| consider whether we will reach the point where ML is actively
| improving its own hardware (after all, we do use ML to design
| next gen ML hardware, but with humans in the loop).
| Voloskaya wrote:
| Sure, you can think a lot of people are too optimistic (and
| as stated in my previous post I agree with you), but
| calling them shysters, hucksters, and charlatans implies a
| hidden motive to lie for personal gains, which isn't there
| (again, in the ML research side). No one working on GPT-2
| thought it would be such a leap on GPT-1, no one working on
| GPT-3 knew that that was the scale at which 0 shot would
| start emerging, no one working on GPT-4 thought it was
| going to be so good, so let's just not pretend we now what
| GPT-5 or 6 scale model will and won't do. We just don't
| know, we all have our guesses but that's just what it is, a
| guess. People making a different guess might be wrong
| ultimately, that doesn't make them charlatans.
| johnthewise wrote:
| >and the folks predicting we're just around the corner have
| been recognized once again as shysters, hucksters, and
| charlatans.
|
| Do you think Sutskever,Hinton or Sutter are charlatans?
| Bjorkbat wrote:
| I wouldn't take it that far myself, but I'm kind of perplexed
| by how the smartest people in the room at any given time have
| very optimistic timelines regarding when we'll have AGI. This
| isn't a modern phenomenon either. Both AI winters were
| preceded by experts more-or-less claiming that AGI was right
| around the corner.
|
| It's as if the easiest people to fool are the researchers
| themselves
| dekhn wrote:
| I think Sutskever is a charlatan outside of his area of
| expertise, Hinton (with whom I worked loosely at Google) is a
| bit of a charlatan (again, outside his area of expertise;
| clearly he and LeCun both did absolutely phenomenal work) and
| I don't know who Sutter is.
|
| If I wanted to predict the next ten years, I'd bring in Demis
| Hassabis, Noam Shazeer, and Vincent Vanhoucke, from what I've
| read of Demis's work, and my interactions with the latter,
| they seem to have very realistic understanding and are not
| prone to hype (Demis being the most ambitious of the three,
| Vincent being the one who actually cracked voice recognition,
| and Noam because ... his brain is unmatched).
| CamperBob2 wrote:
| _I think Sutskever is a charlatan outside of his area of
| expertise, Hinton (with whom I worked loosely at Google) is
| a bit of a charlatan (again, outside his area of expertise;
| clearly he and LeCun both did absolutely phenomenal work)
| and I don 't know who Sutter is._
|
| What do you think of Vizzini?
| exsomet wrote:
| I've been looking at it in the same sense as something like
| Docker. When containers first became a big hype, everyone
| everywhere was using them for everything, including things like
| trying to containerize full desktop environments, which outside
| of a couple niche businesses makes almost no sense.
|
| Similar to containers, my feeling is that the truth is LLMs are
| wildly overkill for almost everything going on today. You don't
| need an LLM to sort some already structured data when a basic
| python library or something will work equally fast,
| predictably, and with less black box magic. There's probably a
| small number of use cases that it makes sense for but for
| everyone else it's just silly to try and force the technology.
| It doesn't help that the people who are selling the shovels in
| this gold rush are extremely good at extending the hype train
| every few months, but eventually when these models stop being
| sold at a loss and businesses have to start facing down with
| the bill to run them and/or make these API calls, it will
| correct itself real fast.
| xmichael909 wrote:
| This interview by Adam Conover, really is a wonderful discussion
| on the topic https://www.youtube.com/watch?v=T8ByoAt5gCA I was
| pretty amazed with GPT when it came up, but increasingly find it
| makes to many mistakes. I full use it as a tool in writing code,
| but it needs to be treated as Intellisense plus, or something to
| that affect, not something that will handle complex tasks. GPT
| and Claude make many mistakes and unless they can solve it from
| completely making up stuff (which I don't think they can) will
| not advance much more beyond waht they currently are.
| pixl97 wrote:
| I take this view, a correct one as "Thank goodness". I don't
| think humanity is ready for a 'correct intelligence' yet,
| especially one that if existed at the level of human
| intelligence would likely rapidly go into the realm of
| superintelligence. Even if it didn't get out of human control,
| the humans that controlled said AI would gain an immense amount
| of power which would present a great destabilization risk.
| zzzbra wrote:
| Heartbreaking: The Worst People You Know Just Made A Great Point
| echelon wrote:
| The music, film, and game industries are about to be completely
| disrupted.
|
| LLMs and AGI might be hogwash, but processing multimedia is
| where Gen AI and especially diffusion models shine.
|
| Furthermore text-to-{whatever} models might produce slop, but
| Gen AI "exoskeletons" (spatial domain, temporal domain editors)
| are Photoshop and Blender from next century. These turbocharge
| creatives.
|
| Hearing and vision are simple operations relative to reasoning.
| They're naturally occurring physical signals that the animal
| kingdom has evolved, on several different occasions, to
| process. This is likely why they're such a low hanging fruit to
| replicate with Gen AI.
| beachandbytes wrote:
| Id agree with you on the creative industry, but disagree that
| that the generative AI's aren't going to do the same to just
| about every other industry. We were at a point where we had
| extremely specialized models that were useful, and now we
| have general models that are EXTREMELY useful in almost all
| contexts. Text, Audio, Video, Data Processing, etc. At least
| in my eyes we are at the same point with LLMs as we were with
| computing when you had a large part of the population that
| was just "not into them". As if it was like choosing any
| other hobby. I'm sure tons of people aren't getting much
| utility out of the space now, but it's not because the
| utility isn't there.
| weweweoo wrote:
| Generative AI appears fantastic aid for many smaller tasks where
| there's enough training data, and correctness of the answer is
| subjective (like art), or easily verifiable by a human in the
| loop (small snippets of code, checking that summary of an article
| matches the contents of the original). Generally it helps with
| the tedious parts, but not with the hard parts of my job.
|
| I don't have much belief in fully autonomous generative AI agents
| performing more complex tasks any time soon. It's a significant
| productivity boost for some jobs, but not a total replacement for
| humans who do more than read from a script, or write clickbait
| articles for media.
| harrisoned wrote:
| I agree with that. At work, we are about to implement a decent
| LLM and ditch Dialogflow for our chatbot. But not to talk
| directly to the client (it's asking for a disaster), just to
| recognize intentions, pretty much like Dialogflow but better.
|
| Right now there are many small but decent models available for
| free, and cheap to use. If it wasn't for the hype, it would
| never have reached that level of optimization. Now we can make
| decent home assistants, text parsers and a bunch of other stuff
| you already mentioned.
|
| But someone paid for that. The companies who believed this
| would be revolutionary will eventually have a really hard
| reality check. Not that they won't try and use it for critical
| stuff, but once they do and it fails spectacularly they will
| realize a lot of money went down the drain.
| elforce002 wrote:
| And we'll thank them for their service.
| ChrisArchitect wrote:
| [dupe]
|
| Please don't post wayback links unnecessarily. Content still
| fresh and available.
|
| Discussion here: https://news.ycombinator.com/item?id=40856329
| anu7df wrote:
| The only question I have is whether Goldman is shorting NVIDIA..
| random3 wrote:
| Ha! +1
|
| Although, I'd be shorting 80% of everyone else _spending_ money
| with NVidia without a clear path to recover. However, given
| that most are (likely?) not listed, there isn 't that much to
| short?
| russellbeattie wrote:
| Pretty sure I read that Goldman itself is currently creating its
| own internal models using its proprietary data to help its
| analysts, IT and investors.
| cpursley wrote:
| Ironically, AI still sucks at accurately parsing PDFs.
| bluelightning2k wrote:
| There is a paradox. To build the future requires irrational
| belief. And to sell that vision.
|
| Perhaps the difference between insanity and visionary, "scam" and
| genius is simply the outcome.
|
| When someone like Sam Altman declares optimistically that we will
| get AGI and talks about what kind of society we will need to
| build... It's kind of hard to tell what mix of those 4 is at
| work. But certainly it will be perceived differently based upon
| the outcome not the sincerity of the effort.
___________________________________________________________________
(page generated 2024-07-05 23:02 UTC)