[HN Gopher] Ask HN: Why is GPT4 better than the other major LLMs?
___________________________________________________________________
Ask HN: Why is GPT4 better than the other major LLMs?
Having used GPT4, PaLM and Claude, it is quite clear to me that
GPT4 is an order of magnitude better than these other LLMs. Google,
Anthropic and everyone else are investing oodles of resources and
the best talent to catch up so why are they (seemingly) not able
to? There is a general sentiment that LLMs have no lasting moat but
OpenAI seems to (for now) have one in terms of a better product.
The big question is why?
Author : takinola
Score : 7 points
Date : 2023-11-25 20:45 UTC (2 hours ago)
| ilaksh wrote:
| I don't think it's 10 times better. It is better. But I think it
| comes down to the size of the model and the training/training
| techniques. OpenAI seems to have invested a lot in human
| reinforcement feedback. Plus they have ways to do automated
| reinforcement I think. Also Google and Anthropic are basically
| deliberately holding back their strongest models because they are
| too expensive and/or they are worried about safety or something.
|
| And note that there are new versions of models from Anthropic
| that have just released or could release within a few months.
| takinola wrote:
| I have a test harness that runs all the LLMs side by side to
| compare the quality of output. I have tried Claude 2.1 (the new
| Anthropic model) and it is better than PaLM but not as good as
| GPT4.
|
| I am skeptical that Google is holding back a better model. It
| would go against every instinct that all the tech companies
| have, especially if you believe that AI is the future, why
| would you hamstring yourself by putting out an inferior product
| when you have a better one gathering dust in the back? Google
| has oodles of cash, so not sure why cost would be an issue. I
| don't get the safety argument so I can't speak to it but unless
| you have AGI it seems premature.
| gtirloni wrote:
| _> Also Google and Anthropic are basically deliberately holding
| back their strongest models because they are too expensive and
| /or they are worried about safety or something._
|
| Any place I can read about this?
| kwant_kiddo wrote:
| I think one clear difference is that they are just so focused on
| shipping compared to the others, and they gain many of the
| benefits that comes with that.
|
| I think it is reflected in both Greg and Sam that they really
| want to ship, and this have made a positive feedback loop into
| the team and what talent they have been able to acquire but also
| build up.
|
| Another point might be that AI chatbots are a first-movers
| market. Even if Grok turned out to be much better I would still
| miss some of the UI features that ChatGPT provide along with my
| chat-history.
|
| In regards to their fast shipping I think it is also reflected in
| their tech-stack. I suspect from reading their job posts (I might
| be very wrong here) that they started just coding everything in
| Python and the tooling/ecosystem that goes along like
| FastAPI/Django etc. maybe a bit C++/CUDA for the training. Then
| when they needed to scale they migrated from Python to Rust in
| the more critical areas of the codebase. They clearly also have a
| monorepo mentioned from [1].
|
| if you look through the their career-page the job description of
| a software engineer for developer-productivity [1] mentions "Our
| current environment relies heavily on Python, Rust, and C++" also
| "Are a proficient Python programmer, with experience using Rust
| in production" I found an earlier one where they mentioned that
| their backend was written in Python. "Proficiency with some
| backend language (we use Python)" [2]:
|
| 1:https://openai.com/careers/software-engineer-developer-produ...
|
| 2:https://openai.com/careers/software-engineer-leverage-engine...
| f0e4c2f7 wrote:
| I don't think this is the only factor but I suspect part of it is
| because GPT-4 had access to better datasets.
|
| After ChatGPT came out a lot of the places GPT was assumed to be
| trained on (reddit, twitter) started closing their APIs.
|
| This alone represents a pretty significant moat.
| seeknotfind wrote:
| Reddit was still pretty easy to get last summer. I saved a
| copy. Twitter has never really been easy. Some data sets are
| getting locked down, but you can still get them if you have
| enough motivation. I think the technical skill first and
| compute access are bigger hurdles.
| DantesKite wrote:
| If we knew, the other LLM's would be better.
|
| The correct answer is: Nobody outside of OpenAI technical staff
| currently knows.
| takinola wrote:
| I don't think this line of reasoning holds. We know why Google
| search is (was?) better. It just so happens there is a self-
| reinforcing feedback loop that is involved. We know why Apple
| is better at making phones. It doesn't follow that their
| success can be easily reproduced. It seems likely to be that we
| should know why OpenAI is better even if it is not an easily
| copied advantage
| makin wrote:
| There is a theory that GPT-4's secret sauce is a combination of
| commissioned high quality training data (which they don't really
| hide) and an unknown implementation of Mixture-of-Experts [1].
|
| No proof, of course, as the other comments have said, they won't
| share. The Open AI name is still ironic. I've also still not seen
| another successful attempt of MoE by any other company, which you
| would expect to if it was true.
|
| [1] https://hkaift.com/the-next-llms-development-mixture-of-
| expe...
| truetraveller wrote:
| I believe it's because it is heavily massaged. On two fronts:
|
| 1) By engineers, who employ tricks on the input, and perhaps the
| output. The input especially. So, when you type into ChatGPT,
| that input gets parsed using non-LLM techniques and/or
| heuristics, etc. This is primarily to get the semantics right.
| Non-LLM techniques can actually be pretty powerful, but the
| synergy of non-LLM and LLM is incredible.
|
| 2) By human farms, who essentially "upvote" and/or add
| corrections to ChatGPT results, and feed this back into the
| system. See OpenAI Kenyan workers (I believe the Kenyan workers,
| on paper, were for "moderation", but nothing stops them from also
| upvoting/correcting).
___________________________________________________________________
(page generated 2023-11-25 23:01 UTC)