[HN Gopher] ChatGPT is not all you need. A SOTA Review of large ...
___________________________________________________________________
ChatGPT is not all you need. A SOTA Review of large Generative AI
models
Author : georgehill
Score : 85 points
Date : 2023-01-20 14:51 UTC (8 hours ago)
(HTM) web link (arxiv.org)
(TXT) w3m dump (arxiv.org)
| Xeoncross wrote:
| A correct anticipation of the future is a big part of all
| successful projects.
|
| However, that is mostly do the effort required to change course.
| If we reach a point where it's easy enough to regenerate
| everything from scratch, will it be so important to correctly
| plan ahead?
| matchagaucho wrote:
| Depends on the temperature of the prompt model. Getting random
| results with each generation is a feature of GPT.
|
| Determinism still requires planning.
|
| (this response generated by GPT ;-) )
| simonw wrote:
| The diagrams in this paper did not fill me with confidence: there
| doesn't appear to be any reason for the layout of the boxes and
| lines in them other than to fill some space.
| haldujai wrote:
| Is it just me or does anyone else cringe when they read "is/is
| not all you need" in the title of an AI-related paper?
|
| Also, what is "SOTA" for review? There isn't exactly a benchmark
| to compare...
|
| In terms of comprehensiveness they don't mention PaLM + variants,
| which probably should be mentioned as it is currently the largest
| LLM with SOTA on several benchmarks (e.g. MedQA-USMLE).
|
| In terms of correctness, I admittedly skipped to the sections I'm
| familiar with (LLMs) but I don't understand why they are
| distinguishing 'text-science' from 'text-text', they're both
| text-text and there is no reason why you can't, for example,
| adapt GPT3.5 to a scientific domain domain (some people even
| argue this is a better approach). A lot of powerful language
| models in the biomedical domain were initialized from general
| language models and use out-of-domain tokenizers/vocabularies
| (e.g. BioBERT).
|
| The authors also make this statement regarding Galactica:
|
| "The main advantage of [Galactica] is the ability to train on it
| for multiple epochs without overfitting"
|
| This is not a unique feature of Galactica and has been done
| before. You're allowed to train LLMs for more than 1 epoch and in
| fact it can be very beneficial (see BioBERT as an example of
| increasing training length).
|
| People GENERALLY don't do this because the corpus used during
| self-supervised training is filled with garbage/noise, so the
| model starts to fit to that instead of what you desire. There is
| nothing special about Galactica's architecture that specifically
| allows/encourages longer training cycles but rather they curated
| the dataset to minimize garbage. As another example, my research
| involves radiology NLP and when doing domain adaptive pretraining
| on a highly curated dataset we have been going up to 8 epochs
| without overfitting.
| sjsdaiuasgdia wrote:
| I like that you ask what SOTA means when the acronym is
| expanded in the title you see as soon as you click the link.
|
| "ChatGPT is not all you need. A State of the Art Review of
| large Generative AI models"
| [deleted]
| zone411 wrote:
| You missed the point. It's obvious what it means to anybody
| in ML. But that's not how the term is used.
| zone411 wrote:
| I agree. This paper also has issues in the areas I'm familiar
| with (e.g. Jukebox: being "limited to English songs" is the
| least of its issues). I don't know why it was randomly
| submitted to HN.
| azinman2 wrote:
| > Is it just me or does anyone else cringe when they read
| "is/is not all you need" in the title of an AI-related paper?
|
| I believe it's a reference to 'attention is all you need,'
| which is a very famous paper now. I'm guessing you probably
| already know that -- did you cringe with it? It was a landmark
| paper so at least it's worth hyperbole in the title. Maybe I'm
| missing that its usage started before that.
|
| Reminds me of my ML prof who would complain about people using
| the word 'optimum' or 'optimized' in titles of papers... one
| could almost always optimize more, and with respect to what is
| not specified by the title.
| nmca wrote:
| It has been a wildly surprising fact over the last few years
| that attenion basically is all you need, in a sense much
| stronger than the one intended by the authors.
| haldujai wrote:
| Yes, aware of the reference and hence the cringe. 'Attention
| is all you need' is a clever play on words for a seminal
| paper whose title directly related to its main scientific
| contribution.
|
| Not sure that every other AI-related paper needs to copy this
| format, this trend is the academic equivalent of click-
| baiting (seemingly trying to associate with the Vaswani et al
| paper) and in my anecdotal experience the usage of this play
| on words seems inversely correlated with the paper's quality.
| earleybird wrote:
| I believe it has an analogue with Heinlein's "there are two
| kinds of jokes; funny always and funny one time".
| Subsequent uses are a bit cringe in the same way that a
| subsequent telling of a funny-one-time joke is.
| PartiallyTyped wrote:
| > Maybe I'm missing that its usage started before that.
|
| Afaik it wasn't used prior to it. While attention (and
| residual connections) were used with recurrent / auto
| regressive models, the paper showed that an encoder-decoder
| (non auto regressive and auto-regressive) architecture with
| just attention (and residuals) is sufficient to achieve great
| results given that you provide positional embeddings.
|
| So prior to transformers it was RNN + Attention (+
| Residuals), but the paper claimed that Attention
| (+Residuals+Pos Embeddings) is all you need.
| la64710 wrote:
| I think the simple chat interface is why chatGPT became popular.
| If other people build relatively simple interfaces to leverage
| other models they might be successful as well.
| winddude wrote:
| I think it's the chat interface combined with memory. The
| second not many people had seen used in practise. The chat bot
| also generates much longer prompts, most other chat models seem
| to be trained for short prompts, and purely conversational.
| amelius wrote:
| > This work consists on an attempt to describe in a concise way
| the main models are sectors that are affected by generative AI
| and to provide a taxonomy of the main generative models published
| recently.
|
| I had trouble parsing this sentence.
|
| And why doesn't the abstract provide at least some basic
| explanation of the title?
| tehsauce wrote:
| I also didn't like the abstract. It feels more like what the
| introduction section of the paper should be. Instead here they
| should summarize what their key finds or results were.
| posharma wrote:
| May be because this paper was generated by ChatGPT :-)
| csande17 wrote:
| "[X] Is All You Need" is a popular trope in AI paper titles,
| sort of like "[X] Considered Harmful".
| basch wrote:
| How it _should_ read if it were concise and clear?
|
| >This work concisely describes the main sectors affected by
| generative AI, along with a taxonomy of recently published
| models.
| [deleted]
| coolspot wrote:
| No OpenScience's Bloom (opensource, 175B params), no Google's T5
| (opensource, 11B).
|
| The AI/ML space moves so fast that this review is already
| outdated.
| anticensor wrote:
| No Riffusion, no VALL-E, no Point-E either.
___________________________________________________________________
(page generated 2023-01-20 23:00 UTC)