[HN Gopher] BERTs Are Generative In-Context Learners
___________________________________________________________________
BERTs Are Generative In-Context Learners
Author : fzliu
Score : 97 points
Date : 2024-11-14 08:18 UTC (14 hours ago)
(HTM) web link (arxiv.org)
(TXT) w3m dump (arxiv.org)
| srameshc wrote:
| As someone who has very limited understanding but tried to use
| BERT for classification, is BERT still relavant when compared to
| LLMs ? Asking because I hardly see any mention of BERTs anymore.
| galeos wrote:
| My understanding is that BERT can still outperform LLMs for
| sentiment classification?
| jsemrau wrote:
| To my understanding yes. But I never found a good use-case
| for sentiment classification.
| ta8645 wrote:
| It seems to be used by Youtube for comment censoring /
| shadow-banning.
| jsemrau wrote:
| That might make sense.
| antononcube wrote:
| I used sentiment analysis a few times in recommender
| systems (for digital media consumption.)
|
| Also for analyzing Trump's tweets (from 2016): https://math
| ematicaforprediction.wordpress.com/2016/11/21/te...
| mynegation wrote:
| For many specialized tasks you can run BERTs (and simpler
| models in general) at scale, with lower latency, at lesser
| cost, with similar or even better results.
| beoberha wrote:
| Depends what you're trying to do. I'm writing a personal
| assistant app (speech to text) and want to classify the user
| input according to the current actions I support (or don't).
| The flagship LLMs are pretty great at it if you include the
| classes in the prompt and they will spit out structured output
| every time. But, man, they are expensive and there's the
| privacy aspect I'd prefer to adhere to. I've only got 24 GB of
| RAM, so I can't run too many fancy local models and things like
| llama3.1:8b don't classify very well.
|
| So I'm trying BERT models out :)
| Tostino wrote:
| Try some of the Quen models. They have some that are slightly
| larger than 8b that will fit on your 24gb quite nicely. They
| have been amazing so far.
| osanseviero wrote:
| Yes, they are still used
|
| - Encoder based models have much faster inference (are auto-
| regressive) and are smaller. They are great for applications
| where speed and efficiency are key. - Most embedding models are
| BERT-based (see MTEB leaderboard). So widely used for
| retrieval. - They are also used to filter data for pre-training
| decoder models. The Llama 3 authors used a quality classifier
| (DistilRoberta) to generate quality scores for documents.
| Something similar is done for FineWeb Edu
| itchyjunk wrote:
| Wait, I thought GPT's were autoregressive and encoder only
| like BERT used masked tokens? You're saying BERT is auto-
| regressive or am I misunderstanding?
| woadwarrior01 wrote:
| You're right. Encoder only models like BERT aren't auto-
| regressive and are trained with the MLM objective. Decoder
| only (GPT) and encoder-decoder (T5) models are auto-
| regressive and are trained with the CLM and sometimes the
| PrefixLM objectives.
| ipsum2 wrote:
| You can mask out the tokens at the end, so its technically
| autoregressive.
| spmurrayzzz wrote:
| They're still very useful on their own. But even more broadly,
| you can often use them in tandem with LLMs. A good example
| could be a classifier that's used as a "router" of sorts; could
| be for selecting a prompt template, directing to a specific
| model, or loading a LoRA or soft prompt vector to be used at
| inference-time.
| deepsquirrelnet wrote:
| They've drowned in the LLM noise, but they're definitely still
| relevant.
|
| - Generative model outputs are not always desirable, and often
| even undesirable
|
| - BERT models are smaller and can run with lower latency and
| serve larger batches with lower vram requirements
|
| - BERT models have bidirectional attention, which can improve
| performance in many applications
|
| LLMs are "cheap" in the sense that they work well generically,
| without requiring fine tuning. Where they overlap with BERT
| models is mostly that they may work better in low training data
| environments due to better generalization capabilities.
|
| But mostly companies like them because they don't "require" ML
| engineers or data scientists on staff. For the lack of care
| given to evaluation that I see around LLM apps, I suspect
| that's going to prove to be a faulty premise.
| antononcube wrote:
| > - BERT models are smaller and can run with lower latency
| and serve larger batches with lower vram requirements
|
| The most recent version of Wolfram Language (aka Mathematica)
| uses by default BERT models for embedding.
|
| (Say, for this function: https://reference.wolfram.com/langua
| ge/ref/CreateSemanticSea... .)
| patelajay285 wrote:
| We found the same result a few years ago in our ICLR paper:
| https://arxiv.org/pdf/2209.14500
|
| We found Google's T5 models which were released in 2019, pre-
| GPT-3, were "secretly" capable of in-context learning with a
| simple inference technique.
|
| Given they use a bidirectional MLM (Masked Language Modeling)
| objective, it wasn't obvious how to do it, but MLM objectives are
| known to produce better language representations than causal
| (next token prediction) objectives. We were able to outperform
| much larger sized GPT-3 models or get very close to their
| performance with far smaller T5 models.
| toxik wrote:
| From that paper it seems the sampling method (SAP) is also
| slower, so that it beats larger models seems expected.
| patelajay285 wrote:
| It's not at all expected. T5 models are not generative models
| by default and they were not thought to be able to perform
| generation, let alone in-context learning. Remember these
| models were released _before_ any of the existing LLMs and
| in-context learning /prompting as a technique became
| popularized with GPT-3.
|
| While the technique requires multiple samples to coax
| generations from this particular model, other LLM training
| schemes have incorporated both unidirectional and
| bidirectional objectives in their training now. However, this
| exploration hasn't been fully resolved as most models are
| still trained only on the causal objective by standard
| practice. There's still a a lot of exploration that can be
| done on pre-training objectives.
| cscurmudgeon wrote:
| Are there any intrinsic dis/advantages of bidirectional models
| over causal models for in-context learning? It seems that
| unidirectional model just have been explored and worked on
| more.
| patelajay285 wrote:
| When you train bidirectionally only, you don't get a
| generative model, that would be the downside. However, you
| can train on a mixture of causal and bidirectional objectives
| as some LLM pre-training has done. As far as I am aware,
| there are no downsides of that, but it is not more common
| simply because the standard practice has been to train causal
| only and there just isn't enough funding/attention to go into
| experimenting on every axis of pre-training (which can be
| very expensive).
| jsenn wrote:
| The "embarrassingly simple inference technique" is to put a bunch
| of [MASK] tokens at the end of the prompt.
|
| I'm having trouble understanding whether this paper is saying
| anything new. The original BERT paper already compared it
| favourably to causal models including GPT. Was there any doubt
| that BERT-style models could be in-context learners?
|
| From what I gather as a non-expert, the problem with BERT is
| scaling/training efficiency: GPT gets C-1 training examples out
| of a training input of length C, but BERT only gets 0.15*C
| examples. Indeed, the author points out that DeBERTa required 3x
| more compute than GPT-3 to achieve the level of performance
| reported, which makes sense.
| Ldorigo wrote:
| Do we a know whether the current SOTA foundation models (Gemini,
| gpt4o, Claude, etc) are actually all GPT-based (as in, causal
| models)?
| dartos wrote:
| GPT-based isn't really a thing outside of openai (it's just the
| commercial name for their models)
|
| But I believe we're confident that all major models are causal
| transformer models right now.
|
| No reason to believe otherwise. If one of them was doing
| something different, they'd let us know in order to stand out.
| Tostino wrote:
| No, they didn't get to co-opt that word.
___________________________________________________________________
(page generated 2024-11-14 23:01 UTC)