[HN Gopher] Gpt2-Chatbot Removed from Lmsys
___________________________________________________________________
Gpt2-Chatbot Removed from Lmsys
Author : synthwave
Score : 29 points
Date : 2024-04-30 18:20 UTC (4 hours ago)
(HTM) web link (lmsys.org)
(TXT) w3m dump (lmsys.org)
| kevingadd wrote:
| I don't understand the connection between the submission title
| and the actual page it links to. Am I missing something obvious?
| Cheer2171 wrote:
| There was a big rumor that this Gpt2 chatbot was ChatGPT 4.5 or
| 5. That model is no longer on lmsys. See big discussion:
| https://news.ycombinator.com/item?id=40199715
| synthwave wrote:
| Sorry, should have clarified. The part of the site that states
| the model is currently unavailable links to their evaluation
| policies which is the link used for this submission :)
| jerbear4328 wrote:
| The news here is that gpt2-chatbot is confirmed to be an
| anonymized, unreleased model. See yesterday's discussion here:
| https://news.ycombinator.com/item?id=40199715 The
| https://chat.lmsys.org page now currently says this:
|
| > *gpt2-chatbot is currently unavailable.* See our model
| evaluation policy [here].
|
| That link takes you to https://lmsys.org/blog/2024-03-01-policy ,
| which is this post's URL. The page was edited yesterday (when
| gpt2-chatbot was added) to include this section:
|
| > Evaluating unreleased models: We collaborate with open-source
| and commercial model providers to bring their unreleased models
| to community for preview testing.
|
| > Model providers can test their unreleased models anonymously,
| meaning the models' names will be anonymized. A model is
| considered unreleased if its weights are neither open, nor
| available via a public API or service. Evaluating an unreleased
| model consists of the following steps:
|
| > 1. Add the model to Arena with an anonymous label. i.e., its
| identity will not be shown to users.
|
| > 2. Keep it until we accumulate enough votes for its rating to
| stabilize or until the model provider withdraws it.
|
| > 3. Once we accumulate enough votes, we will share the result
| privately with the model provider. These include the rating, as
| well as release samples of up to 20% of the votes. (See Sharing
| data with the model providers for further details).
|
| > 4. Remove the model from Arena.
| synthwave wrote:
| It's possible they temporarily removed it because it broke the
| whole anonymised part of that. It literally had information
| about itself in its system prompt and people could deduce it
| was running on an OAI API because of its API error response
| formatting. Either OAI collected all the data they need or
| it'll be back with some changes.
| novia wrote:
| They rate limited it at some point later in the day. The rate
| limiting applied to all the other OpenAI models also, but
| none of the non-OpenAI models. Just another data point of
| validation.
| synthwave wrote:
| Yeah it's also possible they temporarily took it down
| because of the flood of users that were slowing down the
| whole site for everyone. Potential for it to be back but
| the rate limits are really low, something like 2 queries
| per user per day
| Jackson__ wrote:
| > 2. Keep it until we accumulate enough votes for its rating to
| stabilize or until the model provider withdraws it.
|
| > 3. Once we accumulate enough votes, we will share the result
| privately with the model provider. These include the rating, as
| well as release samples of up to 20% of the votes. (See Sharing
| data with the model providers for further details).
|
| So... make the users do the work for free, but never publicly
| publish the data? That seems kinda scummy to me.
|
| I was really motivated with the whole public ranking thing, but
| knowing I'm just doing work for another company with the data
| never being published sucks the fun out of it.
| throwaway4aday wrote:
| You get to use unreleased models for free and compare them to
| others, you are gaining some benefit from this. Why be
| annoyed about it? They aren't making you do anything, you're
| choosing to play around with these models in a "gamified
| evaluation" for whatever reason. If you want to use any of
| the released models without participating in crowdsourced
| training then there are tons of options out there to run many
| of them locally or to pay for private hosting or pay for a
| chatbot as a service. I don't get what the problem is,
| they're footing the bill for you after all.
| Jackson__ wrote:
| What a single person gains from this is extremely little
| unless they keep track of every vote they do, and do >100
| of votes. The public ranking allows you to not have to do
| that.
|
| If there was an opt-out for private model voting, I would
| choose that simply because I don't care to give some
| corporation extremely useful data for so little pay.
___________________________________________________________________
(page generated 2024-04-30 23:02 UTC)