[HN Gopher] CarperAI announces plans for the first open-source "...
___________________________________________________________________
CarperAI announces plans for the first open-source "instruction-
tuned" LM
Author : monkeydust
Score : 98 points
Date : 2022-10-20 07:19 UTC (15 hours ago)
(HTM) web link (carper.ai)
(TXT) w3m dump (carper.ai)
| bilsbie wrote:
| Can anyone explain what's new here? I couldn't quite follow the
| article.
| SpaceManNabs wrote:
| I wonder if this relates to DeepCTRL that came out recently.
| lee101 wrote:
| for now I'd checkout T0 or bloom, lots of models can be
| instructed already.
|
| I created https://text-generator.io and have a whole bunch of
| examples on there of instruction following style prompts
| espadrine wrote:
| How do you instruct BLOOM? Does it need fine-tuning?
| machiaweliczny wrote:
| Has anyone tested OTP 170B from Meta? Seems like it's public
| monkmartinez wrote:
| Just to get OTP up for inference would require a very large
| spend. To use GPT-NeoX (20B parameters) for inference
| requires 45GB of vRAM minimally. Its hard for me to imagine
| using a 170B model for fun somewhere unless one has a large
| GPU farm or lots of money.
| stellaathena wrote:
| GPT-NeoX-20B was specifically targeted to fit on A40s,
| A6000s, and a pair of 3090 Tis. Anything larger than that
| is going to be a real struggle for people who don't own
| computing clusters to use.
| mritchie712 wrote:
| have you considered SQL generation with two inputs:
|
| 1. data from the information_schema of the database
|
| 2. a natural language question
|
| I've played around with gpt3 but not providing a full schema
| for the database is a glaring issue with accuracy of the SQL
| generated.
| petersonh wrote:
| I've done something similar with gpt3 (codex) and had good
| results
| sva_ wrote:
| I wonder how the number of parameters of this will compare to
| GPT-3 - will there be different sizes? Their stated goal is to
| make this available to academics/independent
| researchers/startups, and I think with a 175B parameter model,
| not having the trained model itself isn't necessarily the biggest
| hurdle for most people (having hardware with x hundred GB memory
| is - well maybe not for startups.)
|
| Although there seems to be hope:
|
| _> In prior work, OpenAI found that the outputs from models
| trained with RLHF were preferred to those from 100x larger models
| trained without human feedback._
| machiaweliczny wrote:
| Yeah, they say it will be "chinchilla-optimal", which means
| that it probably will be < 70B, might be actually much less as
| I've seen some recent work that 20B models are able to compete
| with GPT task [0] so I guess it might be using it but probably
| isn't, so there's further room for improvement.
|
| [0]https://www.reddit.com/r/MachineLearning/comments/y4tp4b/r_u
| ...
| wyldfire wrote:
| What's "chinchilla"? is that a codename for a specific
| CPU/GPU?
|
| EDIT: no, it's a competitor with GPT-3 [1]
|
| [1] https://analyticsindiamag.com/deepmind-launches-
| gpt-3-rival-...
| gradys wrote:
| More relevantly, it means that the model parameter count,
| dataset size, and training procedure are balanced such that
| they can achieve equal or better performance as GPT-3 while
| being smaller.
| machiaweliczny wrote:
| Seems like that 20B model is available on HuggingFace [0]
|
| [0] https://huggingface.co/docs/transformers/model_doc/ul2
| jordn wrote:
| This is planned to be 70B but trained in the chinchilla-optimal
| way (more data + training). Scaling laws suggest this should
| outperform the base 175B GPT-3. Then release the base model as
| well as the RLHF-tuned models.
| [deleted]
| visarga wrote:
| I've been waiting for this ever since they trained BLOOM and even
| GPT Neo. Without instruction tuning they are almost worthless. If
| all you want is classification then it's better to use BERT, but
| if you need the generative part then you have to scale up to
| these big guys.
|
| What I need is the instruction following skills of GPT-3, running
| on a single (large) GPU, and not having to pay by token.
| monkmartinez wrote:
| I am very much wishing for the same. I have used T0_3B
| (https://github.com/bigscience-workshop/t-zero) on my M40 with
| 24GB of vRAM. It was fun to get it working, but its not
| remotely comparable to GPT-3 (instructGPT).
| schleck8 wrote:
| > CarperAI will release a chinchilla-optimal large language model
|
| Next-level terminology
| astrange wrote:
| https://www.deepmind.com/publications/an-empirical-analysis-...
___________________________________________________________________
(page generated 2022-10-20 23:02 UTC)