[HN Gopher] Open source implementation for LLaMA-based ChatGPT
___________________________________________________________________
Open source implementation for LLaMA-based ChatGPT
Author : georgehill
Score : 253 points
Date : 2023-02-27 14:30 UTC (8 hours ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| voytec wrote:
| Fake title riding on ChatGPT popularity. I think that it should
| be updated to something like:
|
| Open source implementation for LLaMA-based chat bot*
|
| Open source implementation for LLaMA-based ChatGPT alternative*
| agolio wrote:
| I think the 'alternative' is implied
| threevox wrote:
| Can someone leak the weights please
| bethecloud wrote:
| open-assistant from LAION is in the process of creating an OSS
| RLHF dataset for a personal assistant, may be useful for this
| project
| davidy123 wrote:
| I am very far from an expert on this, but I think domain specific
| conversational AI would be much more useful than these large
| models. It's fun to ask an AI to compose a fresh 600bpm hip hop
| song about the relationship between materials science and the
| breeding habits of mosquitoes, but an open-source medical AI,
| application support AI, or many other applications would be much
| more practical, if they could be accurate enough. And especially
| if they could run "standalone." They could also consult with each
| other, as a network of specialized AI. Is work inching closer to
| more specific, more accurate applications? Or is this just a big
| gimmick/distraction phase around a maybe not so great idea of AI?
| Taek wrote:
| These models are only capable of coherent conversation in the
| first place because they are so large. As soon as you shrink it
| down to be 'domain specific' its ability to form coherent
| sentences even within its own domain greatly reduces.
| pmarreck wrote:
| I think you would still need a large model to train on general-
| purpose knowledge and then train on domain specific things to
| get the specialized knowledge to be truly useful. For example,
| without a general-purpose model, if someone wanted domain-
| specific language "translated" for a neophyte, I doubt it would
| be able to without having also been trained on a general-
| purpose dataset
| nfgrep wrote:
| The way its typically done, AFAIK, is that you train these big
| models on a breadth of information, hoping that it picks up on
| the generalities of the information. In the case of LLMs,
| things like basic inference, for example. You then take these
| big, general models and "fine tune" them for specific
| applications, with specific bits of data. This way, you get
| things like basic inference, and logic, while still having
| something that can answer specific questions.
|
| OpenAI offers the ability to fine-tune some of their models:
| https://platform.openai.com/docs/guides/fine-tuning
|
| There are also other services that will fine-tune an LLM, for a
| specific domain, for you: https://activechat.ai/build-your-own-
| chatgpt/
| Thebroser wrote:
| There are definitely very solid attempts at least to make LLMs
| that encode biomedical knowledge such as BioGPT which is
| trained on Pubmed and other domain specific areas. Source:
| https://arxiv.org/abs/2210.10341
| PeterisP wrote:
| My opinion is entirely opposite - we need conversational AI for
| exploring the multitude of possibilities and identify what
| works, what doesn't work, what is/isn't useful given the
| obvious limitations on accuracy, truthfulness and
| interpretability; but once we can identify a specific narrow
| use case then we can fine-tune a ML system for that isn't
| conversational but is able to provide better results with
| whatever domain-specific structure is required (linking to
| specific sources, including external structured data, providing
| certainty metrics, filtering results according to domain-
| specific criteria instead of the conversational political
| correctness filters which fail some domains, treatment info
| which was correct but has become outdated, etc, etc) that can
| be done better in non-conversational systems.
| panarky wrote:
| The P in GPT is for "pretrained".
|
| The large pretrained model is a prereq for domain-specific
| models.
| PeterisP wrote:
| Not necessarily, you don't need to make a domain-specific
| model from a general model, you can definitely make a large
| pretrained domain-specific model from scratch by training
| it only on domain-specific data, which can result in a
| smaller and more efficient model.
|
| Furthermore, when making task-specific models, an 'encoder'
| architecture (similar to BERT) often works better than a
| 'decoder' architecture (similar to GPTx), so you might want
| to use a similar-but-different architecture than the
| general model intended to be conversational/generative.
| panarky wrote:
| If you want to build a domain-specific classifier that
| determines whether an image is a dog or a cat, and you
| have 50 labeled images of dogs and cats, it's much better
| to start with a large model pretrained on millions of
| images, and then specialize it by training on 50 images
| of dogs and cats.
|
| Try to start with a NN and 50 images of dogs and cats,
| and it won't work very well.
| PeterisP wrote:
| Sure, that's correct, but that's absolutely unrelated to
| what we were talking about; your example is about the
| general concept of transfer learning to task-specific
| annotated data, not about domain-specific pretrained
| models.
|
| For example, if you want a domain-specific model for the
| legal domain, then you can pre-train a large self-
| supervised model on every single legal-related document
| in the world you can get your hands on, instead of a
| general mix of news and fiction and blogs and everything
| else - and that might be a more efficient starting point
| for however many(few) annotated examples you have for
| your task-specific classifier than the general model.
| panarky wrote:
| Legal-related documents are a minuscule fraction of the
| corpus the large model is trained on. The resulting model
| won't have the conceptual fluency that the large model
| has. It's like training a human baby with legal briefs
| and expecting her to be a good lawyer.
| og_kalu wrote:
| Smaller models are typically dumber. Sure you could fine-tune a
| smaller model on say the medical domain and they might even
| perform better on some benchmarks but they won't reason or
| generalize as well. domain finetuned large models >>> domain
| finetuned small models. And because competence in one area
| bleeds over to other areas, you often need much less domain
| specific data to finetune on compared to the smaller models.
|
| You can see instances of this with Minerva, where the finetuned
| 540b version beats the finetuned 62b version despite being
| finetuned on only a quarter of the data the 62b version was
| finetuned on.
| muttled wrote:
| They're claiming the 13b model beats GPT-3 175b which is an
| extraordinary claim requiring extraordinary evidence. If
| that's true, though, it'd be interesting to see if that also
| applies to fine-tuning. Since the claim is predicated on the
| 13b model being better trained (amongst other things?), I
| wonder if limited fine-tuning data handicaps the 13b model
| even if the base model can outperform GPT-3 Davinci, given
| your point about large models handling fine-tuning better
| with limited data.
| og_kalu wrote:
| I mean the benchmarks are there. Can't exactly fake that.
| It should apply to fine-tuning. fine-tuning works off the
| back of the weights. That's why instruction-finetuned
| models even of small models like the T5 converge much
| faster on any additional fine-tuning or training than their
| non instruct counterparts as per the flan paper. Honestly,
| what i'm taking from this paper is that even chinchilla is
| undertrained. 13B was trained on 1T tokens.
| visarga wrote:
| You can already do that. Take your task first to GPT-3 and
| collect a bunch of outputs. Then fine-tune a small model on
| them. Works well, but you need to extend the dataset to cover
| all edge cases because the small model can't draw on the vast
| knowledge GPT-3 has.
| amrb wrote:
| I would care more about LLaMA architecture when I can get hands
| on, honestly this project is more interesting and lighting fast
| on even a 2060 laptop https://github.com/BlinkDL/RWKV-LM
| la64710 wrote:
| News flash : filling a google form to get access to the model
| based on Meta's discretion cannot be called open source .
| MH15 wrote:
| And yet they're trying real hard to.
| iForgotMyPW wrote:
| came here to say this
| nl wrote:
| > I would care more about LLaMA architecture when I can get
| hands on
|
| The architecture is available just fine, both in the paper[1]
| and in code:
| https://github.com/facebookresearch/llama/blob/main/llama/mo...
|
| It's the model weights that you can't get as easily.
|
| [1] https://research.facebook.com/publications/llama-open-and-
| ef...
| UncleOxidant wrote:
| The README there doesn't seem too coherent... how does one
| actually get that running? The Quick Start section isn't very
| helpful.
| lachlan_gray wrote:
| Has anyone tried this on an m1 machine?
| tmptmptmp1 wrote:
| Do you have access to the weights? If so you probably have
| better ML hardware. Wish this model was actually "open".
|
| The perf of a model of that size on the M1 will not be good.
| That is big enough it won't quite fit on a 3090 (24GB)
| without quantization.
| pumanoir wrote:
| I think is feasible. The description even says is designed to
| save on vram[1]. I don't get the other comments about needing
| more vram than a 3090.
|
| Also, Neuralmagic may run their sparsification on ARM cpu's
| in the future, so keep an eye.
|
| 1. _ChatRWKV v2: with "stream" and "split" strategies. 3G
| VRAM is enough to run RWKV 14B :)_
| fswd wrote:
| I've ran it on a AMD 3950 which I think is half the speed of
| a M1, and it's plenty fast. Note I am specifically talked
| about RWKV
| dwheeler wrote:
| The nebullvm projrct and specifically chatllama, by themselves,
| are clearly OSS. They're licensed under Apache-2.0:
| https://github.com/nebuly-ai/nebullvm/tree/main/apps/acceler...
|
| However, many models including LLaMA are _not_ OSS (they have
| terms such as "non-commercial use only"). And you need models.
|
| It can be confusing, it's important to try to make that clear.
| pmarreck wrote:
| very interesting. Do you know of anything that would take
| advantage of the 128 cores of my Ryzen Threadripper even though
| I have a 2080 and a 3080 as well? (Or all three... lol)
| vorticalbox wrote:
| GPU is massively faster than CPUs for this sort of work load.
|
| Even if you could use all 128 cores I would still get the
| 2080 would be faster
| zackmorris wrote:
| Keep in mind that many of the GPU cores go unused, since
| they are dedicated to geometry or ray tracing or whatnot:
|
| https://www.pcworld.com/article/402592/nvidia-turing-gpu-
| gef...
|
| IMHO these chips designed for specialized workloads are
| looking more and more ridiculous. I expect GPU
| manufacturers to keep dragging them out for another decade
| or so, as well as Apple as it explores offshoots of M1. It
| all makes me feel very tired (the ultimate code smell).
|
| A better design would be something like a 256+ core RISC-V
| with local memories in/by each core for data-locality and a
| content-addressable caching scheme for deduplication. Copy-
| on-write languages like Clojure and orchestrating processes
| under something like Docker would make it a breeze to
| program, although it would still support manually managed
| mutability like with Rust for innermost loops in games or
| whatever. It's fairly obvious how it would all work, but IP
| law and gatekeeping ensure that it will not happen anytime
| soon.
|
| Then stuff like CUDA would be just another framework run on
| a symmetric multiprocessor and we could get back to
| exploring alternatives like genetic algorithms like we did
| in the 1990s. Thankfully nobody cares what I think, heck
| even I'm sick of reading my own complaints, so it's easy
| enough to just unsee this.
| dragonwriter wrote:
| > Keep in mind that many of the GPU cores go unused,
| since they are dedicated to geometry or ray tracing or
| whatnot
|
| I mean, "many" is not usually the case; an RTX 3090 has
| 10,496 CUDA cores, 328 Tensor cores, 82 RT (raytracing)
| cores, and 96 render output pipelines. ML apps will uses
| the first and, depending on software, second set. The
| vast majority of the cores being CUDA cores is the norm.
| pmarreck wrote:
| ah ok, that's fair.
| zamalek wrote:
| This is not strictly true. GPUs fare better at these tasks
| for a few reasons:
|
| * The largest contributor is the sheer number of cores.
|
| * The latency between cores and memory.
|
| * FP16 instructions.
|
| 128 cores isn't an insignificant fraction of the number of
| cores on a 1050 (about 600), but CPU cores are individually
| more powerful. That advantage is potentially difficult to
| call. The top-end of Genoa has 96/192 cores, and you can
| slot many of them onto a single board.
|
| AMD is throwing more and more memory into the CPU cache.
| That's very different to a direct path to GBs of HBM, but
| at some point the difference in performance might not
| matter to a novice/dabbler.
|
| I would say give it a go, see what happens.
| amrb wrote:
| There is an example to use multi GPU in the link, Outside of
| this I have read about offloading to cpu/nvme in the case of
| 100GB+ models that don't fit in VRAM tho this will be at the
| expense of performance.
|
| https://pypi.org/project/rwkvstic/
| qwertox wrote:
| I'm using a Ryzen 9 5950X to run some tests with Whisper
| (ASR), and since I have no GPU with more than 4 GB VRAM, I'm
| running it on the CPU. It is slow. It takes between 20
| seconds and 2 minutes to transcribe 1 minute of audio, I'm
| using 8 cores. Adding more cores doesn't seem to improve the
| inferencing time.
|
| ML is really something which should be left to a GPU.
| pumanoir wrote:
| Look into sparsification of ML models:
| https://neuralmagic.com/ If your cpu has avx512 it can make
| it even faster.
| shmatt wrote:
| I don't really understand the benchmarking aspect researchers are
| touting. The public never cared about LLMs until they had a
| proper conversation with one. You can beat GPT3 at any benchmark
| you'd like, but if you can't get people that "feeling" when
| chatting with your model, is it worth anything?
|
| In the future there's going to have to be a way to benchmark the
| "human-ness" or "intrigue" or "feistiness" of a model to show us
| if its getting better at what we want
| antoineMoPa wrote:
| I guess one technique is to train a model on various language
| model outputs to classify these as good/bad/intriguing/robot-
| sounding/repetitive/etc. A human can tag the answers for the
| training dataset.
|
| Then, we can use this model to compare different LLMs and
| optimize new models - could be with genetic algorithms or just
| a human tweaking the model - so that the LLM maximizes whatever
| we want.
| visarga wrote:
| You just rediscovered RLHF - reinforcement learning from
| human preferences. That's the last stage of training for
| chatGPT, but uses RL instead of supervised learning.
| basch wrote:
| The question is, where does this "human-ness" lie? In the
| initial neural network, the training data, or the supervised
| reinforcement?
|
| In theory, a significantly smaller neural network that outputs
| at nearly as good a quality, should be able to chew through
| training data, and its re-enforcement process, much faster and
| cheaper, right? A more generalized, lower parameter model, is
| almost always preferred, as long as it works?
|
| If the human feeling is all boltonable to the neural net later,
| there is no reason to discount this component as lacking
| potential to exceed current models.
| visarga wrote:
| I think it's the feedback from RLHF that is mostly
| responsible, but it only works if the base model is large.
| Never seen a small model doing good conversation. They can do
| ok for classification and open book question answering, but
| generating long form coherent text is hard.
| skybrian wrote:
| Often articles show up on Hacker News that are meant for other
| researchers, not the general public. Not everything is a
| product.
|
| Benchmarks are used as a way to show that a particular machine
| learning technique does better at some task. It's a way for
| researchers to show they're making progress that will be
| legible to other researchers. You can't publish a paper saying
| "we tried it out and we think it's better."
| dragonwriter wrote:
| The techniques to go from a basic GPT-like model to a
| conversational agent are largely published and should be
| reproducible, open-source base models are starting points for
| that work that are unencumbered and available.
|
| This is important for researchers and implementers, not
| (immediately, ay least) end users.
| rnosov wrote:
| GPT-3 performed really well on synthetic benchmarks. It was
| later made palatable for general public consumption. You might
| say that a LLM needs to be good on synthetic benchmarks first
| before you can make public facing chatbots with it.
| visarga wrote:
| This whole debate - if a 13B model can really be as good as GPT3
| - would have been settled if we had a live demo. I am not sure
| their licence allows running public demos, even if you get the
| weights.
| rnosov wrote:
| Looks like they are making ChatGPT clone that would be possible
| to run a single GPU. HN dream come true!
| swyx wrote:
| lol good luck running a 13B model on a single GPU
| ddren wrote:
| Seeing the performance of implementations like FlexGen [1], I
| don't think it would be entirely unreasonable to run a 13B
| model on a single GPU for personal usage purposes. You are
| not going to a run a public service out of it, but it
| probably would be good enough to run your own ChatGPT or
| Copilot locally.
|
| [1]: https://github.com/FMInference/FlexGen
| visarga wrote:
| You need a RTX 3090 24gb
| wcarss wrote:
| ChatGPT's stable diffusion moment?
| Taek wrote:
| Can't have a stable diffusion moment if you refuse to release
| the weights to the general public. Stable diffusion only got
| to where it is because 10,000 people with otherwise zero
| reputation were able to play around with the code and models.
|
| LLaMA is still only available to the elite.
| Jack5500 wrote:
| This seems like a great first step to a truly open source LLM
| visarga wrote:
| If we can confirm the 13B model is as good as GPT3 then at
| least we have a target for reproduction without FB
| restrictions.
| lurquer wrote:
| Indeed. All the weights for all the models will be available
| one way or the other very soon.
|
| The proprietary nature of the weights is not going to be a
| bottleneck for more than a month, if I had to guess.
|
| The other bottle-neck to personal use -- the hardware required
| to run (not train from scratch) the thing - is going to be gone
| within the year I bet. I would assume some clever bloke is
| going to be able to prune the model or decrease the precision
| of the weights and discover you can get good-enough results
| with 1/10th of the memory.
|
| What happens then?
|
| Well, probably some Very Bad Things.
| agolio wrote:
| I'm as much a META hater as anyone - their policies have
| consistently disappointed me in almost every aspect of their
| business - but their stance on this LLaMA project I must say I am
| happy with and seems to mark a turn for the better.
|
| If they follow through on their promise of making the weights
| available and share source code that is a big step in the right
| direction for democratising this technology
| cuuupid wrote:
| The weights are non commercial and while their code is GPLv3
| they've only released inference and have removed anything that
| would give away the training methods :)
| alfalfasprout wrote:
| The paper highlights the training methods pretty clearly
| though including what tweaks they made.
| A4ET8a8uTh0 wrote:
| Yeah, but their history ( and accumulated goodwill ) are
| similar to Microsoft's. They may say it is 'open' or sprinkle
| appropriately sounding corporate speaking all over the press
| release, but the actions will, at best, temporarily prevent
| them from going 'full evil ahead'.
|
| And I like that announcement. I just don't think they will
| actually follow through on this.
| visarga wrote:
| Yann LeCun, who is director of AI research at FB is all over
| the ML Twitter talking about this model. I think they will
| actually release the weights.
| ilaksh wrote:
| When? Why didn't they release them already?
| georgehill wrote:
| For anyone wondering what LLaMA is, here are some useful links.
|
| https://ai.facebook.com/blog/large-language-model-llama-meta...
|
| https://news.ycombinator.com/item?id=34925944
| swyx wrote:
| those who know what it is are questioning why you are working
| on this when the base model itself is not released widely.
| whats the intention here?
| georgehill wrote:
| i am not the author of the repo, i came across this on
| github.
|
| https://twitter.com/nebuly_ai is the author of the repo.
| levesque wrote:
| In what way is this a ChatGPT implementation or equivalent? Seems
| like a chatbot based on a different backend, therefore it has
| absolutely zero link to ChatGPT.
| basch wrote:
| It is an analogy.
|
| ChatGPT:GPT3::ChatLLaMa:LLaMa
| levesque wrote:
| It uses a different engine, so this is as related to ChatGPT
| as a Toyota Corolla is related to a BMW car. This is an
| efficient and open-source chatbot, which is very good news,
| but the authors just wrote a clickbait title and they know
| it.
| f_devd wrote:
| > Toyota Corolla is related to a BMW car.
|
| The analogy is somewhat accurate, but also moot, since
| within the ML community "ChatGPT" can be used either as the
| product or the method (more specifically called RLHF)
| somewhat interchangeably. It's more like Google/Googling,
| where the largest/most popular provider becomes the defacto
| way to refer to a method. As someone who develops DL
| models, the title seems quite apt.
| basch wrote:
| In formal analogies, : is pronounced "is to" and :: is
| pronounced "as".
|
| The purpose here is to use the relationship from a known,
| to describe the relationship between a partial known.
|
| ChatGPT is to GPT3 as ChatLLaMa is to LLaMa. It uses the
| relationship between ChatGPT and GPT3 to extrapolate a
| relationship between an unknown and LLaMa.
|
| see Analogies.pdf https://resources.finalsite.net/images/v1
| 584287027/brockton/...
|
| Corolla:Toyota::3-Series:BMW. If you had heard of a
| Corolla, Toyota, and BMW, but not a 3-Series, you now
| roughly know that a 3-Series is BMWs equivalent to a
| Corolla.
| levesque wrote:
| I think I prefer the other commenter's point, referring
| to ChatGPT as a known learning paradigm for chatbots. But
| thanks for the little crash course on analogies ;)
| basch wrote:
| Isn't that consistent with, and the same as, my original
| comment?
|
| ChatGPT:GPT3::ChatLLaMa:LLaMa::Chatbot-through-RLHF:LLM
|
| aka ChatGPT is a known chatbot implementation, and GPT3
| and LLaMa are known LLMs.
| levesque wrote:
| Yes and no. You wouldn't say GPT to mean large language
| models or autoregressive language models. I would've
| thought the same to be true for ChatGPT instead of
| Chatbots with RL from human feedback (RLHF), perhaps the
| field is moving towards adopting ChatGPT as a paradigm
| name. Note that the title doesn't say a ChatGPT-like
| model based on LLaMa, it says outright opensource
| implementation of ChatGPT.
| basch wrote:
| > You wouldn't say GPT to mean large language models or
| autoregressive language models.
|
| In the analogy, that's exactly what you are saying.
| Identical to Toyota and BMW meaning "the make of the
| car."
|
| Maybe reimplementation is a more precise word, a black
| box re-engineering/cloning. In this case I inferred it by
| knowing it was a different LLM underneath, and that this
| group didn't have access to the chatgpt source code.
| rnosov wrote:
| It is a different backend but it supposedly should be roughly
| comparable to ChatGPT. Also, looks like it's both open source
| and requires a lot less hardware to run and train.
| Taek wrote:
| Its not open source until the weights are available. I have
| the hardware I need to run it but the required files are not
| available unless you receive special access.
|
| You can't use what has been released unless you want to spend
| $500,000 on training.
| pmarreck wrote:
| With only a modicum of trolling-level here, I wonder what
| percentage of that training expense was used to identify
| and avoid "true things that must be muted because they
| offend someone"
| karmasimida wrote:
| Is it of any good?
| rvz wrote:
| > LLaMA is creating a lot of excitement because it is smaller
| than GPT-3 but has better performance. For example, LLaMA's 13B
| architecture outperforms GPT-3 despite being 10 times smaller.
|
| Exactly. Best part is that it is open-source.
|
| That is worth getting excited about. Not a AI SaaS API owned by a
| so-called pseudo-non profit company which struggles on API uptime
| and availablity, just like GitHub.
|
| This is the 'revolution' you are looking for that changes
| everything. Not ChatGPT.
| Ozzie_osman wrote:
| Is it open source though? It looks like Meta was only releasing
| it to academics who apply.
| Taek wrote:
| its not open source, only approved researchers are allowed to
| access the weights. More ivory towering from the AI industry.
| mikie92 wrote:
| right, and I guess there'll be opensource alternatives of
| llama very soon
| eigenvalue wrote:
| It's for your own "safety"...
| holtkam2 wrote:
| I don't have a decent gpu at my disposal... has anyone tried to
| run LLaMA on an EC2 GPU instance? If so, which instance type? (I
| don't wanna overpay)
| alfalfasprout wrote:
| Per the paper the 13B variant runs on a single A100 GPU.
| jstsch wrote:
| Are LLaMA's weights generally available/floating around yet?
| gavi wrote:
| I think you need to apply for it @Meta
|
| https://docs.google.com/forms/d/e/1FAIpQLSfqNECQnMkycAp2jP4Z...
| amrb wrote:
| Nope they pulled a fast one and limited it to academy and
| government, just put the dam thing on hugging face like
| everyone else guys..
| swyx wrote:
| piratebay, but for model weights
| didntreadarticl wrote:
| Have we got any details on the benchmarks that show LLaMa's 13B
| architecture outperforming GPT-3? Because that seems kindsof
| fantastical. Is it just a product of a very specific benchmark or
| does it reflect real world performance?
| rafaelero wrote:
| The GPT-3 they are comparing to is the one that was released on
| 2020. Since then OpenAI made a lot of improvements and nowadays
| I believe GPT-3.5 is competitive to Palm (540b). Still, LLaMa
| is in the same tier, with much less parameters.
| og_kalu wrote:
| Just a heads up with my comparison. Under equivalent
| scenarios, the gap is smaller. davinci-003 gets about 10 more
| points using five shot (which is what the palm comp does)
| og_kalu wrote:
| GPT 3.5 is definitely not comparable to Palm. You can see
| some benchmarks here.
| https://crfm.stanford.edu/helm/latest/?group=core_scenarios
| For example, text-davinci-3 is 56.9 on MMLU, Flan-Palm is 75.
| https://arxiv.org/abs/2210.11416
| tikkun wrote:
| Flan-Palm is 75 with five-shot. text-davinci-003 is it 56.9
| with five-shot or zero-shot?
| og_kalu wrote:
| You're right. Equivalent scenarios, the gap is smaller -
| about a difference of 10. you can check the end of the
| flan paper for some equivalent comparisons.
| rnosov wrote:
| They list 7B, 13B, 33B, 65B architectures. Presumably, they
| compare 65b one to GPT-3 175B. Chinchilla model which is about
| 70B outperformed a much larger GPT-3 model. So not that
| fantastical.
|
| EDIT: I stand corrected. They do compare 13B model with a large
| GPT-3 model which is hard to believe without a bit more
| concrete evidence
| rileyphone wrote:
| You're incorrect.
|
| > For example, LLaMA's 13B architecture outperforms GPT-3
| despite being 10 times smaller.
|
| Of course this is all on benchmarks but it's a big
| improvement if true.
| gersh wrote:
| Is the trained model available to download anywhere?
| ddren wrote:
| Not really, the LLaMA model is only available on request and
| access is granted on a "case by case basis" [1], which for most
| of us is more or less as available as GPT-3 is.
|
| [1] https://ai.facebook.com/blog/large-language-model-llama-
| meta...
| swyx wrote:
| > more or less as available as GPT-3 is
|
| what? GPT3 is available by a public api that anyone can sign
| up and pay for and use for commercial use. how is it "as
| available"?
| ddren wrote:
| I was mostly talking about access to the trained model
| weights. The OpenAI API is certainly better than nothing,
| but it is very restrictive and cost prohibitive for many
| purposes. For instance, you have to adhere to the OpenAI
| usage policies, and while they offer fine-tuning services,
| it is not likely enough to implement techniques like RLHF,
| which is the basis for ChatGPT.
|
| That said, if LLaMa can achieve performance competitive
| with GPT-3 with just 13B parameters, I imagine that it is
| only a matter of time until open source pre-trained models
| based on this architecture become available, which would
| render GPT-3 obsolete.
| SakiWatanabe wrote:
| What is the purpose of this? The model from meta is not available
| to public. Neither this open source "LLaMA-based ChatGPT" nor the
| "open source" LLaMA can be downloaded or actually used by public
| because it would required the actual trained model.
| swyx wrote:
| also LLaMA was released like 2 days ago - how far in advance
| did these folks know about it?
| vdfs wrote:
| At least 2 days for sure
___________________________________________________________________
(page generated 2023-02-27 23:02 UTC)