[HN Gopher] CodeAlpaca - Instruction following code generation m...
___________________________________________________________________
CodeAlpaca - Instruction following code generation model
Author : sahil_chaudhary
Score : 127 points
Date : 2023-03-25 14:17 UTC (8 hours ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| sharemywin wrote:
| I wonder if this
|
| https://techcrunch.com/2022/12/20/petals-is-creating-a-free-...
|
| could be modified to create a system that's trained.
| bestcoder69 wrote:
| Anyone know of a 65B instruct model I can run on llama.cpp? Or if
| it's possible to train one on a 64gb Mac Studio? Haven't come
| across either yet.
| sahil_chaudhary wrote:
| I'm training a 65B model right now, also I believe you can use
| lora-alpaca to train on this data on a much smaller machine.
| cfn wrote:
| Llama has a 65B model and it runs (albeit slowly) on a 64gb Mac
| Studio with llama.cpp.
| pharmakom wrote:
| If someone were to add noise to the llama weights and then
| retrain a little, would anyone be able to tell? Could that org
| then pass it off as their own training, MIT licensed for the good
| of humanity?
| visarga wrote:
| Could be better to permute the neurons and weight matrices in
| such a way as to obtain the exact same network (an isomorphic
| graph).
| c1ccccc1 wrote:
| By looking at the weights? Yes, it would be very easy to tell.
| intalentive wrote:
| >The model is not finetuned to be safe and harmless, so be
| cautious.
|
| Ok I'll be careful, but if it says a bad word, I'm sending you my
| therapy bill.
| gigel82 wrote:
| The results are pretty good; I wish they'd just publish the
| models so we can run the inference locally (not too many people
| have access to 8xA100 to train themselves, though I appreciate
| including the training data and instructions too).
| doctoboggan wrote:
| If a couple of us get together and throw in some money we could
| train it on Lambda Labs hardware like the OP suggests. I would
| volunteer to do it myself but I don't know enough about
| training models to guarantee I am not wasting money with a
| stupid mistake.
| menzoic wrote:
| Anyone with a few hundred bucks to spare can do it by renting
| GPUs from a cloud provider. It only cost Stanford $600 to
| create Alpaca from LLAMA. $100 to generate instructions with
| GPT-3 and $500 to rent cloud GPUs. The license restriction is
| due to the use of GPT-3 output to train a model.
| MacsHeadroom wrote:
| More like $50 or even $5 or less for the cloud GPUs.
| Alpaca-7B's compute costs were close to $50 and that was
| before the 100x cost savings of using LoRA.
|
| A 4bit LoRA fine tune of this project would cost less than $5
| to train even up to 30B/33B.
| nickthegreek wrote:
| I'd love to see a crowdsourcing platform to donate to
| specific fine-tuning projects. I would gladly throw some
| money at someone to do the labor and release the models to
| the public.
| jerrygenser wrote:
| Llama may not be licensed for people to share since you need to
| apply to get one from Facebook for non commercial use. I think
| it's more of a license issue
|
| Hopefully similar work can be done with LoRA so the fine-tuning
| is not as expensive
| sahil_chaudhary wrote:
| Yeah the license issue is the only reason why I didn't
| release the weights yet.
| sahil_chaudhary wrote:
| Given how many people are asking for the weights, I'm working
| to make them available now. So you'll be able to run them
| locally as well.
| wsxiaoys wrote:
| https://github.com/wsxiaoys/awesome-ai-coding
|
| Great time to start an awesome AI coding list!
| nico wrote:
| > The code runs on a 8xA100 80GB, but can also run on 8xA10040GB
| or 4xA100 with lower batch size and gradient accumulation steps.
| To get the GPUs, I suggest using Lambda Labs, best pricing for
| the best hardware.
|
| I wonder how much it was total in $ for the fine-tuning.
|
| Also, does anyone have some sort of table/formula that relates
| MB/GB of training data to $ for fine-tuning?
| menzoic wrote:
| Stanford only spent $500 to fine-tune LLAMA for humam
| instruction with 52k instructions generated by GPT-3. This
| probably costs less. The use of GPT to generate the instruction
| data instead of humans is the massive cost reduction. The
| actual training for fine-tuning on GPUs is relatively cheap.
| IanCal wrote:
| Most of that was getting the data, the training would cost
| something like $100 if memory serves.
| NhanH wrote:
| Probably in the hundreds of dollar for 7B model, and may be a
| thousand or two for the 13B at worst
| MacsHeadroom wrote:
| Far far less. Alpaca-7B's compute cost was around $60-$70 for
| Stanford and around $0.60 (yes 60 cents) for equivalent fine
| tunes using the Parameter Efficient Fine Tuning (PEFT)
| strategy of Low Rank Adapters (LoRA).
|
| The repo above can be replicated for similar costs. Easily
| less than $10 for up to 30B using LoRA (which requires only
| 24GB of VRAM for 30B/33B and smaller).
| fareesh wrote:
| I asked in the issues, let's see
| [deleted]
| sahil_chaudhary wrote:
| All included it costs under 70$ for the 13B model. Training 65B
| now so will report what that will cost.
| sillysaurusx wrote:
| Please do! Also please include how you're calculating the
| costs.
| alex_sf wrote:
| For the 65B fine tune, did you add another A100 node? Or just
| drop batch size?
|
| Any chance you're up to sharing the training parameters?
| sahil_chaudhary wrote:
| Dropping the batch size
| tarruda wrote:
| Didn't explore much, but it seems alpaca-lora has better results
| for coding tasks. One example I've used was: "Implement quicksort
| in python.". This is the result with Code alpaca:
| def quicksort(arr): if len(arr) < 2:
| return arr else: pivot = arr[len(arr)
| / 2] left_index = 0 right_index =
| len(arr) - 1 while left_index <= right_index:
| while arr[left_index] < pivot: left_index
| += 1 while arr[right_index] > pivot:
| right_index -= 1 if left_index <=
| right_index: temp = arr[left_index]
| arr[left_index] = arr[right_index]
| arr[right_index] = temp left_index += 1
| right_index -= 1 return
| quicksort(arr[:left_index]) + [arr[left_index]] +
| quicksort(arr[right_index + 1:])
|
| Alpaca lora result: def quicksort(arr):
| if len(arr) <= 1: return arr pivot =
| arr[0] left = [arr[i] for i in range(1, len(arr)) if
| arr[i] < pivot] right = [arr[i] for i in range(1,
| len(arr)) if arr[i] > pivot] return quicksort(left) +
| [pivot] + quicksort(right)
|
| Shorter and much cleaner, not to mention it works (code alpaca
| version is broken). Also it matches what ChatGPT generates for
| me.
| grph123dot wrote:
| The Alpaca lora result is incorrect since, for example, if all
| elements in array are the same the result in just an array with
| one element.
| sahil_chaudhary wrote:
| Interesting, in my experiments alpaca-lora performed worse but
| definitely possible that isn't the case generally.
| underlines wrote:
| "Model weights aren't part of the release for now, to respect
| OpenAI TOS and LLaMA license."
|
| I feel like the whole Open Source ML scene is slowed down by a
| strong chilling effect. Everyone seems to be afraid to release
| models.
|
| Meanwhile, other models are freely available up to alpaca 30b:
|
| https://github.com/underlines/awesome-marketing-datascience/...
| Tepix wrote:
| Have you applied for the download? The first links were sent
| out pretty quickly.
| politician wrote:
| I don't understand why Facebook hasn't taken down the original
| magnet links in its own repository.
| whazor wrote:
| The people who are maintaining the repository likely have to
| organise meetings with legal and business teams. When no
| agreement can be reached, doing nothing is the easiest way
| forward. They have to do this next to their normal work.
|
| Sending out DCMAs is a different process done, likely done by
| the legal team.
| menzoic wrote:
| Why would they?
| detrites wrote:
| To avoid the embarrassment of having to serve themselves
| with a DMCA?
| dchest wrote:
| The PR is not in their repository, it's in the fork. Pull
| Request refers to "pull this from my repository, here's a
| link", GitHub just presents it in a convenient interface.
|
| (But you're right, those who send DMCAs are likely to
| just a send a link to the original repository :)
| elcomet wrote:
| They could close the PR in their own repo...
| saurik wrote:
| One can assuredly delete the pull request from your own
| list of open pull requests, right? It isn't that the
| information exists in theory on GitHub: it is that it is
| still listed right there when you go to check the health
| of the project's open issues/PRs.
| [deleted]
| sharemywin wrote:
| I'm curious during training if all the weights change compared
| LLaMA?
| sahil_chaudhary wrote:
| Yeah the weights did change.
___________________________________________________________________
(page generated 2023-03-25 23:01 UTC)