[HN Gopher] Show HN: Llama-dl - high-speed download of LLaMA, Fa...
___________________________________________________________________
Show HN: Llama-dl - high-speed download of LLaMA, Facebook's 65B
GPT model
Author : sillysaurusx
Score : 252 points
Date : 2023-03-05 04:28 UTC (18 hours ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| v64 wrote:
| If anyone is interested in running this at home, please follow
| the llama-int8 project [1]. LLM.int8() is a recent development
| allowing LLMs to run in half the memory without loss of
| performance [2]. Note that at the end of [2]'s abstract, the
| authors state "This result makes such models much more
| accessible, for example making it possible to use OPT-175B/BLOOM
| on a single server with consumer GPUs. We open-source our
| software." I'm very thankful we have researchers like this
| further democratizing access to this data and prying it out of
| the hands of the gatekeepers who wish to monetize it.
|
| [1] https://github.com/tloen/llama-int8
|
| [2] https://arxiv.org/abs/2208.07339
| downvotetruth wrote:
| Eagerly awaiting the int8 vs 4 benchmarks. Also, it can run on
| CPU https://github.com/markasoftware/llama-cpu So, an int8
| patch could allow the 65B to run on a standard 128 GB setup
| assuming the 65B model's cache bursts fit, which if I were to
| speculate is why the released models stop @ 65B & meta likely
| already has larger unreleased internal ones.
| v64 wrote:
| early int4 experiments seem to indicate it's possible but you
| do lose performance, see this thread https://www.reddit.com/r
| /MachineLearning/comments/11i4olx/d_...
|
| edit: to clarify, it may be possible to get this loss back
| and there is reason to be optimistic
| CuriouslyC wrote:
| Probably the best method is to just train it on int4 in the
| first place. Fine tuning after quantization would
| definitely help though.
| sp332 wrote:
| Isn't that backwards? You need fairly good resolution
| during training or your gradients will be pointing all
| over the place. Once you've found a good minimum point,
| moving a little away from it with reduced precision is
| probably OK.
| rfoo wrote:
| GP could be mentioning quantization aware training,
| during which the weight and gradient are still computed
| in fp16/fp32.
| brookst wrote:
| I have no idea what the right answer is, but I think the
| argument for int4 training is that the loss measurements
| would take the lower resolution of the model as a whole
| into account.
|
| Is it better to have billions of high resolution
| parameters and quantize them at the end, or to train low
| resolution parameters where the training algorithms see
| the lower resolution? It's beyond me, but I'd love to
| know.
| Scene_Cast2 wrote:
| But by default, training algos don't see the lower
| resolution, your gradient just doesn't work as well.
| There is a body of research on how to make training aware
| of / adapt to the lower precision.
| nl wrote:
| > Probably the best method is to just train it on int4 in
| the first place
|
| Unclear why you think that since experiments show the
| opposite.
|
| In general the gradient seems to get too "bumpy" to do
| good gradient decent at lower levels of precision.
|
| There are some papers showing that making the training
| loop _aware_ of quantitization can help ultimate
| quantizied performance but I 'm not aware of this being
| implemented at large scale.
| causality0 wrote:
| I feel like we're less than a decade away from being able to
| hook LLMs into gaming. How incredible would it be to have NPCs
| driven by LLM?
| pixl97 wrote:
| Honestly I don't think it would be completely impossible now
| in a limited fashion.
|
| Imagine playing a level and doing some particular feats in
| it. They get presented to GPT with a prompt and the story
| gets send to a AI voice model in game where the NPC
| asks/tells the player character about it.
| visarga wrote:
| We'll soon have LLMs in operating systems, LLMs in browsers
| and you are right, probably also in games. LLMs will be the
| platform on which we build almost everything.
| jesusofnazarath wrote:
| [dead]
| bloaf wrote:
| I'd be satisfied plugging a game log/history into a system
| that generates the epic tale of your victory/defeat.
| SloopJon wrote:
| There was an Ask HN post about that idea a couple of months
| ago:
|
| https://news.ycombinator.com/item?id=34478503
|
| I have long wished for less linear stories in video games,
| where branching narrative (a la Choose Your Own Adventure) is
| one possible way to give the player agency. The problem is,
| true branches are expensive, because you end up writing a
| bunch of content the player never experiences.
|
| I see a lot of potential, but it's going to take a different
| kind of craftsmanship, and likely many iterations, to realize
| something more than a novelty.
| causality0 wrote:
| I much prefer handcrafted stories and quests. Characters
| that respond dynamically to the story and the player's
| actions, however, is quite tantalizing.
| swyx wrote:
| why is it that these models tend to be released as float16 and
| converting to int8 is left to the reader? is there something
| special about training that defaults you to float16?
| dspillett wrote:
| Precision, aiming those names refer to standard binary
| numeric types. IEEE754 16-bit floats carry 11 significant
| digits with absolute precision so by coverting to 8-bit
| integers you lose some of that. Depending on the distribution
| of the values in those floats you could be loosing a lot more
| detail then this would imply, which is the reason we use
| floating point numbers for anything in the first place
| (rather than using an int16 where you have greater precision
| at you maximum scale but much less at lower scales).
|
| So if the model is computed using float16s, distribute as-is
| and let the end user choose to user it like that or
| compromise for faster processing of there system can deal
| with many billions of int8s more effectively.
| sillysaurusx wrote:
| They were trained in fp16, and researchers tend to release
| whatever format they trained. It's hard enough to do a large
| release that it's best not to try to have too many goals, for
| the same reason most software projects try not to do too much
| lest their schedule slip.
|
| Still, I'm a little sad they didn't release the optimizer
| weights. It would've given us so much valuable info about the
| dataset, among other benefits.
| charcircuit wrote:
| Quantization and other optimizations are more for
| productionizing models. You start with something accurate and
| then you start making tradeoffs to get the inference time to
| fit into your compute, memory, and time budgets.
| nextaccountic wrote:
| If the model weights are stored as int8, does this mean that
| the floating point capacity of the GPU is wasted? Or the int8
| is converted to float in the GPU?
| woodson wrote:
| Well, tensor cores support int8 instructions (at least from
| Turing onwards), so the hardware is being used, if that's
| your concern.
| rnosov wrote:
| Hmmm, the Github repo suggests that you might be able to run
| the 65B model on a single A100 80gb card. At the moment, the
| spot price on Google cloud for this card is $1.25/hour which
| makes it not so crazy expensive...
| nabla9 wrote:
| $1.25/hour is roughly a year of GPU time until it exceeds the
| price of A100 80GB card.
| metadat wrote:
| I think OP meant that $1.25/hr makes this accessible for
| people try it out themselves cost effectively, without
| having to spend thousands or tens of thousands up front to
| obtain a capable hardware rig.
|
| Obviously $1.25/hr 24/7 does add up quickly, after one
| month the bill would come to $900.
| [deleted]
| notpushkin wrote:
| For even better speeds, perhaps use the link from this script (if
| it ever goes back up) as a webseed for torrent?
| m3kw9 wrote:
| If an AI model like this isn't able to evolve and improve is it
| really useful? Example is code generation or questions that more
| recent training data can teach the AI
| sillysaurusx wrote:
| Update: FB disabled the download link, so I mirrored everything
| to R2 and updated the script to use it. It should be working now
| (though the speed is "only" around 50MB/s).
| toomuchtodo wrote:
| Have you dropped the artifacts in the Internet Archive yet by
| chance?
| sillysaurusx wrote:
| I'm surprised Internet Archive is appropriate for a 220GB
| model weight dump.
|
| Please feel free; it seems like a good idea. I'm not sure I
| have enough weekend left to figure out yet another upload
| service today.
| e12e wrote:
| I've read the readme - but I'm not sure why this is any faster
| than just adding seeds to the torrent? More people downloading
| via torrent than http?
| ahahahahah wrote:
| Are we celebrating theft from tech companies now?
| anaganisk wrote:
| I mean highseas, adblockers, bypassing paywalls, each one of
| them is theft. But on the flipside, companies are constantly
| trying to keep the ownership of data we paid for full price,
| scooping up personal data, selling low quality work behind
| paywall.
| antibasilisk wrote:
| Copying isn't theft. If you bought the ssd it belongs to you in
| its entirety regardless what state you decide to configure it
| into.
| bitL wrote:
| How does LLaMA handle fast fine-tuning? Are they using
| transformer adapters for it?
| loufe wrote:
| It's already been adapted for hugging face transformers[1].
| Apparently that should unlock its full potential. Oobabooga
| integrated the change into text-generation-webui[2] meaning we
| can already access a large chunk of its potential (from what I
| understand).
|
| [1] https://github.com/huggingface/transformers/pull/21955
|
| [2] https://github.com/oobabooga/text-generation-
| webui/commit/90...
| bitL wrote:
| That's absolutely fantastic! Thanks for the links!
| vanillax wrote:
| How or what can someone do with this who isn't a ML expert? Is
| there some docker app that leverages this? To the average dev, is
| this useful to me? I know there's lots of "plug and play" style
| docker apps to get started with Stable Diffusion. I'm curious if
| I can do something fun with this.
| kkielhofner wrote:
| You can shortcut a lot of the steps in these various guides by
| using the Pytorch container from Nvidia[0].
|
| It shouldn't be too hard for someone (me?) to create a
| Dockerfile and Docker hub container FROM this image to get it
| up and running easily.
|
| [0] -
| https://catalog.ngc.nvidia.com/orgs/nvidia/containers/pytorc...
| swalsh wrote:
| Not an expert, but i've downloaded the model, and used it. What
| you get is pretty raw, and not super useful by itself. There
| are some projects to try and do some RLHF on it, and with that
| we might start to get something you can do some useful stuff
| with.
| [deleted]
| arjvik wrote:
| How big is this model? (i.e. disk space to store it)
| kuroguro wrote:
| 65B is ~120GB. All of them combined with the smaller versions
| is ~220GB.
| nextaccountic wrote:
| After converting to int8, does it become smaller? Also, can
| this be further compressed? Like, is there some redundancy a
| special-purpose compressor could exploit?
| rfoo wrote:
| Converting to int8 halves the size.
| swyx wrote:
| thanks for doing this, honestly your writeup seems more valuable
| than the model weights lol
|
| > But for what it's worth, my personal opinion is that LLaMA
| probably isn't OpenAI-grade -- there's a big difference between
| training a model in an academic setting vs when your entire
| company depends on it for wide-scale commercial success. I wasn't
| impressed that 30B didn't seem to know who Captain Picard was.
|
| im new to benchmarking shenanigans but how is it that facebook
| was able to proclaim that it matched GPT3 performance on
| presumably standard LLM benchmarks? is there a good survey paper
| or blogpost on how to think about known deficiencies in
| benchmarks?
| sillysaurusx wrote:
| Because loss != quality. This was one of the most
| counterintuitive discoveries in ML for me. People treat the two
| as interchangeable, and to a certain extent -- a controlled
| extent -- they are.
|
| But if your dataset doesn't include a word about Captain
| Picard, no amount of training will get it to know about the USS
| enterprise. Yet your loss metrics will still reach that magical
| 2.1 value with time. (2.1 is pretty much "excellent" quality;
| below that means you're probably overfitting and need a bigger
| dataset.)
|
| Thanks for the comment friendo. I wasn't sure if this would get
| any attention at all, but that made it worth it. Be sure to DM
| me on Twitter if you'd like to chat about anything ML related:
| basic questions are one of my favorite things to assist with
| too, so feel free.
| nl wrote:
| This isn't really correct.
|
| Loss is a training-time measurement based on performance on
| the training objective.
|
| The training objective is rarely the same as an end user task
| that is being benchmark.
|
| For example, classically language models are training on next
| token prediction. The closest benchmark for that is
| perplexity[1], often reported on the WikiText-103 dataset.
|
| Until around 2019 this was often reported, but since then
| most large language model papers have moved onto reporting
| more useful benchmarks. Some examples of this are question
| answering performance or maybe embedding performance.
|
| Unfortunately there aren't great benchmarks (yet?) for
| generative tasks. Quality is quite hard to measure here in a
| systematic way (see, eg the issues with BLEU benchmarks in
| summarization benchmarks).
|
| [1] https://en.wikipedia.org/wiki/Perplexity
| rnosov wrote:
| You can read the original LLaMA paper which is pretty
| accessible[1]. For example, they claim to outperform GPT-3 on
| HellaSwag benchmark ( finishing sentences ). You can find
| examples of unfinished sentences in the HellaSwag paper [2] on
| page 13. Unfortunately for LLaMA, most people would be probably
| just asking questions about Captain Picard and so on, and on
| this benchmark LLaMA significantly underperforms compared to
| OpenAI models (thats's from their paper).
|
| [1] https://research.facebook.com/file/1574548786327032/LLaMA--
| O...
|
| [2] https://arxiv.org/pdf/1905.07830.pdf
| yunyu wrote:
| Hellaswag is also a deeply flawed benchmark, I wouldn't read
| too much into it: https://www.surgehq.ai/blog/hellaswag-or-
| hellabad-36-of-this...
| nl wrote:
| Because there are many benchmarks that measure different
| things.
|
| You need to look at the benchmark that reflects your specific
| interest.
|
| So in this case ("I wasn't impressed that 30B didn't seem to
| know who Captain Picard was") the closest relevant benchmark
| they performed is MMLU (Massive Multitask Language
| Understanding"[1].
|
| In the LLAMA paper they publish a figure of 63.4% for the
| 5-shot average setting without fine tuning on the 65B model,
| and 68.9% after fine tuning. This is significantly better that
| the original GPT-3 (43.9% under the same conditions) but as
| they note:
|
| > "[it is] still far from the state-of-the-art, that is 77.4
| for GPT code-davinci-002 on MMLU (numbers taken from Iyer et
| al. (2022))"
|
| InstructGPT[2] (which OpenAI points at as most relevant ChatGPT
| publication) doesn't report MMLU performance.
|
| [1] https://github.com/hendrycks/test
|
| [2] https://arxiv.org/abs/2203.02155
| JonathanFly wrote:
| The capability of a language model I care about most is
| probably its ability to represent or simulate Captain Picard.
| In the sense of being good at creative tasks but also Captain
| Picard, specifically. Is OpenAI deliberately doing something
| different on purpose that makes their models better for this,
| or is just that OpenAI has a lot more copyrighted data in
| their dataset, as I noticed just now when skimming the
| Facebook paper for MMLU section and seems be what the
| Facebook folks think?
|
| "A potential explanation is that we have used a limited
| amount of books and academic papers in our pre-training data,
| i.e., ArXiv, Gutenberg and Books3, that sums up to only
| 177GB, while these models were trained on up to 2TB of books.
| This large quantity of books used by Gopher, Chinchilla and
| PaLM may also explain why Gopher outperforms GPT-3 on this
| benchmark, while it is comparable on other benchmarks."
| nl wrote:
| It's unclear exactly why it doesn't work as well for you.
|
| I have two comments that may be useful:
|
| 1) It's very unclear how good the generative capabilities
| of LLAMA are generally. It benchmarks well for code
| generation, but for English there aren't really any good
| benchmarks around. There's good chance the larger model
| performs much better since generative capabilities seem to
| be a partially emergent capability.
|
| 2) If you just want to "make it work" I'd suggest
| downloading all the Star Trek scripts you can that include
| Captain Picard and fine tuning LLAMA using them. It's
| unclear how well this will work, but that is probably about
| as good as you can get.
|
| If you care about this probably deeply, it's probably worth
| trying the same with some of the other open GPT-3 models
| (GPTJ, GPT-NEOX etc)
| ur-whale wrote:
| Is this the full model or just the weights?
|
| [EDIT]: are there checksums available?
|
| [EDIT2]: MD5 signatures seem to be included for all models in
| checklist.chk files next to them
|
| And there's also what the author mentions: the magnet file he
| provides in his README does seed immediately on the download when
| loaded in a bt app which is usually a good sign that the files
| are correct.
| rany_ wrote:
| MD5 signatures don't mean much now that hash collisions could
| be created instantly on consumer hardware. MD5 is only good for
| checking for unintentional data corruption.
| charcircuit wrote:
| >hash collisions could be created instantly on consumer
| hardware
|
| Collisions can be created, but MD5 is still preimage
| resistant. As long as someone with the actual model made the
| hash and Meta didn't generate colliding models themselves.
| You can trust it.
| [deleted]
| [deleted]
| ur-whale wrote:
| Bittorrent uses SHA-1
|
| https://en.wikipedia.org/wiki/BitTorrent
|
| Collisions are possible but not exactly trivial
| yumraj wrote:
| What's the minimum single GPU that'll work for the smallest
| model?
| downvotetruth wrote:
| 3060 12GB
| zargon wrote:
| This reddit post says that the 7B model consumes about 9.7GB of
| VRAM (using int8). I'm sure very soon people will add support
| for using system RAM as swap space, which will allow you to use
| it on an 8GB card, though with a fairly hefty performance
| penalty.
|
| https://www.reddit.com/r/MachineLearning/comments/11h3p2x/d_...
| linearalgebra45 wrote:
| It's been enough time since this leaked, so my question is why
| aren't there blog posts already of people blowing their $300 of
| starter credit with ${cloud_provider} on a few hours'
| experimentation running inference on this 65B model?
|
| Edit: I read the linked README.
|
| > I was impatient and curious to try to run 65B on an 8xA100
| cluster
|
| Well?
| ulnarkressty wrote:
| https://medium.com/@enryu9000/mini-post-first-look-at-llama-...
|
| *later edit - not the 65G model, but the smaller ones.
| Performance seems mixed at first glance, not really competitive
| with ChatGPT fwiw.
| linearalgebra45 wrote:
| > not the 65G model, but the smaller ones
|
| Haha, that's right! I saw that one too
| minxomat wrote:
| > not really competitive with ChatGPT
|
| That's impossible to judge. LLama is a foundational model. It
| has received neither instructional fine tuning (davinci-3)
| nor RLHF (ChatGPT). It cannot be compared to these finetuned
| models without, well, finetuning.
| v64 wrote:
| The compute necessary to run 65B naively was only available on
| AWS (and perhaps Azure, I don't work with them) and the
| required instance types have been unavailable to the public
| recently (it seems everyone had the same idea to hop on this
| and try to run it). In my other post here [1], the memory
| requirements have been lowered through other work, and it
| should now be possible to run the 65B on a provider like
| CoreWeave.
|
| [1] https://news.ycombinator.com/item?id=35028738
| MacsHeadroom wrote:
| I'm running LLaMA-65B on a single A100 80GB with 8bit
| quantization. $1.5/hr on vast.ai
| sillysaurusx wrote:
| Careful though -- we need to evaluate llama on its own
| merits. It's easy to mess up the quantization in subtle
| ways, then conclude that the outputs aren't great. So if
| you're seeing poor results vs gpt-3, hold off judgement
| till people have had time to really make sure the quantized
| models are >97% the effectiveness of the original weights.
|
| That said, this is awesome -- please share some outputs!
| What's it like?
| MacsHeadroom wrote:
| The output is at least as good as davinci.
|
| I think some early results are using bad repetition
| penalty and/or temperature settings. I had to set both
| fairly high to get the best results. (Some people are
| also incorrectly comparing it to chatGPT/ChatGPT API
| which is not a good comparison. But that's a different
| problem.)
|
| I've had it translate, write poems, tell jokes, banter,
| write executable code. It does it all-- and all on a
| single card.
| sillysaurusx wrote:
| That's great to hear. Thank you very much, both for
| reporting this, and especially for the crucial note about
| temperature.
|
| In fact, sampling settings are so important and so easily
| underestimated that I should just pester you to post your
| exact settings. If you get a moment, would you mind
| sharing your temperature, repetition penalty, top-k, and
| anything else? I'll be experimenting with those today,
| but having some known working defaults would be
| wonderful. (You're also the first person I've seen that
| got excellent outputs from llama; whatever you did, no
| one else seems to have noticed yet.)
|
| If you're busy or don't feel like it, no worries though.
| I'm just grateful you gave us some hope that llama might
| be really good. There were so many tweet chains showing
| universally awful outputs that I wasn't sure.
|
| EDIT: I added your comments to the top of the README and
| credited you. Thanks again.
| linearalgebra45 wrote:
| Would you mind publishing your notes/learnings once you
| gain enough understanding of this model?
| sillysaurusx wrote:
| Absolutely! I'll make sure to leave a comment here for
| you whenever something gets written up so you don't miss
| it.
|
| Getting "as good as davinci" on a single A100 is
| groundbreaking work. Facebook and the community should
| both be credited here -- maybe llama-int8 would've been
| created even if the model hadn't leaked, but I don't
| think it would've happened so quickly. Everyone is doing
| phenomenal work, and it's so amazing to see it all come
| together.
|
| But, we'll see. Going to try it myself soon.
|
| Long ago, I cloned OpenAI's API:
| https://github.com/shawwn/openai-server -- my plan is,
| once I get it running, I'll try to host it somewhere so
| that anyone can play with it. I assume it'll be quickly
| swamped, but it's still an interesting challenge; some
| basic load balancing should make it scalable across
| several A100 instances, so there's no reason we can't
| just roll our own OpenAI API.
| rnosov wrote:
| Seconded. Do write it up.
|
| I see vast.ai listing interruptible instance with a
| single A100 80GB at $1/hour which is pretty reasonable.
| ChatGPT plus is $20/month which would be roughly 20 hours
| of use and I wont't be lectured like I'm in a
| kindergarten or something.
|
| A bonus point would be to make the writeup accessible for
| AI challenged developers. Asking for a friend.
| davrosthedalek wrote:
| I would like to support this request for AI challenged
| developers :)
|
| For things like these, I always wonder: How much slower
| would it be to run such a model on a CPU? I mean, clearly
| a lot less interactive, but is it possible at all? Could
| it be chopped up and "streamed" to a GPU with less memory
| halfway efficiently? What is the bottleneck currently on
| GPUs, memory bw or compute?
| [deleted]
| nl wrote:
| On a CPU I'd estimate it would get a maximum of around 5
| tokens per second (a token being a sub-word token, so
| generally a couple of letters). I suspect it'd be more
| like 1 token per second on the large model without
| additional optimisation.
|
| Yes models can be split up. See eg Hugging Face
| Accelerate.
| akreal wrote:
| Which prompt did you use for translation? I'd be curious
| to try it for my task too.
| v64 wrote:
| Note that unlike ChatGPT, these models are pure text
| completers and have not been trained to be prompted. The
| llama FAQ [1] mentions this and gives tips for how to get
| out of the ChatGPT mindset and prompt llama better.
|
| [1] https://github.com/facebookresearch/llama/blob/main/F
| AQ.md#2
| youssefabdelm wrote:
| What's the speed like? How many tokens per second? / Is it
| as fast as say ChatGPT?
| linearalgebra45 wrote:
| What instance are you using?
| linearalgebra45 wrote:
| Are you sure about that? I can't remember where I saw the
| table of memory requirements, but I'm sure some of the larger
| instances here [1] will surely be able to cope (assuming
| they're available!)
|
| Oracle gives you a $300 free trial, which equates to running
| BM.GPU4.8 for over 10 hours - enough for a focused day of
| prompting
|
| [1] https://www.oracle.com/cloud/compute/gpu/
| fswd wrote:
| If you actually try and do this, the sales people will stop
| you due to some internal rule. No GPUs on free credit.
| Unless the situation has changed of course..
| v64 wrote:
| > Are you sure about that?
|
| I'm not. The only way to know it is to try :) thank you for
| the link!
| linearalgebra45 wrote:
| You only get a single month-long window to spend the
| credit! And I'm sure not going to spend any of my own
| money on prompting experiments.
|
| I might be suffering from FOMO to some degree, I've just
| got to tell myself that this won't have been the only
| time model weights get leaked!
| mynameisvlad wrote:
| > And I'm sure not going to spend any of my own money on
| prompting experiments.
|
| This certainly sounds a lot like whining that others
| aren't doing the work you yourself don't want to do.
| linearalgebra45 wrote:
| "prompting experiments" is just my use-case. According to
| v64 a lot of people have had the same idea of spinning up
| a trial instance to run inference, which is unsurprising.
|
| I'm not in a position to put in any meaningful work
| towards optimising this model for lower-end hardware, or
| working on the tooling/documentation/user experience.
| smoldesu wrote:
| Thanks for sharing it! I'm using their "Always Free" tier
| to host an Ampere-accelerated GPT-J chatbot right now.
| Works like a charm, and best of all, it's free!
| damascus wrote:
| Do you have any code from your discord bot you're willing
| to share? I'd be happy to share back any updates I made
| to it. I've been wanting to play with this idea for a
| bit.
| [deleted]
| jocaal wrote:
| I don't understand, the Ampere they refer to in their
| free tier are cpu's not gpu's. How did you manage to do
| that
| smoldesu wrote:
| Custom PyTorch with on-chip acceleration: https://cloudma
| rketplace.oracle.com/marketplace/en_US/listin...
|
| Not as fast as a GPU, but less than 5 seconds for a 250
| token response is good enough for a Discord bot.
| nl wrote:
| This is the most interesting thing I've read in this
| thread. How have I never heard of this accelerator?!
| version_five wrote:
| Thanks for doing what Facebook should have been mature / humble
| enough to have done on their own.
|
| The best outcome of this would be for FB to stop the silliness
| and just release the weights openly themselves.
| cloudking wrote:
| What's up with the domain in the script?
| PRESIGNED_URL="https://agi.gpt4.org...
| arjvik wrote:
| It's pointed at Cloudflare storage right now
| EMIRELADERO wrote:
| I womder, could Facebook take legal action here? While some (most
| of) the data used to train the model is copyrighted, I don't
| think the model is. It's the result of a mathematical process
| applied to a series of facts and works with no more creativity
| put onto them.
| [deleted]
| jeroenhd wrote:
| As far as my understanding of American copyright goes, a
| computer produced work cannot be copyrighted as computers are
| not human, in the same way a photograph taken by a chimp cannot
| be copyrighted no matter who owned the camera that took the
| photo. This is one of the major challenges with the legal
| status of AI as well that will soon be fought over in court.
|
| It's possible that the automated processing of the dataset is
| considered to be non-creative enough that the generated AI
| model cannot be copyrighted. The code to train the model and
| the input dataset (and the works therein) definitely can be,
| but not the model itself.
|
| In that case, Facebook would be out of luck, as long as the
| code to train the model isn't shared. If the courts find AI
| models to be a different type of work that does produce
| copyrightable models, Facebook may follow in the footsteps of
| other copyright giants and start filing lawsuits against anyone
| who they can catch. I very much doubt they'd go so far,
| especially since by the time they can even start a lawsuit
| confidently, the leaked model is probably already outdated and
| irrelevant.
|
| Personally, I expect the model to end up being uncopyrightable,
| as would be the output of the model.
|
| This may or may not have very interesting results. The dataset
| itself is probably copyrightable (a human or set of humans
| composed it, unless that was also done completely
| automatically) but if that copyright is claimed, the individual
| right holders of the included works may demand a licensing fee
| similar to how sound bytes work in music; "you want to use my
| work, pay me a fee".
|
| Or maybe the dataset is considered to be diverse enough that
| individual works cannot be expected to be compensated for their
| inclusion and you can get around copyright law by amassing
| enough content at once, who knows.
| adossi wrote:
| It is intellectual property, regardless of copyright.
| brookst wrote:
| "Intellectual property" is a catch-all for copyright,
| trademark, patent, and trade secrets. There isn't really
| law that protects IP as a general concept, just those four.
| cma wrote:
| It isn't protected as a trade secret if they mostly freely
| shared it with .edu addresses. And once it has been leaked
| out widely publicly it isn't either.
| [deleted]
| digitallyfree wrote:
| There is another angle here besides copyright and that is the
| sharing of prop/trade secret data. This model is only available
| to specific orgs who request it (i.e. it's non-public) and I
| imagine that there are confidentiality terms for the orgs that
| get the access.
|
| Not too familiar with the drama but I believe what happened was
| that someone with access leaked the torrent used to download
| the weights. In a legal sense this would be similar to someone
| say leaking a Google Drive link containing prop information
| that was only intended to be shared with vendors.
| charcircuit wrote:
| You can read the license at this link.
|
| https://docs.google.com/forms/d/e/1FAIpQLSfqNECQnMkycAp2jP4Z.
| ..
|
| There isn't any confidentiality terms.
| kuroguro wrote:
| That definition would apply to almost anything software
| produces ^^;
|
| We can already have different licenses for compiled binaries vs
| the source. Also the output of ML seems to belong to whoever
| pressed the generate button atm.
| EMIRELADERO wrote:
| > That definition would apply to almost anything software
| produces
|
| Not really. The reason software can be copyrighted at all is
| because the actual code (and resulting object code) is
| creative. Courts have named this threshold the "Structure,
| sequence and organization" of the work. ML models don't
| follow any creative SSO the way actual code does.
|
| > Also the output of ML seems to belong to whoever pressed
| the generate button atm.
|
| The output, it seems to me, is uncopyrightable. Copyright
| only cares about who provides the _creativity_ for the work
| at issue, not who put in the effort to make it happen. You
| may own the copyright to your prompt, but the result is
| generated entirely by the AI and thus lacks human autorship.
| AlDante2 wrote:
| I think that copyright law works differently. Source code
| is copyright; the expression as compiled code from that
| source enjoys the same protections. If the model can be
| copyrighted, the expression of the model in the form of its
| weights is probably also protected.
| EMIRELADERO wrote:
| You're correct, but that doesn't disprove my point. I'm
| saying the _model itself_ is uncopyrightable.
___________________________________________________________________
(page generated 2023-03-05 23:02 UTC)