https://lwn.net/Articles/931853/

LWN.net Logo LWN
.net News from the source LWN

  * Content
      + Weekly Edition
      + Archives
      + Search
      + Kernel
      + Security
      + Events calendar
      + Unread comments
      + -------------------------------------------------------------
      + LWN FAQ
      + Write for us
  * Edition
      + Return to the Front page

User: [        ] Password: [        ] [Log in]
|
[Subscribe]
|
[Register]
Subscribe / Log in / New account

Democratizing AI with open-source language models

May 17, 2023

This article was contributed by Koen Vervloesem

When OpenAI made its chatbot ChatGPT available to the public in
November 2022, it immediately became a hit. However, despite the
company's name, the underlying algorithm isn't open. Furthermore,
ChatGPT users require a connection to OpenAI's cloud service and face
usage restrictions. In the meantime, several open-source or freely
available alternatives have emerged, with some even able to run on
consumer hardware. Although they can't match ChatGPT's performance
yet, rapid advancements are occurring in this field, to the extent
that some people at the companies developing these artificial
intelligence (AI) models have begun to worry.

ChatGPT is presented as a bot that users interact with, which
generates human-like text based on their input. OpenAI has fine-tuned
it specifically for engaging in conversations and providing
contextually appropriate responses. It's capable of handling a
variety of tasks, such as generating content or ideas, translating
languages, answering questions, and even providing suggestions for
code in various programming languages. ChatGPT also responds to
follow-up questions, challenges incorrect premises, and rejects
inappropriate requests. How does this work? Under the hood, ChatGPT
uses a neural network trained on vast amounts of text to generate new
content based on the input it receives. Think of it as an advanced
form of autocomplete suggestions.

A neural network is a learning algorithm inspired by how our brains
function. It consists of a large number of nodes, known as neurons,
that receive input from other neurons and perform a mathematical
function to calculate their output, which then goes to other neurons.
Each input has a weight attached to it that determines how much that
value contributes to the result. The neural network's architecture
(how the layers of neurons are connected) and its weights determine
its functionality.

The network starts with random weights; therefore, when it receives
text as input (we're glossing over some details, such as how text is
encoded in numbers), its output is also random. As a result, the
network has to be trained using training data: an input text with a
corresponding output text. Each time the training data input enters
the network, its output is compared with the training data's
corresponding output. The weights are then adjusted to decrease the
difference between the predicted and correct output. In this way, the
network undergoes a learning process until it becomes proficient at
predicting text.

Such a large neural network capable of generating human-like text is
called a large language model (LLM). It typically involves billions
to several hundred billion weights, also known as parameters. ChatGPT
is based on some large language models developed by OpenAI, with
names like GPT-3.5 or (the most recent version) GPT-4. GPT stands for
Generative Pre-trained Transformer and is a specific type of large
language model, introduced by OpenAI in 2018, and based on the
Transformer architecture invented by Google in 2017. Since GPT-3.5,
OpenAI hasn't disclosed the size of its models; GPT-3 (released in
May 2020) had 175 billion parameters and was trained on 570GB of
text.

When large language models are trained on a broad range of data, as
is the case with GPT-3.5 and GPT-4, they are also known as
foundational models. Their broad training makes them adaptable to
various tasks. A foundational model can be fine-tuned by training the
model (or part of it) with new data for a specific task or a specific
subject-matter domain. This is what OpenAI has done with ChatGPT: it
has fine-tuned its foundational GPT models with conversations in
which humans played both the user and the AI role. The result is a
model specifically fine-tuned to follow a user's instructions and
provide human-like responses.

BLOOM

Companies developing large language models lack incentives to
open-source their models and the code to run them since training the
models requires significant computing power and financial investment.
To make the development of these models sustainable, companies need
to be able to build a profitable business around it. OpenAI aims to
do this by offering the paid ChatGPT Plus subscription and its
pay-per-use API access.

Last year, the situation changed with BLOOM (BigScience Large
Open-science Open-access Multilingual Language Model), which is
freely available. This large language model was the result of a
global collaboration involving over a thousand scientists from more
than 250 institutions participating as volunteers in the BigScience
collective. The project was started by Hugging Face, a company that
provides a machine-learning platform, with contributions from NVIDIA,
Microsoft, and the French research institution CNRS.

Development of BLOOM occurred entirely in public. The model was
trained for 3.5 months on the Jean Zay supercomputer in Paris,
utilizing 384 NVIDIA A100 Tensor Core GPUs each with 80GB of RAM. The
data set comprised 1.6TB of text (341 billion words) in 46 human
languages and 13 programming languages, and the model had 176 billion
parameters, which is comparable to GPT-3.

BigScience developed a new license to publish BLOOM: the Responsible
AI Licence (RAIL). Its main purpose is to minimize risks arising from
irresponsible use of the model. According to the license, users are
not allowed to use the model to generate false information with the
intent to harm others; nor may they neglect to disclose that the
generated text was machine-generated. RAIL is not an open-source
license and does not appear on the list of OSI-approved licenses.
Developers can use the BLOOM model with Hugging Face's
Apache-2-licensed transformers library in their own code, subject to
the terms of RAIL.

Smaller large language models

A downside of BLOOM is that it's still too large for convenient local
use. In principle, anyone could download the 330GB model, but using
it would require substantial hardware. BigScience also released
smaller versions of the model, and this trend of smaller models was
continued by others. In February, Meta announced a language model
called LLaMA, which is available in versions with sizes of seven
billion, 13 billion, 33 billion, and 65 billion parameters. According
to the developers, the 13B version performs as well as OpenAI's
GPT-3, while being a factor of ten smaller. And unlike GPT-3, which
requires multiple A100 GPUs to operate, LLaMA-13B needs only one GPU
to achieve the same performance.

Meta trained LLaMA on publicly available data sets, such as Wikipedia
and Common Crawl. The code to run LLaMA is GPLv3-licensed, but to
obtain the full weights of the model, users were required to fill out
a form and agree to a "non-commercial bespoke license". Moreover,
Meta proved to be quite selective in granting access. But within a
week, the weights were leaked on BitTorrent, and LLaMA kickstarted
the development of a lot of derivatives. Stanford University
introduced Alpaca 7B, based on the LLaMA model with seven billion
parameters and supplemented with instructions based on OpenAI's
text-davinci-003 model of the GPT-3.5 family. Both the data set and
the model were released under the CC BY-NC 4.0 license and thus do
not permit commercial use. One reason for this is that OpenAI's terms
of use disallow the development of models that compete with OpenAI.

Subsequently, the open research organization Large Model Systems
Organization published Vicuna, a LLaMA-based model fine-tuned
on 70,000 conversations of users with ChatGPT. This was accomplished
using ShareGPT, which is a browser extension for Google Chrome that
is designed to easily share ChatGPT conversations. According to the
researchers, Vicuna achieves 90% of ChatGPT's quality and outperforms
LLaMA and Alpaca in 90% of cases. Both the code (Apache 2 license)
and the weights (13 billion parameters subject to LLaMA's license)
have been made public. However, since Vicuna is based on LLaMA and on
output from ChatGPT, commercial use is not allowed.

This issue prompted US software company Databricks to develop an
open-source large language model suitable for commercial use: Dolly
2.0. It is based on EleutherAI's Pythia model with 12 billion
parameters and was trained on the Pile text data set, then fine-tuned
on 15,000 instructions with answers. To achieve this, the company
engaged more than 5,000 of its employees. Dolly is trained on
open-ended questions, closed questions, extracting factual
information from texts, summarizing texts, brainstorming,
classification, and creative writing tasks, all of this in English
only. The 23.8GB dolly-v2-12b model can be downloaded from
Databricks' page on Hugging Face. The model is using the MIT license,
while the databricks-dolly-15k data set is published under the CC
BY-SA 3.0 license.

Following this, Stability AI, the creators of the Stable Diffusion
open-source model for generating images, published its own family of
large language models: StableLM, under a CC BY-SA 4.0 license.
Additionally, MosaicML introduced its MPT-7B family of open-source
commercially usable large language models (some of them Apache 2
licensed). Another interesting development is BigCode, a project
kickstarted by ServiceNow Research and Hugging Face to develop large
language models for completing and writing code from other code and
natural language descriptions. Their first model, StarCoder, has been
trained on permissively licensed data from GitHub and is using the
OpenRAIL license, an updated version of the Responsible AI License.

Crowdsourcing open data sets

With new open-source (or freely available) language models emerging
regularly (many of which can be found in the
awesome-totally-open-chatgpt repository), various organizations have
started considering ways to streamline the development of data sets
for training models. One such organization is LAION (Large-scale
Artificial Intelligence Open Network), a non-profit research
organization aiming to democratize AI. With Open Assistant, it plans
to develop large language models capable of running on consumer
hardware.

Open Assistant is still under development, and currently focuses
mainly on collecting data sets with the help of users. The project
already boasts a data set of 600,000 interactions, contributed
by 13,000 volunteers. Everyone can lend a hand in this endeavor, as
explained in the documentation. For example, users are tasked with
grading an answer provided by another person, based on parameters
such as quality or politeness. Another task involves offering an
answer in the role of a chatbot to a user's request. Volunteers may
also be asked to select the best response from two possible answers.
Tasks are available not only in English, but in many other languages
as well. The researchers intend to train language models using the
data set generated by these volunteer tasks. It's worth noting that
OpenAI has a similar approach with ChatGPT: the company pays
(low-wage) contractors to assist in training its language model and
to help identify toxic content.

Running language models on consumer hardware

Running large language models with tens to hundreds of billions of
parameters on consumer hardware is not feasible. However, with the
trend of smaller language models, initiated by LLaMA, operating
ChatGPT-like software on a PC becomes possible. An important project
for running these models is Georgi Gerganov's MIT-licensed llama.cpp.
It enables users to run LLaMa, Alpaca, and other LLaMA-based models
locally on their computers. It runs entirely on the CPU, which is
possible by applying 4-bit quantization of the models. This reduces
the accuracy of the weights to four bits to reduce memory consumption
and computational complexity. Llama.cpp supports Linux, macOS, and
Windows. Users can chat with the model via a command-line interface.

The model has to be downloaded first, for example from Hugging Face's
model hub, and then converted into a specific format. Instructions to
do this are provided in llama.cpp's README file. As a test, I
downloaded the 8.14GB model ggml-vic13b-q4_0.bin, which was already
in the correct format for llama.cpp. The program loads the model
fully in RAM, so it's crucial to ensure the computer has sufficient
free RAM.

After compiling llama.cpp, running the model in interactive mode is
as simple as:

    $ ./main -m models/ggml-vic13b-q4_0.bin -i

On my somewhat dated laptop with four-core Intel Core
i7-10510U 1.8GHz CPU and 24GB RAM, this took 20 seconds to start up.
After posing the question "`What is the most common way of
transportation in Amsterdam?'", the Vicuna model began to generate
its response word by word, taking 15 minutes to complete the task.
The resulting answer was vague but not incorrect, resembling a text
from a travel guide:

    Amsterdam, the capital city of the Netherlands, is known for its
    picturesque canals and bustling bike culture. It's no surprise
    that the most common way to get around Amsterdam is by bicycle.
    However, there are also plenty of other transportation options
    available in the city, including buses, trams, trains, and even
    boats. Whether you prefer two wheels or four, there's a mode of
    transportation in Amsterdam that will suit your needs.

Various projects have emerged that aim to make using open-source
language models more akin to a ChatGPT-like experience. For example,
Nathan Sarrazin has been working on Serge, a web-based interface for
llama.cpp. It runs in a Docker container, allowing users to download
a model from a list, choose its settings, and then initiate a chat.
Another similar undertaking is oobabooga's text-generation-webui,
which supports both CPUs and GPUs for running models. Using GPUs can
significantly improve the performance.

Is open-source AI gaining an edge?

In early May, a document penned by Google's Luke Sernau was leaked: "
`We have no moat, and neither does OpenAI'". The author contends that
the open-source community has been rapidly catching up with
commercial efforts: "`They are doing things with $100 and 13B params
that we struggle with at $10M and 540B. And they are doing so in
weeks, not months.'" Sernau also clearly recognizes the advantages of
the open-source development model. After detailing the numerous
innovations that have occurred within a month of LLaMA's weights
being leaked, he notes that anyone can tinker: "`Many of the new
ideas are from ordinary people.'" The barrier to entry to contribute
to these open-source large language models is just "`one person, an
evening, and a beefy laptop'".

Sernau continues with lessons for Google (and OpenAI), focusing on
LoRA, a technique that accelerates fine-tuning of models and is
already used by a lot of open-source projects in this domain. Thanks
to LoRA, almost anyone with an idea can generate a fine-tuned model
in under a day and for around $100. "`At that pace, it doesn't take
long before the cumulative effect of all of these fine-tunings
overcomes starting off at a size disadvantage'", he said, adding that
focusing on maintaining some of the largest models on the planet
actually puts Google at a disadvantage. At the end, he even made the
case for opening Google's large language models. He perceives Meta as
the clear winner in all of this, because most open-source innovation
is happening on top of its LLaMA model architecture.

If Sernau is right, it means that language models could become a
commodity, fueled by the innovative nature and fast-paced development
model of the open-source AI community. This would enable researchers,
non-profit organizations, and small businesses to access AI
capabilities without depending on cloud-based services or expensive
subscription fees. Looking back at the numerous language models that
have been published during the last few months, we can wonder how
long it will take before we have some usable, completely open-source,
AI assistants to help us with our daily tasks.


Index entries for this article
GuestArticles Vervloesem, Koen


-----------------------------------------
(Log in to post comments)

Democratizing AI with open-source language models

Posted May 17, 2023 19:53 UTC (Wed) by davidstrauss (subscriber, #
85867) [Link]

> nor may they neglect to disclose that the generated text was
machine-generated.

For the license to impose terms on use of the generated result, the
AI model's authors (read: copyright holders/licensors) would need
some claim on generated content. However, in the US, a machine/
algorithm (or its owner) never gains such a claim on content that the
machine has processed or generated.

For generated content, many lawyers are skeptical that OpenAI even
holds the rights that they "grant" back to users. Consensus seems to
be that the generated work is, at most, a derivative work of the
prompts/inputs.

So, I suspect this disclosure requirement is unlikely to be
enforceable in the US, at least to a meaningful level. I could
possibly see it enforced on the first party that directly used the AI
to generate the content, but it's not clear how those terms would
carry over to someone else using the generated content.

This situation is similar to how a proprietary compiler's license
can't require royalties for distribution of the compiled code.

(Standard "I am not a lawyer"/"this is not legal advice" disclaimer,
though I have decades of experience in software licensing topics.)
[Reply to this comment]
Democratizing AI with open-source language models

Posted May 18, 2023 1:43 UTC (Thu) by developer122 (subscriber, #
152928) [Link]

Such licences are also strongly reminiscent of "ethical" licences
that impose restrictions on the use of software, ranging from "no
using for military purposes" to "only use this software for the
greater good" and other such nebulous concepts.

Even if these are enforceable, they are absolute garbage. It's a
colossal reach for a software licence to restrict actives one can
undertake that don't pertain to the software itself. They fall into
the same hole the FSF did with the AGPL, trying to use a licence
(technology) to solve a societal problem.

When all you have is a hammer, everything looks like a nail.
[Reply to this comment]
Democratizing AI with open-source language models

Posted May 17, 2023 22:00 UTC (Wed) by IanKelling (subscriber, #
89418) [Link]

So Dolly 2.0, the "open-source language model" is based on multiple
data sets and the large one is not freely licensed. The authors say
roughly they probably don't have the right to distribute some of it
but training a model is fair use so don't worry about it ( https://
arxiv.org/abs/2304.01373 ).

Up until now, if someone called a non-software work open source, I'd
expect any generated output to have the source data be published
under a free license, as the natural translation of the meaning of
open source, plus the free culture definition specifies something
like that: "all underlying source data should be available alongside
the work itself under the same conditions" https://freedomdefined.org
/Definition.

But now, if you google for "open source language model", it seems to
have quickly become a widespread term that means that at least the
large data set is not under a free software license or open source
license or free culture license. The reason for this seems to
basically be that freely licensed datasets are much smaller and less
useful, so the people making these models have decided to promote a
new definition that is different and degraded in an important way,
and they seem to be doing it successfully. Am I understanding this
right?
[Reply to this comment]
Democratizing AI with open-source language models

Posted May 17, 2023 22:30 UTC (Wed) by IanKelling (subscriber, #
89418) [Link]

And a complicating factor in all this is that the ai models are meant
to be modified, so the idea of source and binary from software and
even most generated free culture works are not equivalent.
[Reply to this comment]
Democratizing AI with open-source language models

Posted May 18, 2023 0:49 UTC (Thu) by flussence (subscriber, #85566)
[Link]

> The reason for this seems to basically be that freely licensed
datasets are much smaller and less useful, so the people making these
models have decided to promote a new definition that is different and
degraded in an important way, and they seem to be doing it
successfully. Am I understanding this right?

Yes, you are. The reason they don't publish the training data is
because they'd open themselves up to be sued for statutory copyright
infringement amounting to some large multiple of the entire GDP of
the planet, and they know it. It was never theirs to use in the first
place.
[Reply to this comment]
Democratizing AI with open-source language models

Posted May 18, 2023 5:32 UTC (Thu) by IanKelling (subscriber, #89418)
[Link]

Well, that is one very common case. But I noticed the case of the
Pythia data set is that they are distributing it and they will just
remove parts of it if someone says to them they have the copyright
and don't permit distribution.

They seem to be distinct cases, and maybe having the data set even
under a nonfree license is like having free source code because, for
example if you have a 10tb data set of individual sentences, for the
purpose of model creation, you don't modify individual sentences, you
add or remove them.
[Reply to this comment]
Democratizing AI with open-source language models

Posted May 18, 2023 13:48 UTC (Thu) by dvrabel (subscriber, #9500) [
Link]

Your brain is a "large language model" trained over years from a huge
body of copyrighted works. Is the output of your brain copyrighted by
those original works? Of course not, so why should a mathematical
model trained in a conceptually similar way be any different?
[Reply to this comment]
Democratizing AI with open-source language models

Posted May 18, 2023 14:41 UTC (Thu) by mb (subscriber, #50428) [Link]

>Is the output of your brain copyrighted by those original works? Of
course not

Of course it can be.
If you learn a pattern and reproduce it, it might be copyrighted by
the original copyright owners, if sufficient amounts are copied.
There are no hard criteria here. Whether it is a copyright violation
has to be decided on a case by case basis.
Neither a brain nor a neural network does magically erase copyright.
[Reply to this comment]
Democratizing AI with open-source language models

Posted May 18, 2023 14:56 UTC (Thu) by smurf (subscriber, #17840) [
Link]

It might be helpful to remember that copyright is a rather recent
invention, and the Disney extensions (copyright shall not be forever,
but extending it by 20 years every 20 years appears to be OK - the
copyright to The Mouse! shall!! not!!! expire!!!1!) even more so.

That being said, the problem seems to be that the language models are
large enough to "remember" specific works, but not structured enough
to also remember the authors, let alone tell the user that use of the
text might be restricted.
[Reply to this comment]
Democratizing AI with open-source language models

Posted May 18, 2023 15:05 UTC (Thu) by epa (subscriber, #39769) [Link
]

Surely after running the model, you fuzzy-string-match its output
against the entire training corpus, and warn the user if it seems to
be parroting an existing work.
[Reply to this comment]
Democratizing AI with open-source language models

Posted May 18, 2023 15:09 UTC (Thu) by mb (subscriber, #50428) [Link]

Correct.

Another problem with judging brains and neural networks equally
w.r.t. copyright is:

If you use a brain do rewrite (re-implement) some pre-existing work
into something that does exactly the same thing, then this is a huge
effort. It is allowed, no copyright is transferred over to the new
work, but it is a huge effort. So it won't be done, unless somebody
really wants to invest time and/or money into it. And then it's
perfectly fine, that the new author owns all copyright on the new
thing.

If you use the Copyright-Eraser-3000 neural network to rewrite
(re-implement) some pre-existing work into something that does
exactly the same thing, then it's just a click of a button,
essentially. It is no effort. It would render copyright useless,
because you could always take any copyrighted material and convert it
into something equivalent with copyrights removed within seconds.

So we must either
- judge brains and neural networks differently.
- or get rid of copyright law entirely.

[Reply to this comment]
Democratizing AI with open-source language models

Posted May 19, 2023 14:35 UTC (Fri) by Lennie (subscriber, #49641) [
Link]

I guess someone has 2 make multiple AI agents that one reads the
original and generates a description and other reads the description
and generates a new work. Aka clean room re-implementation and
copyright bypassed.
[Reply to this comment]
Democratizing AI with open-source language models

Posted May 19, 2023 18:58 UTC (Fri) by mb (subscriber, #50428) [Link]

That would not fix the problem in any way.
Adding an intermediate language step and splitting up the machine
learning part into two independent instances that pass the
intermediate code between them does not increase the effort
significantly. It's still a click of a button in
Copyright-Eraser-3001 (Cleanroom edition).

The copyright concepts around clean-room-reimplementation and simple
reimplementation do not work at all with machine learning.
[Reply to this comment]
Democratizing AI with open-source language models

Posted May 23, 2023 6:50 UTC (Tue) by Lennie (subscriber, #49641) [
Link]

Wikipedia defines: "Typically, a clean-room design is done by having
someone examine the system to be reimplemented and having this person
write a specification. This specification is then reviewed by a
lawyer to ensure that no copyrighted material is included. The
specification is then implemented by a team with no connection to the
original examiners."

So I guess the lawyer step was missing ?

Or are you referring to something else ?

The clean room implementation would get far enough away from the
original copyrighted work assuming it was done correctly, but maybe
you are referring to the situation that the output might be a
re-implementation of the copyright of someone else (because it was
trained on other works).
[Reply to this comment]
Democratizing AI with open-source language models

Posted May 23, 2023 18:47 UTC (Tue) by mb (subscriber, #50428) [Link]

> Or are you referring to something else ?

Yes. The problem is, that the creators of copyright law never has AI
in mind.
So it's not made for it.

It's time to either think of a way to adapt copyright law, or to get
rid of it entirely. Because if we just let machine learning models
remove copyright and be ok with that, it renders copyright law
completely useless. I could then always just take foreign copyrighted
A and make my copyrighted B out of it very quickly and cheaply.
What's copyright good for then?

> So I guess the lawyer step was missing ?

Yes. If you keep a non-trivial human step, I would say it's probably
fine then. The machine steps can be seen as (de-)compilation then.
[Reply to this comment]
Democratizing AI with open-source language models

Posted May 23, 2023 19:24 UTC (Tue) by Lennie (subscriber, #49641) [
Link]

> Because if we just let machine learning models remove copyright and
be ok with that, it renders copyright law completely useless. I could
then always just take foreign copyrighted A and make my copyrighted B
out of it very quickly and cheaply. What's copyright good for then?

That was exactly what I'm implying... did we just figure out that in
a year copyright's current roles will be useless ?
[Reply to this comment]
Democratizing AI with open-source language models

Posted May 24, 2023 5:38 UTC (Wed) by smurf (subscriber, #17840) [
Link]

Huh.

Copyright has, or originally had, a point. The point was not to allow
publishers to rip off authors by printing verbatim copies of their
work without paying them.

The point is not, and never was, to prevent me from reading your work
and then writing something in the same style, or using the general
concepts, or whatever.

So what the heck is the problem? Sure a machine could rewrite Stephen
King's latest blockbuster until it's no longer immediately
recognizeable as a copy. So could I. Approximately nobody would buy
that, however.

So what's the problem we're trying to solve here?
[Reply to this comment]
Democratizing AI with open-source language models

Posted May 24, 2023 7:24 UTC (Wed) by mb (subscriber, #50428) [Link]

If I can remove Copyright from a program by clicking a button in
Copyright-Remover-3000 AI app, the the rip-off-protection is useless
and uneffective.
Rewriting by human takes significant amounts of ressources. That is
the only reason why the ripoff protection works today.
[Reply to this comment]
Democratizing AI with open-source language models

Posted May 24, 2023 8:42 UTC (Wed) by paulj (subscriber, #341) [Link]

The original point of copyright was for publishers - note *not*
authors - to re-assert the oligopoly they previously enjoyed over the
printing of books.

The argument "won't someone think of the poor authors?" was made by
the *printers*, but for completely self-serving reasons. The printers
never cared for the authors before when they had their oligopoly by
guild and royal decree, nor did they once they got copyright. They
just made authors sign over the rights.
[Reply to this comment]
Democratizing AI with open-source language models

Posted May 24, 2023 11:37 UTC (Wed) by farnz (subscriber, #17727) [
Link]

If you're interested in the background to this comment, look up the
"Statute of Anne", the first copyright law. As paulj says, it wasn't
about the authors at all - it was a consequence of the Stationers'
Company trying to recreate their monopoly on printing after it became
legal for anyone to own a printing press.
[Reply to this comment]
Democratizing AI with open-source language models

Posted May 21, 2023 4:46 UTC (Sun) by NYKevin (subscriber, #129325) [
Link]

> the copyright to The Mouse! shall!! not!!! expire!!!1!

PSA: The (oldest) copyright to The Mouse will expire on January 1,
2024, and neither Disney nor Congress have shown the slightest
interest in doing anything about it. Disney has even publicly
acknowledged[1] that it will happen. It was a useful example for a
while, but we will soon need to start using a new one.

[1]: https://www.nytimes.com/2022/12/27/business/mickey-mouse-...
[Reply to this comment]
Democratizing AI with open-source language models

Posted May 21, 2023 7:52 UTC (Sun) by smurf (subscriber, #17840) [
Link]

Well, they have done plenty in the past.

The Disney example will serve. After all, while another extension
seems to be off the table, nobody is actually working on reverting
(some of) this atrocity either.

NB, according to Wikipedia the worst example is Mexico. Life+100
years. Ugh.
[Reply to this comment]
Democratizing AI with open-source language models

Posted May 19, 2023 14:11 UTC (Fri) by Lennie (subscriber, #49641) [
Link]

If you read a page and can reproduce it word for word then you've
created a work that is copyrighted by the original author(s).
[Reply to this comment]
Democratizing AI with open-source language models

Posted May 18, 2023 4:42 UTC (Thu) by pabs (subscriber, #43278) [Link
]

I wonder if any of these open models meet the Debian Deep Learning
Team's Machine Learning Policy:

https://salsa.debian.org/deeplearning-team/ml-policy/
[Reply to this comment]
Democratizing AI with open-source language models

Posted May 18, 2023 18:14 UTC (Thu) by IanKelling (subscriber, #
89418) [Link]

That is a good relevant link. Thanks.
[Reply to this comment]
Democratizing AI with open-source language models

Posted May 18, 2023 11:28 UTC (Thu) by wtarreau (subscriber, #51152)
[Link]

Nice summary of the current situation!

Another point that makes opensource models progress faster is that
their performance depends a lot on their size, because a lot of
parameters have to be evaluated for each produced word. And while the
early players had deep pockets to use hundreds or thousands of GPUs,
individuals do not have an A100 GPU to play with this code, so
they're willing to make compromises on the model's size so that it
fits in their GPU or main memory and runs fast enough to be usable.
That's how we're now seeing a lot of work being done to try to
optimize the accuracy of small models. Vicuna 13B is exceptionally
good even in 4 bits and can be run on a 8GB machine with a moderate
CPU. A few weeks ago, TogetherComputer released RedPajama 3B (3
billion parameters) which is amazing for this size, and they're still
finishing to train the 7B model.

What I suspect is that models will more and more specialize on
certain knowledge areas to be smaller, faster, and more accurate on
such models, with less hallucinations, and they will complement each
other better (exactly like humans in fact, you don't expect an
electrician to be a good plumber and nobody would want such a
dual-competence person if it made them work slower). We'll see, but
these are very interesting times. I think that these new language
models will have at least as high an impact on the society as
smartphones had, if not even more.
[Reply to this comment]
Democratizing AI with open-source language models

Posted May 18, 2023 11:29 UTC (Thu) by karim (subscriber, #114) [Link
]

Thanks for this well explained and researched article.

I think it's wonderful that the open source M.O. is enabling the
commoditization of generating and running LLMs/AI. That's awesome.
It'll eventually (short-?, mid-?, long-? term) make sure that the
cost of such capabilities is as low as possible. In and of itself I
think this is a comforting and worthwhile goal. Both at a personal
level and a org/corp level, it'll enable local use of LLMs on private
data-sets. That's huge.

Yet, still, the most "powerful" models are likely to come from those
organizations with the largest of data sets. This will likely mean
the existing dominant consumer-facing mass-market behemoths,
including Google, FB/Meta, Twitter, etc. will remain on top.
Regardless of whether it open sources its underlying tech or not,
think of the models that Google, for example, can actually generate
by training its AIs on just the collective set of training videos
found on YouTube ... keeping in mind that it also has access to data
from Gmail, Android devices, Gdrive, maps/navigation, etc. Same with
FB/Meta. Training discussions just based on social network
interaction (no matter what your opinion might be about this) is
going to be quite interesting as well.

In short, it's likely that the commoditization of LLM engines through
open source is going to ensure that the ensuing battle is about data
sets. This will be interesting to watch.
[Reply to this comment]
Democratizing AI with open-source language models

Posted May 20, 2023 17:16 UTC (Sat) by rra (subscriber, #99804) [Link
]

I think there's a widespread assumption right now that larger is
better, but part of what I gathered from the article is that it's far
from clear this is truly the case. Those large commercial models from
companies like OpenAI and Google have a massive problem with
inventing totally fake information (which their marketing departments
are desperately trying to get everyone to use the term
"hallucination" for instead of the more accurate term "bullshit"). I
suspect there are a lot of causes for that and some of them are
inherent in the structure of the model, but it seems likely that part
of the problem is that shoveling everything on the Internet, true or
false, into the model produces a model that is not weighted for truth
over falsity.

Smaller models aren't as splashy because they are less likely to be
able to hold a general conversation about any topic while being
confidently wrong. But unless the goal is to replace business
executives and politicians with AI, it's not clear that application
is useful for more than entertainment. Smaller models carefully
trained on more accurate and vetted data may be more useful for
practical applications that don't make for good viral social media
but that do help solve real-world problems.
[Reply to this comment]
Democratizing AI with open-source language models

Posted May 20, 2023 17:46 UTC (Sat) by Wol (subscriber, #4433) [Link]

Or in other words, "Garbage In Garbage Out" is as true for AI as it
is for anything else. :-)

Cheers,
Wol
[Reply to this comment]
Democratizing AI with open-source language models

Posted May 20, 2023 18:06 UTC (Sat) by smurf (subscriber, #17840) [
Link]

Most people recognize obvious garbage/fantasy/lies. They've been
labelled as such, implicitly, since early childhood. Somehow, your
fantasy pet dragon doesn't elicit quite the same reaction from your
parents as a real-world cat, esp. when it has kicked the vase off the
living room table; also, you might have noticed that grown-ups are
much more concerned about the monsters at the IRS than the ones under
your bed.

There are no labels whatsoever on the relevant parts of the AI
training data. It's all a heap of text scraped off the WWW. At best,
some labels are haphazardly glued on after the model has been
trained, but that's much too superficial. You obviously can't get
deep categorization that way, and thus can't prevent the model from
hallucinating. Or spewing bullshit. Take your pick.
[Reply to this comment]
Textual AIs beyond chatting

Posted May 18, 2023 12:12 UTC (Thu) by smurf (subscriber, #17840) [
Link]

Language models are just the first step, though. Is there anything
out there beyond the ubiquitous chatbot?

For images we have the "here's a rough outline of the contents, and
some hints about the style I want, now draw a pretty picture" AIs,
Midjourney and Stable Diffusion and such, with selective refinement
like inpainting.

Is there anything out there that does the same thing with text? For
example, takes an outline and some examples of style to produce some
text that's actually coherent? Like, highlight a terse description of
a scene or two and ask the machine to "expand this to ten pages and
introduce Joe Evilmonger"?

[Reply to this comment]
Textual AIs beyond chatting

Posted May 18, 2023 14:46 UTC (Thu) by hkario (subscriber, #94864) [
Link]

> Is there anything out there that does the same thing with text? For
example, takes an outline and some examples of style to produce some
text that's actually coherent? Like, highlight a terse description of
a scene or two and ask the machine to "expand this to ten pages and
introduce Joe Evilmonger"?

That's what ChatGPT can already do. You just have to ask it to do it.
Tell it to give you short answers, and it will provide you with
sentence or two for an answer. Tell it to generate long paragraphs
based on a concept and it will do that instead.

Same error checking, grammar, typos, etc. Ask it to proofread
following text and it will give you suggestions.
[Reply to this comment]
Democratizing AI with open-source language models

Posted May 19, 2023 2:27 UTC (Fri) by aiethics.abhishek (subscriber,
#164948) [Link]

Something else to consider in this discussion is how we think about
democratization in the first place. We mostly focus on underlying
data and whether trained weights are released. But, democratization
can take on other meanings, for example, on profits, use, and
governance, as articulated here
[Reply to this comment]
Democratizing AI with open-source language models

Posted May 19, 2023 9:32 UTC (Fri) by tzafrir (subscriber, #11501) [
Link]

From what I understand, making "AI" a commodity means that it is not
a product you can control, gatekeep and profit from.

(Or rather: higher-end will still require beefy servers, but "good
enough" will become available to jut about any free software and
startup, just like Linux is)

Will that actually happen? That's to be seen. But that is the sense
of the word here.
[Reply to this comment]
Democratizing AI with open-source language models

Posted May 21, 2023 1:15 UTC (Sun) by NightMonkey (subscriber, #
23051) [Link]

    A neural network is a learning algorithm inspired by how our
    brains function.

So, yeah. I know, you might think this is semantics, but, it should
really say:

    A neural network is a learning algorithm inspired by a model of
    how brains function.

Yes, there are still too many unknowns for us to just declare we know
how the brain works. :) So, computational models of neurological
processes are very likely wrong insofar as how they mimic the brain
and its components. I just have to say this because the hubris in
this field of computing is breathtakingly large, outstripping what
medical and biological sciences support. Cheers, and thanks for a
nice read. :)
[Reply to this comment]
Democratizing AI with open-source language models

Posted May 21, 2023 1:24 UTC (Sun) by NightMonkey (subscriber, #
23051) [Link]

Don't miss that there are also lots of poorly paid humans, hired in
countries without a history of worker protection, assisting the
proprietary "A.I." shops to trim the rough edges of their bots
output...

https://time.com/6247678/openai-chatgpt-kenya-workers/

Can open frameworks and corpus help here? Not so sure myself... But I
think this should be considered in comparing the two licensing and
"intellectual property" models.
[Reply to this comment]
Democratizing AI with open-source language models

Posted May 24, 2023 16:24 UTC (Wed) by jezuch (subscriber, #52988) [
Link]

I'm late to the party but I have to note that reading this,
especially the last part, eerily reminded me of https://qntm.org/
mmacevedo
[Reply to this comment]

                  Copyright (c) 2023, Eklektix, Inc.
   Comments and public postings are copyrighted by their creators.
          Linux is a registered trademark of Linus Torvalds