[HN Gopher] Kyutai AI research lab with a $330M budget that will...
___________________________________________________________________
Kyutai AI research lab with a $330M budget that will make
everything open source
Author : vasco
Score : 242 points
Date : 2023-11-19 11:48 UTC (11 hours ago)
(HTM) web link (techcrunch.com)
(TXT) w3m dump (techcrunch.com)
| treprinum wrote:
| $330M is like 33 training runs of a large LLM. Not sure that
| would bring open source anywhere near GPT-3.5.
| abiraja wrote:
| Have you tried Mistral?
| dannyw wrote:
| Mistral is genuinely groundbreaking, for a fast, locally-
| hosted model without content filtering at the base layer. You
| can try it online here: https://labs.perplexity.ai/ (switch
| to Mistral)
| anonzzzies wrote:
| Are more companies/teams than the creating team working to
| get this to copilot/chatgpt standards?
| js4ever wrote:
| Wow I was not expecting this, It's really something else in
| terms of speed, and results are not bad! Will test it more
| eropple wrote:
| It's very fast, but it doesn't seem very good. It doesn't
| take instruction well (acknowledges and spits back the same
| wrong stuff) and doesn't seem to have much of a corpus or
| it's dropping most of it on the floor because it
| successfully answers zero of my three basic smoke-test
| questions.
| sigmar wrote:
| >doesn't seem to have much of a corpus
|
| what do you mean by 'corpus'? It is only 13GB so
| questions that require recalling specific facts aren't
| going to work well with so little room for 'compression',
| but asking mistral to write emails or perform style
| revisions works quite well for me
| mark_l_watson wrote:
| Are you running mistral-7B or mistral-7B-instruct?
| audessuscest wrote:
| Thanks for the link, do you know any other similar services
| that support fine-tuning ?
| gwervc wrote:
| Maybe at today cost. But this is a lab about AI not just LLM.
| Also given that French research is underfunded since decades,
| our researchers are accustomed to producing good results at a
| budget cost.
| nwoli wrote:
| Also has incredibly good math institutions
| umanwizard wrote:
| "French research is underfunded since decades" is a French-
| ism, just so you know. Normally native English speakers would
| say "French research has been underfunded for decades".
| Obscurity4340 wrote:
| Depuis ?
| umanwizard wrote:
| That can be used with either a duration ("depuis des
| decennies") or a point in time ("depuis 1995"). English
| "since" can only be used with a point in time.
| Obscurity4340 wrote:
| Merci, francais ami!
|
| Edit: is that correct or is there an apostrophe Im
| missing? It would better if you were francaise, n'est-ce
| pas?
| umanwizard wrote:
| I think "merci, mon ami francais" would be more correct,
| but at any rate, I'm not French!
| trealira wrote:
| I've noticed a lot of comments on Hacker News that use this
| construction. I don't point it out because often people
| dislike unsolicited advice about their English.
|
| Spanish speakers also make this mistake. I haven't met
| German speakers, but they also say the equivalent of "is
| (adjective) since (time)" in German. It makes me wonder if
| English is just unusually strict about this distinction
| compared to other European languages.
| vunderba wrote:
| I am also not a huge fan of unsolicited grammatical
| advice, because I feel like it's both low effort and
| derails the conversation. If the grammar error is more
| syntactical in nature and doesn't obscure the meaning of
| the thoughts intended to be conveyed, then let's just all
| _move on_.
|
| This is an international forum, we're well accustomed to
| being able to parse meaning despite a few inconsequential
| grammatical issues.
| umanwizard wrote:
| I pointed it out because when I lived in France, I
| personally appreciated people pointing out grammatical
| mistakes I made in French. If the OP doesn't care about
| sounding more like a native English speaker, that's
| perfectly fine and he can just ignore my comment.
| bee_rider wrote:
| People can just not engage in conversations they don't
| like. This forum has a branching structure, it is easy to
| hide a topic that you don't like.
|
| If someone was to be really odious and hateful, the
| presence of that sort of thing could be harmful, but this
| seems like a polite correction of a common problem. And
| it is an international forum, there are plenty of people
| who might get something from language-chat.
| lucubratory wrote:
| As someone learning a language that's not my native one,
| people who correct my Chinese kindly are the absolute
| gold standard. People who correct it rudely are rare but
| still more helpful than people who say nothing and just
| move on. This is very common in language learning
| communities. My default assumption is that other people
| who are learning a language that I know natively probably
| feel similar, and would appreciate kind, contextual
| corrections. I am happy to adjust if someone lets me know
| they don't want that or if it's against the rules of a
| specific forum.
| umanwizard wrote:
| I think most European languages lack the distinction
| between "he did X" and "he has done X". Actually, a lot
| of languages (including Spanish, German and French) have
| a distinction like this syntactically but it doesn't
| really mean the same thing as it does in English.
| SamBam wrote:
| Huh, I just had a fairly long and fruitless conversation
| with ChatGPT 4 trying to understand whether this was
| true. ChatGPT kept insisting that romance languages
| distinguish between the preterite ("he did," simple past
| tense) and the present perfect ("he has done"), but every
| time I asked it to give me specific examples it would
| start translating them, find that they were the same, and
| then say "well, in Italian [or French] it's actually more
| about context..."
|
| Finally it was able to give me the Spanish "El hizo esto"
| vs "El ha hecho esto," but admitted that whether one used
| one vs the other was quite regional.
| bee_rider wrote:
| I wonder if it is just the result of applying some
| transformation to "was underfunded for decades" in an
| attempt to make it cover the present.
| boisgerault wrote:
| A Gallicism :)
| nmfisher wrote:
| If they're researching avenues other than LLMs, that money goes
| a lot further.
|
| We can only hope.
| espadrine wrote:
| During the conference where it was announced, they indicated
| they have a partnership with Scaleway to access their Nabu 2023
| supercomputer[0]. I expect it will be similar to the
| relationship between OpenAI and Microsoft Azure, with free
| credits as part of their investment in exchange for being at
| the forefront of AI datacenter design. Indeed, Xavier Niel is
| one of three investors in Kyutai, and founded Scaleway.
|
| [0]: https://www.scaleway.com/en/news/scaleway-releases-the-
| detai...
| qwertox wrote:
| Maybe related:
|
| EU is starting a program where AI startups will get access to
| their supercomputers and also get a chance to win prizes.
|
| 16 November 2023: _Commission opens access to EU supercomputers
| to speed up artificial intelligence development_ [0]
|
| - Launch of the Large AI grand challenge: This competition -
| launched today, is a collaboration led by the EU funded project
| AI-BOOST, with access to the European Supercomputers being
| facilitated by the EuroHPC JU. It encourages the wide
| participation from European start-ups with experience in large-
| scale AI models. The winners are expected to release the
| developed models under an open-source license for non-
| commercial use, or through publishing their research findings.
| The challenge will select up to four promising European AI
| start-ups that will be given access to EuroHPC supercomputing
| facilities to foster the development of large-scale AI models
| in Europe and a EUR1 million prize will be distributed among
| the winners.
|
| - Opening up European supercomputer capacity: Access will be
| established for ethical and responsible AI start-ups, enabling
| them to efficiently train their models using European
| supercomputers.
|
| - Enhanced activities and services: the EuroHPC JU will advance
| activities and services powered by High-Performance Computing
| to foster trustworthy AI in Europe. These efforts will aim to
| facilitate increased accessibility for AI communities and
| promote the optimal and efficient use of HPC technologies for
| scientific and industrial innovation.
|
| [0]
| https://ec.europa.eu/commission/presscorner/detail/en/ip_23_...
| konschubert wrote:
| First they set up massive barriers such that you'd be insane
| to start an AI company in Europe. And then they dole up
| subsidies.
|
| Makes no sense. That money will go to waste.
| zirgs wrote:
| What massive barriers are there?
| JumpCrisscross wrote:
| > _What massive barriers are there?_
|
| Starting and scaling a business in Europe is harder than
| in America. This is true for fundamental reasons, like
| language and national borders. It's also true due to
| regulation and the culture of work and towards commerce
| in general.
|
| Some of those come with reasonable tradeoffs, _e.g._ in
| respect of employee protections. Many do not,
| particularly when it comes to licensing, bureaucracy and
| the peculiar way most European tax law and regulation
| tries to compensate for its licensing and bureaucracy by
| adding more bureaucracy in front of a balancing subsidy.
| carstenhag wrote:
| Opinion of a german here: Build & break things (without
| caring for laws or implications) is what the US and their
| companies do. Often it works, and sometimes breaking the
| law makes a new law pop up to legalize it, but that's not
| a good thing.
|
| Yes, it's a big annoying to start a company in some EU
| countries. No, it's not as bad as HN claims.
| 8n4vidtmkvmk wrote:
| Working for a big American company.. we have to abide by
| a lot of laws, including the EU ones.
|
| Small companies can get away with it for awhile, but not
| forever
| imjonse wrote:
| Requiring companies to offer more than two weeks of paid
| time off per year.
| gentleman11 wrote:
| Sorry, could somebody show a source for that? An $11M bill,
| that's nuts and just need a reference lol
| tyingq wrote:
| There's some compute also, which may not be counted in the 330M
|
| _" Co-founded by the Iliad group...To do this, the laboratory
| will use the computing power made available to it by Scaleway,
| an iliad Group subsidiary. Scaleway's supercomputer has the
| highest-performance computing power for AI applications
| deployed to date in Europe."_
|
| https://kyutai.org/CP_Kyutai_AI_EN.pdf
| coding123 wrote:
| Seems like it could be unlimited if people just bought hardware
| logicchains wrote:
| > $330M is like 33 training runs of a large LLM. Not sure that
| would bring open source anywhere near GPT-3.5.
|
| Open source is already quite near GPT-3.5; it's reaching GPT-4
| level that is the challenge.
| eschluntz wrote:
| Hopefully it's commercial open source, not for research only
| kaliqt wrote:
| Yeah, very tired of everyone trying to pass off "source
| available" licenses as "open source" licenses and getting away
| with it.
| yieldcrv wrote:
| the "and" in FOSS is actually doing alot of heavy lifting, as
| open source really does only mean source available
|
| yes, I want FOSS too
| babyeater9000 wrote:
| I just want free software. The oss is redundant to me.
| gchamonlive wrote:
| Why is it redundant? Redundant to what? What would
| account for what opensource brings without opensource
| itself?
| meiraleal wrote:
| The F, free software. opensource(r) isn't needed but it
| became a bigger brand than free software and everything
| with source available is called open source nowadays
| thinkmassive wrote:
| Free Software is inherently open source
| meiraleal wrote:
| Yes. The problem is that open source isn't free software.
| babyeater9000 wrote:
| You have free software. Free software is pretty rigidly
| defined. You also have open source software, which people
| also seem to think is defined. I'm my opinion, the
| concept of open source software is vague enough that its
| definition is open to interpretation. Look at the people
| claiming that source available software is open source.
| Source available software is, in fact, open source
| software, even if it's not compatible with copyleft. Free
| software is not open to interpretation. Open source
| software can be free software, but some software can
| rightfully be called open source software even if it
| isn't free software. So, if we are using the terms
| interchangeably because they are the same thing, then
| open source is a redundant term. If open source software
| and free software are not the same, which might be the
| case sometimes, then I want free software. I'm not a
| programmer. I don't care to make money from software and,
| frankly, I don't care about the money making aspect of
| software. Open source stuff, to me, reeks of corporate
| capture. I don't want telemetry, or to be bled
| financially to use a product. I don't believe that
| software is or can ever be a product. Algorithms
| shouldn't be copyrighted even if they are wrapped in a
| programming language. I don't care about implementation.
| I think this is a case of A is B and B is sometimes A.
| It's the sometimes case that really bothers me.
| https://www.gnu.org/philosophy/open-source-misses-the-
| point....
| JumpCrisscross wrote:
| > _I don 't believe that software is or can ever be a
| product_
|
| You must mean should be, since we have decades of
| evidence to the contrary.
| babyeater9000 wrote:
| I mean that software, in it's written form, is the
| documentation of knowledge from software development, a
| service. I view sciencing as another service that
| produces knowledge. Knowledge has zero cost of
| duplication and, as such, cannot be considered a product.
| Artifacts that are produced by the application of
| knowledge are products because they have a non-zero cost
| of duplication. Computer hardware is an example of a
| product. I don't view intellectual property as property
| either. Software, in my opinion, isn't a product.
| Software is knowledge. I don't claim to be correct. I'm
| attempting to share my point of view. Anything with zero
| cost of duplication isn't a product in my mind because
| these things are infinitely copyable once created. Once a
| mathematician discovers a math they don't retain rights
| to it. Charging money for software is, in my view, no
| different than trying to make people pay for secret
| knowledge. You might be able to keep the secret locked
| down for a while, but it will get out eventually.
| Knowledge is the closest thing we have to magic, and if
| we choose to view it through the zero sum lens of
| capitalism, I think that does society a disservice in the
| long run. If I were a wizard I would share the magic, not
| try to charge money to teach people a spell or two. It
| might be the case that all products are knowledge given
| form, but keep in mind that knowledge exists before and
| after discovery and its fruits/artifacts must be created
| with work.
| ben_w wrote:
| The open source[0] is the only reason anyone has time to
| make most of the valuable free software.
|
| We can't all be like Donald Knuth or Simon Tatham making
| TeX and PuTTY as personal projects.
|
| [0] specifically the freedom to fork, to develop further,
| and to make new releases that others can also build upon,
| which means I aver that many of the public AI models are
| sufficiently open that they're de facto open source even
| if the licensing isn't there.
|
| Even if it's a de jure violation of the copyright to make
| a derivative, I'm not sure you could prove that had
| happened when all the weights are floating point numbers
| you can randomise slightly as a first step -- if training
| just happens to move them back to the original values,
| well, that's just evidence the optimiser was working.
| gentleman11 wrote:
| Money is important to find ventures, but the open source
| aspect is important for guaranteeing user freedom in the
| long run in our society
| Lapha wrote:
| >as open source really does only mean source available
|
| The definition and history of the term as a licence is
| unambiguous in that the only restriction on redistribution
| is that it contains the source code under the same licence.
| There are senior engineers alive today that weren't even
| born when this was the commonly understood meaning of the
| term, it's not a new concept.
|
| The term and usage is being co-opted these days but that's
| bound to happen when it's not a legally protected
| definition. Give it another 10-20 years and I'm sure we'll
| be having the same argument over whatever term ends up
| replacing 'open source'.
| yieldcrv wrote:
| I'm all for "language evolving"
|
| Language exists to convey a shared concept
|
| In this case, the evolved version of open source fails to
| convey a shared concept in comparison to the prior term,
| "free and open source software" or FOSS for a shorthand
| adjective
|
| Here, people with knowledge of the lexicon are using it
| accurately, and people without knowledge of the lexicon
| or its etymology are complaining when they should be
| pushing for FOSS instead of getting surprised everytime
| ekianjo wrote:
| you can thank the "open source" movement for muddying the
| waters
| eropple wrote:
| The OSD is plenty clear.
| ekianjo wrote:
| except in the name
| Lapha wrote:
| 'Free' software has problems with its name too. The ones
| muddying the waters are people and companies releasing
| source code with a proprietary licence while trying to
| latch onto the open source branding.
| Kiro wrote:
| Getting away with what? The default is closed source.
| anonzzzies wrote:
| Which is basically the same for most companies. Too
| dangerous to even _read_ the source.
| sp332 wrote:
| Getting away with calling it "open source".
| ctxc wrote:
| Source available but with a pricing for commercial use seems
| to be the most sustainable way for open source (pun intended)
| though.
| api wrote:
| But that gets in the way of open source as free labor for
| giant companies.
| lsaferite wrote:
| I feel like the license used on Llama covers that, right?
| Carve out an exclusion for large companies so they have
| to obtain a commercial license and let all the ones below
| that access it for free.
| JumpCrisscross wrote:
| > _Carve out an exclusion for large companies so they
| have to obtain a commercial license_
|
| And this is why open source means source available, look
| at the license. There are infinite variations between
| free software and classic source available.
| KolmogorovComp wrote:
| You can (and most likely should) still use a OSI-approved
| license if you want, such as AGPL/GPLv3, while providing
| the possibility of case-to-case commercial licenses.
|
| gifski [0] for example is a successful open-source project
| doing that.
|
| [0] https://gif.ski/license.html
| BrianHenryIE wrote:
| I just see now, there are two Gifskis, one Rust, one
| Swift. It's worth looking at the maintainer of the Swift
| one, he has 1000+ repos, many with 1000+ stars.
|
| * https://github.com/ImageOptim/gifski
|
| * https://github.com/sindresorhus/Gifski
|
| * https://github.com/sindresorhus
| nativeit wrote:
| He appears to be the original creator of the "Awesome X"
| repo: https://github.com/sindresorhus/awesome
| KolmogorovComp wrote:
| Both gifski are maintained by @sindresorhus, as you can
| see by commit counts. (Even if he's not the main
| maintainer of the swift version).
| JumpCrisscross wrote:
| > _very tired of everyone trying to pass off "source
| available" licenses as "open source"_
|
| This train has passed. Sort of like crypto/crypto. Open
| source functionally means source available; the rigid
| definition remains free software.
| edouard-harris wrote:
| I've heard the licenses Meta releases its AI models under
| described as "open access", to differentiate them from the
| stricter "open source" definition.
| gumballindie wrote:
| If it's out there then it's fair use isnt it? Or does that only
| apply to non ai people's ip?
| MacsHeadroom wrote:
| ML models are probably not actually copyrightable, for the
| same reasons their outputs aren't. So you can probably just
| ignore model licenses, yes.
| nwoli wrote:
| I mean I'm happy with just the weights if that's all they can
| do. If it's actually useful then you should be able to use it
| to build something important, not just make middlemen "shovels
| during goldrush" saas apps
| Obscurity4340 wrote:
| Can you provide a small snippet of what is meant by
| "weights", I assume that refers to some probabillistic ratios
| or something?
| IanCal wrote:
| The weights refers to the trained model.
|
| Lots of neural nets are pretty standard, and work by
| multiplying lots of numbers by other ones. The key is to
| figure out _which_ numbers to use. Those are the weights of
| each connection between the artificial neurons in the
| model.
|
| With them you can run the model yourself.
| layer8 wrote:
| https://deeplizard.com/lesson/ddd2drizla
| lukeschlather wrote:
| They've clearly said they are truly making this open source,
| the hope you're replying to is not about data vs. weights
| it's about licensing. Having all the weights and training
| data isn't really useful if the license prevents you from
| using it. In fact it's a problem since you might open
| yourself up to trouble just by reading it.
| YetAnotherNick wrote:
| I would want an open source company which has clear one time
| pricing for commercial uses. I think if a company could release
| state of the art open source model, 1000 compnaies could
| definitely pay $5000, making it much more sustainable.
| elorant wrote:
| If there was a black box of ChatGPT that doesn't need to
| connect to the Internet companies would gladly pay
| $50k/yearly to use it. There's huge demand but privacy
| concerns keep a lot of corporations away.
| modernpink wrote:
| What does the name mean? Presumably it's Japanese but not seeing
| anyone Japanese in the company
| thibaut_barrere wrote:
| I read somewhere else:
|
| > the answer is called Kyutai (pronounced "Cute AI" and meaning
| "sphere" in Japanese),
| lowkey_ wrote:
| Maybe not Japanese, but the French love Japanese culture.
| jpgvm wrote:
| The love appears mutual from my time in Japan.
| mrits wrote:
| That money will go fast with trying to poach out of industry ,
| buying nvidia hardware, and no clear direction. Hopefully it is
| just the fault of the author and not actually another case hoping
| a bunch of phds will come up with something given time and money.
| amelius wrote:
| Is the complete code open source? Or only the models?
| htrp wrote:
| How are they going to continue past the first set of model
| releases?
___________________________________________________________________
(page generated 2023-11-19 23:01 UTC)