Post AVJuLsb6vps1hld4YS by stablehorde@sigmoid.social
(DIR) More posts by stablehorde@sigmoid.social
(DIR) Post #AVJgxpWRPoiloSeAvA by simon@fedi.simonwillison.net
2023-05-04T16:16:57Z
0 likes, 0 repeats
Leaked Google document: “We Have No Moat, And Neither Does OpenAI”The most interesting thing I've read recently about LLMs - a purportedly leaked document from a researcher at Google talking about the huge strategic impact open source models are havinghttps://simonwillison.net/2023/May/4/no-moat/
(DIR) Post #AVJhJg1nOaKCjnjKm8 by adr@mastodon.social
2023-05-04T16:20:44Z
0 likes, 0 repeats
@simon holy shit this is *terrific*. and I mean just your blog post. Gonna dig into that document.
(DIR) Post #AVJhYvatnquHMKQDBo by luis_in_brief@social.coop
2023-05-04T16:23:11Z
0 likes, 0 repeats
@simon Pairs interestingly with Zuckerberg on open models in their earnings call: https://s21.q4cdn.com/399680738/files/doc_financials/2023/q1/META-Q1-2023-Earnings-Call-Transcript.pdf
(DIR) Post #AVJhp0WRFSjrHD0E9g by adamchainz@fosstodon.org
2023-05-04T16:25:41Z
0 likes, 0 repeats
@simon wow, open source wins again. Thanks for excerpting!
(DIR) Post #AVJi4cRcXK3NEaC32O by matt@toot.cafe
2023-05-04T16:27:50Z
0 likes, 0 repeats
@simon Does all of the work on top of LLaMA actually count? After all, that model was leaked out of Facebook.
(DIR) Post #AVJiQBoCFTXAzm2SLg by simon@fedi.simonwillison.net
2023-05-04T16:32:55Z
0 likes, 0 repeats
@matt it proved that it was all possible to run on end-user hardware - and the openly licensed trained-from-scratch LLaMA alternatives are already starting to emerge https://simonwillison.net/2023/May/3/openllama/
(DIR) Post #AVJifcFYDtSBLSBniC by matt@toot.cafe
2023-05-04T16:34:32Z
0 likes, 0 repeats
@simon Oh damn, I hadn't seen that post yet. Things are definitely heating up.
(DIR) Post #AVJirCWjgZp52fkfBI by nelson@tech.lgbt
2023-05-04T16:35:01Z
0 likes, 0 repeats
@simon thank you for highlighting this and summarizing some interesting points. I really appreciate the view you're giving in to current AI developments.
(DIR) Post #AVJjgEpasWtuxOuOqe by frijolito@infosec.exchange
2023-05-04T16:47:18Z
0 likes, 0 repeats
@simon I’m not understanding why this is a surprise if the larger companies are milking the models they have since it’s clearly providing a ROI and the open source communities are getting excited to innovate the underlying components
(DIR) Post #AVJkEB25VX3T1Qw5Qm by simon@fedi.simonwillison.net
2023-05-04T16:53:33Z
0 likes, 0 repeats
@frijolito until recently I thought that the cost involved in training a model would mean the open source community would always be several steps behind OpenAI and Google - apparently at least one person inside Google doesn't think that's true
(DIR) Post #AVJl04zdGXoSBsd3wW by overbyte@gamepad.club
2023-05-04T17:01:36Z
0 likes, 0 repeats
@simon This actual solves one of my fundamental problems with the current LLM tools like chatGPT and CoPilot: that you have to basically stream all of your content / code to Microsoft to use their tool. This seems to indicate that running an open source server would be entirely feasible. If the models are also trained using only correctly licenced material as well (rather than Microsoft buying github and ignoring the licences for the model) then we have a full house
(DIR) Post #AVJliNZjXSJcZ2dfSC by piccolbo@toot.community
2023-05-04T17:09:59Z
0 likes, 0 repeats
@simon The reading list alone is gold.
(DIR) Post #AVJp96ZPAhj3VKv8XA by jeancf@noc.social
2023-05-04T17:48:24Z
0 likes, 0 repeats
@simonLoRA is clearly a great tool but, to use an open source analogy, it feels like applying a kernel patch downstream: it gets the job done but at some point, if it is generic enough, it needs to be upstreamed. And that part is not possible with LoRA. To integrate the modification in the model, a full retraining is inevitable.
(DIR) Post #AVJqf10JV8a3SkHplw by matt@toot.cafe
2023-05-04T18:03:12Z
0 likes, 0 repeats
@simon After thinking about this a little more, I wonder if OpenAI still has a moat in GPT-4's ability to work with image inputs. The applications of that for accessibility sound really promising, though most of us don't actually have access to that feature yet, so I suppose it could turn out to be smoke and mirrors.
(DIR) Post #AVJqf1VVd9XB1UskOe by simon@fedi.simonwillison.net
2023-05-04T18:05:27Z
0 likes, 0 repeats
@matt they still haven't shipped that! Meanwhile there are already open models that can do that surprisingly well: https://simonwillison.net/2023/Apr/19/llava-large-language-and-vision-assistant/
(DIR) Post #AVJtTQWR6EAezdTfYu by matt@toot.cafe
2023-05-04T18:36:56Z
0 likes, 0 repeats
@simon Wow, yeah, that _is_ impressive. Can't wait to see what could be done with a model like that but fine-tuned for accessibility (e.g. render the UI in this image as something like an accessibility tree).
(DIR) Post #AVJuLsb6vps1hld4YS by stablehorde@sigmoid.social
2023-05-04T18:46:35Z
0 likes, 0 repeats
@simon and yet Google is instead tightening their grip harder!
(DIR) Post #AVJwsrtkkixd06CtOK by movonw@chaos.social
2023-05-04T19:14:57Z
0 likes, 0 repeats
@simon bazaar strikes back! 💥
(DIR) Post #AVK0CREJujGhwcqmOm by numist@xoxo.zone
2023-05-04T19:50:15Z
0 likes, 0 repeats
@simon tbh it's nice to see groups of researchers taking the lead on AI. it's not fun to imagine what the world would have been like had the Internet been the product of a race between two corporations
(DIR) Post #AVK1YIoE0G8y9WdMfI by erica_sea55@mastodon.social
2023-05-04T20:07:27Z
0 likes, 0 repeats
@simon oh wow, this is incredible, thanks!
(DIR) Post #AVK28UCZ7hMhuYGyVk by eichin@mastodon.mit.edu
2023-05-04T20:14:09Z
0 likes, 0 repeats
@simonFor an anonymous doc, isn't "Having read through it, it looks real to me" a point in favor of it being LLM-written? (Not quite a "tell" but a cause to go Hmmmm.)
(DIR) Post #AVK5SuWbGPjrCm4gXw by ppatel@mstdn.social
2023-05-04T20:51:17Z
0 likes, 0 repeats
@simon That document was the best reading of this week by far.
(DIR) Post #AVK9DYxbYbpPnTdvBw by jimgar@fosstodon.org
2023-05-04T21:33:02Z
0 likes, 0 repeats
@simon Hey Simon, I’ve been holding off the use of ChatGPT, Bard, etc., even though I think they could be useful. This is because I can see (especially with ChatGPT) the horrible unethical behaviour that the companies are using in their arms race to deploy deploy deploy. With all the talk in this leaked doc about open source alternatives, do you know of any LLMs that are “ethically sourced” and available for the average punter to use? I don’t want to be left behind :/
(DIR) Post #AVKFM0rqHanOOIMO2q by simon@fedi.simonwillison.net
2023-05-04T22:42:20Z
0 likes, 0 repeats
@jimgar the ethics of this stuff is incredibly complicatedI'm very optimistic about the models being trained on the RedPajama data - there's one out already and evidently more to follow very shortly https://simonwillison.net/tags/redpajama/
(DIR) Post #AVKFZAcTKtM0OvRdNQ by grandfunk@fosstodon.org
2023-05-04T22:43:06Z
0 likes, 0 repeats
@simon enjoyed this and your blog generally. Keep it up.
(DIR) Post #AVKFkMBsegRhTQLCHQ by simon@fedi.simonwillison.net
2023-05-04T22:44:28Z
0 likes, 0 repeats
Claude is an interesting option that's one of the most promising closed alternatives to ChatGPT - they have an interesting approach to AI safety which they call "constitutional AI" https://www.anthropic.com/index/introducing-claude
(DIR) Post #AVKUH8GUqk2hCO1T7Y by MudMan@mas.to
2023-05-05T01:29:23Z
0 likes, 0 repeats
@simon We don't talk enough about how one of the big bugbears at the start of the ML explosion was the assumption that these models would be stuck under corporate control forever because the tech would be proprietary and expensive to run.There is no correlation with the likelihood of the other risks, but I admit I was on board with that one but it didn't quite materialize.
(DIR) Post #AVKYRxOU0aHGU6Avx2 by shajith@mastodon.social
2023-05-05T02:16:08Z
0 likes, 0 repeats
@simon Excellent doc there. I keep thinking Google should respond to Meta’s stroke of luck with Llama by shipping a LLM browser API and local model work Chrome.
(DIR) Post #AVL0RRfujgG2pJsBkG by yogsototh@ieji.de
2023-05-05T07:29:27Z
0 likes, 0 repeats
@simon I am so glad because I read it cost about $80k to learn a full model. I expected, on the opposite, that open source could never reach the same quality. That really is a relief.
(DIR) Post #AVL1kNOixk7D18QJcm by jimgar@fosstodon.org
2023-05-05T07:44:27Z
0 likes, 0 repeats
@simon thank you so much, l’ll give these a look. Everywhere I look in tech it’s one ethical nightmare after another 😵💫
(DIR) Post #AVMR6UjF4xPw8Xckgy by resing@social.coop
2023-05-06T00:03:08Z
0 likes, 0 repeats
@simon what's your take on the copyrighted material included in RedPajama through CommonCrawl? It seems to me that one could train a model on only text that has been shared freely and that might be more ethical. cc @jimgar
(DIR) Post #AVMSWIgIarnCFu7GvA by simon@fedi.simonwillison.net
2023-05-06T00:19:12Z
0 likes, 0 repeats
@resing @jimgar I'm not convinced it's possible to train a usable LLM without including copyrighted material in they raw pretraining dataAs such, personally think it's a necessary evil to avoid a monopoly on LLM technology belonging to organizations that are willing to train against crawler data
(DIR) Post #AVMmVDpP9N15zu8evo by resing@social.coop
2023-05-06T04:03:05Z
0 likes, 0 repeats
@simon @jimgar not sure I follow. Are you saying that crawler data, which includes copyrighted material shouldn’t be used by commercial companies and LLMs are inherently flawed because of that? If so, I’m not saying you’re wrong, just trying to understand.
(DIR) Post #AVMnzsxWtG5uCl7lhY by simon@fedi.simonwillison.net
2023-05-06T04:19:52Z
0 likes, 0 repeats
@resing @jimgar I'm saying I'm not sure it's possible to build a useful LLM without including copyrighted data in the training setThe ethics of this entire field are incredibly murky - I wrote about that last year https://simonwillison.net/2022/Aug/29/stable-diffusion/#ai-vegan
(DIR) Post #AVNFLRqx33GfGnYOxM by jimgar@fosstodon.org
2023-05-06T09:26:12Z
0 likes, 0 repeats
@simon @resing it *all* feels fundamentally wrong, so long as the results rely on indiscriminate harvesting of people’s work without permission. Literally the only compelling argument I have heard is the “necessary evil” Simon mentions - doing it anyway but making it open source. I just find it sad that this is the position we’re in at all, and worse, how little the majority of people seem to care about providence and permissions full stop.
(DIR) Post #AVNihx6sPiynFk5NhY by joapen@masto.ai
2023-05-06T14:55:18Z
0 likes, 0 repeats
@simon very interesting post Simon, thanks for sharing
(DIR) Post #AVNqesBr32lyl8SNtI by simon@fedi.simonwillison.net
2023-05-06T16:24:23Z
0 likes, 0 repeats
@jimgar @resing search engines work by indiscriminately harvesting people's work without their permission, and have done for decadesWhat's different here isn't how the things are built, it's what they can be used forPeople mostly tolerated search engines because they saw them as useful - they helped people's work be found, they didn't (appear to) threaten their livelihoods
(DIR) Post #AVNqpxHyQos3l3bTJA by simon@fedi.simonwillison.net
2023-05-06T16:25:33Z
0 likes, 0 repeats
@jimgar @resing note that I'm not saying that search engines were morally/ethically pure here either!The ethics around this are deeply complicated - there are no easy or obvious answers
(DIR) Post #AVOJavhKxgTB9PedCC by miki@dragonscave.space
2023-05-06T21:48:18Z
0 likes, 0 repeats
@simon OpenAI could get a moat if they were willing to do more investments into the ChatGPT plugin ecosystem, especially if they added some kind of (embeddings-based) long-term memory.
(DIR) Post #AVOTm3eSjIX0XtFrJw by resing@social.coop
2023-05-06T23:42:34Z
0 likes, 0 repeats
@simon @jimgar the legal issue might be resolved soon. if @binarybits@instinctd.com is right, Stable Diffusion could lose the lawsuit against them. I buy his argument in favor of that. If that's the case, LLMs trained on sets that only allow that use might really take off https://arstechnica.com/tech-policy/2023/04/stable-diffusion-copyright-lawsuits-could-be-a-legal-earthquake-for-ai/
(DIR) Post #AVQCRbtGvQyDkXlXE0 by demiurg@fosstodon.org
2023-05-07T19:37:56Z
0 likes, 1 repeats
@simon You are mentioned in an article of Spiegel :) https://www.spiegel.de/wissenschaft/mensch/kuenstliche-intelligenz-es-rollt-ein-tsunami-auf-uns-zu-kolumne-stoecker-a-2410efbd-ab92-4c09-9cde-7d66ab4629c9