Post AT2lneOh2igSAP2Oky by abathur@hachyderm.io
(DIR) More posts by abathur@hachyderm.io
(DIR) Post #AT2aD0ofcz4FUIW0K8 by simon@fedi.simonwillison.net
2023-02-25T14:26:59Z
0 likes, 0 repeats
My favourite idea from this excellent @tomscott video about the new breed of AI tools is represented by this snapshot - where Tom asks if we are at the beginning, middle or end of the curve - and proposes that if we are just at the beginning, things are about to accelerate in very surprising ways https://youtu.be/jPhJbKBuNnA
(DIR) Post #AT2cChCYkqMsYufM92 by JackDeeth@mastodon.online
2023-02-25T14:49:22Z
0 likes, 0 repeats
@simon @tomscott tom scott dieshttps://www.youtube.com/watch?v=CsCS5BOjdgk
(DIR) Post #AT2cWzywdgYMbGylDU by gisiger@nerdculture.de
2023-02-25T14:53:05Z
0 likes, 0 repeats
@simon Interesting opinion piece. I just have one question: who on earth wants to archive this nuisance called email? I mean, if it is important information, I will put it somewhere. Sure. But I have a strict 5-year policy. Every January, all emails older than that are deleted. Gone. I have been doing this for over 10 years now and have never regretted it.
(DIR) Post #AT2cocA1wcGfvsZ26S by rolle@mementomori.social
2023-02-25T14:56:12Z
0 likes, 0 repeats
@simon Just watched this couple of days ago and posted about it in so similar manner than it kinda feels like some algorithm is watching me after all, heh. What a great insight that is. And Tom, he is a genious in so many ways. @tomscott
(DIR) Post #AT2d7Pok3q3X9VZUno by simon@fedi.simonwillison.net
2023-02-25T14:59:27Z
0 likes, 0 repeats
@gisiger I'm somewhat fascinated by the difference between archivists (like myself and Tom Scott) and - I'm not sure what the right word is here - non-archivists?The idea of throwing away anything I've spent even a second of time thinking about in the past feels so wrong to me!
(DIR) Post #AT2e9lohL4oDuoEngW by gisiger@nerdculture.de
2023-02-25T15:11:10Z
0 likes, 0 repeats
@simon Well, I am a big archiver and backuper. I am also an avid daily diarist. But emails? Why should I archive an email asking my plumber for a quote? Or one confirming a boring meeting? To me, email is what phone calls were decades ago: a daily nuisance, generally not worth preserving. But I get it, different people see things differently.
(DIR) Post #AT2ezWWDShkzWlGqtk by simon@fedi.simonwillison.net
2023-02-25T15:20:42Z
0 likes, 0 repeats
@gisiger to be honest I'm absolutely terrible at email - but I know Tom Scott runs his entire business out of it, so I imagine for him having a thread where he arranged a tour of an engineering facility with some expert eight years ago is super valuable to him
(DIR) Post #AT2h0adPqMo6UWH5e4 by bingaman@mastodon.social
2023-02-25T15:43:09Z
0 likes, 0 repeats
@simon @tomscott I wonder if there is something to being our age. Another revolution has absolutely started
(DIR) Post #AT2hCAVmnJKKwmE8R6 by college_physics@defcon.social
2023-02-25T15:43:31Z
0 likes, 0 repeats
@simon @tomscott the thing is, it is not digital tech as such, it is what one might "legal tech" that will shape how the information universe will evolve. In one sentence: "who gets legal access to what data (including code as data) and what can they legally do with it". The pattern of using a stream of data to predict an expected value is old, generic and has wide applicability. That is what banks have been doing with credit scoring (a sort of low IQ AI) for decades. It is a highly regulated business because it messes up with peoples lives. From adtech to crypto the tech industry in its various recent incarnations has been getting away messing with people's lives with impunity. We have been in sort of stasis for more than a decade not because there weren't good enough algos. The "system" is simply unable to find a consistent legal framework that is compatible with earlier notions of ownership, privacy and agency so it is simply kicking the can...
(DIR) Post #AT2hNfEktmM9QQk6Nc by 11thJeff@mastodon.social
2023-02-25T15:45:20Z
0 likes, 0 repeats
@simon @tomscott @tchambers What if we're at the other end? We've fed LLMs and transformer AIs one human internet's worth of knowledge, and we got out a few semi-cogent chatbots. We fed them all our artwork, and we got some models that can paint in a few styles. We fed them millions of hours of traffic telemetry, and got a car that can mostly drive itself until something weird or unexpected happens. But each of these has a hard limit well above an average human, well below a professional.
(DIR) Post #AT2hZEgtQFtOZKQCHY by simon@fedi.simonwillison.net
2023-02-25T15:48:26Z
0 likes, 0 repeats
@11thJeff @tomscott the thing I particularly liked about Tom's argument in the video is that the accelerating line on the curve is mainly about regular people figuring out new things they can do with the new technology, which then creates a wave of new innovation inspired by those discovered use-casesChatGPT only became available to the public on November 30th last year
(DIR) Post #AT2hxBnHqQMsXBpluy by simon@fedi.simonwillison.net
2023-02-25T15:51:18Z
0 likes, 0 repeats
@11thJeff @tomscott I expect that if we stopped training new models entirely today and just spent the next 12 months continuing to explore the capabilities of existing models like GPT-3.5 and Stable Diffusion and the various open source LLMs we would still see a huge increase in our understanding of ways we could use those models
(DIR) Post #AT2ifwwntFpxWId3WC by abathur@hachyderm.io
2023-02-25T16:01:50Z
0 likes, 0 repeats
@simon@11thJeff @tomscott if I had to find pearls to clutch, I think it'll be over whether these models will further ruin incentives on the internet. Like: make people leery of providing training data for their replacement, make it ~easier to launder bullshit/disinfo, create exponentially more synthetic content that makes the work of other mortals harder to find...Can the internet bear the weight of the models, and can they exist if that pro-social surplus dries up?
(DIR) Post #AT2jWBI1oKmH3nSj9E by simon@fedi.simonwillison.net
2023-02-25T16:11:30Z
0 likes, 0 repeats
@abathur @11thJeff @tomscott I like Tom's comparison to Napster as a harbinger of what was to come: MP3 sharing less to streaming which had massive societal implications - almost completely wiling out video rental stores for example
(DIR) Post #AT2jlHVsxE16hp90hE by 11thJeff@mastodon.social
2023-02-25T16:08:41Z
0 likes, 0 repeats
@abathur @simon @tomscott Also ChatGPT wasn't the first; GPT-3 has been out two and a half years; it was just too unwieldy to release to the public, and hadn't been pruned by as many human labelers working at $2 a day. So to me this feels more like the bend of the logistic curve, the peak of the hype cycle after the accel, where science journalists keep asking, what can't we do?Take a look at some old space science videos... we may have a generation or more to wait...https://www.youtube.com/watch?v=EgrdAUFFMrA
(DIR) Post #AT2jlHuLUHaBvgaXp2 by simon@fedi.simonwillison.net
2023-02-25T16:13:57Z
0 likes, 0 repeats
@11thJeff @abathur @tomscott maybe original GPT-3 was the FTP and newsgroups era of MP3 sharing - nerds could use it, but it was inaccessible to everyone elseChatGPT was the Napster moment when suddenly anyone sufficiently motivated but without deep technical skills could start using it
(DIR) Post #AT2jyyBLcl38nz463U by abathur@hachyderm.io
2023-02-25T16:14:48Z
0 likes, 0 repeats
@simon@11thJeff @tomscott oh--absolutely. But what if that video store is, say, stackoverflow or code blogging?
(DIR) Post #AT2kP23Pqmu6LLma36 by simon@fedi.simonwillison.net
2023-02-25T16:17:09Z
0 likes, 0 repeats
@abathur @11thJeff @tomscott right, exactly! The societal implications of this stuff look like they could be vast, unpredictable and very disruptive
(DIR) Post #AT2ke8yAQCSE8PX7SK by 11thJeff@mastodon.social
2023-02-25T16:15:57Z
0 likes, 0 repeats
@abathur @simon @tomscott @tchambers And the limiting factor in that space video, it turns out, was mass to orbit. We had no economic way to get building materials that large or bulky into orbit for less than millions of dollars, thousands per kg. Recent years, we've shrunk that considerably, but still nowhere near what's needed for those designs.The corollary with AI is processing power and training data. Already at the limit of Moore's law, working in single nm, crunching a whole internet..
(DIR) Post #AT2ke9TMYDPLhA8252 by simon@fedi.simonwillison.net
2023-02-25T16:18:52Z
0 likes, 0 repeats
@11thJeff @abathur @tomscott @tchambers some of the most interesting recent papers have been about massive optimizations to enable existing models to run on a tenth of the hardware - these for example:- https://simonwillison.net/2023/Feb/21/flexgen/- https://simonwillison.net/2023/Feb/24/introducing-llama/
(DIR) Post #AT2lOF67y6zmSjEpea by 11thJeff@mastodon.social
2023-02-25T16:32:26Z
0 likes, 0 repeats
@simon @abathur @tomscott @tchambers See the biggest change might just be that I'll ask a chatbot to generate an overview of compact transformer implementations, instead of going to a private blog written by a human. Automated content upon request, instead of human content on contingency. And it probably won't be as good, but it'll be good enough, like a Google search is good enough not to bother a reference librarian. Wonder if we're already there, if we're shrinking the implementation already
(DIR) Post #AT2lneOh2igSAP2Oky by abathur@hachyderm.io
2023-02-25T16:36:50Z
0 likes, 0 repeats
@simon@11thJeff @tomscott I guess I like the question of whether the internet can bear their weight and whether they can exist without its surplus of ~free ~well-intentioned-if-not-correct human work to train on worth pearl-clutching because it asks how they'll influence the kind of pro-social work that has brought us here and vice-versa.It seems at least plausible that their existence and quality will be in tension with the conditions needed to create them?
(DIR) Post #AT2mm6micytjaH6VYe by misc@mastodon.social
2023-02-25T16:47:45Z
0 likes, 0 repeats
@simon @11thJeff @abathur @tomscott It's a weird analogy because Napster didn't enable the creation of new tools. It awakened and highlighted a hunger for unlimited music libraries on demand, which first developers and then capitalists responded to. It also wasn't that unpredictable.
(DIR) Post #AT2n22k3QXopg7SKZc by simon@fedi.simonwillison.net
2023-02-25T16:50:45Z
0 likes, 0 repeats
@misc @11thJeff @abathur @tomscott Napster was the earliest indication that there was massive demand for on-demand availability of media online - maybe unsurprising now, but I don't think many people were confidently predicting it before it happened
(DIR) Post #AT2nKq2mrU1fnaBF0y by abathur@hachyderm.io
2023-02-25T16:51:25Z
0 likes, 0 repeats
@simon@11thJeff @tomscottThere's probably a good story in a 24th-century where AI research funding got choked out by the collapse of the open resources they need for training and never bootstraps to something like single-shot learning. There are still AIs, but they're all trained on early 21st century data sets and understanding them is like reading Lord Byron.
(DIR) Post #AT2nKqeMbm5Fg7lFaK by simon@fedi.simonwillison.net
2023-02-25T16:54:23Z
0 likes, 0 repeats
@abathur @11thJeff @tomscott I'm still looking forward to seeing what happens if someone trains a language model entirely on out-of-copyright data, so it has a decidedly 19th century tone of voice (and deeply upsetting social biases)
(DIR) Post #AT2nZddLHl8FeVKsnw by misc@mastodon.social
2023-02-25T16:56:56Z
0 likes, 0 repeats
@simon @11thJeff @abathur @tomscott Yes I agree. But if the argument is simply that ChatGPT is spotlighting a demand, which entrepreneurs will arise and compete to meet - well first of all, I don't yet see an analogous new market demand that's nearly as clear cut. And second, AI skeptics get accused of moving goal posts, but on the scale of AI hype, this seems like a pretty humble claim.
(DIR) Post #AT2nlOfgjxASit2YAi by abathur@hachyderm.io
2023-02-25T16:58:15Z
0 likes, 0 repeats
@simon@11thJeff @tomscottThis reminds me of the work Franco Moretti did in Distant Reading, where he's using statistical methods to "read" 10s of thousands of novels (iirc maybe 19th century? It's been a bit) that humans rarely if ever read anymore.
(DIR) Post #AT2qOrfn2wHxs82EvA by simon@fedi.simonwillison.net
2023-02-25T17:28:17Z
0 likes, 0 repeats
@misc @11thJeff @abathur @tomscott Tom's video asks the question "are we at the beginning, the middle or the end of the curve" and states that we will be able to look back at this moment in time with certainty in a few years to answer that question, even if we can't answer it nowI'm currently betting on beginning of the curve and investing my research efforts appropriately - would be very happy to be wrong about this
(DIR) Post #AT2rHtix7eAY9zkwYS by misc@mastodon.social
2023-02-25T17:38:00Z
0 likes, 0 repeats
@simon @11thJeff @abathur @tomscott It's an interesting question, and I agree that the answer is yet unclear. I just think the specific analogy was confused.
(DIR) Post #AT2rTTWiLj0ENxYJA8 by jaythvv@infosec.exchange
2023-02-25T17:39:07Z
0 likes, 0 repeats
@simon @tomscott middle/end, i reckon. current architectures are refinements of ideas that aren't new - there is just a lot more compute poweri reckon not too much too worry about when a model that takes millions to train just makes some pretty (and weird) pictures and text
(DIR) Post #AT2sjzEcKL0VJWWano by simon@fedi.simonwillison.net
2023-02-25T17:54:46Z
0 likes, 0 repeats
@jaythvv @tomscott I think we are only just beginning to see what regular people use these models for - even if we stopped training new models entirely I think we've only just begun to see how people are going to put them to work
(DIR) Post #AT2svF7JcLuDl0TcKO by 11thJeff@mastodon.social
2023-02-25T17:55:55Z
0 likes, 0 repeats
@simon @misc @abathur @tomscott If you're wrong, I don't think you'd be quite as happy. Because it means you're late. Happiest of you're right, anyway.
(DIR) Post #AT2tJQQHEakjltsQUq by jaythvv@infosec.exchange
2023-02-25T17:56:41Z
0 likes, 0 repeats
@simon @tomscott that's fair. but that's an adoption acceleration curve, rather than a technology acceleration
(DIR) Post #AT2tVACdMkP8wP5oIq by simon@fedi.simonwillison.net
2023-02-25T18:03:20Z
0 likes, 0 repeats
@jaythvv @tomscott yeah in Tom's video he's talking about adoption more than development - though he argues that increased adoption leads to increased investment which leads to accelerated development effort
(DIR) Post #AT2yweGgBwJbmBuUaW by MudMan@mas.to
2023-02-25T19:04:01Z
0 likes, 0 repeats
@simon @tomscott I'm late to this, but to me the key image is this.Guy's entire premise is that Napster was the first major "disruptor" in 1999, a year after Google launched, mid-way through the dotcom crash.So maybe the thought that we don't know where in the curve we are doesn't hold that much water, considering.The guy is great at talking, but there's some irony in how much video essay discourse in general mirrors some of the "confidently wrong" stuff about AI sometimes.
(DIR) Post #AT2zLoGc9XBi1jsT4K by simon@fedi.simonwillison.net
2023-02-25T19:08:41Z
0 likes, 0 repeats
@MudMan @tomscott what's wrong with his argument that Napster represented a major disruptive moment?Prior to the Napster revolution the only way to obtain music was to go to a record shop and buy a physical CD!
(DIR) Post #AT2zX4bfk6lttB68nY by MudMan@mas.to
2023-02-25T19:08:31Z
0 likes, 0 repeats
@simon @tomscott In 1999 we were having serious conversations about algorithmic filters in search engines impacting the political landscape. By 2001, when Napster's legal trouble picked up people on TV were broadly discussing the possibility of VoD as a service.ML-based tools have been in play in academia and art for several years (DLSS 1 is what? 2019?). When Youtubers notice we're in the middle of the curve. That's a decent assumption. Doesn't mean you know where the crest is, though.
(DIR) Post #AT2zX5gJkK57DsHMg4 by simon@fedi.simonwillison.net
2023-02-25T19:10:33Z
0 likes, 0 repeats
@MudMan @tomscott "Doesn't mean you know where the crest is, though."I thought that was the main argument made in the video: no one can know today what point of that curve we are on right now... but if it turns out we are right at the start of it then things are going to be very unpredictable and weird for the next few years
(DIR) Post #AT2zqoEcK3bou7j0nQ by MudMan@mas.to
2023-02-25T19:14:23Z
0 likes, 0 repeats
@simon @tomscott For the music industry, sure. It was a moment where the already existing MP3 trading went mainstream and it was obvious that it was time to migrate it, which Apple, not Spotify, did first.But everybody knew the transition was gonna happen at that point, the old music industry just didn't have an incentive.Napster wasn't ChatGPT, it was Tesla. We all knew cars needed to go electric, the old school just didn't want to invest in going first. It's a bad analogy.
(DIR) Post #AT304MykSLcgQRliK0 by MudMan@mas.to
2023-02-25T19:16:06Z
0 likes, 0 repeats
@simon @tomscott Bit of nuance, though. His argument is we don't know if we're at ground floor. We do. We know where this tech was a year ago. Two years ago. The applications have been defined, we are in iteration mode.We don't know how good we'll get during iteration, so of his three scenarios, two remain in play. And there may yet be another big breakthrough before we're done, as there can be at any point.But it's not the pit of uncertainty he describes, we know how fast we're iterating.
(DIR) Post #AT3SyvcrcjxyoZI2fg by mskeggs@mastodon.social
2023-02-26T00:40:34Z
0 likes, 0 repeats
@simon @tomscott It’s funny watching comments decay into whether the analogy about MP3 sharing is apt. In my job, we use tools that could have been put together with PHP in 2000, but only emerged a few years ago, as it takes time to have all the ideas (even when people will say in the future “it was obvious”). Even with the well publicised issues around AI search, it seems like there is plenty still to be invented with the current gen AI tools, let alone next gens.
(DIR) Post #AT3YAgLakqUsftdQrg by smy20011@m.cmx.im
2023-02-26T01:36:00Z
0 likes, 0 repeats
@simon @tomscott The curve question is interesting. The GPT* technology maybe at the end of curve. GPT3 and GPT3.5 are nearly the same model with different training process. The reason why ChatGPT is widely adopted is because the better UX/UI and free for anyone to use.Stable diffusion is a different story. They propose a new model and a new way of generating images. A huge amount of work built on stable diffusion and improve its ability to generate images. However, you need to change the model or retrain it more data in order to achieve that.For LLM (generate one word at a time), we may at the end of curve since we are using it for 10+ years. A new architecture may needed to solve all the problem that LLM have.A lot of work is trade off between accuracy & speed & creativity of LLMs. The real breakthrough are the ones that push the trade off curve.
(DIR) Post #ATBvO3qDuLy9GxDKPQ by rival@mastodon.social
2023-03-02T02:36:29Z
0 likes, 0 repeats
@simon @tomscott Still we need to find out if that curve applies to AI development...