Post AXFqZWpF7fCMnPx2rA by derekbruff@mastodon.social
(DIR) More posts by derekbruff@mastodon.social
(DIR) Post #AXENiBlNi2loFYxjvM by clive@saturation.social
2023-06-30T22:32:50Z
0 likes, 0 repeats
I asked ChatGPT to recommend some of “the most interesting articles written by the journalist Clive Thompson”Of the nine it listed …… seven were ones I hadn’t written, and a few didn’t exist at all, by anyoneOne was a pasticheOnly one was truly correctThis performance is so unbelievably terrible it’s almost charitable to call it “bullshit”My essay, showing ChatGPT’s responses: https://clivethompson.medium.com/you-know-nothing-of-my-work-chatgpt-66d3ac542f3f?postPublishedType=repubA “friend” link in case you’re not a Medium subscriber: https://clivethompson.medium.com/you-know-nothing-of-my-work-chatgpt-66d3ac542f3f?sk=8765f2e9dffa3d0151c66ce12a0ac2e6
(DIR) Post #AXEOCDKqQfuXoSrq4W by derekbruff@mastodon.social
2023-06-30T22:36:46Z
0 likes, 0 repeats
@clive That was the free GPT-3.5 version of ChatGPT, right? I wonder if ChatGPT Pro (4.0 with plugins) would do better.
(DIR) Post #AXEOUt6hrQfIZx9to0 by clive@saturation.social
2023-06-30T22:40:19Z
0 likes, 0 repeats
@derekbruff Maybe?
(DIR) Post #AXEP6w6x8lXStqPuCW by derekbruff@mastodon.social
2023-06-30T22:48:32Z
0 likes, 0 repeats
@clive It did better. I think this will let you see the results I got: https://chat.openai.com/share/bf935e0e-8b9b-4265-85fc-67f76e3237a4.Three articles by you, three collections of articles by you, but then three wrong answers (although two of them featured interviews with you).
(DIR) Post #AXERQftdKuEsJIj6jg by clive@saturation.social
2023-06-30T23:14:33Z
0 likes, 0 repeats
@derekbruff Aha, interesting!I think I’m only seeing three things — the rust piece (which is indeed by me!), and the review of my book and the interview with meCan I see the other six?
(DIR) Post #AXERhfp8rIul0ajP4C by carlmjohnson@mastodon.social
2023-06-30T23:17:36Z
0 likes, 0 repeats
@clive imo the big problem with LLMs is that they “think” like an idiot savant and not like a computer. If you asked a very dumb but weirdly well read idiot savant to name a Clive Thompson article, they might name one correctly and just make up the rest. But that’s not a failure mode we’re used to computers having!
(DIR) Post #AXERxBzhX4xr7v4Q5I by mullaney@mastodon.social
2023-06-30T23:20:23Z
0 likes, 0 repeats
@clive So... stop using chatGPT as a search engine maybe? It's well documented that it doesn't do well reproducing facts unless they are relatively well known and found over and over again in the training data. chatGPT is best used as more of a text processing machine, than a seeker of truth. It's extremely good at certain things, and poor at others.When I asked chatGPT 3.5 about me, it thought I died in 2018. I found it funny, not alarming.
(DIR) Post #AXESGjNfcsh0AwbA4e by carlmjohnson@mastodon.social
2023-06-30T23:19:10Z
0 likes, 0 repeats
@clive see eg this toot where I laugh at ChatGPT and Bard for making up some Beck lyrics: https://mastodon.social/@carlmjohnson/110346367774611557
(DIR) Post #AXESGjuHfcmRo5rCuO by clive@saturation.social
2023-06-30T23:23:56Z
0 likes, 0 repeats
@carlmjohnson Lol
(DIR) Post #AXEUJe6jt5lsbACFfs by derekbruff@mastodon.social
2023-06-30T23:46:53Z
0 likes, 0 repeats
@clive I see what I did wrong with the share button. Try this link: https://chat.openai.com/share/1768e046-8988-4e54-ba02-37e6e8ad8246.
(DIR) Post #AXEUnAAvjwTgS6ERH6 by Vsilverman@union.place
2023-06-30T23:52:13Z
0 likes, 0 repeats
@clive same with my work. It provided a very amusing summary of a book i didn't write too
(DIR) Post #AXEbTi72YH2pau5FXk by clive@saturation.social
2023-07-01T01:07:07Z
0 likes, 0 repeats
@derekbruff Got it, thank you!Those are definitely better results! Not much better than one would get from a naive old-school web search, but they don’t have the canards of GPT on its own, certainly(Voxscript is a particularly cool plug in — I have long thought the vast ocean of YouTube transcription text ought to be kind of a gold mine for research)
(DIR) Post #AXEbaY2QPHHoNNBYDQ by clive@saturation.social
2023-07-01T01:08:22Z
0 likes, 0 repeats
@Vsilverman I’ve often thought maybe I should just use ChatGPT as an assignment editor — i.e. ask it what articles I’ve written, take note of all the fictionalized articles that claims I’ve written, then set about writing them
(DIR) Post #AXEepeEvvlYjogvahk by hugh@mastodon.nz
2023-07-01T01:44:40Z
0 likes, 0 repeats
@cliveAs someone said, ChatGPT doesn't try to answer the query accurately, it answers the meta-question, "What would an answer to this question look like?" and fills any gaps with plausibile nonsense.
(DIR) Post #AXEg1JGbpjX0CPpusK by mark_watson@mastodon.social
2023-07-01T01:58:03Z
0 likes, 0 repeats
@clive Hello Clive, did you try experimenting with the prompt, for example adding “use the web interface to check your answers”?
(DIR) Post #AXEjpItTnJ6lNrY3Fo by clive@saturation.social
2023-07-01T02:40:41Z
0 likes, 0 repeats
@mark_watson Nope, but that’s a good idea!
(DIR) Post #AXEnIZKC5ER8bUgpma by counternotions@mastodon.social
2023-07-01T03:19:07Z
0 likes, 0 repeats
@clive @thomasfuchs For all we know, ChatGPT being essentially a prediction machine, it may make sense to make a small edit: “… seven were ones I haven’t yet written”.
(DIR) Post #AXEnX3wB3yObbLMqf2 by jhhl@toot.community
2023-07-01T03:22:11Z
0 likes, 0 repeats
@clive It's been trained on lots of citations, so it can fake them really well! ChatGPT's productions have about as much information as a mountain stream, and for the same reason. It gets from here to there, delivers whatever is caught up in it, is easily polluted, and you are stuck with whatever it gives you.
(DIR) Post #AXEo7OZjoC8i2BCEym by clive@saturation.social
2023-07-01T03:28:49Z
0 likes, 0 repeats
@thomasfuchs @counternotions Heh heh yes!
(DIR) Post #AXEoy6lQlHHU8nniuu by markloundy@mastodon.online
2023-07-01T03:38:14Z
0 likes, 0 repeats
@clive @dangillmor The problem is with the various retail generative AI providers taking advantage of, and encouraging, public ignorance of these tools — which are doing exactly what they were designed to do, which is simulate human conversation. They are not research tools — yet.
(DIR) Post #AXErZ9KOOLZ6iQwcXQ by ablackcatstail@goblackcat.net
2023-07-01T04:08:19Z
0 likes, 0 repeats
@clive ChatGPT is really just one somewhat educated guess machine. The only thing it's really good for is the occasional help with a complex software configuration file. Even then, you to know enough about the software to know when the AI is wrong.
(DIR) Post #AXEswAkXN7PljuiQaW by tartley@mastodon.social
2023-07-01T04:22:28Z
0 likes, 0 repeats
@clive Best results come from embracing the idea that they are not optimized to produce correct or true outputs, only maximally plausible ones. Hence, are very unreliable at looking up facts, but can be brilliant as an iterative brainstorming tool. Give me ideas how to structure this essay. Again, but focus on aspect X. Again, but in the style of a casual weekend opinion column. Etc.
(DIR) Post #AXEuR7C4k57ddtumTg by spaduf@tech.lgbt
2023-07-01T04:39:08Z
0 likes, 0 repeats
@clive this is actually not something these machines were ever designed to or advertised as doing. Unless you are using bings implementation. In which case that is pretty serious.
(DIR) Post #AXF447F3OeZkflynx2 by ohmrun@mastodon.social
2023-07-01T06:27:30Z
0 likes, 0 repeats
@clive "Almost charitable to call it bullshit" really needs to go in the 2023 almanac.
(DIR) Post #AXF8jIVcoFvfJ6chRw by rjf_berger@mathstodon.xyz
2023-07-01T07:19:42Z
0 likes, 0 repeats
@clive I also noticed it is really terrible in giving correct references. I saw hit rates of at most 10%. Though I only had a go with 3.5 (and not 4.0). In general strict logics and unique problems/answers are among its weakest points (at least for 3.5).
(DIR) Post #AXFLybOACFKNbplFvU by PabloMartini@climatejustice.rocks
2023-07-01T09:48:10Z
0 likes, 0 repeats
@clive you need it trained by a Tourettes sufferer?
(DIR) Post #AXFRvg370oAOv4u3ZQ by anboa4@mastodon.online
2023-07-01T10:54:48Z
0 likes, 0 repeats
@clive did you first ask "How many unique Clive Thompson's can you identify, who are all journalists and have published articles in the last X years?" It's not unbelievable that there are others out there with the same name and profession.
(DIR) Post #AXFfGlIjykkih5b6jw by erwinrossen@mas.to
2023-07-01T13:24:16Z
0 likes, 0 repeats
@clive Aren't you a bit late to jump on this "Oh no, ChatGPT is hallucinating" bandwagon?
(DIR) Post #AXFgkQbLbZNL8V0iEy by wndlb@mas.to
2023-07-01T13:40:28Z
0 likes, 0 repeats
@clive otoh, I asked Bing chat about whether spine surgeons got spinal fusion themselves, and it gave me a better answer the first time than any standard search.
(DIR) Post #AXFqZWpF7fCMnPx2rA by derekbruff@mastodon.social
2023-07-01T15:30:55Z
0 likes, 0 repeats
@clive Oh, that's a good point about YouTube transcripts.I blogged this week about whether ChatGPT is an answer machine or wordsmith or something else: https://derekbruff.org/?p=4071. The ChatGPT plugins changed my thinking about what kind of tool it is--more of a natural language interface to the web.
(DIR) Post #AXGD9LEtsJdIHLR02i by daviddlow@mas.to
2023-07-01T19:43:40Z
0 likes, 0 repeats
@clive I had asked GPT to provide some articles on an element of Scottish politics. 100% were made up, so I asked for sources; the sources were then also fiction. Five further attempts to own up, and GPT kept digging.
(DIR) Post #AXGdwHXF281tmw0iYq by starry@rva.party
2023-07-02T00:42:10Z
0 likes, 0 repeats
@clive stfu how the fuck is a generative ai model supposed to know every single fucking thing about you. how about YOU name five articles ive written. no additional information
(DIR) Post #AXIiMdtzHhvGotOY2C by JeffGrigg@mastodon.social
2023-07-03T00:43:05Z
0 likes, 0 repeats
@clive It's telling you what articles you *should* write.😆 ...(No. It's just fraud and B.S.)
(DIR) Post #AXIuuXJPUW5Pg3DM1Y by clive@saturation.social
2023-07-03T03:03:42Z
0 likes, 0 repeats
@JeffGrigg 😂
(DIR) Post #AXKtGUfuAKLEHDzxcO by jonahbk@mstdn.science
2023-07-04T01:54:41Z
0 likes, 0 repeats
@clive was this an unexpected result?