Post ATmMklN4g6P1grGD68 by jsit@social.coop
(DIR) More posts by jsit@social.coop
(DIR) Post #ATmI5UvKWc8mzQLozQ by simon@fedi.simonwillison.net
2023-03-19T15:36:52Z
0 likes, 0 repeats
What are common misunderstandings you've seen people have with regards to tools like ChatGPT that cause them to then use those tools in ways that mislead them with convincing false output?I want to collect more common mistakes like the "assuming ChatGPT can read content from URLs you give it" one
(DIR) Post #ATmIr5TMSGCVxExeb2 by mnl@hachyderm.io
2023-03-19T15:42:26Z
0 likes, 0 repeats
@simon not understanding that your prompt is not "asking" it something, but instead of steering it into providing a useful answer based on the corpus it has been trained on. I in fact just linking to this in another thread: https://arxiv.org/abs/2212.03551Everything useful I got these models to do are based on understanding their probabilistic nature.
(DIR) Post #ATmIr8KFqZhOnrivJo by mnl@hachyderm.io
2023-03-19T15:48:00Z
0 likes, 0 repeats
@simon another one, and this is opaque to me as well, is that your past conversational context gets "minced up" and as such, discussions tend to get weirder and weirder over time, and it's often best to just start fresh after 20 messages.
(DIR) Post #ATmJ36iLfDr5HShdtA by mschfr@mastodon.social
2023-03-19T15:42:36Z
0 likes, 0 repeats
@simon Asking for sources and trusting the answer. ChatGPT will just make up some convincing sounding literature
(DIR) Post #ATmJXQvT59PeeZIfLc by vanderwal@mastodon.social
2023-03-19T15:55:32Z
0 likes, 0 repeats
@simon one of the things I hit when drilling into understanding coding methods for new versions is ChatGPT answers across versions, which make its answers unusable.
(DIR) Post #ATmK8GGffftMJRmKrA by olepbr@snabelen.no
2023-03-19T15:58:29Z
0 likes, 0 repeats
@simon i guess just blindly trusting it; last week i had to explain to some fellow students that a Python library which it dreamed up and used in an example in fact does not exist
(DIR) Post #ATmKLYNvcjIsGZVlZI by vanderwal@mastodon.social
2023-03-19T15:59:10Z
0 likes, 0 repeats
@simon looking for explanations around the edges of things popular, but common grad school fodder and a domain I know well it confidently drifts into really wrong answers very quickly. Math, information science, social sciences, etc.
(DIR) Post #ATmKvTHC5wA2qaUKky by jvschrag@hachyderm.io
2023-03-19T16:01:12Z
0 likes, 0 repeats
@simon Believing that when an exact, correct answer to a question exists in the training data that the LLM will necessarily spit out that answer when asked the question. (It may, it may not)
(DIR) Post #ATmL6yfzgsePPyZhtw by simon@fedi.simonwillison.net
2023-03-19T16:03:13Z
0 likes, 0 repeats
@mschfr yeah I've had fun asking it for quotes too - it enthusiastically invents quotes by people (including by me) that they've definitely never said
(DIR) Post #ATmLb34DvjaXiuFNSK by osma@sigmoid.social
2023-03-19T16:08:43Z
0 likes, 0 repeats
@simonMy feeling is that many people think that ChatGPT will learn something from the prompts people send it, and thus get smarter over time. Which might be true in some sense (OpenAI probably uses some of the inputs to improve next generations of GPT) but definitely doesn't mean that it will remember details of past discussions.Not sure if this could lead to convincing, but false inputs, though.
(DIR) Post #ATmLocNnwDxV1AsOEi by starr@ruby.social
2023-03-19T16:09:27Z
0 likes, 0 repeats
@simon the big ones I see over and over are people thinking it is an AGI, that it understands what it’s talking about and that it has state beyond the last n tokens.
(DIR) Post #ATmM03q0Can1LwWQ7c by smach@fosstodon.org
2023-03-19T16:12:38Z
0 likes, 0 repeats
@simon Ask a librarian https://blacktwitter.io/@bibliotecaria/109650353375080864
(DIR) Post #ATmMDZrOSFpMQvcMdM by simon@fedi.simonwillison.net
2023-03-19T16:19:24Z
0 likes, 0 repeats
@osma yeah that's a really good misconception - I've seen tutorials that assume it can remember things from previous chat sessions, which it absolutely cannot
(DIR) Post #ATmMaGMX749mcf05NA by simon@fedi.simonwillison.net
2023-03-19T16:23:59Z
0 likes, 0 repeats
@smach oh that's a fantastic anecdote, thank you! The library equivalent of what's been happening to @opencage https://blog.opencagedata.com/post/dont-believe-chatgpt
(DIR) Post #ATmMklN4g6P1grGD68 by jsit@social.coop
2023-03-19T16:24:32Z
0 likes, 0 repeats
@simon “Why did you say something that was incorrect?”
(DIR) Post #ATmMxzse2osySEL09Q by simon@fedi.simonwillison.net
2023-03-19T16:28:07Z
0 likes, 0 repeats
@jsit hah, yeah that pattern where people argue with it and it apologizes and then contradicts itself and then keeps on doing that as they argue more is always interesting
(DIR) Post #ATmN9aWW99P3HoVyz2 by smach@fosstodon.org
2023-03-19T16:35:38Z
0 likes, 0 repeats
@simon You're welcome! And ugh hadn't heard about that one. Looks like you'll be collecting an interesting list.
(DIR) Post #ATmNmxErcqgJ1w1qrY by mschfr@mastodon.social
2023-03-19T16:44:39Z
0 likes, 0 repeats
@simon It's the same with biographies and sightseeing tips. Had ChatGPT generate mostly correct, but kind of bland recommendations, but then invent totally not existing sights.
(DIR) Post #ATmO1Dr2e4TecKgp6G by jsit@social.coop
2023-03-19T16:44:43Z
0 likes, 0 repeats
@simon I think it’s an amazing language-generation tool, but entrusting it to be correct about facts is the wrong way of looking at it. But this is seemingly what people are most trying to use it for.
(DIR) Post #ATmOGhG1blHa9WkogS by simon@fedi.simonwillison.net
2023-03-19T16:48:31Z
0 likes, 0 repeats
@mschfr as an experiment I asked one model (Claude) to write me the script for a walking tour of Año Nuevo state park - deliberately giving it a pretty obscure targetIt wrote something that looked great, and entirely invented a historic lighthouse in the middle of it!
(DIR) Post #ATmOdwiuGxbOmOLjGK by mschfr@mastodon.social
2023-03-19T16:54:07Z
0 likes, 0 repeats
@simon Had the same experience with several other places - f.e. I asked it to give me recommendations for Annecy in France - most was correct like a SEO page where the author never was in the city, but ChatGPT also invented a cable car which never existed. So I think everything where you're not writing texts, but texts with facts is something for your list
(DIR) Post #ATmPaW1vA3Hx4qAgVc by darrel_miller@mastodon.social
2023-03-19T17:04:39Z
0 likes, 0 repeats
@simon What makes it more confusing is that Bing Chat can use the content of referenced URLs to provide answers. I have used it to summarize IETF drafts with reasonable success.
(DIR) Post #ATmQCCefe4ILzUGcE4 by simon@fedi.simonwillison.net
2023-03-19T17:09:52Z
0 likes, 0 repeats
@darrel_miller right - Bing DOES have the ability to reference some content from URLs - provided they've been snapshotted by the Bing search engine crawler
(DIR) Post #ATmQcELfyG6ylxnEhM by vanderwal@mastodon.social
2023-03-19T17:15:38Z
0 likes, 0 repeats
@simon One other area where it (to me) falls into false outputs is asking about a question about a perception of a subject, which early in the chat may include the person behind that perception and allude to their work, but then mangles what the actual work says. This is odd as the concepts and framings were on the right path leading to the work, but then fabricates a summary while not following the path it was laying out.
(DIR) Post #ATmQyGSenTLhhBZ6EC by josephholsten@mstdn.social
2023-03-19T17:17:09Z
0 likes, 0 repeats
@simon I’m looking forward to this list, because it’ll be interesting to compare it to human failure modes. I see people frequently decrying the flaws of an article or publication that they’ve only read the title or summary or tertiary sources about (eg Cochrane report on physical interventions against upper respiratory infections). Some people will even claim to have read the source when challenged.
(DIR) Post #ATmRC6GxTVne9Mgs76 by vanderwal@mastodon.social
2023-03-19T17:20:31Z
0 likes, 0 repeats
@simon @darrel_miller I tried Bing yesterday and found this where it would correctly point to a source for the things the chat was roughly understanding, but asking for a summary of that it pointed to it either would say it couldn’t summarize it or would come up with something not close to what the source states.I had Bing on the most conservative mode so it quickly would say it didn’t know something or couldn’t provide an answer (I think I prefer that).
(DIR) Post #ATmRa5wWLtay4sZYqO by maxtappenden@me.dm
2023-03-19T17:22:44Z
0 likes, 0 repeats
@simon https://phpc.social/@ramsey/110013008198408095
(DIR) Post #ATmaZhj5HwvOYgAzhI by phillmv@hachyderm.io
2023-03-19T19:07:45Z
0 likes, 0 repeats
@simon on a related note that gave me some dismay,i sent an acquaintance a link to your “chatgpt can’t access the internet” post, plus a link to an openai faq, and he still refused to believe it since the fake summary it had confabulated (based on the current chat session) was so superficially plausible that it must be right
(DIR) Post #ATmfJxtVuJX5qBMvr6 by gpshead@infosec.exchange
2023-03-19T20:01:02Z
0 likes, 1 repeats
@simon People throughout history have mistaken the ability to speak or write well with possessing knowledge, logic, and reason. LLMs exploit this human animal fallacy.
(DIR) Post #ATmizwfl8kIg4me2LY by simon@fedi.simonwillison.net
2023-03-19T20:41:56Z
0 likes, 0 repeats
@phillmv challenge them to try the experiment themselves: make up a similar but entirely invented URL and see what it does with that
(DIR) Post #ATmk2AD3sdNaatqLSK by budgibson@me.dm
2023-03-19T20:52:20Z
0 likes, 0 repeats
@gpshead @simon @simon I think more generally, we respond to things we perceive as human as though they were full functioning humans. Witness how we interact with babies, the infirm, those on their deathbeds, those in catatonic states. Part of the ethics here has to be that the #LLM fully disabuses people of this tendency. I think GitHub CoPilot is a good example of an #LLM that succeeds on that score and is quite useful.
(DIR) Post #ATmk2AqPWKr4YwFlmy by simon@fedi.simonwillison.net
2023-03-19T20:53:46Z
1 likes, 0 repeats
@budgibson @gpshead I really like this line from an Emily M. Bender paper:"We call on the field to recognize that applications that aim to believably mimic humans bring risk of extreme harms. Work on synthetic human behavior is a bright line in ethical Al development, where downstream effects need to be understood and modeled in order to block foreseeable harm to society and different social groups."https://simonwillison.net/2023/Mar/15/emily-m-bender/
(DIR) Post #ATml7ECSZVVcN7XZnE by dahukanna@mastodon.social
2023-03-19T20:55:09Z
0 likes, 0 repeats
@gpshead @simon mainly western society. In other cultures, lived experience, mastery and being able to teach others rather than only oratory skills are also signs of intelligence, knowledge and reasoning.
(DIR) Post #ATml7EmcP4QsBGSS9Y by simon@fedi.simonwillison.net
2023-03-19T21:06:03Z
0 likes, 0 repeats
@dahukanna @gpshead that's a fascinating observation!Could it be that cultures that equate a high quality of writing to intelligence could be more susceptible to assuming LLMs are genuine "intelligences"?
(DIR) Post #ATmlzcGdJ2QkeXQWRs by hans@social.gerwitz.com
2023-03-19T21:14:31.325801Z
0 likes, 0 repeats
@simon @dahukanna @gpshead even in western traditions, there is a long history of philosophers (sometimes) dismissing the rhetoricians as mere masters of language rather than wisdom. Thought I am not really qualified to comment on it.
(DIR) Post #ATmlzd3COTHL5A9J9E by simon@fedi.simonwillison.net
2023-03-19T21:15:35Z
0 likes, 0 repeats
@hans @dahukanna @gpshead I like that description of language models as fluent rhetoricians but without any underlying wisdom at all
(DIR) Post #ATmmN2IcgYm83BWc8e by budgibson@me.dm
2023-03-19T21:17:23Z
0 likes, 0 repeats
@simon @gpshead I’m a fan of her reasoning on this score. It’s a sort of insight that drives readily from sociolinguistics. A large proportion of our utterances are phatic.
(DIR) Post #ATmpiwx3Y689hJZuk4 by sminnee@mastodon.nz
2023-03-19T21:57:30Z
0 likes, 0 repeats
@simon @hans @dahukanna @gpshead I think of it as an incredibly articulate black-out drunk person who has no idea what they’re saying even though they can string a good sentence together on autopilot
(DIR) Post #ATn0swSLLQMwYLBCRE by lukevanin@noc.social
2023-03-20T00:02:30Z
0 likes, 0 repeats
@simon That it can understand algorithms and write code that produces a specific behaviour.It can sometimes do this for simple code and for commonly used algorithms where there are many examples, but it often fails for algorithms and languages that are not as widely used. It will implement a different algorithm, or combine parts of different but similar or related algorithms. The code it produces is a weird mashup that looks like it might be doing the right thing, but is complete wrong.