Post AXhfLfGhvi59SMwcN6 by Jackivers@mastodon.social
 (DIR) More posts by Jackivers@mastodon.social
 (DIR) Post #AXhf5nhymbDn008VGa by Jackivers@mastodon.social
       2023-07-15T01:29:05Z
       
       0 likes, 0 repeats
       
       Did GPT-4 Code Interpreter Escape From Its Sandbox? @simon any thoughts on this little mystery?https://craftycto.com/micro/did-code-interpreter-escape/
       
 (DIR) Post #AXhf5oOAFkxv6psC1I by simon@fedi.simonwillison.net
       2023-07-15T01:33:08Z
       
       0 likes, 0 repeats
       
       @Jackivers my best guess is it's a hallucination - it started imagining content of the documents for some reason, like when ChatGPT hallucinates the contents of a URL https://simonwillison.net/2023/Mar/10/chatgpt-internet-access/
       
 (DIR) Post #AXhfLfGhvi59SMwcN6 by Jackivers@mastodon.social
       2023-07-15T01:36:08Z
       
       0 likes, 0 repeats
       
       @simon Ah, hadn’t thought of that. But if so, the hallucinations had no visible connection to my uploaded source docs (proposal templates!) … aren’t hallucinations usually contextual? Like “the API that ought to exist”?
       
 (DIR) Post #AXhfesUN07gLxdUqWG by Jackivers@mastodon.social
       2023-07-15T01:37:42Z
       
       0 likes, 0 repeats
       
       @simon Oh wait, it’s definitely not hallucinating — I found several of the sources for real
       
 (DIR) Post #AXhfet8QbBizxsEpxQ by simon@fedi.simonwillison.net
       2023-07-15T01:39:34Z
       
       0 likes, 0 repeats
       
       @Jackivers right, but that could mean they are in the training data and it somehow associated them with the other content Can you share a link to a transcript?
       
 (DIR) Post #AXhfrD62CzlXLgziIS by Jackivers@mastodon.social
       2023-07-15T01:41:37Z
       
       0 likes, 0 repeats
       
       @simon Just updated it with two links where I found the sources https://craftycto.com/micro/did-code-interpreter-escape/
       
 (DIR) Post #AXhg1brSzIMP3I8gdc by Jackivers@mastodon.social
       2023-07-15T01:43:43Z
       
       0 likes, 0 repeats
       
       @simon Oops. Misunderstood your request, adding link to transcript.
       
 (DIR) Post #AXhgJDbbuOWZ2L7RWy by Jackivers@mastodon.social
       2023-07-15T01:47:03Z
       
       0 likes, 0 repeats
       
       @simon Added link to transcript https://craftycto.com/micro/did-code-interpreter-escape/
       
 (DIR) Post #AXhgWZteLIw9w3vXbU by Jackivers@mastodon.social
       2023-07-15T01:49:31Z
       
       0 likes, 0 repeats
       
       @simon The other thing is, what I had it doing was extracting text from PDF files and merging that text … coding more than creating. Doesn’t seem like a place where hallucination would happen.
       
 (DIR) Post #AXi86DJqSaSB6GOf0S by simon@fedi.simonwillison.net
       2023-07-15T06:58:20Z
       
       0 likes, 0 repeats
       
       @Jackivers I'm confident that's all halllucinations - it looks like the function it wrote to extract content produced garbage binary data, which for some reason inspired it to hallucinate the file contents based just on the filename
       
 (DIR) Post #AXiWcWfwQWB1IcflNQ by Jackivers@mastodon.social
       2023-07-15T11:33:10Z
       
       0 likes, 0 repeats
       
       @simon bingo—hallucination based on the file name alone fits all the details. It’s funny to watch it shift in and out of reality as the pull of hallucination ebbs and flows. I’ll update my post.