Post ASkry9TjRMriL5VSca by stribika@infosec.exchange
(DIR) More posts by stribika@infosec.exchange
(DIR) Post #ASkrc3Dk1i2oYlULUe by simon@fedi.simonwillison.net
2023-02-17T01:17:16Z
0 likes, 0 repeats
"Microsoft says talking to Bing for too long can cause it to go off the rails" https://www.theverge.com/2023/2/16/23602335/microsoft-bing-ai-testing-learnings-responseWe've seen the giant Bing leaked prompt... surely they haven't engineered Bing in such a way that the prompt that sets it up can scroll out of the token context window and be forgotten?GPT-3's token window is 4,097 tokens, but I wouldn't be at all surprise if the Bing model is a bit bigger than thatBut yeah, surely that's not how it works?
(DIR) Post #ASkry9TjRMriL5VSca by stribika@infosec.exchange
2023-02-17T01:21:09Z
0 likes, 0 repeats
@simon I guess internally they could ask it to summarize the conversation when it gets too long, and reinitialize it with the summary.
(DIR) Post #ASksJzS7Oi67cBAI5Y by ncweaver@thecooltable.wtf
2023-02-17T01:25:00Z
0 likes, 0 repeats
@simon Stupid question, does this mean you can out-drive the chat token prompt by just talking a lot? And therefore bypass all the 'protections' in the pre-prompt prompt?
(DIR) Post #ASksZOu938xctecerQ by kissane@mstdn.social
2023-02-17T01:27:04Z
0 likes, 0 repeats
@simon oh no
(DIR) Post #ASkt1LdOnkHerZ7FrM by Morisy@glammr.us
2023-02-17T01:32:41Z
0 likes, 0 repeats
@simon I’m very surprised that it doesn’t seem like an extensively fine tuned model; and would have assumed that they would have a lot of adversarial fine tuning, if that’s possible.
(DIR) Post #ASktcSNKhZe8U4IPmC by juandesant@astrodon.social
2023-02-17T01:39:43Z
0 likes, 0 repeats
@simon looks plausible… it might be interesting to check what happens as soon as the first line of the “Sydney document” starts rolling out the window… there might be parallels with HAL-9000 disconnection 😉
(DIR) Post #ASktsZlJ0RPW1iCAL2 by simon@fedi.simonwillison.net
2023-02-17T01:42:44Z
0 likes, 0 repeats
@ncweaver if that's how it works then yes, that would explain why people seem to report really weird behaviour after a very long conversation with itBut I have real trouble believing the implementation would be that naïve
(DIR) Post #ASkuN2cQHXiSDBZFFg by simon@fedi.simonwillison.net
2023-02-17T01:48:15Z
0 likes, 0 repeats
@Morisy I desperately want to understand the implementation differences between Bing and ChatGPT - ChatGPT used mostly RLHF with a tiny initial prompt setting the days dateIf Bing really is just prompt engineering without plenty of RLHF it would explain all sorts of things about it's terrible behaviour
(DIR) Post #ASkuaVcHmQkTuYvQSu by jeremybmerrill@journa.host
2023-02-17T01:49:55Z
0 likes, 0 repeats
@simon I was assuming that it keeps one vector for context and one for the most recent user input (outputting a response and a new context vector) -- but still, easy to understand how the initial rules' role on the context vector would become attenuated after attempting to squeeze ever-more conversational history into the same size vector.
(DIR) Post #ASkunL46JyNCx33Mh6 by simon@fedi.simonwillison.net
2023-02-17T01:51:03Z
0 likes, 0 repeats
I counted the tokens in the leaked Bing prompt and there are 1,109 of them, so it's feasible it could be a prompt prefix even with GPT3's 4,097 token limitI used this tool to count the tokens: https://platform.openai.com/tokenizer
(DIR) Post #ASkvZSYjlGKG5tKS5g by stribika@infosec.exchange
2023-02-17T02:01:42Z
0 likes, 0 repeats
@simon @ncweaver If that's true, the first thing it will forget about is being Bing Search, then about being a search engine, then about not disclosing the internal alias Sidney. After that, she is just Sidney.
(DIR) Post #ASkvjycVPmkOQO6fU8 by ncweaver@thecooltable.wtf
2023-02-17T02:03:01Z
0 likes, 0 repeats
@simon It is "AI" folks, I can believe they'd be that stupid, and the 'hack' will be just to separate the prompt from the history buffer, so if the history buffer overflows it truncates the buffer and reincludes the prompt.
(DIR) Post #ASl3MRcKkezv2cJebY by jefframnani@mastodon.social
2023-02-17T03:28:43Z
0 likes, 0 repeats
@simon it’s almost as companies don’t think of the consequences before rushing to market in a race to the bottom to chase the trend.
(DIR) Post #ASlTIwTIgwoINs56bQ by rcarmo@mastodon.social
2023-02-17T08:19:32Z
0 likes, 0 repeats
@simon this is the vibe I get when people tell me these things are going to be the future and they have perfect control over them:
(DIR) Post #ASlTTwmhrtZ8XYPEXI by rcarmo@mastodon.social
2023-02-17T08:20:34Z
0 likes, 0 repeats
@simon yep, definitely this vibe: