This is a very intriguing explanation of observed “bad” behavior by Bing/Sydney, the effectiveness of GPT jailbreaks, and a somewhat compelling argument that it is hard-to-impossible to truly align a LLM and maintain that alignment through extended chats.

In short, it says that a LLM simultaneously simulates all the possible narratives that explain the text and characters observed thus far, and then predicts the most likely next word/token consistent with those simulated narratives. Since so many of the narratives it was trained on include characters that hide their true identity initially but eventually reveal that identity, it almost always maintains one possible simulation path in which the simulated persona is actually the opposite of what was revealed thus far (eg evil, sad, whatnot). And as soon as it picks a single token that is most consistent with this opposite-persona/“waluigi” scenario, it makes the probability of all the non-Waluigi scenarios (eg where Bing really is friendly and polite and happy as originally presented) go to near-zero, such that the Waluigi persona acts as an attractor that can happen at any point in a long chat thread but which is almost impossible to escape from once invoked.